1. Testing Ember Apps
Jo Liss
https://twitter.com/jo_liss
Hi, Iām Jo, Iām on the
http://solitr.com/blog
Capybara and Konacha
core teams, and Iām an
Ember contributor.
This talk will be more an
architectural over view
than a tutorial. Iāll
assume Rails in some
places, but the concepts
apply to other backends
as well.
Slides licensed under CC BY 3.0.
2. This presentation has 2 parts:
Part 1: Full-Stack Integration Tests
Capybara similar: PhantomJS
3. Part 1: Full-Stack Integration Tests
Capybara
Part 2: Client-Side Integration Tests
Konacha similar: QUnit, Mocha, Jasmine
I like to test apps with a
combination of Capybara
and Konacha tests.
I noticed that having a
good test suite makes me
*much* more productive.
4. Part 1: Full-Stack Integration Tests
Capybara
Itās not very hard to get
Capybara working right.
So I want to focus on
high-level architecture
instead. I think thatās
sometimes under-
appreciated.
5. Capybara
Quick refresher:
āSelenium for Railsā
You can plug in different Q: Who has used Capybara to
drivers, like WebKit. All test a JavaScript app?
of the following applies ------->
to any driver. Q: Who ran into problems with
either performance, or
flickering tests (brittleness)?
7. Capybara
These pains are architectural.
Performance :-(
Brittleness :-(
There are probably limits So letās talk about why
to how fast you can this happens.
make Capybara tests.
And you can hunt down
flickering tests (and you
should), but they will still
pop up occasionally.
8. So you have a test suite
written in Ruby, and it
talks to a DB backend.
Test suite
DB
9. Firefox + So when you have a
Capybara test, Capybara
will automatically spin up a
Selenium Firefox for you, and your
test suite will send requests
to that Firefox.
Test suite
But Selenium commands are mostly
non-blocking, i.e. they return before
they finish. So when you have a
DB sequence of commands, even with
Ajax, Capybara does some intelligent
polling to make them appear
synchronous.
This works transparent 99% of the
time, and it makes for very readable
test code, but very occasionally, youāll
end up with a race condition, and then
you have to understand whatās going
on underneath.
So let me double that line to indicate
that itās async.
10. So who does the Firefox make
Firefox + HTTP requests to? Clearly, Ruby is
busy with the test suite.
Selenium Capybara actually spins up
another ser ver thread.
Test suite
DB
11. Firefox +
Selenium
Server Test suite
So now Firefox has a
server to load the pages DB
from.
If thereās Ajax requests,
this whole thing becomes
asynchronous. So thatās
another source of race
conditions. Let me double
that line...
12. Firefox +
Selenium
Server Test suite
And of course the server
talks to the database -- DB
another line...
13. Firefox +
Selenium
Server Test suite
Which means you also This is clearly fundamentally quite complex
have to be careful not to
have the t wo Ruby DB and error-prone.
threads interfere with Iām not trying to convince you that Capybara
each other while they is a bad tool. But I think itās actually useful to
access the database. understand this architecture.
By the way, just to be clear, thatās not
Seleniumās fault. If you drop in WebKit or
PhantomJS into Capybara, it suffers from the
same issues.
14. Firefox +
Selenium
At the moment, this kind of architecture is
the only solution to give us a full-stack
integration test, database to DOM. So itās
actually very useful.
Server If you donāt use Rails, Iād definitely Test suite
recommend finding a comparable solution,
or hand-rolling one.
DB
16. ... is limiting yourself to
exercising one happy path for
Capybara
each model, to make sure that
stuff in the DB ends up in the
DOM, and vice versa.
Powerful but clunky
Strategy: Only test every model once
DB-to-DOM, read and write
17. And then we can use pure client-
side integration testing to get
Capybara
more fine-grained testing.
Powerful but clunky
Strategy: Only test every model once
DB-to-DOM, read and write
Move ļ¬ner-grained tests to client side
18. Part 2: Client-Side Integration Tests
with Konacha
The idea is to limit the
architectural complexity.
No backend server, no DB,
test runs directly in the
browser, not in a separate
process.
This makes it faster, and
extremely reliable.
19. Konacha
Rails gem packaging
Konacha is pretty
Mocha.js testing framework + simple, so if youāre not on
Rails, you can easily
hand-roll an equivalent
test setup.
Chai.js assertion lib
Let me show you what
Mocha and Chai do...
20. Mocha + Chai
describe 'todo list', ->
it 'can add items', ->
Mocha is a testing
framework, like QUnit or
Jasmine, or RSpec on
Ruby.
But Mocha doesnāt do
assertions, so you
typically combine it with
the Chai assertion
library.
22. Interlude: Konacha Video
Before I go on, let me show you
what weāre trying to achieve.
http://www.youtube.com/watch?v=heK78M6Ql9Q
So we basically just hit Cmd+R on
Konachaās development ser ver.
Konacha automatically runs your tests in
an iframe, and loads your application.css.
Thatās also really easy to do yourself if
you donāt use Rails with Konacha.
This is also really useful to ramp up people
to Ember, because you can visually see
what your test does. And if a test fails,
you can just click into the iframe and
interact with the application.
24. Why is Konacha fast?
No page loads You donāt have to serve and parse the entire
stack of assets for every test case.
25. Why is Konacha fast?
No page loads
100% synchronous No polling, no waiting.
26. Why is Konacha fast?
No page loads
100% synchronous
No stack Most expensive thing is the DOM.
27. Unit vs. Integration
What you just saw I call āclient-side
integration testsā. They donāt involve the
backend, but on the frontend, they exercise
the entire Ember app.
28. Unit vs. Integration
Lots of simple layers ==> integration
tests win.
On Ember, I avoid unit tests. The reason is, the
individual layers of an Ember app are very
simple. What I care about is whether they play
together. Unit tests tend to be very āweakā, i.e.
theyāll just keep passing even when your app
Okay, breadānābutter
breaks.
time.
As a rule of thumb, I tend to do zero unit
testing for Ember apps.
29. Ember setup
No āofļ¬cialā support for testing.
So we wing it. A bunch of pieces are
missing to make this
easy like with Rails.
(Notably, good fixture
handling, package
manager, ...)
Hereās some practical
tips:
30. Thereās no repeated app
instantiation yet, so I
like to start up the
Starting the app
application at the
beginning of the entire
test suite.
31. Starting the app
# Parse time: We call
App.deferReadiness, and
advance readiness it
App.deferReadiness() once you want the app to
run.
# Global before all:
before -> UPDATE Feb 26 2013:
App.advanceReadiness() Instead you can now Enable
Ember.testing *before*
parsing your app code, and
kick it off with ābefore ->
App.initialize()ā; no more
advanceReadiness.
32. Router
We tell the router not to
mess with the URL.
App.Router.reopen
location: 'none'
33. Router
App.Router.reopen
location: 'none'
beforeEach: ->
App.router.transitionTo('...')
Before each test, UPDATE Feb 26 2013:
transition back to the
root state. You can now use
ābeforeEach ->
This is a pretty hackish App.reset()ā.
way to reset app state,
but for now itās
surprisingly reliable.
35. Konacha: Keep body
Konacha.reset = function() { }
Konacha by default
cleans the <body>
element before each test.
But our app keeps
running bet ween tests,
so we disable this by
nuking Konacha.reset.
36. Runloop & setTimeout
Ember automatically
schedules runloops with
setTimeout. But setTimeout is
Ember.testing = true
the enemy of deterministic
tests. So we disable
automatic runloop creation
by enabling the Ember.testing
flag.
You want to put this before
loading (parsing) your app
modules.
37. Runloop & setTimeout
Ember.testing = true
Itās OK to have Em.run everywhere:
Em.run => foo.set ...
Em.run => $(...).click() Most actions have their
effects deferred to the
end of the runloop. In test
code you need the effects
immediately, so you wrap
things in Ember.run.
Looks funny, but itās
nothing to worry
about. :-)
38. Animations
jQuery.fx.off = true
Nothing like this for D3. :-( This is not Ember-related,
but you donāt want
animations, because they
are asynchronous. For
D3,, this may require
support in your
production code. :-(
Probably the trickiest
thing is data fixtures:
39. Model Fixtures
1. Client-side ļ¬xtures. You have a choice
whether you write them
in JavaScript or
2. Server-side ļ¬xtures. generate them from the
server side, and itās kind
of a trade off.
40. (1) Client-Side Fixtures
FixtureAdapter (immature)
App.TodoList.FIXTURES = [
{ ... }, { ... }
For client-side fixtures,
] there is a FixtureAdapter
in Ember. It still needs
some love, but we can
probably get it there
pretty soon.
And basically you just
define an array of
fixtures for every model.
42. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
But you donāt know if
your fixtures actually
represent the reality of
your backend.
43. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
:-( Fragile You can even have bugs
in your fixtures, where
you donāt set up
bidirectional belongsTo
and hasMany
relationships properly.
44. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
:-( Fragile
:-( Server-side computed attributes
JavaScript is still not very powerful. Oftentimes itās So the alternative to all
easier in practice to implement computed properties on the this is...
backend side and ser ve them out as read-only attributes.
In one backend I worked on, half of the properties were DB
columns, and half were just methods on the Rails models.
Trying to keep these properties manually updated in your
fixture data is obviously painful.
45. (2) Server-Side Fixtures
rake test:fixtures
1. Write ļ¬xtures to DB
2. Generate JSON to ļ¬xtures.js
46. (2) Server-Side Fixtures
rake test:fixtures
1. Write ļ¬xtures to DB
2. Generate JSON to ļ¬xtures.js
Load through RESTAdapter
Load that data in fixtures.js before every test
case, perhaps using your RESTAdapter so you
translate the JSON correctly.
48. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain Compact definitions, esp. w/ FactoryGirl; stuff doesnāt
go out of sync.
49. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain
:-( Usability Generated fixtures file can go stale and you have to regenerate. Itās not bad,
just bothersome.
50. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain
:-( Usability
:-( Complex to set up Work to set up. You end up with some
custom code, and it ties tightly into
backend.
I personally think itās generally worth it,
but it also depends on your specific app.
51. Fixture Woes
Global ļ¬xtures :-( Both of these techniques
mean that you have the same
fixture set for the entire test
suite.
want FactoryGirl Ideally weād build something
like FactoryGirl, but I donāt
think weāre quite there yet.
52. Bonus: JS-driven?
Capybara but in JavaScript?
One thing that
sometimes comes up is,
can we have a full-stack
integration test written
in JavaScript?
53. Firefox +
So instead of having the
Selenium Test suite written in
Ruby...
Server Test suite
DB
54. Firefox w/
JS tests
... we push them into the
browser, and so we avoid
the t wo-threaded problem.
Server
I think that would be
really awesome -- and the
truth is, we just donāt have
the tooling yet to make
that work easily.
Perhaps the biggest hurdle
DB is that we donāt have a way
to reset the database, add
fixtures, and perhaps
query database records
from JavaScript.
But it is probably a good
direction to move towards
55. Q &A
Notes from the talk, thanks to @cgcardona:
Q: With Konacha do you see opportunity for TTD?
A: Not quite TDD (hard with visual stuff), but continuous.
When working with Konacha I wrote tests as I went.
Q: Can you test views in isolation?
A: Itās really tricky to instantiate a view in isolation, Ember
wants a whole appā¦ Itās too āunit testyā. It might be
possible.
56. Q &A
Comment: We use āRosieā (https://github.com/bkeepers/
rosie) for Javascript factories.
Comment: Have you tried VCR for server side things? Itās a
Ruby library that will record request responses so you can
play them back later. We run VCR against a live production
server and generate response data that the tests use.
57. Q &A
Q: A lot of bugs come from asynchronicity. Have you tried
to test that speciļ¬cally?
A: No I havenāt. My hope is that a lot of these bugs
disappear with ember-data. Mocha allows for asynchronous
tests where you set a callback for when your test is
complete.