DevEX - reference for building teams, processes, and platforms
Integration Testing With Cucumber How To Test Anything J A O O 2009
1. Integration testing
with Cucumber:
How to test anything
Dr Nic Williams
mocra.com
drnicwilliams.com
@drnic
$ sudo gem install tweettail
$ tweettail jaoo -f
2.
3.
4. Ruby on Rails scenarios
Scenario: Login via OpenID
Given I am on the home page
And OpenID "http://drnicwilliams.com" maps to "Dr Nic"
When I follow "login"
And I fill-in "OpenID" with "http://drnicwilliams.com"
And I press "Login"
Then I should see "Welcome, Dr Nic"
5. Not just Rails
newgem - package + deploy RubyGems
tabtab - DSL for tab completions
choctop - package + deploy Cocoa apps
Scenario: Build a DMG with default custom DMG config
Given a Cocoa app with choctop installed called 'SampleApp'
When task 'rake dmg' is invoked
Then file 'appcast/build/SampleApp-0.1.0.dmg' is created
6.
7. tweettail
$ sudo gem install tweettail
$ tweettail jaoo
mattnhodges: Come speak with me at JAOO next week - http://jaoo.dk/
Steve_Hayes: @VenessaP I think they went out for noodles. #jaoo
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
8. New Gem in 2min
newgem tweet-tail
cd tweet-tail
script/generate executable tweettail
rake manifest
rake install_gem
9. Rakefile
$hoe = Hoe.new('tweettail', TweetTail::VERSION) do |p|
p.developer('FIXME full name', 'FIXME email')
...
end
$hoe = Hoe.new('tweettail', TweetTail::VERSION) do |p|
p.developer('Dr Nic', 'drnicwilliams@gmail.com')
...
end
11. New Gem in 2min
cont...
newgem tweet-tail
cd tweet-tail
script/generate executable tweettail
rake manifest
rake install_gem
tweettail
SUCCESS!! To update this executable,
look in lib/tweet-tail/cli.rb
12. User story
Feature: Live twitter search results
on the command line
In order to reduce cost of getting
live search results
As a twitter user
I want twitter search results
appearing in the console
13. Describe behaviour in plain text
Write a step definition in Ruby
Run and watch it fail
Fix code
Run and watch it pass!
15. features/cli.feature
Feature: Live twitter search results on command line
In order to reduce cost of getting live search results
As a twitter user
I want twitter search results appearing in the console
Scenario: Display current search results
Given twitter has some search results for "jaoo"
When I run local executable "tweettail" with arguments "jaoo"
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week...
Steve_Hayes: @VenessaP I think they went out for...
theRMK: Come speak with Matt at JAOO next week...
drnic: reading my own abstract for JAOO presentation...
"""
16. Running scenario
$ cucumber features/command_line_app.feature
...
1 scenario
2 skipped steps
1 undefined step
You can implement step definitions for missing steps
with these snippets:
Given /^twitter has some search results for "([^"]*)"$/ do |arg1|
pending
end
17. features/step_definitions/
twitter_data_steps.rb
Given /^twitter has some search results for "([^"]*)"$/ do |query|
FakeWeb.register_uri(
:get,
"http://search.twitter.com/search.json?q=#{query}",
:file => File.dirname(__FILE__) +
"/../fixtures/search-#{query}.json")
end
19. features/fixtures/search-jaoo.rb
{
"results": [
{
"text": "reading my own abstract for JAOO presentation",
"from_user": "drnic",
"id": 1666627310
},
{
"text": "Come speak with Matt at JAOO next week",
"from_user": "theRMK",
"id": 1666334207
},
{
"text": "@VenessaP I think they went out for noodles. #jaoo",
"from_user": "Steve_Hayes",
"id": 1666166639
},
{
"text": "Come speak with me at JAOO next week - http://jaoo.dk/",
"from_user": "mattnhodges",
"id": 1664823944,
}],
"refresh_url": "?since_id=1682666650&q=jaoo",
"results_per_page": 15,
"next_page": "?page=2&max_id=1682666650&q=jaoo"
21. Running scenario
$ cucumber features/command_line_app.feature
...
Scenario: Display current search results
Given a safe folder
And twitter has some search results for "jaoo"
When I run local executable "tweettail" with arguments "jaoo"
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week...
Steve_Hayes: @VenessaP I think they went out for noodles...
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
"""
1 scenario
1 failed step
3 passed steps
23. fetching JSON feed
def initial_json_data
Net::HTTP.get(URI.parse("http://search.twitter.com/search.json?q=#{query}"))
end
24. fakeweb failure?!
Scenario: Display current search results
Given a safe folder
And twitter has some search results for "jaoo"
When I run local executable "tweettail" with arguments "jaoo"
getaddrinfo: nodename nor servname provided, or not known
(SocketError)
from .../net/http.rb:564:in `open'
...
from .../tweet-tail/lib/tweet-tail/tweet_poller.rb:24:in
`initial_json_data'
from .../tweet-tail/lib/tweet-tail/tweet_poller.rb:9:in `refresh'
from .../tweet-tail/lib/tweet-tail/cli.rb:39:in `execute'
from .../tweet-tail/bin/tweet-tail:10
Then I dump stdout
25. features/step_definitions/
common_steps.rb
When /^I run local executable "(.*)" with arguments "(.*)"/ do |exec, arguments|
@stdout = File.expand_path(File.join(@tmp_root, "executable.out"))
executable = File.expand_path(File.join(File.dirname(__FILE__), "/../../bin", exec))
in_project_folder do
system "ruby #{executable} #{arguments} > #{@stdout}"
end
end
26. Can I ignore a
There’s probably always something
you can’t quite test
Minimise that layer of code
Test the rest
bin main lib
27. Can I ignore a
There’s probably always something
you can’t quite test
Minimise that layer of code
Test the rest
bin main lib
1x sanity check
28. Can I ignore a
There’s probably always something
you can’t quite test
Minimise that layer of code
Test the rest
bin main lib
1x sanity check
all other integration tests
on internal code
29. bin/tweettail
Do I ne ed to test this?
#!/usr/bin/env ruby
#
# Created on 2009-5-1 by Dr Nic Williams
# Copyright (c) 2009. All rights reserved.
require 'rubygems'
require File.expand_path(File.dirname(__FILE__) + "/../lib/tweet-tail")
require "tweet-tail/cli"
TweetTail::CLI.execute(STDOUT, ARGV)
30. features/cli.feature
...
Scenario: Display current search results
Given twitter has some search results for 'jaoo'
When I run local executable 'tweettail' with arguments 'jaoo'
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week...
Steve_Hayes: @VenessaP I think they went out for...
theRMK: Come speak with Matt at JAOO next week...
drnic: reading my own abstract for JAOO presentation...
"""
31. features/cli.feature
...
Scenario: Display some search results
Given a safe folder
And twitter has some search results for "jaoo"
When I run local executable "tweettail" with arguments "jaoo"
Then I should see some twitter messages
Scenario: Display explicit search results
Given a safe folder
And twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo"
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week
Steve_Hayes: @VenessaP I think they went out for
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
"""
32. end result
$ rake install_gem
$ tweettail jaoo
JAOO: Linda R.: I used to be a mathematician - I couldn't very well have started...
bengeorge: Global Financial Crisises are cool: jaoo tix down to 250 for 2 days.
kflund: First day of work at the JAOO Tutorials in Sydney - visiting the Opera House
wa7son: To my Copenhagen Ruby or Java colleagues: Get to meet Ola Bini at JAOO Geek Nights
ldaley: I am going to JAOO... awesome.
jessechilcott: @smallkathryn it's an IT conference. http://jaoo.com.au/sydney-2009/ .
scotartt: Looking forward to JAOO Brisbane next week - http://jaoo.com.au/brisbane-2009/
scotartt: JAOO Brisbane 2009 http://ff.im/-2B5ja
gwillis: @tweval I would give #jaoo a 10.0
rowanb: Bags almost packed for Sydney. Scrum User Group then JAOO. Driving there
mattnhodges: busy rest of week ahead. Spking @ Wiki Wed. Atlassian booth babe @ JAOO
conference Syd Thurs & Fri. Kiama 4 Jase's wedding all w'end #fb
pcalcado: searching twiter for #jaoo first impressions.
kornys: #jaoo has been excellent so far - though my tutorials have been full of
Steve_Hayes: RT @martinjandrews: women in rails - provide child care at #railsconf
CaioProiete: Wish I could be at #JAOO Australia...
33. ‘I run executable internally’ step
defn
When /^I run executable internally with arguments "(.*)"/ do |arguments|
require 'rubygems'
require File.dirname(__FILE__) + "/../../lib/tweet-tail"
require "tweet-tail/cli"
@stdout = File.expand_path(File.join(@tmp_root, "executable.out"))
in_project_folder do
TweetTail::CLI.execute(@stdout_io = StringIO.new, arguments.split(" "))
@stdout_io.rewind
File.open(@stdout, "w") { |f| f << @stdout_io.read }
end
end
34. Many provided steps
Given /^a safe folder/ do
Given /^this project is active project folder/ do
Given /^env variable $([w_]+) set to "(.*)"/ do |env_var, value|
Given /"(.*)" folder is deleted/ do |folder|
When /^I invoke "(.*)" generator with arguments "(.*)"$/ do |generator, args|
When /^I run executable "(.*)" with arguments "(.*)"/ do |executable, args|
When /^I run project executable "(.*)" with arguments "(.*)"/ do |executable, args|
When /^I run local executable "(.*)" with arguments "(.*)"/ do |executable, args|
When /^I invoke task "rake (.*)"/ do |task|
Then /^folder "(.*)" (is|is not) created/ do |folder, is|
Then /^file "(.*)" (is|is not) created/ do |file, is|
Then /^file with name matching "(.*)" is created/ do |pattern|
Then /^file "(.*)" contents (does|does not) match /(.*)// do |file, does, regex|
Then /^(does|does not) invoke generator "(.*)"$/ do |does_invoke, generator|
Then /^I should see$/ do |text|
Then /^I should not see$/ do |text|
Then /^I should see exactly$/ do |text|
Then /^I should see all (d+) tests pass/ do |expected_test_count|
Then /^I should see all (d+) examples pass/ do |expected_test_count|
Then /^Rakefile can display tasks successfully/ do
Then /^task "rake (.*)" is executed successfully/ do |task|
35. ‘I should see...’
features/step_definitions/common_steps.rb
Then /^I should see$/ do |text|
actual_output = File.read(@stdout)
actual_output.should contain(text)
end
Then /^I should not see$/ do |text|
actual_output = File.read(@stdout)
actual_output.should_not contain(text)
end
Then /^I should see exactly$/ do |text|
actual_output = File.read(@stdout)
actual_output.should == text
end
36. ‘When I do something...’
features/step_definitions/common_steps.rb
When /^I run project executable "(.*)" with arguments "(.*)"/ do |executable, args|
@stdout = File.expand_path(File.join(@tmp_root, "executable.out"))
in_project_folder do
system "ruby #{executable} #{arguments} > #{@stdout}"
end
end
When /^I invoke task "rake (.*)"/ do |task|
@stdout = File.expand_path(File.join(@tmp_root, "tests.out"))
in_project_folder do
system "rake #{task} --trace > #{@stdout}"
end
end
37. ‘Given a safe folder...’
features/support/env.rb
Before do
@tmp_root = File.dirname(__FILE__) + "/../../tmp"
@home_path = File.expand_path(File.join(@tmp_root, "home"))
FileUtils.rm_rf @tmp_root
FileUtils.mkdir_p @home_path
ENV["HOME"] = @home_path
end
39. features/cli.feature
Scenario: Poll for results until app cancelled
Given twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo -f"
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week
Steve_Hayes: @VenessaP I think they went out for
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
"""
When the sleep period has elapsed
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week
Steve_Hayes: @VenessaP I think they went out for
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
CaioProiete: Wish I could be at #JAOO Australia...
"""
When I press "Ctrl-C"
...
40. adding -f option
$ cucumber features/cli.feature:22
...
Scenario: Poll for results until app cancelled
Given twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo -f"
invalid option: -f (OptionParser::InvalidOption)
lib/tweet-tail/cli.rb
module TweetTail::CLI
def self.execute(stdout, arguments=[])
options = { :polling => false }
parser = OptionParser.new do |opts|
opts.on("-f", "Poll for new search results each 15 seconds."
) { |arg| options[:polling] = true }
opts.parse!(arguments)
end
app = TweetTail::TweetPoller.new(arguments.shift, options)
app.refresh
stdout.puts app.render_latest_results
end
end
41. features/fixtures/
search-jaoo-
{
"results": [{
"text": "Wish I could be at #JAOO Australia...",
"from_user": "CaioProiete",
"id": 1711269079,
}],
"since_id": 1682666650,
"refresh_url": "?since_id=1711269079&q=jaoo",
"query": "jaoo"
}
42. features/step_definitions/
twitter_data_steps.rb
Given /^twitter has some search results for "([^"]*)"$/ do |query|
FakeWeb.register_uri(
:get,
"http://search.twitter.com/search.json?q=#{query}",
:file => File.expand_path(File.dirname(__FILE__) +
"/../fixtures/search-#{query}.json"))
since = "1682666650"
FakeWeb.register_uri(
:get,
"http://search.twitter.com/search.json?since_id=#{since}&q=#{query}",
:file => File.expand_path(File.dirname(__FILE__) +
"/../fixtures/search-#{query}-since-#{since}.json"))
end
43. hmm, sleep...
$ cucumber features/cli.feature:22
...
Scenario: Poll for results until app cancelled
Given twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo -f"
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week - http://jaoo.dk/
Steve_Hayes: @VenessaP I think they went out for noodles. #jaoo
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
"""
When the sleep period has elapsed
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week - http://jaoo.dk/
Steve_Hayes: @VenessaP I think they went out for noodles. #jaoo
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
CaioProiete: Wish I could be at #JAOO Australia...
"""
44. features/cli.feature
Scenario: Poll for results until app cancelled
Given twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo -f"
and wait 1 sleep cycle and quit
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week
Steve_Hayes: @VenessaP I think they went out for...
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
CaioProiete: Wish I could be at #JAOO Australia...
"""
45. features/step_definitions/executable_steps.rb
When /^I run executable internally with arguments "([^"]*)" and
wait (d+) sleep cycles? and quit$/ do |args, cycles|
hijack_sleep(cycles.to_i)
When %Q{I run executable internally with arguments "#{args}"}
end
features/support/time_machine_helpers.rb
module TimeMachineHelper
# expects sleep() to be called +cycles+ times, and then raises an Interrupt
def hijack_sleep(cycles)
results = [*1..cycles] # irrelevant truthy values for each sleep call
Kernel::stubs(:sleep).returns(*results).then.raises(Interrupt)
end
end
World(TimeMachineHelper)
46. using mocha
require "mocha"
World(Mocha::Standalone)
Before do
mocha_setup
end
After do
begin
mocha_verify
ensure
mocha_teardown
end
end
features/support/mocha.rb
47. working!
$ cucumber features/cli.feature:22
Feature: Live twitter search results on command line
In order to reduce cost of getting live search results
As a twitter user
I want twitter search results appearing in the console
Scenario: Poll for results until app cancelled
Given twitter has some search results for "jaoo"
When I run executable internally with arguments "jaoo -f" and wait 1 sleep cycle and
quit
Then I should see
"""
mattnhodges: Come speak with me at JAOO next week - http://jaoo.dk/
Steve_Hayes: @VenessaP I think they went out for noodles. #jaoo
theRMK: Come speak with Matt at JAOO next week
drnic: reading my own abstract for JAOO presentation
CaioProiete: Wish I could be at #JAOO Australia...
"""
1 scenario (1 passed)
3 steps (3 passed)
48.
49. Rakefile
task :default => [:features]
$ rake
(in /Users/drnic/Documents/ruby/gems/tweet-tail)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -w -
Ilib:ext:bin:test -e 'require "rubygems"; require "test/unit"; require "test/
test_helper.rb"; require "test/test_tweet_poller.rb"'
Started
....
Finished in 0.002231 seconds.
4 tests, 10 assertions, 0 failures, 0 errors
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I "/Library/
Ruby/Gems/1.8/gems/cucumber-0.3.2/lib:lib" ...
.................
5 scenarios (5 passed)
17 steps (17 passed)
Run unit tests + features
66. Integration testing
with Cucumber:
How to test anything
Dr Nic Williams
mocra.com
drnicwilliams.com
twitter: @drnic
drnic@mocra.com
Notas del editor
Abstract: You can write a small Ruby library in only a few lines. But then it grows, it expands and then it starts to break and become a maintenance nightmare. Since its open source you just stop working on it. Users complain that the project has been abandoned. Your project ends up causing more grief for everyone than if you'd never written it at all.
Instead, we will learn to write all Ruby libraries, RubyGems with tests.
This session is NOT about "how to do TDD". More importantly this session will teach you:
* the one command you should run before starting any new Ruby project
* the best way to write tests for command-line apps, Rake tasks and other difficult to test code
* how to do Continuous Integration of your Ruby/Rails libraries with runcoderun.com
Once you know how to write tests for all Ruby code, you'll want to do it for even the smallest little libraries and have the confidence to know your code always works.
The tools and attitudes to acceptance testing and unit testing your applications and libraries have been evolving and improving in all languages and frameworks. In the Ruby space, Cucumber is the one tool has dominated all conversations about acceptance testing.
Most Ruby developers write Rails applications, so the most common usage for Cucumber is describing user stories for a web application.
But its entirely possible to write acceptance tests for libraries, command line applications and generators too.
Since we&#x2019;re at JAOO, we want to track conversations on twitter about JAOO.
Since we&#x2019;re nerds, let&#x2019;s pull these into to the command line and print them out.
Perhaps we&#x2019;ll also add a &#x201C;tail&#x201D; feature to sit there watching for new tweets as they come along.
A twitter terminal client. Very nerdy. Very JAOO.
This would be our sample (truncated) output from the previous example page.
It will then sit there pulling down search results and printing new search results when they appear.
We&#x2019;ll use Ctrl-C to cancel.
&#x2018;newgem&#x2019; is the one command you should run before starting any new Ruby project or Rails plugin.
And we have a working command line application!
In agile development we describe a user story. What is it that the user wants from our software system? And what is the value to them for it.
story_text is the content from the &#x201C;Basic user story&#x201D; slide
If this search query is called then the contents of the fixtures fill will be returned.
If this search query is called then the contents of the fixtures fill will be returned.
Save a real copy of actual data from the target feed into your project.
This wires fakeweb into cucumber and asks it to throw errors if we ever ask for remote content that hasn&#x2019;t been setup via FakeWeb.request_uri
So when we run our feature scenarios again we just get the error.
Why? We haven&#x2019;t written any code yet; but we&#x2019;ve finished setting up our integration test.
Ben Mabey took 137 slides to discuss when and why to start with Cucumber and then progress to unit tests of sections of your code.
Somewhere in our solution code we call out to the json feed, parse the JSON, and print out the results.
If I disable my internet and run my tests then I should definitely see fakeweb coming into effect. But it doesn&#x2019;t. Bugger.
This is because &#x201C;When I run local executable...&#x201D; invokes the executable in new Ruby process. It knows nothing about fakeweb.
This is the default implementation of &#x201C;run local executable&#x201D;. We actually make a pure external system call to run the app, just like a user.
If you do need to stub out something - a remote service, change the clock, speed things up, then its a lot easier to do it within the same Ruby process.
So we&#x2019;ll break up our integration test: 1 sanity check to test that the bin/tweettail executable is wired up correctly and pulls down some twitter data.
The rest of our integration tests will invoke the library code directly within the same Ruby process.
If you do need to stub out something - a remote service, change the clock, speed things up, then its a lot easier to do it within the same Ruby process.
So we&#x2019;ll break up our integration test: 1 sanity check to test that the bin/tweettail executable is wired up correctly and pulls down some twitter data.
The rest of our integration tests will invoke the library code directly within the same Ruby process.
If you do need to stub out something - a remote service, change the clock, speed things up, then its a lot easier to do it within the same Ruby process.
So we&#x2019;ll break up our integration test: 1 sanity check to test that the bin/tweettail executable is wired up correctly and pulls down some twitter data.
The rest of our integration tests will invoke the library code directly within the same Ruby process.
There is a very thin wrapper around the library code which was auto-generated. This helps you trust that it should &#x201C;Just Work&#x201D;. So let&#x2019;s just test the final part instead to test the ultimate result
Original version that we couldn&#x2019;t fake out the remote data...
New version where we can.
The first scenario is a thin sanity check: does our app actually pull down the twitter search data feed and print out some message. We have no idea what it will print out, just the structure.
The second and all subsequent scenarios will start testing our executable within the ruby runtime, so we can stub out the remote HTTP calls.
Now we go and write some code, install the gem locally, and run it.
New version where we can.
The first scenario is a thin sanity check: does our app actually pull down the twitter search data feed and print out some message. We have no idea what it will print out, just the structure.
The second and all subsequent scenarios will start testing our executable within the ruby runtime, so we can stub out the remote HTTP calls.
If you&#x2019;ve used cucumber with rails you&#x2019;ll have seen the provided steps for webrat like &#x201C;When I follow &#x2018;link&#x2019;&#x201D; and &#x201C;When I select &#x2018;some drop down&#x2019;&#x201D;. If you use the cucumber in a RubyGem you get many provided step definitions too.
The basic relationship between then is STDOUT and the file system. Do something which prints to STDOUT or modifies the file system, and then test the output or files.
To explore how this is happening, let&#x2019;s look at it in reverse. When we want to test the STDOUT, we need to be able to view and explore it. So we&#x2019;re reading the STDOUT from previous steps from a file. The file name is stored in @stdout.
The reason I store STDOUT into files is so that when a scenario fails, I can easily peruse each generated STDOUT file and look at what was output myself. If I just kept STDOUT in memory between steps then I&#x2019;d lose that.
It is all the When steps that run commands or rake tasks or generators, which in turn creates new files and prints things out to STDOUT.
We run each of these commands from an external system command, just like the user would do, and then store the STDOUT to a file. Its file name is always stored in @stdout.
If we&#x2019;re saving STDOUT to files, if we&#x2019;re testing generators or other applications or libraries that create new files, where is a safe place to do that?
All scenarios get a &#x201C;safe folder&#x201D;. You get a tmp folder within your project source folder.
You even get a fake HOME folder and the $HOME env variable wired to it, incase your application wants to manipulate dot files or other content within a user&#x2019;s home folder.
Let&#x2019;s write a scenario first for the -f option
The aim is to only very vaguely care about &#x201C;how the hell am I going to implement &#x2018;the sleep period has elapsed&#x2019;
Let&#x2019;s drive development now. First, adding a -f option.
We&#x2019;ll need some more sample data from twitter. Here&#x2019;s what the JSON might look like on a subsequent call to the API. The since_id value in the file name comes from the original sample JSON data - the refresh_url value.