Talk given at RailsClub Moscow 2015

1. RAILSCLUB 2015
Testing
and Software Writer a year later
Simon Bagreev
Head of Ruby development
Rambler&Co

2. RAILSCLUB 2015
whoami
status_200
semmin
Simon Bagreev

3. RAILSCLUB 2015
US

4. RAILSCLUB 2015
RailsConf 2014

5. RAILSCLUB 2015
TDD
● doesn’t test anything
● complicates design
● holy trinity (speed, coverage, ratio)
● adds cost, not value

6. RAILSCLUB 2015
Software Writer
● does the whole thing work?
● testing at the higher level (system tests)
● adequate testing

7. RAILSCLUB 2015
More problems
● no testing
● false confidence
● holy wars
● ignored failing tests
● endless test suite (due to units)
● “virtual” QA department

8. RAILSCLUB 2015
No testing
● no proof that features
work
● no way to catch
regressions
● impossible to maintain

9. RAILSCLUB 2015
No testing - TODO
● start with user stories
● write acceptance specs
● move to system tests
● create “visual” tests

10. RAILSCLUB 2015
False confidence
● many (unit) tests
● runs fast
● 100% coverage

11. RAILSCLUB 2015
About testing
“Testing shows the presence, not the
absence of bugs.”
Edsger W. Dijkstra
NATO Software Engineering Conference, 1969

12. RAILSCLUB 2015
Value of unit testing
“Most software failures come from the interactions between
objects rather than being a property of an object or method in
isolation.”
“ Functional testing typically finds twice as many bugs as unit
testing ”
James Coplien
Why Most Unit Testing is Waste

13. RAILSCLUB 2015
Coverage
class Event
def initialize(started_at)
@started_at = started_at
end
def today?
(Time.now.beginning_of_day..Time.now.end_of_day).cover? @started_at
end
end

14. RAILSCLUB 2015
100% coverage
RSpec.describe Event do
subject { Event.new(start_time) }
context 'today' do
let(:start_time) { Time.now }
it 'todays event' do
expect(subject).to be_today
end
end
end

15. RAILSCLUB 2015
100% coverage
no expectation
RSpec.describe Event do
subject { Event.new(start_time) }
context 'today' do
let(:start_time) { Time.now }
it 'todays event' do
subject.today?
end
end
end

16. RAILSCLUB 2015
● mutation testing
● higher level testing
● “Create system tests with
good feature coverage
(not code coverage)”
James Coplien
Why Most Unit Testing is Waste
False confidence TODO

17. RAILSCLUB 2015
Holy wars
● RSpec vs MiniTest
● Capybara vs Cucumber
● AR fixtures vs FactoryGirl
● stubs vs mocks
● PhantomJS vs Selenium
● etc...

18. RAILSCLUB 2015
Ignored failures
“Oh, you have to invoke
Maven with this flag that
turns off those tests — they
are tests that no longer
work!”
James Coplien
Why Most Unit Testing is Waste

19. RAILSCLUB 2015
“Rotten” tests
“If you want to reduce your test mass, the number
one thing you should do is look at the tests that have
never failed in a year and consider throwing them
away.”
James Coplien
Why Most Unit Testing is Waste

20. RAILSCLUB 2015
rspec-rotten, FTW
“rotten” – haven’t changed status (failed/passed /pending)
in extended period of time
https://github.com/rambler-digital-solutions/rspec-rotten

21. RAILSCLUB 2015
rspec-rotten
● uses custom rspec formatter
● subscribes to rspec notifications
● after every test run saves stats into json file:
{
"id": "./spec/controllers/calendar_controller_spec.rb[1:1:1]",
"state": "failed",
"date": "2015-09-21T22:04:39.063+03:00",
"location": "./spec/controllers/calendar_controller_spec.rb:13",
"description": "render the :show template"
}

22. RAILSCLUB 2015
rspec-rotten
Notifies you about “rotten” tests:

23. RAILSCLUB 2015
Endless unit tests
Weinberg’s Law of Decomposition

24. RAILSCLUB 2015
Endless unit tests
Acceptance test

25. RAILSCLUB 2015
Endless unit tests
Unit testing

26. RAILSCLUB 2015
Endless unit tests
(Testing) is identifying and resolving problems that arise in
the system as a whole rather than in individual
components. Testing is done by groups, as an individual
programmer lacks the total knowledge needed to test
large sections of the system. The emphasis in testing
should never be on individual modules.
Edsger W. Dijkstra
NATO Software Conference, 1969

27. RAILSCLUB 2015
Additionally
● hard to maintain (tests are code too, may contain
bugs)
● hard to change (tends to become “binary”)
● may never occur in prod

28. RAILSCLUB 2015
Many unit tests TODO
● mutation testing
● rspec-rotten
● “Software Writer’s” approach: “If this test fails,
what business requirement is compromised?”

29. RAILSCLUB 2015
“Virtual” QA dept
“My unit tests pass, QA / PM / Product Owner will
take care of the rest” ... Nope
another team

30. RAILSCLUB 2015
“Virtual” QA TODO
● focus on acceptance specs (outside-in)
● “visual” tests for end-user regressions

31. RAILSCLUB 2015
rspec-visual, FTW
"visual" testing with rspec via screenshot comparison
https://github.com/rambler-digital-solutions/rspec-visual

32. RAILSCLUB 2015
rspec-visual step 1
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Create “visual” test

33. RAILSCLUB 2015
rspec-visual step 2
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Take a screenshot

34. RAILSCLUB 2015
rspec-visual step 3
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Use look_like matcher to compare with “stable”

35. RAILSCLUB 2015
rspec-visual step 3
# rspec/visual/.../look_like_matcher.rb
RSpec::Matchers.define :look_like do |expected|
# ... set file paths
match do |actual|
# ... copy to stable folder if not found
system "compare -metric PAE -subimage-search -dissimilarity-threshold 1
#{expected_image_file_path} #{actual_image_file_path} #{diff_file_path}"
end
# ... customize failure message
end
look_like matcher

36. RAILSCLUB 2015
rspec-visual result
!=

37. RAILSCLUB 2015
rspec-visual diff

38. RAILSCLUB 2015
rspec-visual result
!=

39. RAILSCLUB 2015
rspec-visual diff

40. RAILSCLUB 2015
rspec-visual gotchas
● dependencies (Capybara, Poltergeist + PhantomJS,
imagemagick)
● only supports Poltergeist
● dynamic data
● speed

41. RAILSCLUB 2015
To conclude
As a Software Writer:
● focus on what really matters
● throw away tests that have no business
value
● do “visual” testing
● use Thought-Driven Development

42. RAILSCLUB 2015
Contacts
В группе компаний Rambler&Co всегда есть открытые вакансии для тех, кто
хочет профессионально расти и развиваться, занимаясь тем, что по-
настоящему нравится
hr@rambler-co.ru
www.rambler-co.ru/jobs

43. RAILSCLUB 2015
Links
RailsConf 2014 Keynote: Writing Software by DHH: http://bit.ly/1LwrENX
Why Most Unit Testing is Waste, James Coplien: http://bit.ly/1kBMor1
Why Most Unit Testing is Waste part 2, James Coplien: http://bit.ly/1OuzrOc
NATO Software Engineering Conference 1969: http://bit.ly/1iwANKT
rspec-visual: https://github.com/rambler-digital-solutions/rspec-visual
rspec-rotten: https://github.com/rambler-digital-solutions/rspec-rotten

44. RAILSCLUB 2015
Thanks
Questions?