"Turning TDD upside down - For bugs, always start with a passing test" - Common workflow on TDD is to write failed tests. The problem with this approach is that it only works for a very specific scenario (when fixing bugs). This presentation will present a different workflow which will make the coding and testing of those tests much easier, faster, simpler, secure and thorough' Presented at LSCC (London Software Craftsmanship Community) http://www.meetup.com/london-software-craftsmanship on sep 2016.

1. Start with passing Tests
(TDD for bugs)
London, 22nd Sep 2016, v0.5

2. Me
• Developer for 25 years
• AppSec for 13 years
• Day jobs:
• Leader OWASP O2 Platform
project
• Application Security Training
for JBI Training
• Part of AppSec team of:
• The Hut Group
• BBC
• AppSec Consultant and Mentor

3. • @Leanpub (buy for 0$ )
• http://leanpub.com/u/DinisCruz
–
Books Published

4. • published at Leanpub (buy for 0$ )
– http://leanpub.com/u/DinisCruz
Books under development
Major revision with lots of new content
(based on Maturity Models app)
Ideas shown in this presentation
and a lot more

6. See also:
http://blog.diniscruz.com/2016/03/new-era-of-software-with-modern.html

8. See also:
http://blog.diniscruz.com/2016/03/new-era-of-software-with-modern.html

10. TDD

11. • For bugs, always start with a passing test
– Common workflow on TDD is to write failed tests.
– The problem with this approach is that it only
works for a very specific scenario (when fixing
bugs).
– This presentation will present a different workflow
which will make the coding and testing of those
much much easier, faster, simpler, secure and
thorough'
Key concept: Turning TDD upside down

12. • For TDD to be productive you need
– Real time unit test execution (when hands lift)
– Real time code coverage
• TDD focus needs to be on
– making developers more productive
– preventing developers from switching context
• You need something like Wallaby or NCrunch
• If 99% code coverage doesn’t happen ‘by default’
TDD workflow is not working
TDD

13. 9 9 % C O D E C O V E R A G E
…is not the destination
…it is ‘base camp’

15. TDD in WebStorm with WallabyJS

16. What happens when you increase attack surface

17. You want a test to fail

18. TDD in WebStorm with WallabyJS
• … but is a topic for another talk :)

19. TDD WORKFLOW FOR A BUG

20. • Bug is found
• Issue created and documented
• Issue prioritised and scheduled
• Fix
• QA
• Deploy
– hope is has been fully fixed?
Normal flow for bugs and security issues

21. • Bug found, Issue created, Scheduled
• TDD loop
– Write test that fails
– Fix Issue
– Test passes
– All done?
• QA
– why is this needed?
With TDD

22. • Very few do this loop in pure
TDD
– very hard to do without
realtime test execution and
code coverage
• Needs test API & Infrastructure to be there already
• are you able to spend an extra 4h, 8h,16h fixing a blind spot
of your test APIs?
• End result tends to be very shallow tests
• Root cause of issue is usually not identified (or fixed)

23. • Given these bugs:
• Login + open page X + enter data + submit data =
BUG happens
• Create object + populate with data + do action = BUG
happens
• call service A + inner call to Service B + bad data
returned from Service B = BUG happens
• If you write a failed test, what happens when a
problem occurs before we reach the Bug happens
How do you know what failed?

24. • Tests are only written while in ‘fix mode’
• Very little time to improve test APIs for test execution
environment
– There are really hard problems in Testing, that will need time
to be solved
• Creating test proxies
• Creating ‘hot’ run-time environments
• Improving test execution and code coverage visualisation (when
Wallaby or NCrunch is not available)
• Will break the build
• Will create a high threshold of what can be tested
The problem with failed tests

25. • Missing Headers on Production (HSTS for example)
• Debug messages shown on production
• SQL Injection on user controller field
• Wrong product shown on page
• Wrong link generated on page and email
• Performance issues (for example: slow search)
• Inconsistent error messages returned from Web Service
• Message Queues accepting message with bad data
• Expose user info via parameter manipulation
• Over-posting on Controller’s Model bindings
• Use of Strings on models to hold data (‘strings should to be banned’)
Example of bugs

26. • Aren’t they the current behaviour of the
application?
• Isn’t that how the application was designed
• If you are writing a test to replicate the issue,
should that test: Pass or Fail?
• My definition of feature is:
Once the code hits the main ‘develop’ branch, it’s
behaviour is a feature (regardless if the code is in
production)
Are these bugs or features?

27. Until you write the (passing) test,
you don’t really know the root
cause of the issue

28. 1.Find issue
2.Create PoC (Proof-of-Concept)
3.Report issue (on PDF or Bug tracking)
4.Dev ‘fixes’ issue
5.Fix pushed to production
6.Is it really fixed? If yes, end of story
7.Find variation of issue
8.go back to 2.
Bug/Security fix merry-go-round

29. • Why is QA finding issues that the devs are not aware
of, or have been fixed before?
• Most QA teams are clicking on things and running
scripts
• QA teams:
– should be reviewing the Tests created/changed since the
last release
– need to have developers since they need to write Tests
– need to have senior developers since they need to improve
the Test’s APIs and execution/development Environments
QA - Most are not being efficient

30. WRITING PASSING TESTS

31. • Write Test first
• Understand the issue
• Improve test APIs and Frameworks
• Write one Test per affected layer (or call stack)
– start with e2e (end-to-end) and end with pure Unit Tests (on the root
cause of the issue)
– For affected each method, write tests for it’s callers (creating a reverse
tree)
• By definition, the fix of original issue(s) will break these tests
• Transform these ‘passing bug tests’ into ‘passing regression tests’
• QA team (and manager/architect/security) reviews the test’s diff
Suggested workflow

32. • Team members need to communicate using Tests
• Tests should be easier to write than explanations
(text, screenshot, videos)
• Teams collaboration should be done via tests
(from dev to QA, from QA to dev, from Team A to
team B)
• Managers need to write tests (and demand
powerful/easy-to-use APIs)
Tests need to be the lingua franca

33. • It is crazy that we tend to promote the best developers
into management position (since that is the ‘only’ career
path) and then make them do everything except coding
• These ‘senior devs, now managers/architects’ tend to
be the best programmers in the building and usually
love coding
• The solution is to make them read and write tests
• Teams need to communicate using tests
• Printing tests (on paper) is a very effective way to code
review
Managers need to code

34. When creating tests on the ‘Fix’ stage,
the focus (& time allocated) is on
fixing the bug (not on testing it)
When creating tests on the ‘Issue Creation’
stage, the focus (& time allocated) is on
how to test it and what is its root cause

35. CASE STUDY: WHEN I CREATED A
VULNERABILITY

36. • Here is the code I wrote (at the Data Layer)
• This method is designed to be called by the
controller (i.e. rest api endpoint):
Feature request: Allow data editing on UI

37. Feature request: Allow data editing on UI

38. Regression test that passes on issue

43. Fix for Path transversal

44. Regression test

45. LET’S SEE HOW IT LOOKED IN
THE CODE

46. …before the vuln is created

47. …when the vuln is created

48. … adding comments

49. …after issues are created

50. …improving comments

51. …updating issues after 1st fix

52. … after final fix

53. Thanks, any questions
@diniscruz
dinis.cruz@owasp.org