Presentation at BOSC2012 by B Temperton - The Bioinformatics Testing Consortium

1. The Bioinformatics Testing Consortium:
Improving Bioinformatics software one manuscript at a time
Better Science through Superior Software - Titus Brown
Ben Temperton
Giovannoni Lab
Department of Microbiology
Oregon State University
btemperton@gmail.com
@bentemperton

2. Bioinformatics - Dealing with the data deluge
Data problem
identified

3. Bioinformatics - Dealing with the data deluge
Data problem
identified
Software written to
analyze data

4. Bioinformatics - Dealing with the data deluge
Data problem
identified
Software written to
analyze data
Solution published
and shared

5. Bioinformatics - Dealing with the data deluge
Data problem
identified
Software written to
analyze data
Solution published
and shared

6. The Problem - Insufficient codebase quality checks
• Software is released having been tested only by those who wrote it

7. The Problem - Insufficient codebase quality checks
• Software is released having been tested only by those who wrote it

8. The Problem - Insufficient codebase quality checks
• Software is released having been tested only by those who wrote it
• Software rarely checked as part of the manuscript review process

9. Not a new problem in IT
Developer ‘Client’
Unit testing
Integration testing
System testing
User Acceptance testing

10. Not a new problem in IT
Developer Tester ‘Client’
Unit testing Integration testing
System testing
User Acceptance testing

11. Not a new problem in IT
Developer Tester ‘Client’
Unit testing Integration testing Installs
System testing Runs
User Acceptance testing Meets Requirements

12. Not a new problem in IT
Developer Tester ‘Client’
Unit testing Integration testing Installs
System testing Runs
User Acceptance testing Meets Requirements

13. Not a new problem in Science - Peer Review
Manuscript Peer Review:
Improving quality for free*!
* not free

14. The Bioinformatics Testing Consortium
Developer places code in Does it compile?
Open Source Repository Does it run?
and registers project Testers run
through standard Does the test data output match?
tests ...
Testers assigned to
project by BTC
Minimum
Developer fixes
bugs/issues
BTC standards
met?
NO
YES
BTC
APPROVED
Testers raise
bugs/issues in
discussion with
developer

15. Who Benefits?
• New Users
• Reduced activation energy when using BTC-approved software
• Compulsory inclusion of a test data set with known results

16. Who Benefits?
• New Users
• Reduced activation energy when using BTC-approved software
• Compulsory inclusion of a test data set with known results
• Developers
• Rapid awareness of issues
• Bugs raised in a consistent manner with enough details to replicate

17. Who Benefits?
• New Users
• Reduced activation energy when using BTC-approved software
• Compulsory inclusion of a test data set with known results
• Developers
• Rapid awareness of issues
• Bugs raised in a consistent manner with enough details to replicate
• BTC Testers
• Using students as testers teaches standards in early career

18. Who Benefits?
• New Users
• Reduced activation energy when using BTC-approved software
• Compulsory inclusion of a test data set with known results
• Developers
• Rapid awareness of issues
• Bugs raised in a consistent manner with enough details to replicate
• BTC Testers
• Using students as testers teaches standards in early career
• Journal Editors
• Assurance that software described in a manuscript is fit-for-purpose and
key analyses can be reproduced

19. What’s Needed?
Feedback

20. What’s Needed?
Feedback
Participation

21. What’s Needed?
Feedback
Participation
Test Cases

22. What’s Needed?
Feedback
Participation
Test Cases
@BioTestConsort

23. With thanks to:
Iddo Friedberg Stephen J. Giovannoni
J. Cameron Thrash Bank Beszteri
Jonathan Eisen
Pascale Gaudet Jason Stajich
Titus Brown
Hilmar Lapp
Cameron Neylon
Chris Sullivan