Bof4162 kovalsky

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1
Insert Picture Here
15 Lessons Learned from
15 Years of Quality Control
Jiří Kovalský
NetBeans Community Manager, Oracle, Inc.
Glenn Holmer
Web Development Lead, Weyco Group, Inc.
Thomas Kruse
Java Developer, JUG Muenster

Program Agenda
 NetBeans > About
 15 Lessons Learned
●
Problem
●
Solution
●
Proof
 Q & A

Graphic Section Divider

NetBeans > About
●
Integrated Development Environment
●
Rich Client Platform
●
Plugins
●
Community

NetBeans > About
●
almost 35 000 000 downloads
●
~4 000 000 lines of code
●
680 code contributors
●
since 1998
Number of code contributors
●
last issue ID: #236132

NetBeans Experience with Quality Control
You are a lucky bug. I'm seeing that you'll
be shipped with the next five releases.

●
Recurring problems with lack of time
for necessary stabilization at the
end of release cycles
●
Unplanned slips
# 1 – Quality is feature
Problem
Fig 1. Preview of new java.awt.RealStuffLayout in NetBeans 8.0 GUI Builder

Lesson
●
Add quality into feature set and keep that in mind when planning the release
●
Get support from developers

NetBeans 7.0
 quality didn't get enough
attention right from the start
 release date twice moved
 longest release ever (10
months)
July 2010:
“We're smart. Let's release on 20101213!”
October 2010:
“Well, if we send QA to Hawaii for a month
we could release on 20110208!”
November 2010:
“Hm, 20110308 is a nice release date
too, isn't it?”
NB 4.0 NB 5.5 NB 6.1 NB 6.8 NB 7.1
0
10
20
30
40
50
releaselengthinweeks

●
How to set the right Release Criteria?
●
How to determine whether the product
is ready for shipment?
# 2 - Release Criteria
Problem
Fig 2. NetBeans engineers shortly before going on vacation

Lesson
●
Listen to your customer
●
Find a metric matching satisfaction of the customer
●
Base Release Criteria on the metric
●
Reevaluate Release Criteria periodically

NetBeans 3.0 – 7.0
●
guessed overall bugs target
NetBeans 7.1 – 7.3
●
calculated bugs target considering:
●
historical data
●
expected size of changes
●
planned release dateNetBeans 7.4
●
set number of bugs for 10k LOC per areas
●
agreement from development

Notice:
 promising trend
 decreasing difference
 6.9 goal changed on the fly
 7.0 goal too aggressive
Recommendation index:
NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 NB 7.4
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Bugs target Opened bugs
NetBeans 7.0 86 %
NetBeans 7.1 91 %
NetBeans 7.2 94 %
NetBeans 7.3 95 %
numberofbugsatCodeFreeze

Lack of structured view on complex data
in issue tracking system:
●
bug trends
●
quality status of individual areas
●
actual distance from Release Criteria
●
contribution of individuals
# 3 – Metrics and dashboards
Problem
Fig 3. NetBeans managers in the NetBeans Release Control Center

Lesson
Create presentation layer on raw data from your issue tracking system and use it to
●
predict progress of stabilization
●
estimate whether Release Criteria is achievable

●
Relying only on absolute bug count
is like planning a bike trip on 2D
map. If you ignore hills, you will
never reach destination on time.
●
Meeting Release Criteria before
FCS does not mean anything!
# 4 – Income vs. total bug count
Problem
Fig 4. NetBeans bugs desperately trying to escape attention of NetBeans testers

Lesson
●
Income of bugs for brand new features is equally important as actual bug count
●
Having 10 open bugs at the end of every week however with income of 10 new
bugs daily means that product is far from stable
●
Release only if you get bug income under control (i.e. 50 % bug count = bug
income)

JavaScript Editor in
NetBeans 7.3:
 week #2: high income → Code
Freeze to slip by 2 weeks
 week #4: income below 40% →
desired quality reached
 week #7: income still under
control (33%)
1 2 3 4 5 6 7
0
20
40
60
80
100
120
140
Total Income
week
#ofbugsinJavaScriptEditor

●
Testing on artificial projects & use cases
●
Serious problems found after release
●
Not covered features and peaks
●
Subjective view on new features
# 5 – Engagement of end users
Problem
Fig 5. NetBeans developer fixing serious memory leak in High Resistance mode

Lesson
●
Engage real users of your product during its stabilization
●
Ask for concrete help and provide your support
●
Give appropriate acknowledgement

NetCAT 7.3:
 71 tested configurations
 116 reviewed tutorials
 456 bugs fixed in NetBeans 7.3
Time investment:
Members
Reported
defects
Productivity Fixed
NetCAT 7.3 112 1363 12,2 43 %
Community 1686 3273 1,9 32 %
147
1490
Invested hours
Gained hours

Low quality bug reports
●
have low fix rate
●
consume more time
# 6 – Quality of bug reports
Problem
Fig 6. Main telescopes near Prague for communication on NetBeans mailing lists

Lesson
●
Encourage users to follow bug reporting guidelines such as:
●
one problem = one bug
●
meaningful summary
●
detailed description
●
steps to reproduce
●
use attachments
●
Try to automate reporting of suitable kind of bugs

#151578 #213130
34months
Day #1: bug reportedDay #1: bug reported
Day #2: not reproducible,
more information required
Day #1: not reproducible,
more information required
Day #36: silence,
bug closed as invalid
Day #2: exact steps to
reproduce bug provided
Day #1073: reopened, exact steps
to reproduce provided
Day #2: reproduced & reassigned
Day #3: bug fixed
Day #4: bug verifiedDay #1081: reproduced &
bug fixed

●
Bug reports lack important information
●
Difficult & annoying bug submission
●
Subjectivity of reporters
●
Duplicated bugs slow down stabilization
# 7 – Automated bugs reporting
Problem
Fig 7. Two NetBeans engineers parking their vehicles in NetBeans garage

Lesson
●
Automate bugs reporting as much as possible
●
Send all relevant information as part of the automatic report
●
Automate postprocessing of the reports

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
0
5
10
15
20
25
30
35
Automated exception reports All exception reports
Notice:
 50 % increase in exception
reports since 2009
Comments:
 2006 – automated exception
reporting introduced
 2011 – automated exception
reporting further improved
%ofexceptionreportsinalldefects

Manual testing:
●
not suitable for complex matrix of
supported configurations
●
boring in longterm projects
Automated testing:
●
expensive maintenance
●
stability is never 100%
# 8 – Automated vs. manual testing
Problem
Fig 8. Building machines in full swing trying to compile NetBeans sources

Lesson
●
Find the right balance between manual and automated testing
●
NetBeans experience:
●
Define basic functionality as features used almost by everyone
●
Develop automated tests only for such basic functionality

NetBeans 3.5 NetBeans 7.3
Sources are compilable
Basic functionality always works
Length of full certification 13 weeks 5 weeks
NetBeans 3.6
 decision to invest into test
automation
 basic functionality in daily
builds guaranteed
 full certification shortened by
8 weeks

Typical problems of test specifications:
●
outofdate instructions
●
vague or not descriptive statements
●
wrong granularity/formatting
●
unknown execution time
# 9 – Quality of test specifications
Problem
Fig 9. Proposal of NetBeans 8.0 splash screen

Lesson
●
Assign each test specification its owner
●
Review and update test specifications on a release basis by other volunteers
●
Measure execution time regularly

NetBeans 6.9:
 testing of some areas behind
 execution time measured
NetBeans 7.0:
 priorities changed to cover
important areas primarily
 invest into test specifications
NetBeans 7.1 NetBeans 7.2 NetBeans 7.3 NetBeans 7.4
0
50
100
150
200
250
300
34
51
71
129
NetCAT team NetBeans QE team
Having quality test specifications pays off greatly
#oftestedconfigurations

●
How to find not tested code?
●
Do we have some dead code?
# 10 – Code coverage
Problem
Fig 10. Closeup on broken chip responsible for outages of NetBeans 7.0 daily builds

Lesson
●
Evaluation of code coverage results is very expensive
●
Use wisely or not at all
●
Cheaper way for finding not tested code is to review and update test specifications by
experienced users

NetBeans 6.0
 measurement and evaluation
of actions code coverage
took 5 months
 results too late for NetBeans
6.1
Tools to find dead code:
 FindBugs
 PMD
NetBeans 6.0 released NetBeans 6.1 released
Actions code coverage

●
Staff is a very expensive tool
●
They are lazy to wake up at 2:30 am
and test at least basic features in
that nice cold lab!
# 11 – Testing infrastructure and tools
Problem
Fig 11. Survival knife every NetBeans engineer is equipped with when leaving office

Lesson
●
Don’t reinvent the wheel
●
Evaluate existing frameworks and tools, do a cost analysis and implement what
suits your needs
●
Prefer open source solutions

NetBeans 3.5
 own tool but extremely expensive
 not maintainable after departure of its author
NetBeans 5.0
 attempt to stabilize the tool
 expensive, failed
NetBeans 5.5
 switch to a different tool
 worked for several years
NetBeans 6.0
 tool slowly on decline
NetBeans 7.3
 switch to open source tool
NetBeans 7.4
 satisfaction with the move lasts

Destabilized codebase after merge with
major feature developed in parallel on a
branch
# 12 – Pre-integration testing
Problem
Fig 12. Dedication of NetBeans test engineers is often admirable

Lesson
●
Pretest big changes before their integration to main repository

NetBeans 3.5
 first attempt to modernize
project infrastructure totally
failed
 codebase not compilable for 8
weeks
NetBeans 4.0
 new Ant based project system
preintegration tested
 no broken builds at all NetBeans 3.5 NetBeans 4.0
0
1000
2000
3000
4000
5000
6000
Projects bugs All bugs
Impact of 8 weeks of missing build on bugs income

●
Lack of intensive communication of
development towards quality
assurance
●
Even relatively small undocumented
change can have a big impact on
quality and influence testing
process
# 13 – Developers proximity
Problem
Fig 13. NetBeans test engineer surrounded by two NetBeans developers

Lesson
●
Close communication between development and quality assurance teams
especially about:
●
future plans
●
expected code changes
●
planned integrations into main branch
●
Frequency and intensity can vary case by case however AlwaysInformQEAnd
LetThemDecide approach is a good start which can be reduced later.

Java Projects
 54.000 lines of code
 developed and tested by
colleagues in the same office
 very stable functionality
GlassFish support
 56.000 lines of code
 always developed by distant
developer in different location
 problems graduated during
NetBeans 7.4 release
Java Projects GlassFish support
Lines of code 54 000 56 000
Changed lines of code 17 % 15 %
Income of bugs 68 121
Income of P1/P2 bugs 7 51
Normalized income of P1/P2 1.3 9.1
Income of bugs in NetBeans 7.4 (6 months)

●
You have never enough resources,
have you?
●
So, should highly exposed features
be tested more intensively or do all
features deserve equal attention?
●
Same question applies to new vs.
existing functionality
# 14 – Resource balancing
Problem
Fig 14. Wife of NetBeans web master explaining new family model to her husband

Lesson
●
No! Testing of new or heavily used features is a must of course. Surprised? :)
●
Rule of 3 steps:
●
test new or changed code
●
test unchanged but highly used code
●
test the rest of the code if you still have resources left or let it be

NetBeans 6.8
 1 tester for 2 developers
NetBeans 6.9
 1 tester for 8 developers
 still tried to full test everything
NetBeans 7.0
 the same approach didn't work
 gave up to test everything
 employed community to help cover the missing resources
NB 6.8
NB 6.9
NB 7.0
NB 7.1
NB 7.2
NB 7.3
NB 7.4
0
1000
2000
3000
4000
5000
NB 6.8
NB 6.9
NB 7.0
NB 7.1
NB 7.2
NB 7.3
40
50
60
70
80
satisfactionwithqualityin%#ofbugsatCodeFreeze

●
Serious bugs found in product
shortly after its release
●
How to eliminate them?
# 15 – Go/NoGo includes end users
Problem
Fig 15. NetBeans engineer eagerly completing appraisal of his manager

Lesson
●
Engage end users to decision process whether product is ready for release or not
●
Side effect: end users will then consider the product as “theirs” and help advertise it

Community Acceptance
 two weeks long survey
 to gather feedback on RC
 to identify potential stoppers
 to approve FCS
RC – Release Candidate
FCS – First Customer Ship
NB 6.5 NB 6.7 NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3
0
5
10
15
20
25
Bugs nominated as stoppers in CA survey
numberofpotentialstoppers

Questions & Answers

●
How to overcome operational
blindness?
●
How to control quality before
Feature Freeze?
# 16 – Bonus lessons
Problem
Fig 16. Incident which convinced NetBeans team to change their parking system

Lesson – Operational Blindness
Make the work more diverse
●
different types of tests
●
automated testing
●
tools development / maintenance
●
shift responsibilities
Motivate testers suitably
●
give them responsibility
●
reward best contributors (gifts, publicity, credit)

Lesson – Quality before Feature Freeze
What did not work:
●
milestones driven development
●
4+1 cycles
What worked:
●
have a good buffer in your plan
●
quality (not date) driven releases
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
0
100
200
300
400
500
600
700
800
NetBeans 6.7 NetBeans 7.3
M1 M2 M3
numberofreportedbugs
No effect of 4+1 development on overall quality

Insert Picture Here
Thank you!
Jiří Kovalský
NetBeans Community Manager at Oracle, Inc.
Glenn Holmer
Web Development Lead at Weyco Group, Inc.
Thomas Kruse
Java Developer, JUG Muenster

Bof4162 kovalsky

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Bof4162 kovalsky

Similar a Bof4162 kovalsky (20)

Más de Geertjan Wielenga

Más de Geertjan Wielenga (20)

Último

Último (20)

Bof4162 kovalsky