2010 GTAC Crowd Source Testing Mozilla Style

Crowd Source Testing
Mozilla Community Style
Matt Evans
Mozilla QA Director
Wednesday, November 10, 2010

about:talk
• Background and Scale of Mozilla
• Crowdsourcing: Crowd vs. Community
• History of Mozilla Community testing
• How Mozilla succeeds with Community
testing

• I have been in the testing biz for most my career
• 9 mos as QA Director @ Mozilla
• Manage a team of ~30 Global Test Engineers
about:me

about:mozilla

about:mozilla
Mozilla’s Mission:
is to promote openness,
innovation, and opportunity
on the web

I’m still getting my head around
scale at Mozilla
• The scale of information ﬂow
• The scale of the user base

Mozilla Firehose

Mozilla Firehose
• Many sources of information
• bugzilla (200+bugs/day)
• forums (50+)
• blogs (50+)
• tweets (100s/day)
• wikis (1000s pages)
• irc (50+ channels)

Mozilla Scale
400+
Developers
10,000+
Nightly Testers
2,500,000
Beta Testers
400,000,000
Users
400,000,000
350,000,000
300,000,000
250,000,000
200,000,000
150,000,000
100,000,000
50,000,0000

Story Time

Big decision: Ship or
No-Ship to Beta
A Recent Firefox product
team meeting

Protector of the tester
community

Get feedback from
tester community

We were both wrong
right!

It was a big change for
my perspective

Respecter of the Tester
Community as QA
Team Partner

We now work much
better together

Crowd Sourcing: It involves a crowd
working towards a common goal

What is new, the virtual crowd is
enabled by internet technologies

• Building a global community of testers ~30k+
• Stewardship of the test community is their
secret sauce
Community approach to
software testing

Mozilla Labs Crowdsource
Crowdsourcing Project

Key Findings
• there are two types users: heavy
users and casual users.
• 90-9-1 Principle
• crowd motivation is separated into
two forms: intrinsic and extrinsic
• crowdsourcing relies on people, and
thus works when participants are
treated as people.
Mozilla Labs Crowdsource
Crowdsourcing Project

VS
Community Crowd

Mozilla uses a hybrid of crowd based
feedback mechanisms and community
management
“Crowd sourced
testing Mozilla Style”

The History of Mozilla
Testing Community

Mozilla.org is
launched
by Netscape
1998 2002
Mozilla 1.0
is released
Phoenix 0.1
released
2004
Firefox 1.0
released
2005-2009 2010
Firefox
1.5-3.5
released
Firefox 4.0
expected
release
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3

1998 2002 2004 2005-2009 2010
1 2 3
1998 Netscape open sourced the Netscape
browser source code.

History of Mozilla’s Tester Community
•A lot of passionate people without a development
background wanted to get involved.
Asa Dotzler
First Mozilla Community Tester

The Rise of the
Nightly Build

•A great success
•The community of testers
grew to the 1000s

Too much of good thing

•As tester community
members increased so did
the noise

1998 2002 2004 2005-2009 2010
1 2 3
• 2002+
• The rise of Firefox code base
• Split from AOL/Netscape
• 2003 Mozilla Foundation is formed

1998 2002 2004 2005-2009 2010
2

1998 2002 2004 2005-2009 2010
1 2 3
• 2007+
• The rise of Continuous Integration
and unit tests

Continuous Integration
~3.3 million automated tests
executed for every committed
code change
•The largest CI system built
on Buildbot and Tinderbox
technologies
•For every code change
•11 full builds of the
browser across
supported platforms
•~300k unit/perf tests per
build

Mineﬁeld is the result of
Continuous Integration
Continuous Integration and Unit Testing
is the Foundation of Today’s Mozilla
Community Testing Success

Community Feedback

Mozilla Runs On
Bugzilla

Crash Data Analysis

Crash Data Analysis
5 years of crash data
FF1.5 FF2.0 FF3.0 FF3.5
Filed
Fixed
Dups

Firefox 4 Beta
Feedback

Firefox 4 Beta
Feedback
LCD Screens of Feedback and Twitter mentions are
displayed throughout the Mozilla Ofﬁce

Engaging the Mozilla
Tester Community

Manual Testing
Litmus - Test Case Management

Test and Bug Days

Mozmill Test Automation Framework
Community Focused
Automated Testing
• Javascript framework for
testing gecko based
applications
• Includes an IDE
• Installs as an Firefox
extension
• Better suited for
community contribution

Mozilla Community Based Automated Testing

Overview of How we succeed
with Community Testing
• Continuous Integration is the foundation of
today’s community testing
• Utilizing Community Feedback
• Engage in Manual Testing efforts
• Engage Community focused automated
testing in the near future

Community
Stewardship

Community Stewardship

Community Stewardship
quality.mozilla.org

Reaching Out to the Community
quality.mozilla.org
quality.mozilla.org

A call out to you!

Thank you
Questions?

2010 GTAC Crowd Source Testing Mozilla Style

Recomendados

Recomendados

Más contenido relacionado

Similar a 2010 GTAC Crowd Source Testing Mozilla Style

Similar a 2010 GTAC Crowd Source Testing Mozilla Style (20)

2010 GTAC Crowd Source Testing Mozilla Style