SlideShare una empresa de Scribd logo
1 de 51
Descargar para leer sin conexión
Queues: An Invisible
Money Drain
Phil Sarin
VP Engineering, GameChanger Media
@philsarin
http://216ways.net
“It ain’t what you don’t know that gets you into
trouble. It’s what you know for sure that just ain’t so.
— Mark Twain
Things I once believed
You shouldn’t release often because releases have high overhead.
You need to track all of your bugs.
Slack is bad. Your team should be busy.
Specialists will always be more productive than generalists.
The Biggest Misunderstanding:
We think queues are free.
What we’ll cover tonight
What a queue is
Identifying the queues in your organization
Tactics for handling queues
Some war stories
Part 1: Queues and why
they matter
Flickr: Jeff Turner
“Queues are the root
cause of the majority of
economic waste in product
development.”
— Donald G. Reinertsen
Examples of queues
Feature Queues
Waiting to get designed
Waiting to get developed
Waiting for testing
Waiting to get released
Waiting to get validated
Bug Queues
Waiting for triage
Waiting to get fixed
Waiting to get verified
Waiting for release
What’s wrong with big queues?
Big queues increase cycle time and delay cost
Big queues increase variability
Big queues require management
Big queues delay feedback
Big queues hurt morale
The conservative team
vs
The aggressive team
Variability
Flickr: State Farm
Flickr: State Farm
Flickr: State Farm
Flickr: State Farm
Batch Sizes
Flickr: kbeil
1 year
3 months
vs
3 months 3 months 3 months
vs
Why small batches are (usually) better
Realize benefits sooner
Faster feedback
Higher motivation
Less schedule slippage
Old School: Releases are expensive so let’s batch up work for
release.
New School: Let’s drive down the cost of releases so that we can
release more often.
Queue takeaways
Queues introduce delay.
Queues have economic and psychological costs.
High capacity utilization drives up queue size.
Small batches can reduce the cost of queues.
Part 2: How we apply this
theory at GameChanger
Flickr: Jeff Turner
How we’ve managed queues
1. Swarm
2. Generalize queue handling
3. Obliterate queues
4. Reduce testing batch sizes
1. Swarm
Flickr: State Farm
The stages of swarming
1. Skepticism/suspension of disbelief
2. Guilt and busywork
3. Scope blows up
4. The “surge capacity effect”
Announcements
Invite Tools
Alerts
Announcements Invite Tools Alerts
vs
No silver bullet
2. Generalize
Flickr: Kaleb Fulgham
Multiple queues,
1 server each
Single queue,
Single server
Single queue,
pool of servers
Multiple queues,
1 server each
Single queue,
Single server
Single queue,
pool of servers
Sustained huge queues
are least likely
Pressures to specialize
A specialty differentiates your business
“I don’t want to touch that code!”
Initial builder owns it forever.
Early adopters become de-facto specialists.
Some people want specialist careers.
Resisting specialist queues
Outsource non-core specialties
Despecialize
Challenge someone
Staff up
Generalist teams and morale
3. Obliterate queues
http://provocateurs.ca/2014/11/16/delmar-dont-be-afraid-to-hit-delete/
4. Reduce testing batch sizes
That time we missed our
deadline by four months
2 months development
2 weeks
testing
2 months development
2 weeks
testing
4 months bug-driven development
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
Can we reduce this batch size further?
Our first batch size reduction
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
2 weeks
dev
test
Can we reduce this batch size further?
2 weeks
dev+test
T
2 weeks
dev+test
T
2 weeks
dev+test
T
2 weeks
dev+test
T
2 weeks
dev+test
T
2 weeks
dev+test
T
Continuous pre-release delivery?
It didn’t work
Nobody
used
these
How we fixed it
More automated test coverage
Instituted small batch manual testing process
Hired more testers
Release Days
5.15 21
5.14 14
5.13 26
5.12 21
5.11 23
5.10 4
5.9 66
5.8 7
5.6 21
It worked, mostly, eventually
Parting Thoughts
Flickr: Jeff Turner
Things I once believed
You shouldn’t release often because releases have high overhead.
You need to track all of your bugs.
Slack is bad. Your team should be busy.
Specialists will always be more productive than generalists.
Mindset shifts
Being busy vs being productive
Individual vs team
Rituals vs principles
Takeaways
Queues are real and expensive
You should know where your queues are
There are a bunch of ways to reduce the cost of queues
Further Reading
The Principles of Product Development Flow, Donald G. Reinertsen
“Software Inventory” by Joel Spolsky: http://www.joelonsoftware.com/
items/2012/07/09.html
“How we fixed more bugs by deleting our bug DB” and “Building
around generalists” on my blog (216ways.net)

Más contenido relacionado

Destacado

Lean Simulation Activity Ideas using a ream of copy paper
Lean Simulation Activity Ideas using a ream of copy paperLean Simulation Activity Ideas using a ream of copy paper
Lean Simulation Activity Ideas using a ream of copy paper
Tom Curtis
 

Destacado (14)

FACES: questions for effective visual boards
FACES: questions for effective visual boardsFACES: questions for effective visual boards
FACES: questions for effective visual boards
 
Effective Teams
Effective TeamsEffective Teams
Effective Teams
 
Attack Sources of Variability to Improve Predictability
Attack Sources of Variability to Improve PredictabilityAttack Sources of Variability to Improve Predictability
Attack Sources of Variability to Improve Predictability
 
Lean UX Workshop
Lean UX WorkshopLean UX Workshop
Lean UX Workshop
 
Multitaskers Anonymous
Multitaskers AnonymousMultitaskers Anonymous
Multitaskers Anonymous
 
Checklist Checklist
Checklist ChecklistChecklist Checklist
Checklist Checklist
 
Consumer Behavior Ads
Consumer Behavior AdsConsumer Behavior Ads
Consumer Behavior Ads
 
Lean Simulation Activity Ideas using a ream of copy paper
Lean Simulation Activity Ideas using a ream of copy paperLean Simulation Activity Ideas using a ream of copy paper
Lean Simulation Activity Ideas using a ream of copy paper
 
Design Thinking and the Facilitation Process
Design Thinking and the Facilitation ProcessDesign Thinking and the Facilitation Process
Design Thinking and the Facilitation Process
 
Strategic UX Workshop - UX Bristol
Strategic UX Workshop - UX BristolStrategic UX Workshop - UX Bristol
Strategic UX Workshop - UX Bristol
 
Efficient or Just Busy?
Efficient or Just Busy?Efficient or Just Busy?
Efficient or Just Busy?
 
Lean Lego Game
Lean Lego GameLean Lego Game
Lean Lego Game
 
A Workplace with No Managers
A Workplace with No ManagersA Workplace with No Managers
A Workplace with No Managers
 
Healthcare Napkins All
Healthcare Napkins AllHealthcare Napkins All
Healthcare Napkins All
 

Similar a Queues: An Invisible Money Drain

ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
Aberla
 
Bringing Science to Software Development
Bringing Science to Software DevelopmentBringing Science to Software Development
Bringing Science to Software Development
Arty Starr
 

Similar a Queues: An Invisible Money Drain (20)

Discovering your inner tester
Discovering your inner testerDiscovering your inner tester
Discovering your inner tester
 
Exploratory testing workshop
Exploratory testing workshopExploratory testing workshop
Exploratory testing workshop
 
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010
 
Mini-Training: Using root-cause analysis for problem management
Mini-Training: Using root-cause analysis for problem managementMini-Training: Using root-cause analysis for problem management
Mini-Training: Using root-cause analysis for problem management
 
How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014How To (Not) Open Source - Javazone, Oslo 2014
How To (Not) Open Source - Javazone, Oslo 2014
 
Improving your Agile Process
Improving your Agile ProcessImproving your Agile Process
Improving your Agile Process
 
Things Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression TestingThings Could Get Worse: Ideas About Regression Testing
Things Could Get Worse: Ideas About Regression Testing
 
Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing
Pushing the Bottleneck: Predicting and Addressing the Next, Next ThingPushing the Bottleneck: Predicting and Addressing the Next, Next Thing
Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing
 
Slack's Ali Rayl on Scaling Support for User Growth
Slack's Ali Rayl on Scaling Support for User GrowthSlack's Ali Rayl on Scaling Support for User Growth
Slack's Ali Rayl on Scaling Support for User Growth
 
Selenium Users Anonymous
Selenium Users AnonymousSelenium Users Anonymous
Selenium Users Anonymous
 
Improve Testing with a Zone Defense
Improve Testing with a Zone DefenseImprove Testing with a Zone Defense
Improve Testing with a Zone Defense
 
Get Faster - While You're Getting Better
Get Faster - While You're Getting BetterGet Faster - While You're Getting Better
Get Faster - While You're Getting Better
 
DevOps Game at SGZA
DevOps Game at SGZADevOps Game at SGZA
DevOps Game at SGZA
 
Testers in an agile world
Testers in an agile worldTesters in an agile world
Testers in an agile world
 
Discount Usability Testing for Agile Teams
Discount Usability Testing for Agile TeamsDiscount Usability Testing for Agile Teams
Discount Usability Testing for Agile Teams
 
ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
ESEconf2011 - Caine Matthew: "Creating an Environment of Teamwork, Quality, I...
 
Devops at scale is a hard problem challenges, insights and lessons learned
Devops at scale is a hard problem  challenges, insights and lessons learnedDevops at scale is a hard problem  challenges, insights and lessons learned
Devops at scale is a hard problem challenges, insights and lessons learned
 
Requirementless testing
Requirementless testingRequirementless testing
Requirementless testing
 
Root Cause Analysis
Root Cause AnalysisRoot Cause Analysis
Root Cause Analysis
 
Bringing Science to Software Development
Bringing Science to Software DevelopmentBringing Science to Software Development
Bringing Science to Software Development
 

Último

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

Último (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 

Queues: An Invisible Money Drain