My presentation at Agile-Scrum International Summit, Bangalore on a 'Kanban' implementation we did in 2004-05.

1. www.scrumindia.in
Applying ‘Kanban’ in Enterprise-Class
Products Sustaining Engineering
~ An Experience Report ~
Tathagat Varma
@tathagatvarma
http://managewell.net
http://slideshare.net/managewell
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
1
Name: TV

2. www.scrumindia.in
Discussion Topics
• Enterprise-class Software Products
• Our Product
• Old Process
• Problems with Old Process
• New Process
• Did this move the needle?
• What is Kanban?
• Kanban in Software Engineering
• What did we learn?
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
2
Name: TV

3. www.scrumindia.in
What characterizes Enterprise Software?
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
3
Name: TV

4. www.scrumindia.in
Microsoft Windows timeline
https://en.wikipedia.org/wiki/Timeline_of_Microsoft_Windows
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
4
Name: TV

5. www.scrumindia.in
Other major OS and tools timeline
https://en.wikipedia.org/wiki/Solaris_(operating_system) https://en.wikipedia.org/wiki/Bugzilla
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
5
Name: TV

6. www.scrumindia.in
As a contrast, what are consumer internet companies doing?
• Continuous Integration -> Continuous Delivery ->
Continuous Deployment
• On ‘good days’, Flickr releases a new version every half
an hour (Jun 20, 2005)
• IMVU pushes a revision of code to the website every nine
minutes (Feb 10, 2009)
• The other day we passed product release number 25,000
for WordPress. That means we’ve averaged about 16
product releases a day, every day for the last four and a
half years! (May 19, 2010)
• A new version of Google Chrome now due every six
weeks (Jul 22, 2010)
• Facebook does code push twice a day (Aug 4, 2012)
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
6
Name: TV

7. www.scrumindia.in
Desktop OS Market share (Nov 2012)
http://marketshare.hitslink.com/operating-system-market-share.aspx?qprid=10&qpcustomd=1
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
7
Name: TV

8. www.scrumindia.in
Our Product
• Network Management domain
• Windows-based specialized hardware (“Appliances”)
• Installed in data centers for traffic monitoring,
analysis and network troubleshooting
– but not generally on production network
• Typical users are technical folks – CIO, Network
Manager, Network Engineers
• Selling cycles typically align with quarterly or annual
budget cycles
• Many sales require implementing customer specials
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
8
Name: TV

9. www.scrumindia.in
Old Process, circa 2003
• Customer Bugs prioritized based on multiple business
parameters, including (partial list) -
– Severity
– Impact on Revenue, Volume, Competitive, etc.
– Case age
• PMO would prepare Maintenance Release Plan of Record (MR
POR) and get buy-in for various types of MRs -
– Service Packs – bunch up ~50-60 bugs typically every quarter
– Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP
– Patches – workaround for customer-specific issues
• SPs would have
– Above The Line (ATL) requirements – must fix
– Below The Line (BTL) requirements – fix if time permits
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
9
Name: TV

10. www.scrumindia.in
A typical support timeline
http://itconvergence.blogspot.in/2012/10/oow-12-elison-extradata-oracle-r122.html
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
10
Name: TV

11. www.scrumindia.in
Problems with Old Process
• Dev team had no bandwidth to take on maintenance releases
• Huge pile of customer escalations without “home”
• Compounded by high incoming field rate
• Low closure rate (largely due to no dedicated resources)
• Large wait for customers to get bug fixes
• Tech Support often tasked team directly and broke the process
• Hot fixes not always available to all customers
• Sometimes, a new bug fix might break a hot fix
• If a hot fix failed in the field, rollbacks would be very difficult
• Difficult to estimate time to resolve a bug and give an ETA
• High-priority bugs could arrive at any time
• Customer specials could arrive anytime with top priority
• High internal rejection rate of bug fixes by Tech Support
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
11
Name: TV

12. www.scrumindia.in
New Process, 2004-05
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
12
Name: TV

13. www.scrumindia.in
Our Kanban Process in action…
Dev Team = 15
Tech Support
GUI
GUI
PMO
Product A
Product A
W
IP
=
Queue = 0
3 Backend
Backend WIP = 1wk
W
WIP = 15 Product B
Product B
IP
=
QA Team
QA Team
4
Queue = 0
WIP = 2
Product C
Product C
CST Manager
Product D
Product D WIP = 1wk
3
=
Protocols
Protocols
IP
W
3
=
IP
W
Decide Drivers
Decide Drivers
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
13
Name: TV

14. www.scrumindia.in
Process improvement…the beginnings…
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
14
Name: TV

15. www.scrumindia.in
Cumulative Hot Fix Process
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
15
Name: TV

16. www.scrumindia.in
Weekly Build Process
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
16
Name: TV

17. www.scrumindia.in
So, what is happening?
• Though not an originally stated vision or goal, the “Work in
Progress” (WIP) is being limited to # of team members
• At any time, one developer is assigned only one piece of
work, thereby achieving “One-Piece Flow”
• New work is only assigned when current work is
completed (or cancelled/stalled), and a team member is
available
• No wait state or switching costs at an individual level
• Smaller lead time for bugs (in contract to lead time for SP)
• The process is allowing ‘continuous deployment’ of each
of the hot fixes – even though it is only being practiced in a
limited manner
• Finally, the flexibility gained is not a zero-sum game – there
is no penalty on performance in rest of the process
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
17
Name: TV

18. www.scrumindia.in
Did this move the needle?
• Bugs addressed each quarter
• Quality of bug fixes
• “Homes” for bugs
• Total bugs open
• Open days open
• People motivation
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
18
Name: TV

19. www.scrumindia.in
Shift from SPs to Cumulative Hot Fixes while maintaining High Quality
Maintenance Releases (Service Packs, Patches, Hot Fixes)
Q3 2003 Through Q4 2006 (Fiscal Year)
100 100
96 97 96
70 66 94 100
92 92 93
91
88
87
85 90
60 80
Percentage of released 80
Maintenance Releases
50 (Service Packs,
Patches, Hots Fixes) 70
that addressed
customer reported
Total Maintenance 60
40
Releases (Service
Packs, Patches, Hot
62 32 50
Fixes) for this
30 30
Hot Fixes quarter. 28
30 27
26
25 25 25 40
Patches 22
Service 30
20
Packs 15 28
16 27
12 18 28
20 26
26 24 20
21
10 7 11
8
2 5 10
3
1 0
2 1 1
5 4 4 4 4 0 1 0 0
2 3 2 2 2 0 2 3 0
0 0 1 1 0
Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 06
Service Packs Patches Hot Fixes % Successful
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
19
Name: TV

20. www.scrumindia.in
Increase in bugs with “homes”
Customer Direct and Indirect % w/Homes - Historical
Week Ending February 03, 2006
100
Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)
90
87
80 82 82 83 82 81 80 82 80
75 76 76 75 75 76 78 76
72 74 72
71 72 73 71 75
70
67 68 69 68 68 69 69 71
70 69 68
63 65 64
60 59 60 61 61 60 61
55 54 55 52 55 55 54 56
54 54 53 55 55 56
50 51 51 50 51 51 51 50 49 53 52 51 51 52
51 51 53
48
47 48 48 48 48 48
46 45 47 44 45 45 44 44 47 47 47 47 46
45 44 43 42 43
40 39 38 41 41 42
40 40
35
30 31
27 27 25
20 21
18 16
15
10 11
0
5
5
5
8/ 5
5
9/ 5
5
6
06
10 1 /2 5
8 5
11 8 /2 5
5 5
/2 05
12 3 /2 5
0/ 5
6/ 5
20 05
27 05
3/ 5
17 05
24 05
1/ 5
22 05
29 05
5/ 5
19 05
26 05
2/ 5
23 05
30 05
10 7/ 2 5
10 4 /2 5
11 4/ 2 5
11 1 /2 5
12 9/ 2 5
12 6 /2 5
20 06
27 06
3/ 6
13 0
10 0
7/ 00
15 0
12 0
9/ 00
16 0
13 0
7/ 00
8/ 00
9/ 00
/ 0
/1 00
/2 00
/2 00
/ 0
/1 00
/1 00
/2 00
/ 0
/1 00
/2 00
/3 00
1/ 00
2/ 00
6/ 00
5/ /2 0
6/ 2 0
7/ 2 0
8/ 2 0
9/ 2 0
1/ 2 0
20
5/ / 20
5/ / 20
6/ / 20
6/ / 20
7/ / 20
7/ / 20
8/ / 20
8/ / 20
9/ / 20
9/ / 20
10 / 20
11 /20
12 /20
12 / 20
1/ / 20
1/ / 20
2
2
2
/2
/2
/2
/2
/2
6
5/
Direct Indirect Goal (80% w/Homes) Combined
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
20
Name: TV

21. www.scrumindia.in
Total Bugs Open
Severity Across Products (Historical)
Week Ending March 31, 2006
130 260
243
236
120 230 240
222 224
110 213
21 21
0 1 220
201
199 1 1 1
969795
100 191 200
90 180
161
80 160
1 1 1 1
47464646 1 1
4850
70 132 131 140
127 1 1
2725
122 121
60 11
1 120
50 100
40 747575 7473 80
71 69707069
65 66676768 6866 68
6058 6264 60
63 6160
57 57 5555
30 53 52 51 60
4947 50494950 4749 49
46 44
20 3538 33 343536 40
32 32
10 20
0 0
01
29
21
06
27
17
08 -0 5
29 05
19 05
21
13
31
19 - 04
10
31
11 -0 5
04
25
15
09
30
11
02
23
03 -0 6
24
17
-O 3
-O
-J
-M
-M 5
-J
-J
-O 5
-J
-J
-A
-D
-N
-D
-D 4
-F
-M 5
-M
-A 5
-S 5
-S
-N
-D 5
-D
-F
-F
-M 6
an
un
an
ul
ul
ug
eb
pr
ep
ep
eb
eb
ay
ct
ct
ay
ct
ec
ec
ec
ov
ec
ov
ec
ar
ar
ar
-
-
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
4
5
5
4
5
5
4
5
5
6
6
5
Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
21
Name: TV

22. www.scrumindia.in
Average Days Open
Severity: Average Days By Severity (Historical)
Week Ending March 31, 2006
299
500 300
475 27575
2
268 275
450 260
425 248
244 24646
2
242 244 244 250
400 23435 232
2 232
226
223
375 225
210
207
350 203 201
200
325 1 1
8382
179 180 1 1
7779
300 172
168
165 1 63 1 1
6366 1 1 1
646265 175
162
275 155 156 156 155 1 1 1
575656 1 1 1
545557 154
1 52 152 149 149
147 148 45 148
250 1 42 1 43 1 1 44474644
1 1 1 1 144 1 1
4447 146
150
139 40 140 141
136 1 35 135 133 1 1
3434
132
225 123 123
17
1 125
200
175
100
150
125 75
100
50
75
50
25
25
0 0
01
29
21
06
27
17
08 -0 5
29 05
19 05
21
13
31
19 -04
10
31
11 -0 5
04
25
15
09
30
11
02
23
03 -0 6
24
17
-O 3
-O
-J
-M
-M 5
-J
-J
-A
-O 5
-J
-J
-D
-N
-D
-D 4
-F
-M 5
-M
-A 5
-S
-N
-D 5
-D
-F
-F
-M 6
-S 5
an
un
ul
ul
an
ug
eb
pr
ep
eb
ep
eb
ct
ct
ay
ay
ct
ec
ov
ec
ec
ov
ec
ec
ar
ar
ar
-
-
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
-0
4
5
5
4
4
5
5
5
5
6
6
5
Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
22
Name: TV

23. www.scrumindia.in
People motivation
• Started with 16 people dev team
• We had zero attrition in the team
• Once the backlog started coming down, engineers were
ramped off the team to do new features
• Eventually dismantled the team and rolled-up engineers into
dev teams when backlog came down to single digits
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
23
Name: TV

24. www.scrumindia.in
What is Kanban?
• Kanban (literally signboard or billboard) is a scheduling system
for lean and just-in-time (JIT) production. According to its
creator, Taiichi Ohno, kanban is one means through which
JIT is achieved.
• Kanban is not an inventory control system; it is a scheduling
system that helps determine what to produce, when to
produce it, and how much to produce.
• The need to maintain a high rate of improvement
led Toyota to devise the kanban system. Kanban became an
effective tool to support the running of the production
system as a whole.
• In addition, it proved to be an excellent way for promoting
improvements because reducing the number of kanban in
circulation highlighted problem areas.
https://en.wikipedia.org/wiki/Kanban
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
24
Name: TV

25. www.scrumindia.in
A Kanban System at my Toyota dealership
https://twitpic.com/het3u
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
25
Name: TV

26. www.scrumindia.in
How does it work?
http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
26
Name: TV

27. www.scrumindia.in
How Kanban helps achieve “Just-in-Time”
• For example, to efficiently produce a large number
of automobiles, which can consist of around 30,000
parts, it is necessary to create a detailed production
plan that includes parts procurement. Supplying
"what is needed, when it is needed, and in the
amount needed" according to this production plan
can eliminate waste, inconsistencies, and
unreasonable requirements, resulting in improved
productivity.
http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
27
Name: TV

28. www.scrumindia.in
Kanban for Software
• Visualize the Workflow: Represent the work items
and the workflow on a card wall or electronic board
• Limit Work-in-Progress (WIP): Set agreed upon
limits on how many work items are in progress at a
time
• Measure and Manage Flow: Track work items to
see if they are proceeding at a steady, even pace
• Make Process Policies Explicit: Agree upon and
post policies about how work will be handled
• Use Models to Evaluate Improvement
Opportunities: Adapt the process using ideas from
Systems Thinking, Deming, etc.
Kanban: Successful Evolutionary Change for your Technology Business – David Anderson
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
28
Name: TV

29. www.scrumindia.in
Why Kanban in Software Engineering?
https://leanandkanban.files.wordpress.com/2009/04/kanban-for-software-engineering-apr-242.pdf
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
29
Name: TV

30. www.scrumindia.in
What did We learn?
• Process improvement should be
driven by business needs – and
NOT because some process looks sexy!
• Don’t let a process limit your potential –
think beyond gurus!
• Don’t let absence of a process limit your
potential – do whatever it takes to
serve customer better!
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
30
Name: TV

31. www.scrumindia.in
Thanks guys!
• To the wonderful Customer Sustaining Team (Dev and QA),
Program Management and Tech Support at Network
Associates / Network General in Bangalore and San Jose for
some great stuff !! 
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
31
Name: TV

32. www.scrumindia.in
Resources
• Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and
http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166
• http://refcardz.dzone.com/refcardz/getting-started-kanban
• Ship early and ship twice as often, https://www.facebook.com/notes/facebook-
engineering/ship-early-and-ship-twice-as-often/10150985860363920
• How we built Flickr,
http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/
• Continuous Deployment at IMVU: Doing the impossible fifty times a day,
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-
times-a-day/
• A New Version of Google Chrome now due every six weeks,
http://techcrunch.com/2010/07/22/google-chrome-versions/
• In praise of continuous deployment: The WordPress.com story,
http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/
• CONWIP, https://en.wikipedia.org/wiki/CONWIP
• Kanban applied to Software Development: from Agile to Lean,
http://www.infoq.com/articles/hiranabe-lean-agile-kanban
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
32
Name: TV

33. www.scrumindia.in
Q&A
Questions and Answers?
Date: 7-9 Dec 2012 ScrumIndia.In Proprietary Information
33
Name: TV