Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Monte Carlo presentation for FM
Similar a Monte Carlo presentation for FM (20)
Último
Último (20)
Monte Carlo presentation for FM
- 1. Evening the Odds: Monte Carlo Forecasting Hunter Tammaro / @htammaro excella.com | @excellaco
- 2. excella.com | @excellaco How do you forecast today?
- 3. excella.com | @excellaco Whydo you make forecasts?
- 4. excella.com | @excellaco Setting the Stage
- 5. excella.com | @excellaco GettingStarted • Team funded for one year • Only high-level, long-term goals • Built a“releasable” product over a few months • Started to improve ad-hoc,based on feedback
- 6. excella.com | @excellaco GettingFinished • Question of when to move on • What does it mean to be “complete?” • When willit happen? • How sure can we be?
- 7. excella.com | @excellaco Forecasting with (or without) MonteCarlo
- 8. excella.com | @excellaco • Estimate allbacklog items • Add up to get the total backlog size • Divide by average velocity • That’syour release date! • Thatisyour release date, right? Forecasting with Averages
- 9. excella.com | @excellaco • How long did forecasting take? • How accurate is the forecast? • How sure are you? • What happens if you miss the forecast? BUT…
- 10. excella.com | @excellaco Determine a range of future outcomes and their probabilities based onrepeated, random sampling of existingdata Enter the Monte Carlo method
- 12. excella.com | @excellaco Acknowledgeand estimateuncertaintyin theforecast Weunderstandthisimplicitlyforother types of predictions! Embracing Variability
- 16. excella.com | @excellaco Creating a Monte Carlo forecast
- 17. excella.com | @excellaco Decide whatto forecast • Fixed backlog orfixed date? • Entire product or smaller project? • What arethe work items in the backlog?
- 18. excella.com | @excellaco Gather Data • Measure the variability in what you’re trying topredict • Takttimes –the time between completed PBIs • Takt accounts foractual output of the whole team
- 19. excella.com | @excellaco Gather Data For software teams, measure takt in weekdays between completed items M T W T F M T W S S 3 days 1 day 0 days 1 day 2 days
- 20. excella.com | @excellaco Run one simulation Randomly choose takttimes for each item in your backlog Add up the total time PBI Takt 1 2 1 3 3 4 0 5 1 6 2 PBI Takt 7 8 9 10 11 12 Total Measured Simulated 3 2 1 0 1 3 10 days
- 21. excella.com | @excellaco Run a lot more simulations Repeat a few dozen times… PBI Takt 7 1 8 0 9 0 10 3 11 0 12 2 Total 6 PBI Takt 7 3 8 2 9 1 10 3 11 3 12 3 Total 15 PBI Takt 7 1 8 3 9 3 10 1 11 3 12 0 Total 11 PBI Takt 7 3 8 3 9 3 10 3 11 1 12 0 Total 13 PBI Takt 7 3 PBI Takt 7 1 PBI Takt 7 2 PBI Takt 7 2
- 22. excella.com | @excellaco Results Count the number of simulations of a given duration Determine cumulative percentages Days N Runs Running % 0-4 4 4% 5-6 24 28% 7-8 27 55% 9-10 24 79% 11-12 15 94% 13-14 5 99% 15+ 1 100%
- 23. excella.com | @excellaco And graph it 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 10 15 20 25 30 0-4 5-6 7-8 9-10 11-12 13-14 15+ Days N Runs Cumulative %
- 24. excella.com | @excellaco Let a computer do it!
- 25. excella.com | @excellaco Number of takt inputs Simulations, 10 takts Simulations, 30 takts
- 26. excella.com | @excellaco Range of takt inputs Simulations, narrow range Simulations, broad range
- 27. excella.com | @excellaco Waiting for sprint end Simulations, continous delivery Simulations, delivery at sprints
- 28. excella.com | @excellaco Adding an outlier Simulations, normal range Simulations, with outlier
- 29. excella.com | @excellaco How we used the forecast
- 30. excella.com | @excellaco • Understand how sure we are that we’ll finish on time • Explicitly acknowledges risk • Tailor the forecast you communicate to your audience Uncertainty
- 31. excella.com | @excellaco Release planning Gave us the information we needed to plan a release Made the significance of good prioritization more obvious
- 32. excella.com | @excellaco Pace setting • Understand impact of scope changes throughout the project • Gave us a better feel for our status than “red/yellow/green” • Easy to update once created
- 33. excella.com | @excellaco Tracking over time 10/27/18 12/16/18 2/4/19 3/26/19 5/15/19 7/4/19 8/23/19 8/18/18 9/7/18 9/27/18 10/17/18 11/6/18 11/26/18 12/16/18 1/5/19 forecast complete date date of simulation forecast evolution 50% confidence 80% confidence 95% confidence
- 34. excella.com | @excellaco Tracking over time
- 35. excella.com | @excellaco Limitations and how to work with them
- 36. excella.com | @excellaco The usual suspects Planning is guessing • Don’t mistake math for certainty • Even if you’re 80% sure, you’re wrong 1 in 5 times You need historical data • Take a few weeks to track if you need to • A “gut” estimate may get you over this hump
- 37. excella.com | @excellaco The past dictates the future Will the team have the same capacity? • Keep the team clear of impediments • Keep team membership consistent Will work items be the same size? • Keep an eye on story splitting and estimation • Discuss when work items are unusually large
- 38. excella.com | @excellaco What are we building? Does the backlog we’re simulating adequately represent the project? • Build in a buffer of nice-to-haves so you have features to trade out if you need to • Adjust your projected backlog using historical data
- 40. excella.com | @excellaco Use the Monte Carlo technique to predict the size of your backlog: • Create epics for the entire product • Break a handful into stories • Use Monte Carlo to model story counts for the others Use Monte Carlo again to simulate the simulated backlog Double Monte Carlo
- 41. excella.com | @excellaco • Estimate the PBIs you can complete by a date, not the date to complete the PBIs • Choose simulated times until you hit your forecast date • Useful for fixed timelines Fixed-Date Monte Carlo
- 42. excella.com | @excellaco Other sources of variability
- 43. excella.com | @excellaco Your turn! Activity
- 44. excella.com | @excellaco A simple forecasting tool Input tab: takt times, start date, size of forecasted backlog
- 45. excella.com | @excellaco A simple forecasting tool Simulation tab: simulated PBIs and their takt times, totals for each simulated backlog
- 46. excella.com | @excellaco A simple forecasting tool Results tab: Distribution of simulations by date range
- 47. excella.com | @excellaco Things to try • Add more input takt times in the same range of values • Replace some of the input takt times to use a broader range of values • Replace just one input takt time with an outlier
- 48. excella.com | @excellaco • What do you notice about your forecasts? • How could you use it with your current team, or when could you have used it before? • What would you need to start using this technique? Discuss with a partner
- 49. excella.com | @excellaco Takeaways • Understand all possibilities • Validates Agile concepts • Replaces a data point with a conversation
- 50. How to connect hunter.tamaro@excella.com @htammaro https://linkedin.com/in/wmhunter Excella Consulting @excellaconsulting Connect With Us! @excellaco
Notas del editor
- Average velocity per sprint? Average throughput per sprint? Look at a burnup chart of either of the above? Others…?
- Plan releases and related activities Plan when to take on future projects Project budgeting and staffing decisions Others…?
- Wanted to know when to go live to help roll out public engagement We needed a forecast!
- “Divide by average velocity, or plot on a burnup chart”
- Probably not! Our estimates that far out are really bad, and only gives you one day. What are the odds you deliver on that day?
- Centuries old, but took off in 1940s as computers came into use One of the first Monte Carlo algorithms ran on the ENIAC! Especially useful for modeling random processes or those with high uncertainty
- Examples on individual slides: weather forecast, retirement “It’s hard to make predictions, especially about the future” – Yogi Berra (?)
- Weather forecasters do the same thing with storm tracking, creating “spaghetti plots” of where storms might go based on different inputs, and seeing which ones are most likely…
- Financial planning uses Monte Carlo
- Here is a Monte Carlo forecast being used to track Godzilla.
- Let’s step through the process!
- Many team have a backlog that looks like this: some number of well-defined user stories at the top, with bigger, loosely-defined epics or features below that haven’t yet been decomposed. In this presentation, I’ll step through forecasting this well-defined portion up there. In my story, this was our product; we had a good sense of what was left by thet time we started forecasting. For bigger products, you can think of it as the “project” our team is working on – these product backlog items represent a new capability for our product, and we want to know when we’ll release it. Later in the presentation I’ll cover some more complex cases.
- We’re predicting a date, so we’ll use takt time, but we’ll look at other possibilities later Cycle time looks at time to complete an individual item, not the rate at which the team works through the backlog Takt times roll up a lot of other factors in variability – impediments, vacations, etc
- For software teams this is probably in weekdays
- Emphasize: we are using takt times, not point estimates or other – that’s rolled up in time
- How many times? Hard to say, because it depends on your inputs and the size of the backlog you’re trying to simulate, but in general, more runs will get you more precision in your results. There are a few approaches and even academic papers that attempt to find a formula for the ideal number of runs. But nowadays, computers are fast enough that you can just add more and more runs until you get a reasonable-looking output.
- Percentages are cumulative
- The graph is a good way to tell if you have enough simulations – this one has a pretty smooth graph with a nice recognizable bell curve. If it’s jumping all over the place, that’s a tell that you need more simulations. Grouping the outputs like we have here – rather than single days – is an easy way to smooth out a curve without having to add simulations.
- Point of this activity: make it very simple, to demystify it; make it clear how it works, so you know how to adapt it yourself; make it useable on a phone
- Tripling the number of inputs doesn’t really affect the outcome! From Adam Yuret’s talk on Monday: you have a decent sense of the range of inputs after just seven samples.
- Broader range of inputs leads to broader range of outputs – obvious, but stresses the value of consistent sizing for enhancing predictability
- Let’s say you have a team that only completes work on the last day of each sprint. Here I’m comparing two teams that complete the exact same amount of work each sprint, but one completes individual stories throughout the sprint, and the other completes everything on the last day. The team the only finishes work at sprint boundaries has a lot more uncertainty due to all the risk it’s carrying within sprints. That expresses itself as variability in takt times – they’re all either 10 days (the first item completed) or zero (all the other items, finished the same day). You could get a more normal-looking graph by forecasting the second team based on throughput per sprint instead of takt time, but you’d have the same uncertainty – your graph would only be able to forecast at the sprint interval.
- Adding just one outlier is enough to push out your median timeline, but also adds a “long tail,” making it take longer and longer to add certainty
- Need to start this section by about 3:38 pm (10-12 mins left)
- Turned out the devs’ gut estimate lined up with the “most likely” date, but with just a 50/50 chance of making it Accepted that internally, but communicated the 80% date externally
- Not just when we were releasing, but what to include – and what other factors to consider. If this is the date we’re feature complete, what else do we need to do? How much overlap? Validates the “gut check” from the devs
- Watch the forecast change over time – going up or down as the team gets faster or slower, or as the backlog evolves; cone of possibility narrowing as the team gets more reliable, or as real time closes in on the prediction
- Another view
- Will the team keep working at roughly the same rate? Will PBIs be broken down to roughly the same size? No PO so longer stories; shifts after she starts and we start doing feedback New Agile teams not as good at breaking things down, but get better In both cases, use recent data, not necessarily more, to make sure it’s accurate
- Mixture of refining stories and scope expansion
- If you don’t have a well-developed backlog, or any backlog, use Double Monte Carlo to forecast what the ultimate size will be
- Useful for fixed-date events, like trade shows or prescheduled releases
- Like throughput per sprint, or bugs assigned to team, or stuff like that