Presented by Valentina Grigoreanu at CHI2010. Demonstrates how tools can be designed around users' strategies. StratCel is an add-in for Excel that supports spreadsheet users' debugging strategies. Includes evaluation and design guidelines for supporting spreadsheet debugging.

1. Valentina Grigoreanu, Margaret Burnett, George Robertson Oregon State University & Microsoft Corporation 04.13.10 – CHI 2010 Talk Strategy-Centric Approach:Design of End-User Debugging Tools Valentina Grigoreanu (M.A.X.) 1

2. Agenda Problem + Theoretical Findings StratCel: New Strategy-Centric Debugging Tool Tactics+Moves Stratagems Strategies Valentina Grigoreanu (M.A.X.) 2

3. Research Methods: Eight Studies Factors to consider in future studies (Moves and Tactics) Generalizing across environments, populations, & study setups (Stratagems) A Sensemaking perspective on end-user debugging (Strategies) Designing end-user debugging tools (All four levels) Valentina Grigoreanu (M.A.X.) 3

5. Problem Addressed Not only is EUP code buggy… Problem amplified by: Tens of millions of end-user programmers [Scaffidi et al. 2005] Lack of training in software engineering techniques [Study Participants] Increasing size and complexity of spreadsheets [Whittaker 1999] Valentina Grigoreanu (M.A.X.) 5

6. Related Work Previous research / tools to improve end-user debugging: Automatic bug detection Mapping techniques from professional SE Feature usage A critical stone has been left unturned… Valentina Grigoreanu (M.A.X.) 6

7. CHI Paper/Dissertation RQ How can we design tools such that they support end-user programmers’ existingeffectivedebugging strategies? What are they? How are they used? Are there gender differences? Do the strategies generalize across EUP environments? How can tools be designed around strategies? What impact would strategy support have on success? Valentina Grigoreanu (M.A.X.) 7

8. Definitions Has Impact on… Iteratative improvements to Existing Features. New feature innovation. Overview. Stumbling Blocks. Checks. Note: All three of these are needed for strategy-centric design. Valentina Grigoreanu (M.A.X.) 8

9. Outline Valentina Grigoreanu (M.A.X.) 9

10. The Stratagems Code Inspection Control Flow Dataflow Feedback Following Help Proceeding as in Prior Experience Spatial Specification Checking Testing To-do Listing Valentina Grigoreanu (M.A.X.) 10

11. Outline Valentina Grigoreanu (M.A.X.) 11

12. Outline Valentina Grigoreanu (M.A.X.) 12

13. StratCel 21 implications for design At 3 levels of abstraction: Strategies (high) Stratagems (medium) Moves/tactics (low) Valentina Grigoreanu (M.A.X.) 13

14. StratCel Valentina Grigoreanu (M.A.X.) 14

15. 1. Strategies Implication Observed two strategies Comprehensive strategy: Thorough, found easy bugs quickly, lots of fixes within 30 minutes But… Needed help keeping track of cells to return to Selective strategy: Fixed the hardest bug, found some of the easier ones in the process But… Needed help keeping track of cells not checked Valentina Grigoreanu (M.A.X.) 15

16. Strategies Implication Sounds a lot like support for the to-do listing stratagem might help: Observed in Forms/3, PowerShell, professional devs. Poorly supported in all: often requires repurposing Shows promise for improving debugging success Valentina Grigoreanu (M.A.X.) 16

17. StratCel:Strategies and To Dos Does not impose an order Valentina Grigoreanu (M.A.X.) 17

18. 2. Stratagems Implication Support all strategems in the context of each to-do item (specification checking, testing, code inspection, etc.). Each provides additional relevant information for debugging. Valentina Grigoreanu (M.A.X.) 18

19. StratCel:Multi-Stratagem Support Valentina Grigoreanu (M.A.X.) 19

20. 3. Moves & Tactics Implications Highlighting formula cells Too many warnings is as bad as none at all Interworksheet dependencies Valentina Grigoreanu (M.A.X.) 20

21. Good Tactics, Bad MovesImplication 3 Following dependencies between spreadsheets How many of our participants do you think understood this icon when they saw it? Implication for design: at least a tooltip, would be nice. Though a method for navigating to that/those cell(s) would be ideal. Valentina Grigoreanu (M.A.X.) 21

22. StratCel: Highlight Formulas Valentina Grigoreanu (M.A.X.) 22

23. StratCel: Error Checking Valentina Grigoreanu (M.A.X.) 23

24. StratCel: Interworksheet Dependencies Valentina Grigoreanu (M.A.X.) 24

25. Evaluation: Study Setup 8 Control participants / 5 Treatment participants Experience with formulas Think-aloud study Debugged a grade-book spreadsheet in Excel Seeded with 10 bugs Or so we thought... There were actually 12! Our StratCel participants found 2 more bugs that the researchers, the spreadsheet’s creator, and the Excel-only participants all had overlooked! Valentina Grigoreanu (M.A.X.) 25

27. Support for selective and comprehensive strats:Selective: used status borders to easily find areas on which to focus attention. Comprehensive: walked through the entire to-do list item-by-item. Bugs Found p<0.005 Valentina Grigoreanu (M.A.X.) 26

29. Contextually-provided information on additional stratagems:

30. Specification

31. Recently used function library

33. Time to first bug find: p<0.005

34. Time to first bug fix: p<0.003Valentina Grigoreanu (M.A.X.) 27

35. StratCel Results:Triangulation Found CF: 4.5, TF: 9.0 CM: 5.0, TM: 9.0 Fixed: CF: 3.5, TF: 9 CM: 2.5, TM: 8.5 “I feel like [StratCel] would be extra useful for someone like me who, well, I can use Excel and I can figure it out, but I'm definitely not an expert at Excel. […] I think the only problems I had were with the Excel functions I hadn't learned. This is like a really good way of helping me keep track of what I've done and not get lost." Valentina Grigoreanu (M.A.X.) 28

36. Outline Valentina Grigoreanu (M.A.X.) 29

37. Parallels to IR Research - This design approach can be used beyond end-user debugging. - [T]he goal has been formed in much IR research to have the system do the searching for us, but that is not the only alternative. It should also be possible to design search interfaces that harmonize with and make easy the prosecution of good search strategy, systems that make it easy for novices to move quickly into good searching because the system promotes it” [Bates 1990]. Valentina Grigoreanu (M.A.X.) 30

38. StratCel Results:Usability Was StratCel intuitive – a.k.a. did we really support end-user programmers’ debugging strategies? "And this is straight-forward and makes a lot of sense. When you look at it, you know what it is. There are lots of tools, where you can tell that people said, 'well… there's just a workaround and you can just do it this way'. But this one, it just seemed very straightforward and it builds on everything from Excel.” “This was really helpful because it has a way to say these are all your formulas… These are the ones you need to go look at. And I like this part [the address field] which shows me where I can find all of the formulas, so I can see them. For example, on this one, I could see there was a gap for E16and I could go back and look specifically at that cell, because I expect it to be the same, and see what's going on.” Valentina Grigoreanu (M.A.X.) 31

39. StratCel Results:Real-World Applications Applicable to real-world tasks: “I think this would be useful for my complex accounting spreadsheets. If you would like to share the tool, I would love to try it on those.” “Looking at [StratCel], I was thinking I have to have a way of tracking my [knitting] patterns. So things that… Ok. I have a pattern and I have steps I have to go through. And I need a way to track them.” "So, can I use your tool? You should sell this and make a million dollars!” Valentina Grigoreanu (M.A.X.) 32

40. Validated Design Guidelines Automatic error detection tools default: Value quality (low number of false-positives) over quantity (detecting more possible types of errors). Important information about cells (e.g., to-do status) should be overlaid onto the spreadsheet to give the user a quick overview of the to-do status of both individual cells and of the overall spreadsheet. Some users are comprehensive, whereas others will start by trying to fix apparent bugs right away. Support both, since both have pros and cons. Strategy-based tools should provide explicit support for to-do listing. To improve debugging of end-user programs, it helps to automatically generate a list of items to check so that all areas of the code are given equal attention. Information about the remaining strategems should be provided in the context of each to-do item to provide more information on which to base a bug fix. Viewing formulas related to an item (e.g., the consistent formulas, recently used formulas, or formulas used in files in a certain directory) may be particularly useful for improving debugging success. 33