1. Capturing and Analyzing Low-Level Events
from the Code Editor
YoungSeok Yoon (youngseok@cs.cmu.edu)
Institute for Software Research (ISR)
Brad A. Myers (bam@cs.cmu.edu)
Human-Computer Interaction Institute (HCII)
School of Computer Science
Carnegie Mellon University

2. Motivation
• When trying to understand what problems the
developers have, researchers investigate…
how frequently when / how do
do those the developers use
problems occur? what kinds of certain features?
problems do
developers
have?
in what context in what sequence do the
does a certain developers complete
event occur? certain tasks?
PLATEAU 2011 2

3. Motivation
• When trying to evaluate existing tools…
how how much
do the developers time do they
frequently do
actually use those spend using
developers
tools? the tools?
use the tools?
do the what types of
exactly how developers use errors do
they are used the tools as they make?
in detail? expected?
PLATEAU 2011 3

4. How do we answer these questions?
• Essentially, we are collecting the usage data
from the developers
• The usage data can be collected by:
▪ asking the developers
▪ observing the developers
▪ mining software repositories
▪ logging the developers’ behavior
• However, previous tools make it difficult or impossible to
get / analyze fine-grained code editing history
PLATEAU 2011 4

5. Asking the developers
• Interviews, surveys, focus groups
• Limitations
▪ Responses may not be reliable
▪ Developers perform many operations quite automatically
 Maybe they might not remember the specific occasions
PLATEAU 2011 5

6. Observing the developers
• Contextual inquiries, lab studies
• Think aloud + videotape
• Limitations
▪ Requires manual inspection of the videotape
 Can be time-consuming and error-prone
• [Coman2008], [Ko2003], [Ko2005], ...
PLATEAU 2011 6

7. Mining software repositories
• Has been used for studying software evolution
• Limitations
▪ Cannot know exactly what happened between two
consecutive revisions
 we may miss some important user behavior
▪ Need to infer from the check-in comments and other clues
• [Aversano2007], [Bettenburg2009], [Kim2005], [Murphy-Hill2009], …
PLATEAU 2011 7

8. Eclipse Usage Data Collector (UDC)
• Logging tool for aggregate data
• Collects usage information from all the Eclipse users
who consented to upload their data to UDC
• Limitations
▪ Does not capture the sequences of events
▪ Misses some important commands executed
• e.g., UDC ignores backspace, moving cursor with arrow keys, …
▪ Command-specific parameters are not captured
PLATEAU 2011 8

9. Other Logging Tools
• Mylyn Monitor [Kersten2006]
▪ Textual level changes are not captured
▪ Focuses on more abstract user interaction data on the
entire IDE
• Syde / Replay [Hattori2011]
▪ Only commands which change the source code are logged
▪ AST-level, not textual level
PLATEAU 2011 9

10. Our new logging tool: FLUORITE
Full of
Low-level
User
Operations
Recorded
In
The
Editor
PLATEAU 2011 10

11. Our new logging tool: FLUORITE
• FLUORITE is a publicly available Eclipse plug-in that
captures low-level code editing events and
produces XML log files
• This is a tool for the researchers
• Can be used to overcome the previously listed limitations
• For each event, FLUORITE logs:
▪ timestamp
▪ event type / command ID
▪ event-specific parameters
PLATEAU 2011 11

12. Three types of events
• FLUORITE logs three different types of events:
▪ Commands: all the user events in the code editor
• (e.g., typing new text, moving cursor, copying, …)
▪ Document Changes
• logged whenever the active file is modified
• contain the actual inserted/deleted code, resulting code length, …
• makes it possible to reproduce snapshots of the files at any point
▪ Annotations
• logged when the developer manually adds annotations to help the researcher
PLATEAU 2011 12

13. An example
Click
PLATEAU 2011 13

14. An example
Shift ↓
PLATEAU 2011 14

15. An example
Del
PLATEAU 2011 15

16. An example
PLATEAU 2011 16

17. Resulting FLUORITE log
<Command __id="2" _type="MoveCaretCommand" caretOffset="142"
docOffset="142" timestamp="3977"/>
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN"
timestamp="5598"/>
<DocumentChange __id="4" _type="Delete" docASTNodeCount="22"
docActiveCodeLength="125" docExpressionCount="10" docLength="151"
endLine="9" length="39" offset="142" startLine="8" timestamp="7186">
<text>
<![CDATA[ System.out.println("Hello World!");
]]>
</text>
</DocumentChange>
<Command __id="5" _type="EclipseCommand"
commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>
PLATEAU 2011 17

18. ① Cursor was
Resulting FLUORITE log moved by the
mouse clicking
<Command __id="2" _type="MoveCaretCommand" caretOffset="142"
docOffset="142" timestamp="3977"/>
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN"
timestamp="5598"/>
<DocumentChange __id="4" _type="Delete" docASTNodeCount="22"
docActiveCodeLength="125" docExpressionCount="10" docLength="151"
endLine="9" length="39" offset="142" startLine="8" timestamp="7186">
<text>
<![CDATA[ System.out.println("Hello World!");
]]>
</text>
</DocumentChange>
<Command __id="5" _type="EclipseCommand"
commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>
PLATEAU 2011 18

19. Resulting FLUORITE log
② One line of code
<Command __id="2" _type="MoveCaretCommand" caretOffset="142"
was selected by
docOffset="142" timestamp="3977"/>
SHIFT + ↓
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN"
timestamp="5598"/>
<DocumentChange __id="4" _type="Delete" docASTNodeCount="22"
docActiveCodeLength="125" docExpressionCount="10" docLength="151"
endLine="9" length="39" offset="142" startLine="8" timestamp="7186">
<text>
<![CDATA[ System.out.println("Hello World!");
]]>
</text>
</DocumentChange>
<Command __id="5" _type="EclipseCommand"
commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>
PLATEAU 2011 19

20. Resulting FLUORITE log
<Command __id="2" _type="MoveCaretCommand" caretOffset="142"
docOffset="142" timestamp="3977"/>
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN"
timestamp="5598"/>
<DocumentChange __id="4" _type="Delete" docASTNodeCount="22"
docActiveCodeLength="125" docExpressionCount="10" docLength="151"
endLine="9" length="39" offset="142" startLine="8" timestamp="7186">
<text>
<![CDATA[ System.out.println("Hello World!");
]]>
</text> ③ “Delete” key
</DocumentChange> was pressed
<Command __id="5" _type="EclipseCommand"
commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>
PLATEAU 2011 20

21. Resulting FLUORITE log
<Command __id="2" _type="MoveCaretCommand" caretOffset="142"
docOffset="142" timestamp="3977"/>
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN"
timestamp="5598"/>
④ The actual
<DocumentChange __id="4" _type="Delete" docASTNodeCount="22" code
deleted by the
docActiveCodeLength="125" docExpressionCount="10" docLength="151"
endLine="9" length="39" offset="142" startLine="8" timestamp="7186">
“delete” command
<text>
<![CDATA[ System.out.println("Hello World!");
]]>
</text>
</DocumentChange>
<Command __id="5" _type="EclipseCommand"
commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>
PLATEAU 2011 21

22. FLUORITE log files
• The log files are written in XML format
▪ Anyone can build their own automatic analyzer!
• Is FLUORITE practical enough to use?
▪ Already has been useful for a couple of studies
• Our own exploratory study with 12 developers
• Dörner’s evaluation study of Euklas system
▪ Size of the logs
• Average log size: 236.8KB / hr = 9.25MB / week
• Could be reduced to 1MB / week if the logs were compressed
▪ Performance
• There was no measurable performance loss during our study
PLATEAU 2011 22

23. Built-in analyses
• We also provide a log analyzer which has
several built-in analyses
• Our study focused on when and how the developers
backtrack while editing code
▪ The analyses were built for this purpose
PLATEAU 2011 23

24. Example Analysis:
Command distribution report
PLATEAU 2011 24

25. Example Analysis:
Command distribution report
PLATEAU 2011 25

26. Example Analysis:
Command distribution report
PLATEAU 2011 26

27. Example Analysis:
Command distribution report
PLATEAU 2011 27

28. Example Analysis:
Keystroke distribution report
PLATEAU 2011 28

29. Example Analysis:
Keystroke distribution report
PLATEAU 2011 29

30. Example Analysis:
Keystroke distribution report
PLATEAU 2011 30

31. Example Analysis:
Keystroke distribution report
PLATEAU 2011 31

32. Example Analysis:
Code size growth graph
PLATEAU 2011 32

33. Events View
PLATEAU 2011 33

34. Example Analysis:
Code editing pattern detection
• FLUORITE logs enable us to detect code editing patterns
composed of sequences of events
• Examples (not all of them are implemented)
▪ Typo correction
▪ Parameter tuning
▪ Commenting out / uncommenting
▪ Cutting/copying and pasting within a project
▪ Manual refactoring (e.g., rename variable)
• Preliminary implementations
Pattern Counts Rate Precision
Typo correction 274 / 288 13.6 / hr 95.14%
Parameter tuning 52 / 98 2.6 / hr 53.06%
PLATEAU 2011 34

35. Conclusion
• FLUORITE web page:
http://www.cs.cmu.edu/~fluorite/
• FLUORITE is publicly available, open-sourced tool for
Eclipse which can be used when conducting studies
• FLUORITE turned out to be useful for our own study,
and we hope that it will help you too!
PLATEAU 2011 35

36. Questions?
• FLUORITE web page:
http://www.cs.cmu.edu/~fluorite/
• Acknowledgements
▪ National Science Foundation (NSF) CCF-0811610
▪ Korea Foundation for Advanced Studies (KFAS)
PLATEAU 2011 36

37. References
• [Aversano2007] Aversano, L., Cerulo, L. and Di Penta, M. 2007. How Clones are Maintained: An Empirical
Study. In Proc. 11th European Conf. on Soft. Maint. and Reengineering (CSMR’07). 81-90.
• [Bettenburg2009] Bettenburg, N., Weyi, S., Ibrahim, W., Adams, B., Ying, Z. and Hassan, A. E. 2009. An
Empirical Study on Inconsistent Changes to Code Clones at Release Level. In Proc. 16th Working Conf. on
Reverse Eng. (WCRE’09). 85-94.
• [Coman2008] Coman, I. D. and Sillitti, A. 2008. Automated Identification of Tasks in Development Sessions.
In Proc. 16th IEEE Int’l Conf. on Program Comprehension (ICPC’08). 212-217.
• [Kersten2006] Kersten, M. and Murphy, G. C. 2006. Using task context to improve programmer
productivity. In Proc. 14th ACM SIGSOFT Int’l Symp. on Foundations of Soft. Eng. (FSE’06). 1-11.
• [Kim2005] Kim, M., Sazawal, V., Notkin, D. and Murphy, G. 2005. An empirical study of code clone
genealogies. In Proc. 10th Euro. Soft. Eng. Conf. & 13th ACM SIGSOFT Int’l Symp. on Foundations of Soft.
Eng. (ESEC/FSE’05). 187-196.
• [Ko2003] Ko, A. J. and Myers, B. A. 2003. Development and evaluation of a model of programming errors.
In Proc. IEEE Symp. on Human Centric Computing Languages and Environments (HCC’03). 7-14.
• [Ko2005] Ko, A. J., Aung, H. H. and Myers, B. A. 2005. Design requirements for more flexible structured
editors from a study of programmers’ text editing. In Proc. Extended Abstracts of CHI'2005. 1557-1560.
• [Hattori2011] Hattori, L., D’Ambros, M., Lanza, M. and Lungu, M. 2011. Software Evolution
Comprehension: Replay to the Rescue. In Proc. 19th IEEE Int’l Conf. on Program Comprehension (ICPC’11).
161-170.
• [Murphy-Hill2009] Murphy-Hill, E., Parnin, C. and Black, A. P. 2009. How we refactor, and how we know it.
In Proc. 31st Int’l Conf. on Soft. Eng. (ICSE’09). 287-297.
PLATEAU 2011 37

38. BACKUP SLIDES
PLATEAU 2011 38

39. Longer example log
<Command __id="2" _type="MoveCaretCommand" caretOffset="103" docOffset="103"
timestamp="11073"/>
<Command __id="3" _type="EclipseCommand"
commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN" timestamp="13001"/>
<DocumentChange __id="4" _type="Delete" docASTNodeCount="15" docActiveCodeLength="86"
docExpressionCount="4" docLength="112" endLine="8" length="39" offset="103" startLine="7"
timestamp="14969">
<text><![CDATA[ System.out.println("Hello World!");
]]></text>
</DocumentChange>
<Command __id="5" _type="EclipseCommand" commandID="org.eclipse.ui.edit.delete"
timestamp="14985"/>
<DocumentChange __id="6" _type="Insert" docASTNodeCount="22" docActiveCodeLength="125"
docExpressionCount="10" docLength="151" length="39" offset="103" timestamp="17564">
<text><![CDATA[ System.out.println("Hello World!");
]]></text>
</DocumentChange>
<Command __id="7" _type="UndoCommand" timestamp="17570"/>
PLATEAU 2011 39

40. Example Analysis:
Command distribution report
PLATEAU 2011 40

41. Example Analysis:
Command distribution report
Command Count (Percentage)
InsertString 5797 (31.48%)
LINE_DOWN 5693 (10.67%)
DELETE_PREV 4495 (10.48%)
MoveCaret 3586 (8.63%)
LINE_UP 2751 (8.27%)
PLATEAU 2011 41

42. Example Analysis:
Keystroke distribution report
PLATEAU 2011 42

43. Example Analysis:
Keystroke distribution report
Key Count (Percentage)
↓ 5797 (12.64%)
Backspace 5693 (12.41%)
↑ 4495 (9.80%)
→ 3586 (7.82%)
← 2751 (6.00%)
PLATEAU 2011 43