This document discusses extracting insights from data exhaust, or unused data. It provides 10 lessons for doing so: choose a meaningful problem, find relevant data, raw data is better than processed, guide user input, solve easier problems first, create a quick baseline model, test on sample data, use continuous integration, pick the right tools, and prioritize developer productivity. As a case study, it analyzes skills data from attendees of the Strata conference to understand the audience and identify skills clusters. Visualization tools like Gephi are used to analyze similarities between attendees based on their skill vectors.

1. Distilling Data
Exhaust
How to Surface Insights &
Build Data Products
Feb 2, 2011
Peter Skomoroch
LinkedIn
@peteskomoroch

2. What is Data Exhaust?

3. What is Data Exhaust?
My Delicious Tags

4. What is Data Exhaust?
Words I use on Twitter

5. What can you do with it?
•Data has value
•I’ll share some lessons I’ve
learned about how to extract
that value
•We’ll go through a case study

6. Part 1) 10 Lessons Learned

7. 1) Choose a meaningful problem
http://www.flickr.com/photos/aloshbennett/
•Find pain points
•Work on stuff that
matters
•Look for
underutilized data

8. 2) Find or collect relevant data
•DataWrangling
•InfoChimps
•Pete Warden
•Factual, SimpleGeo
•Mechanical Turk

9. 3) Raw is better than processed
•Normalization could
be incorrect
•Data might be lost
or corrupted
•Good approach:
public.resource.org
http://www.flickr.com/photos/nedraggett/347280918/

10. 4) Guide user input when you can
•Auto suggest
•Validate inputs
•Collect tags, votes
•Makes data
scrubbing easier

11. 5) Solve easier problems first
http://where2conf.com/where2010/public/schedule/detail/12400

12. 6) Build a baseline model quickly
•Iterate rapidly after
baseline is done
•Measure accuracy
on hold out test set

13. 7) Test code on sample data
build logical sample data

14. 8) Use Continuous Integration

15. 8) Use Continuous Integration
https://github.com/matthayes/azkaban

16. 9) Pick the right tool for the job

17. 10) Developer productivity is key
•Fast Iterations: Python, Ruby, Pig
•Convention over configuration
•Embrace Github, DevOps, & EC2
•Currently using JRuby & Sinatra

18. SNA Team: sna-projects.com

19. Part 2) Case Study: Strata

20. Conference Insights
•I’d like to understand the audience
at Strata
•What companies do we work for?
•What are the top skills at Strata?
•Do attendees cluster together
based on skill?

21. Round up a Data Viz team

22. Use the right tools
•Data Crunching: Hadoop, Pig
•Statistical Work: Python, NumPy
•Visualization: Gephi

23. Find Some Data: Attendees

24. Add LinkedIn data

27. Extract Skills from Profiles
What are skills?
Extract

28. Build Hadoop Skill Graph
Discover
Core Talent Graph for “Hadoop”
Igor Perisic

29. The Talent Graph

40. We can combine skills with the
attendee directory to better
understand Strata

41. What are skills @Strata?

42. Extract skills for attendees

43. Top Skills @Strata

44. Information Overload

45. Relevance Measures
Jaccard Similarity
TFIDF

46. Relevant Skills @Strata

47. Do attendees cluster together
based on skills?

48. •Compute similarity of
attendees based on skill vector
distance
•Cluster similarities in Gephi

50. More analysis on the way
•DJ Patil has a session tomorrow
•We’ll blog about additional
Strata insights soon

51. Questions?
Peter Skomoroch
LinkedIn
@peteskomoroch
http://linkedin.com/in/peterskomoroch
Blog: DataWrangling.com

52. Appendix