A Quick Tour of OpenRefine
•Download as PPTX, PDF•
2 likes•3,073 views
A quick feature tour of OpenRefine, using an example open spending data dataset from Birmingham City Council.
Recommended
More Related Content
Similar to A Quick Tour of OpenRefine
Similar to A Quick Tour of OpenRefine (20)
More from Tony Hirst
More from Tony Hirst (20)
Recently uploaded
Recently uploaded (20)
A Quick Tour of OpenRefine
- 1. Wrangling Data with OpenRefine Tony Hirst Computing and Communications The Open University @psychemedia
- 2. “It’s … a great joy to learn a technique, because as soon as you learn it, you start thinking in it. When I learn a new technique my imaginative possibilities have expanded.” Grayson Perry, Playing to the Gallery
- 3. Real data is often dirty and messy
- 5. This is a hands-on workshop, so fire up your laptops…
- 7. Loading – import source
- 11. Exploring – text facets
- 12. Exploring – text facets
- 13. Exploring – text facets
- 15. Cleaning – text facets
- 16. Checking – no blanks
- 17. Cleaning – tidying columns
- 18. Cleaning – tidying numbers
- 19. Cleaning – tidying numbers
- 20. value.replace(‘£’, ‘’).replace(‘,’ , ‘’) Cleaning – tidying numbers
- 21. Cleaning – tidying numbers
- 22. Exploring – number facets
- 23. Exploring – filtering number ranges
- 24. Exploring – sorting columns
- 25. Cleaning – making dates
- 26. value.toDate( ‘d/M/y’ ) Cleaning – making dates
- 27. http://bit.ly/javadateformat Cleaning – making dates
- 28. Exploring – filtering date ranges
- 32. Exploring – filter and facet
- 34. Cleaning – cluster / make alike
- 35. Cleaning – good practice
- 36. We need a small dataset for the next example…
- 42. value.replace( / LTD.?/, ‘ LIMITED’) Cleaning – normalisation
- 43. Annotating – reconciled data
- 46. Annotating – reconciled data
- 48. Annotating – URL based data
- 50. Annotating – URL based data
- 51. JSON['results’]
- 54. value.parseJson['results']['company']['registered_address_in_full'] Annotating – parsed JSON data
- 55. split(value, ‘,’) Annotating – parsed JSON data
- 56. split(value, ‘,’)[-1] split(value, ‘,’)[-1].strip() Annotating – parsed JSON data
- 57. Saving – annotated data
- 59. Reuse – exporting your action list
Editor's Notes
- "Wrangling Data With OpenRefine" We've all found datasets that hold the promise or riches inside them, but the data itself is in a terrible state. Commas and percentage signs that mess up number formats, dates that we can see are dates but the computer can't, lists of items in single cells, data spanning multiple rows where we want pivot one column's values across multiple columns, and so on. OpenRefine is a powerful browser based tool that helps you work with tabular data and tidy it up as you go along. As well as editing functions (and yes, you will get to see some regular expressions string matching action!) we can also cluster similar words and phrases, spotting that "J. Smith & Sons Ltd" is probably the same as "J Smith and Sons Limited" and being able to group them as such. Once you've seen it in action you'll wonder why working with data ever needed to be so difficult before. 45 mins x 3
- Dirty in the sense that the data may contain errors; messy in that it may be in the wrong shape, form or layout. (Not messy in statistical sense – statisticians are well know for the way they appropriate words and make them mean something no-one else means. In stats, messy data is often regarded as data that is not normally distributed). OpenRefine as a tool helps us work with data that can be represented in tabular form, and generally just tidy it up. Open Refine can help us with inconsistently formatted data, or data that is incorrectly represented. Open Refine can help us get an overview of a dataset, as well as splitting it up in a variety of ways. And Open Refine can help us annotate our data with data from an external source.
- https://dl.dropboxusercontent.com/u/1156404/466161Payments_over_%C2%A3500_November_2014.csv - http://bit.ly/brumspend
- http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html