Central Pennsylvania Open Source Conference, October 17, 2015 Data is a hot topic in the tech sector with big data, data processing, data science, linked open data and data visualization to name only a few examples. Before data can be processed or analyzed it often has to be cleaned. OpenRefine is an open source interactive data transformation tool for working with messy data. This presentation will begin with a short overview of the features of OpenRefine. To demonstrate basic concepts of data cleaning, manipulating, faceting and filtering with OpenRefine, Pennsylvania Heritage magazine subject index data will be used as a case study.

1. INTRODUCTION TO
OPENREFINE
CENTRAL PA OPEN SOURCE CONFERENCE
OCTOBER 17, 2015
Heather Myers / @privatestorm
https://www.linkedin.com/in/heathercmyers

2. ABOUT ME
Web administrator in the government and cultural heritage
sectors.
Currently working at the Pennsylvania Historical and
Museum Commission.

3. OPENREFINE
"a powerful tool for working with messy data: cleaning it;
transforming it from one format into another; and extending it
with web services and external data."

4. GETTING STARTED
Choose dataset
Decide what you want to accomplish with data
Install OpenRefine
Run OpenRefine
http://openrefine.org/download.html
http://127.0.0.1:3333

5. ABOUT THE DATASET
Pennsylvania Heritage magazine subject index.
Index of 12,000+ magazine terms for issues dated 1975–
2002.
http://bit.ly/1Udha8D

6. DATA TO DO LIST
Create lists
of terms for
specific
issues
Extract list
of terms

7. IMPORT ALL DATA
File upload
Web
download
Copy from
clipboard
Google data
import

8. CONFIGURE PARSING OPTIONS
Choose
options
Update
preview
Name
project
Create
project »

9. OPENREFINE PROJECT

10. APPLY A TEXT FILTER
Click down
arrow on
column
Choose text
filter from
menu

11. APPLY A TEXT FILTER
Type text
Choose
additional
options

12. TEXT FILTER OPTIONS
Add
multiple
filters
Reorder
filters

13. TEXT FILTER OPTIONS
Case
sensitive
Regular
expressions

14. EXPORT DATA
Click export
button
Choose
export type
Save file

15. DATA TO DO LIST
Create lists
of terms for
specific
issues
Extract list
of terms

16. IMPORT SELECTION OF DATA
Choose
CSV /TSV/
separator-
based files
Use
semicolon
as custom
separator

17. DATA SEPARATED INTO COLUMNS

18. SPLIT INTO SEVERAL COLUMNS
Click down
arrow on
column
Choose edit
column
Choose
split into
several
columns

19. SPLIT INTO SEVERAL COLUMNS
Split by
separator
or field
length
Choose
after
splitting
options

20. EDIT SINGLE CELL
Hover over
cell & click
edit
Update
data type
Update text
Click apply

21. TRANSFORM TEXT
Click down
arrow on
column
Choose edit
cells
Choose
transform

22. TRANSFORM TEXT
Type
expression
Choose
options
Choose OK

23. TERMS SPLIT INTO COLUMNS

24. MOVE COLUMN
Click down
arrow on
column
Choose edit
column
Choose
move
column left

25. TRIM WHITESPACE
Click down
arrow
Choose edit
cells,
common
trans-
formations
Choose
trim leading
& trailing
whitespace

26. JOIN CELLS
Custom
text
transform
GREL
expressions
plus space
in between

27. HIDE COLUMNS
Click down
arrow
Choose
view
Choose
collapse all
other
columns

28. SORT VALUES
Click down
arrow
Choose sort
Choose
additional
options

29. UNDO / REDO
Choose
undo/redo
tab in left
column
Click
previous
step to
undo

30. FILTER UNDO / REDO
Type
keywords in
filter box

31. EXTRACT OPERATION HISTORY
Choose
extract
button
Choose
steps to
save in left
column
Copy JSON
in right
column

32. APPLY OPERATION HISTORY
Open or
create new
project
Click apply
button in
undo/redo
Paste JSON
Click
perform
operations

33. EXPORT DATA
Click export
button
Choose
export type
Save file

34. LEARN MORE
Website: OpenRefine
http://openrefine.org/
Book: Using OpenRefine
http://bit.ly/1QC0oNS
Course: Big Data University
http://bit.ly/1QC1sl1

35. THE END