Vivan Zhang of NYSE DataScience Academy describes how to tidy data with TIDYR

1. Data Tidying with
tidyr

2. Outline
❖ Three rules of tidy datasets
❖ Why tidyr?
❖ Why do we need tidy data?
❖ spread() and gather() – reshape the layout
❖ separate() and unite() – split and combine cells
2

3. The three rules of tidy datasets
❖What dataset will we consider a tidy dataset?
❖Your data will be easier to work with R if it follows three rules:
➢ Each variable in the dataset is placed in its own column
➢ Each observation is placed in its own row
➢ Each value is placed in its own cell
3

4. The three rules of tidy datasets
4http://r4ds.had.co.nz/tidy-data.html

5. Why do we need tidy data?
❖Tidy data works well with R because R is a vectorized programming
language.
❖A data frame is a list of vectors that R displays as a table. When your
data is tidy, the values of each variable fall in its own column vector.
❖R's operations are optimized to work with vectors
❖Tidy data takes advantage of both of these traits.
5

6. Why tidyr?
❖ Prerequisites
install.packages("tidyr")
library(tidyr)
6

7. Four examples of data formats
table1 # tidy example
❖ Tidy data
## country year cases population
## 1 Afghanistan 1999 745 19987071
## 2 Afghanistan 2000 2666 20595360
## 3 Brazil 1999 37737 172006362
## 4 Brazil 2000 80488 174504898
## 5 China 1999 212258 1272915272
## 6 China 2000 213766 1280428583
7

8. Four examples of data formats
table2
❖ Messy data
## country year key value
## 1 Afghanistan 1999 cases 745
## 2 Afghanistan 1999 population 19987071
## 3 Afghanistan 2000 cases 2666
## 4 Afghanistan 2000 population 20595360
## 5 Brazil 1999 cases 37737
## 6 Brazil 1999 population 172006362
## 7 Brazil 2000 cases 80488
## 8 Brazil 2000 population 174504898
## 9 China 1999 cases 212258
## 10 China 1999 population 1272915272
## 11 China 2000 cases 213766
## 12 China 2000 population 1280428583
8

9. Four examples of data formats
table3
❖ Messy data
## country year rate
## 1 Afghanistan 1999 745/19987071
## 2 Afghanistan 2000 2666/20595360
## 3 Brazil 1999 37737/172006362
## 4 Brazil 2000 80488/174504898
## 5 China 1999 212258/1272915272
## 6 China 2000 213766/1280428583
9

10. Four examples of data formats
table4a # cases
❖ Messy data (“longitudinal” or “panel” data)
## country 1999 2000
## 1 Afghanistan 745 2666
## 2 Brazil 37737 80488
## 3 China 212258 213766
table4b # populations
## country 1999 2000
## 1 Afghanistan 19987071 20595360
## 2 Brazil 172006362 174504898
## 3 China 1272915272 1280428583
10

11. spread() and gather() – reshaping the layout
❖ The two most important functions in tidyr are gather() and spread(). Each
relies on the idea of a key value pair.
❖ Key value pairs: Similar to the concept of a list in R, a key value pair is a
simple way to record information. Keys can be regard as variable names,
while values are the data associated with the specific variables.
❖ When the data contains one column of variable names (the keys) and other
columns of corresponding values, we should use spread() and gather().
11

12. spread() – Spread rows into columns
❖ spread() turns a pair of key:value columns into a set of tidy columns. To
use spread(), pass it the name of a data frame, then the name of the key
column in the data frame, and then the name of the value column.
❖ Pass the column names as they are; do not use quotes.
12
table 2
## country year key value
## 1 Afghanistan 1999 cases 745
## 2 Afghanistan 1999 population 19987071
## 3 Afghanistan 2000 cases 2666
## 4 Afghanistan 2000 population 20595360
## 5 Brazil 1999 cases 37737
## 6 Brazil 1999 population 172006362
...

13. spread() – Spread rows into columns
spread(table2, key=key, value=value)
# A tibble: 6 × 4
country year cases population
* <chr> <int> <int> <int>
1 Afghanistan 1999 745 19987071
2 Afghanistan 2000 2666 20595360
3 Brazil 1999 37737 172006362
4 Brazil 2000 80488 174504898
5 China 1999 212258 1272915272
6 China 2000 213766 1280428583
13

14. spread() – Spread rows into columns
❖spread() returns a copy of your data set that has had the key and value
columns removed.
❖In their place, spread() adds a new column for each unique value of the
key column. These unique values will form the column names of the new
columns.
❖ spread() distributes the cells of the former value column across the cells
of the new columns and truncates any non-key, non-value columns in a
way that prevents duplication.
❖ In the other words, spread() will create columns to contain values, while
the column names are corresponding keys.
14

15. gather() – Gather columns into rows
❖ gather() does the reverse of spread().
❖ gather() collects a set of column names and places them into a single key
column. It also collects the cells of those columns and places them into a
single value column. You can use gather() to tidy table4a.
15

16. gather() – Gather columns into rows
❖ To use gather(), pass it the name of a data frame to reshape.
❖ Then pass gather() a character string to use for the name of the key
column that it will make, as well as a character string to use as the name
of the value column that it will make. Finally, specify which columns
gather() should collapse into the key value pair (here with integer
notation).
16
table4a
## country 1999 2000
## 1 Afghanistan 745 2666
## 2 Brazil 37737 80488
## 3 China 212258 213766

17. gather() – Gather columns into rows
gather(table4a, key="year", value="case", 2:3)
Source: local data frame [6 x 3]
country year case
(chr) (chr) (int)
1 Afghanistan 1999 745
2 Brazil 1999 37737
3 China 1999 212258
4 Afghanistan 2000 2666
5 Brazil 2000 80488
6 China 2000 213766
17

18. separate() and unite() – split and combine cells
❖ spread() and gather() help you reshape the layout of your data to place
variables in columns and observations in rows.
❖ separate() and unite() help you split and combine cells to place a single,
complete value in each cell.
18

19. separate() – Separate one column into several
❖ To use separate, pass the name of a data frame to reshape and the name of
a column to separate. Also give separate an argument called into, which
should be a vector of character strings to use as new column names.
table3
## country year rate
## 1 Afghanistan 1999 745/19987071
## 2 Afghanistan 2000 2666/20595360
## 3 Brazil 1999 37737/172006362
## 4 Brazil 2000 80488/174504898
## 5 China 1999 212258/1272915272
## 6 China 2000 213766/1280428583
19

20. Using basic R
❖ table3 combines the values of cases and population into the same cell. It
may seem that this would help you calculate the rate, but that is not so.
You will need to separate the population values from the cases values if
you wish to do math with them.
❖ Let’s see how we can do this using basic R.
temp <-
data.frame(t(matrix(unlist(strsplit(table3$rate,split='/')),
ncol=length(table3$rate), nrow = 2)))
table3_new <- cbind(table3[,-3], temp)
colnames(table3_new)[3:4] <- c("case","population")
20

21. separate() – Separate one column into several
separate(table3, rate, into=c("cases","population"), sep="/")
# A tibble: 6 × 4
country year cases population
* <chr> <int> <int> <int>
1 Afghanistan 1999 745 19987071
2 Afghanistan 2000 2666 20595360
3 Brazil 1999 37737 172006362
4 Brazil 2000 80488 174504898
5 China 1999 212258 1272915272
6 China 2000 213766 1280428583
21

22. separate() – Separate one column into several
❖ By default, separate() will split values wherever a non-alphanumeric
character appears. Non-alphanumeric characters are characters that are
neither a number nor a letter.
❖For example, in the code on the previous slide, separate() splits the
values of rate at the forward slash characters.
22

23. unite() – Unite several columns into one
❖ unite() does the opposite of separate(): it combines multiple columns
into a single column.
❖ Give unite() the name of the data frame to reshape, the name of the new
column to create (as a character string), and the names of the columns
to unite. unite() will place an underscore (_) between values from
separate columns. If you would like to use a different separator, or no
separator at all, pass the separator as a character string to sep.
23

24. unite() – Unite several columns into one
table5
## Source: local data frame [6 x 4]
## country century year rate
## (chr) (chr) (chr) (chr)
## 1 Afghanistan 19 99 745/19987071
## 2 Afghanistan 20 00 2666/20595360
## 3 Brazil 19 99 37737/172006362
## 4 Brazil 20 00 80488/174504898
## 5 China 19 99 212258/1272915272
## 6 China 20 00 213766/1280428583
24

25. unite() – Unite several columns into one
unite(table5, "new", century, year, sep="")
## # A tibble: 6 x 3
## country new rate
## * <fctr> <chr> <chr>
## 1 Afghanistan 1999 745/19987071
## 2 Afghanistan 2000 2666/20595360
## 3 Brazil 1999 37737/172006362
## 4 Brazil 2000 80488/174504898
## 5 China 1999 212258/1272915272
## 6 China 2000 213766/1280428583
25

26. unite() – Unite several columns into one
❖ unite() returns a copy of the data frame that includes the new column,
but not the columns used to build the new column.
❖ If you would like to retain these columns, using the argument
remove=FALSE
26