Presentation given at the November meeting of the Sheffield R Users Group on the data.table package in R.

1. data.table talk
January 21, 2015

2. The data.table package
author: Pete Dodd date: 4 November, 2014

3. dataframes in R
What is a dataframe?
default R objects for holding data
can mix numeric, and text data
ordered/unordered factors
many statistical functions require dataframe inputs

4. dataframes in R
Problems:
print!
slow searching
verbose syntax
no built-in methods for aggregation
Which is most annoying depends on who you are. . .

5. Constructing data.tables
myDT <- data.table(
number=1:3,
letter=c('a','b','c')
) # like data.frame constructor
myDT2 <- as.data.frame(myDF) #conversion
The data.table class inherits dataframe, so data.tables (mostly) can
be used exactly like dataframes, and should not break existing code.

6. Examples
WHO TB data:
D <- read.csv('TB_burden_countries_2014-09-10.csv')
names(D)[1:10]
## [1] "country" "iso2" "iso3"
## [5] "g_whoregion" "year" "e_pop_num"
## [9] "e_prev_100k_lo" "e_prev_100k_hi"

7. Examples
WHO TB data:
head(D[,c(1,6,8)])
## country year e_prev_100k
## 1 Afghanistan 1990 327
## 2 Afghanistan 1991 359
## 3 Afghanistan 1992 387
## 4 Afghanistan 1993 412
## 5 Afghanistan 1994 431
## 6 Afghanistan 1995 447

8. Examples
Mean TB in Afghanistan
mean(D[D$country=='Afghanistan','e_prev_100k'])
## [1] 397.6087
As data.table:
library(data.table)
E <- as.data.table(D) #convert
E[country=='Afghanistan',mean(e_prev_100k)]
## [1] 397.6087

9. Examples
dataframe multi-column access:
D[D$country=='Afghanistan',
c('e_prev_100k','e_prev_100k_lo',
'e_prev_100k_hi')]
data.table multi-column means, renamed:
E[country=='Afghanistan',
list(mid=mean(e_prev_100k),
lo=mean(e_prev_100k_lo),
hi=mean(e_prev_100k_hi))]
## mid lo hi
## 1: 397.6087 187.913 684.7391

10. Examples
Means for each country? data.table solution:
E[,list(mid=mean(e_prev_100k)),by=country]
## country mid
## 1: Afghanistan 397.60870
## 2: Albania 29.52174
## 3: Algeria 133.95652
## 4: American Samoa 15.09130
## 5: Andorra 30.71304
## ---
## 215: Wallis and Futuna Islands 117.86957
## 216: West Bank and Gaza Strip 11.14783
## 217: Yemen 180.30435
## 218: Zambia 501.39130
## 219: Zimbabwe 386.30435

11. Examples
A more complicated example:
E[,
list(lo=mean(e_prev_100k_lo),
hi=mean(e_prev_100k_hi)),
by=list(country,
century=factor(year<2000)
)]

12. Examples
Output:
## country century lo hi
## 1: Afghanistan TRUE 189.20000 749.80000
## 2: Afghanistan FALSE 186.92308 634.69231
## 3: Albania TRUE 13.20000 65.40000
## 4: Albania FALSE 10.59231 47.53846
## 5: Algeria TRUE 49.40000 212.80000
## ---
## 427: Yemen FALSE 62.69231 218.38462
## 428: Zambia TRUE 291.60000 1024.90000
## 429: Zambia FALSE 197.00000 733.76923
## 430: Zimbabwe TRUE 14.81000 1074.60000
## 431: Zimbabwe FALSE 56.07692 1219.61538

13. Examples
eo <- E[,plot(sort(e_prev_100k))]
0 1000 2000 3000 4000 5000
050010001500
Index
sort(e_prev_100k)
(1-
line combination with aggregations

14. Fast insertion
A new column can be inserted by:
E[,country_t := paste0(country,year)]
head(E[,country_t])
## [1] "Afghanistan1990" "Afghanistan1991" "Afghanistan1992
## [5] "Afghanistan1994" "Afghanistan1995"

15. Keys: fast row retrieval
Need to pre-compute (setkey line)
setkey(E,country) #must be sorted
E['Afghanistan',e_inc_100k]
## country e_inc_100k
## 1: Afghanistan 189
## 2: Afghanistan 189
## 3: Afghanistan 189
## 4: Afghanistan 189
## 5: Afghanistan 189
## 6: Afghanistan 189
## 7: Afghanistan 189
## 8: Afghanistan 189
## 9: Afghanistan 189
## 10: Afghanistan 189
## 11: Afghanistan 189
## 12: Afghanistan 189

16. Gotchas: column access
E[,1]
## [1] 1
E[,1,with=FALSE]
## country
## 1: Afghanistan
## 2: Afghanistan
## 3: Afghanistan
## 4: Afghanistan
## 5: Afghanistan
## ---
## 4899: Zimbabwe
## 4900: Zimbabwe
## 4901: Zimbabwe
## 4902: Zimbabwe
## 4903: Zimbabwe

17. Gotchas: copying
E2 <- E
E[,foo:='bar']
head(E2[,foo])
## [1] "bar" "bar" "bar" "bar" "bar" "bar"

18. Gotchas: copying
This is because copying is by reference.
Use:
E2 <- copy(E)
instead.

19. Summary
more compact
faster (sometimes lots)
less memory
great for aggregation/exploratory data crunching
But: - a few traps for the unwary
Good package vignettes & FAQ,

20. Related
aggregate in base R
plyr: use of ddply
sqldf: good if you know SQL
RSQLlite: ditto
other: - RODBC etc: talk to databases - dplyr: nascent, by Hadley,
internal & external