Taivo Pungas at ML Estonia Meetup (29.03.2016)

1. Data science
for everyone
Taivo Pungas
29.03.2016

3. For everyone?
● Manual/one-off analyses
○ Not production-level code
● Personal interest
● No deep learning
Survey:
● PhD [student]?
● >1 stats/ML class?
● Can code?

4. Finding
interesting data#1

5. select * from users
Job Experience

6. 1. Existing datasets
int’l organisations
Kaggle
opendata.riik.ee
→ R library (WIP)

7. 1. Existing datasets
2. Scrape
kv.ee/2724720 kv.ee/2724721

8. 1. Existing datasets
2. Scrape
check ToS
don’t DoS
KV, Postimees, Auto24, Osta, …
1M Estonian real estate ads

9. 1. Existing datasets
2. Scrape
3. Quantified Self

10. QS: how did I feel today?
very goodvery bad
count
OK

11. Process & tools for
analysis#2

12. Get data
Clean
Explore /
visualise
Publish
My process
Sleep on it
feedback

13. Python
general purpose data processing
Hadleyverse
Rvs
pandas
matplotlib
scikit-learn

14. R libraries: Hadleyverse
Step Libraries
Get data rvest, xml2, readxl
Clean dplyr, tidyr, stringr
Explore / visualise ggplot2
Publish
... and many others
from Hadley Wickham.

15. Alternatives
Excel / Google Sheets
● External data sources
● Google Apps Script
○ Google Translate API
○ Sending e-mail
○ …
● Not easy to reproduce analyses
Tons of other software
R & Python worth the learning curve

16. R is easy: reading data
# Read data
apartments <- read.csv2("data/apartment_rent_tartu.csv",
sep=";", header=TRUE)

17. R is easy: dplyr
library(dplyr)
# Find average price by part of city
apartments %>%
group_by(Linnaosa) %>%
summarise(KeskmineHind=mean(HindKohandatud)) %>%
arrange(desc(KeskmineHind))

18. R is easy: lin. regression
# Build linear model
fit <- lm(HindKohandatud ~ Tube, data=apartments)
summary(fit)

19. Presenting your
results#3

20. interactive > static

21. D3.js: powerful web visualisations

22. Easy to useHard to use
Limited
Powerful
ggplot2
D3.js
D3 derivates
Excel
GSheets
AI

23. How to reach an audience
● Social media
● Start a blog
○ stat24.ee
○ pungas.ee
● Offer free content
○ Newspapers (tip lines)
○ Guest posts on blogs
● Push to Estonian data science community
○ TODO: FB group? Community blog?

24. Putting it together
##

25. Examples
Apartment prices: R + D3.js 18k hits
Salaries of public servants: R + D3.js 38k hits
Study data: R + D3.js 3k hits
Election promise calculator: D3.js 42k hits
Bondora: R
Alcohol deaths: Illustrator

26. News & inspiration
Mailing lists:
Information is Beautiful, Data Science Weekly, Data Elixir
Blogs:
FiveThirtyEight, R-bloggers, Stat24, Mike Bostock

27. Long-term motivation
Flickr: ucirvine / CC BY-NC-ND

28. pungas.eetaivo@
tpungas