Data Tools cosystem_for_non_programmers

Data Tools Ecosystem
for non-programmers

Óscar Marín Miró
@outliers_es
www.outliers.es

1

Contents at a glance
The Data Process
‣Why mess with Data?
‣A Data Pipeline

Data Acquisition Tools
‣Data Sources
‣Google Forms & Google Docs
‣Scraping tools
‣Network acquisition tools
‣PDF tools
‣Open Reﬁne

Data Analysis
‣Excel capabilities
‣Beyond excel

Data Visualization Tools
‣Timeline.js
‣Wordle
‣Plot.ly
‣Google Fusion Tables
‣Gephi
‣CartoDB
‣MapBox
‣Tilemill

Developers needed
‣Size
‣Real-Time
‣Analysis beyond Basics
‣Interactivity*

Resources
2

The Data Process

http://en.wikipedia.org/wiki/Data-driven_journalism
3

Why mess with data?

Cognitive Complexity

Wisdom

•Non transferable
•Ethic knowledge application

Knowledge

•Personal Absortion
•Action: Information application
•‘Insights’

Information

•Structured Data
•Semantics
•Narrative
•Spatialized Data

Data

•Register observations

4

A Data Pipeline
Data

Data

Acquisition
•API’s
•Scraping
•Files
•DB’s
•Providers

Information

Analysis

•Text Mining
•Machine Learning
•Network Metrics
•Stats
•Filtering

Presentation

•Visualization
•Story

5

Data Formats
• JSON (Javascript Object Notation) - Dev
• CSV, XLS - Tabular
• XML - Old, but still used
• Web data

7

Data Acq Techniques
• Ask the user: Forms
• Download from Data Repositories
• Web data: Scrape
• Ask the provider: API access
8

Google Forms

http://www.encuesta15m.cc/subject.html

12

Google Forms

http://www.encuesta15m.cc/subject.html

13

Google Forms

http://data.elperiodico.com/

14

Google Spreadsheet

http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/
15

Scrape w/ Chrome
Scraper

https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd
18

Scrape w/ Chrome
Scraper

http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/
21

Scrape w/ import.io

http://import.io/

22

PDF Extraction: Tabula

http://tabula.nerdpower.org/

23



24



25

Network Data: Flocker

http://ﬂocker.outliers.es/
26

Network Data: Flocker

http://ﬂocker.outliers.es/
27

Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
28

Data

29

Data

30

Data

31

Data

32

Data

33

Data

https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions
https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions

34

Data Analysis Tools
• Numeric analysis
• Network analysis
• Language analysis

36

Data Analysis w/ MS
Excel
•
•
•
•
•
•
•

Sort
Basic Stats: Mean, Median, Sum...
Filter
Charts
Pivot tables
What-if
Data Analysis Tool

37

Data Analysis:MS Excel
‘Pivot Tables’

38

Data Analysis:MS Excel
‘What-If’

39

Data Analysis: MS Excel

Correlations, Covariance,
Statistic Summary, Regression,
Histogram, Factor Analysis,
Fourier Analysis....
http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf
40

Data Analysis: R

http://www.r-project.org/

41


http://well-formed.eigenfactor.org/
44

• Timelines
• Words
• Chart tools
• Networks
• Maps
45

Timeline JS

http://timeline.knightlab.com/
46

Timeliner (OKFN)

http://timeliner.okfnlabs.org

47

Wordle

http://www.wordle.net/create

48

Wordle

http://www.wordle.net/create

49

Plot.ly

https://plot.ly/
50

Plot.ly

https://plot.ly/
51

Plot.ly

https://plot.ly/
52

Plot.ly

https://plot.ly/
53

Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
54


55


56


57


58


http://www.tableausoftware.com/public/

59

Gephi

http://gephi.org/
60

Gephi

http://gephi.org/
61

Gephi

http://gephi.org/
62

CartoDB

http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com/
63

CartoDB
“Shapeﬁle”

http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com/
64

CartoDB

http://cartodb.com/
65

MapBox

http://www.mapbox.com//
66

MapBox

67

MapBox

68

MapBox

69

MapBox

70

MapBox

71

MapBox

72

When you’ll need a
developer
‣Size
‣Real-Time
‣Analysis beyond Basics
‣Interactivity
‣API access

74

OKFN
http://okfn.org/
http://schoolofdata.org/
http://ckan.org/
http://2014.okfestival.org/

76

Other
http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers
http://www.cccb.org/ca/marcperiodisme_de_dades-44600
http://www.visualisingdata.com/index.php/
resources/
81

Data Sources
http://ckan.org/
http://www.theguardian.com/data
http://www.google.com/publicdata/
http://data.worldbank.org/
http://datahub.io/
http://www.quora.com/Data/Wherecan-I-ﬁnd-large-datasets-open-to-thepublic
82

Any questions?
Thanks for your attention!

83

Data Tools cosystem_for_non_programmers

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (19)

Destacado

Destacado (9)

Similar a Data Tools cosystem_for_non_programmers

Similar a Data Tools cosystem_for_non_programmers (20)

Más de itnig

Más de itnig (20)

Último

Último (20)

Data Tools cosystem_for_non_programmers