As-is state of WDMMG/OpenSpending project, 2011-06 (OKCON2011)

1. OpenSpending.org
Current State
Stefan Urbanek
stefan.urbanek@gmail.com June 2011
@Stiivi

2. Data

3. Data Store

4. Collections
data metadata
dataset
entry
entity classifier dimension

5. Dimension types
classifier collection
entity collection
string scalar field
float scalar field
date hash/dictionary
no implicit hierarchy

6. Time
■ time is a range from to
■ compound object
■ dataset specifies date for analytics
■ in cubes it is split into year + manth + day
■ date hierarchy is hardwired in app logic

7. amount
the only one measure

8. Extraction,
Transformation, Loading

9. OpenSpending.org ETL – overview
mapping wdmmg
openspending pylons application
paster pylons managed data
CSV file csvimport store with ORM
ckan-datapkg
datapkg paster load
"raw" data locally cached
resource resource
wdmmg-ext
extraction modules
loading
loading
scripts
loading
scripts
scripts
cofog still runing from here

10. classification
–
entity
–
date
from
to + +
amnt
CSV file
entry

11. Aggregations

12. ≈
dataset* cube
*still as dimension

14. roll-up is aggregated
“on the fly”