This document discusses using HBase to store and query multi-dimensional data cubes. It describes how data cubes can be modeled in HBase by storing projections instead of distinct dimensions, and pre-computing all aggregations. An example row key and column family structure is given to store pageview counts by country, user agent, domain, and time. The document also introduces Zohmg, a distributed, multi-dimensional, time-series data store built on HBase that can be configured and mapped using YAML and Python scripts.

1. Hypercubes in HBase
Fredrik Möllerstrand <fredrik@last.fm>
Hadoop User Group UK, April 14 2009

2. Hello.
• Per Andersson, Fredrik Möllerstrand
• Chalmers University of Technology, Sweden
• Master thesis at last.fm

3. stats.last.fm
• Web statistics for in-house use.
• Served out of mysql.

4. stats.last.fm
• y-axis: pageviews.
• x-axis: time.
• also: countries.

5. stats.last.fm
SELECT pageviews, country
FROM webstatistics
GROUP BY country;

6. SQL: Star schema
• Facts table & dimension tables.
• Joins!

7. Data Cube Foundations
• n-dimensional cube
• attribute => dimension
• attribute value => measurement

8. Data Cube Foundations
• dimensionality reductions
• projections

9. Data Cube Foundations
• Aggregation: sum, count, average, &c.
• Data cubes modeled as:
in RDBMSs, modeled as star schemas.
in HBase, modeled with column-families.

10. Data Cubes in HBase
• Store projections,
not distinct dimensions.
• Pre-compute *everything*.

11. Data Cubes in HBase
• Rowkey: unit + time
i.e. ‘pageviews-20090414’
• One column-family for every projection.
i.e. ‘country-useragent’
• One qualifier per point in n-space.
i.e. ‘US-safari’, ‘NO-opera’, &c.

12. The SQL-DB Problem
• Too much data to keep in memory.
• Plenty of joins makes queries complex.
• Can’t serve at mouse click rate.

13. The Solution;
A data store that is:
• Distributed
• Multi-dimensional
• Magnetic(!)
• Just general enough

14. Enter: Zohmg.

15. Zohmg;
A data store that is:
• Distributed
• Multi-dimensional
• Time-series-based
• Magnetic(!)
• Just general enough

16. Tech
• Rides on the back of Dumbo.
• Stores aggregates in HBase.
• Serves JSON.

17. Zohmg
• $> setup.py
# create hbase database.
• $> import.py --mapper weblogs.py
# run dumbo job.
• $> serve.py
# start web server.

18. Developers, developers.
• Configuration - yaml.
• Mapper - python.

19. User’s configuration.
project_name: webmetrics
dimensions:
- country
- domain
- useragent
- usertype
units:
- pageviews
projections:
country:
- country
domain-usertype:
- domain
- usertype
country-domain-useragent-usertype:
- country
- domain
- useragent
- usertype

20. User’s mapper.
def map(key, value):
from lfm.data.parse import web
log = web.parse(value)
dimensions = {'country' : geoip(log.host),
'domain' : log.domain,
'useragent' : classify(log.useragent),
'usertype' : ("user", "anon")[log.userid == None]
}
values = {'pageviews' : 1}
yield log.timestamp, dimensions, values

21. Example.

22. Dimensions in HBase
• Column-family:
country-useragent-domain
• Qualifier:
US-firefox-last.fm

23. Questions?