For data, and data science, to be the fuel of the 21th century, data driven applications should not be confined to dashboards and static analyses. Instead they should be the driver of the organizations that own or generates the data. Most of these applications are web-based and require real-time access to the data. However, many Big Data analyses and tools are inherently batch-driven and not well suited for real-time and performance-critical connections with applications. Trade-offs become often inevitable, especially when mixing multiple tools and data sources. In this talk we will describe our journey to build a data driven application at a large Dutch financial institution. We will dive into the issues we faced, why we chose Python and pandas and what that meant for real-time data analysis (and agile development). Important points in the talk will be, among others, the handling of geographical data, the access to hundreds of millions of records as well as the real time analysis of millions of data points.

1. GoDataDriven
PROUDLY PART OF THE XEBIA GROUP
Real time data driven applications
Giovanni Lanzani
Data Whisperer
Using Python + pandas as back end

2. Who am I?

3. Who am I?
2008-2012: PhDTheoretical Physics
2012-2013: KPMG
2013-Now: GoDataDriven

4. Feedback
@gglanzani

5. Real-time, data driven app?
•No store and retrieve;
•Store, {transform, enrich, analyse} and retrieve;
•Real-time: retrieve is not a batch process;
•App: something your mother could use:
SELECT attendees !
FROM!pydataberlin2014 !
WHERE password = '1234';

6. Get insight about event impact

7. Get insight about event impact

8. Get insight about event impact

9. Get insight about event impact

10. Get insight about event impact

11. Is it Big Data?

12. Is it Big Data?
•Raw logs are in the order of 40TB;
•We use Hadoop for storing, enriching and pre-
processing;
•(10 nodes, 24TB per nodes)

13. Challenges
1. Privacy;
2. Huge pile of data;
3. Real-time retrieval;
4. Some real-time analysis.

14. 1. Privacy

15. 1. Privacy

16. 3. Real-time retrieval
•Harder than it looks;
•Large data;
•Retrieval is by giving date, center location + radius.

17. 4. (Some) real-time analysis

18. Architecture
AngularJS app.py
helper.py
REST
Front-end Back-end
JSON

19. JS - 1

20. JS - 2

21. Flask
from flask import Flask!
app = Flask(__name__)!
!
@app.route("/hello")!
def hello():!
return "Hello World!"!
!
if __name__ == "__main__":!
app.run()

22. Flask
from flask import Flask!
app = Flask(__name__)!
!
@app.route("/hello")!
def hello():!
return "Hello World!"!
!
if __name__ == "__main__":!
app.run()

23. app.py example
@app.route('/api/<postcode>/<date>/<radius>', methods=['GET'])!
@app.route('/api/<postcode>/<date>', methods=['GET'])!
def datapoints(postcode, date, radius=1.0):!
...!
stats, timeline, points = helper.get_json(postcode, date, radius)!
return … # returns a JSON object for AngularJS

24. data example
date hour id_activity postcode hits delta sbi
2013-01-01 12 1234 1234AB 35 22 1
2013-01-08 12 1234 1234AB 45 35 1
2013-01-01 11 2345 5555ZB 2 1 2
2013-01-08 11 2345 5555ZB 55 2 2

25. helper.py example
def get_json(postcode, date, radius):!
...!
!
lat, lon = get_lat_lon(postcode)!
postcodes = get_postcodes(postcode, radius)!
!
data = get_data(postcodes, dates)!
!
stats = get_statistics(data, sbi)!
timeline = get_timeline(data, sbi)!
!
return stats, timeline, data.to_json(orient='records')

26. helper.py example
def get_json(postcode, date, radius):!
...!
!
lat, lon = get_lat_lon(postcode)!
postcodes = get_postcodes(postcode, radius)!
!
data = get_data(postcodes, dates)!
!
stats = get_statistics(data, sbi)!
timeline = get_timeline(data, sbi)!
!
return stats, timeline, data.to_json(orient='records')

27. helper.py example
def get_statistics(data, sbi):!
sbi_df = data[data.sbi == sbi] # filter by sbi!
!
hits = sbi_df.hits.sum() # sum the hits !
delta_hits = sbi_df.delta.sum() # sum the delta hits!
!
if delta_hits:!
percentage = (hits - delta_hits) / delta_hits!
else:!
percentage = 0!
!
return {"sbi": sbi, "total": hits, "percentage": percentage}

28. helper.py example
def get_statistics(data, sbi):!
sbi_df = data[data.sbi == sbi] # filter by sbi!
!
hits = sbi_df.hits.sum() # sum the hits !
delta_hits = sbi_df.delta.sum() # sum the delta hits!
!
if delta_hits:!
percentage = (hits - delta_hits) / delta_hits!
else:!
percentage = 0!
!
return {"sbi": sbi, "total": hits, "percentage": percentage}

29. helper.py example
def get_timeline(data, sbi):!
df_sbi = data.groupby([“date”, “hour", "sbi"]).!
aggregate(sum)!
return df_sbi

30. helper.py example
def get_timeline(data, sbi):!
df_sbi = data.groupby([“date”, “hour", "sbi"]).!
aggregate(sum)!
return df_sbi

31. helper.py example
def get_json(postcode, date, radius):!
...!
!
lat, lon = get_lat_lon(postcode)!
postcodes = get_postcodes(postcode, radius)!
dates = date.split(';')!
!
data = get_data(postcodes, dates)!
!
stats = get_statistics(data)!
timeline = get_timeline(data, dates)!
!
return stats, timeline, data.to_json(orient='records')

32. Who has my data?

33. Who has my data?
•First iteration was a (pre)-POC, less data (3GB vs
500GB);
•Time constraints;
•Oeps:

34. Who has my data?
•First iteration was a (pre)-POC, less data (3GB vs
500GB);
•Time constraints;
•Oeps:
import pandas as pd!
...!
source_data = pd.read_csv("data.csv", …)!
...!
def get_data(postcodes, dates):!
result = filter_data(source_data, postcodes, dates)!
return result

35. Advantage of “everything is a df”
Pro:
•Fast!!
•Use what you know
•NO DBA’s!
•We all love CSV’s!
!
!
!

36. Advantage of “everything is a df”
Pro:
•Fast!!
•Use what you know
•NO DBA’s!
•We all love CSV’s!
!
!
!
Contra:
•Doesn’t scale;
•Huge startup time;
•NO DBA’s!
•We all hate CSV’s!

37. If you want to go down this path
•Set the dataframe index wisely;
•Align the data to the index:
!
•Beware of modifications of the original dataframe!
source_data.sort_index(inplace=True)

38. If you want to go down this path

39. If you want to go down this path

40. If you want to go down this path

41. If you want to go down this path
The reason pandas is faster is because I came
up with a better algorithm

42. If you don’t…
data = get_data(postcodes, dates)
AngularJS app.py
helper.py
REST
Front-end Back-end
JSON

43. If you don’t…
data = get_data(postcodes, dates)
database.py
Data
psycopg2
AngularJS app.py
helper.py
REST
Front-end Back-end
JSON

44. If you don’t…
data = get_data(db, postcodes, dates)
database.py
Data
psycopg2
AngularJS app.py
helper.py
REST
Front-end Back-end
JSON

45. Handling geo-data
def get_json(postcode, date, radius):!
"""!
...!
"""!
lat, lon = get_lat_lon(postcode)!
postcodes = get_postcodes(postcode, radius)!
dates = date.split(';')!
!
data = get_data(postcodes, dates)!
!
stats = get_statistics(data)!
timeline = get_timeline(data, dates)!
!
return stats, timeline, data.to_json(orient='records')

46. Issues?!
•With a radius of 10km, in Amsterdam, you get 10k
postcodes.You need to do this in your SQL:
!
!
!
•Index on date and postcode, but single queries
running more than 20 minutes.
SELECT * FROM datapoints !
WHERE !
date IN date_array !
! ! AND !
! ! ! postcode IN postcode_array;

47. Postgres + Postgis (2.x)
PostGIS is a spatial database extender for PostgreSQL.
Supports geographic objects allowing location queries
SELECT *!
FROM datapoints!
WHERE ST_DWithin(lon, lat, 1500)!
AND dates IN ('2013-02-30', '2013-02-31');!
-- every point within 1.5km !
-- from (lat, lon) on imaginary dates

48. Other db’s?

49. Steps to solve it
1. Align data on disk by date;
2. Use the temporary table trick:
!
!
!
!
3. Lose precision: 1234AB→1234
4. (Compression)
CREATE TEMPORARY TABLE tmp (postcodes STRING NOT NULL PRIMARY KEY);!
INSERT INTO tmp (postcodes) VALUES postcode_array;!
!
SELECT * FROM tmp!
JOIN datapoints d!
ON d.postcode = tmp.postcodes!
WHERE!
d.dt IN dates_array;

50. GoDataDriven
We’re hiring / Questions? / Thank you!
@gglanzani
giovannilanzani@godatadriven.com
Giovanni Lanzani
Data Whisperer