Restats

1. r restats Processing real-time data to cubes on data structure servers

2. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons lea scaling scales

3. theProblem r Rails mySQL Redis Application Warehouse Statistics Trackings Calculate every X hours Doesn’t Scale! 800 M+ 500 GB+

6. r theSolution Rails Redis r Application Statistics Trackings Process restats Real-Time

9. what is restats? r Redis Statistics advanced key-value store data cubes data structure server geo-location cubes

11. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons lea scaling scales

12. r how does it work: tracking { Rails "created_at": "2012-10-18T21:34:56+02:00", "type": "adrequest", Produces "location_id": 1, "campaign_id": 1, "publisher_id": 1, JSON "latitude": "53.36669921875", "longitude": "-1.5", Application "bannersize": "320x50" } At 2012-10-18, there was an adrequest for campaign 1 at 53.366,-1.5 for the bannersize 320x50

13. r how does it work: dimensions Attribute Geo Time type location_id day latitude campaign_id month longitude publisher_id year bannersize

14. how does it work: cubing r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Maps 1 Attribute 1 Redis Key Combination Redis INCR Time Attribute day-month-year type Maps INCR 1st Depth day-month-year-type: 18-10-2012 adrequest Redis INCR 18-10-2012-adrequest

15. how does it work: 1st depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attribute day-month-year type Maps INCR day-month-year-type: 18-10-2012 adrequest Redis INCR 18-10-2012-adrequest Time Attribute month-year type Maps INCR month-year-type: 10-2012 adrequest Redis INCR 10-2012-adrequest Time Attribute year type Maps INCR year-type: 2012 adrequest Redis INCR 2012-adrequest

16. how does it work: 2nd depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attribute day-month-year type location_id Maps INCR day-month-year-type-location_id: 18-10-2012 adrequest 1 18-10-2012-adrequest-1 Redis INCR Time Attribute day-month-year type campaign_id Maps INCR day-month-year-type-campaign_id: 18-10-2012-adrequest-1 ... 18-10-2012 adrequest 1 Redis INCR Combinations

17. how does it work: 3rd depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attribute day-month-year type location_id campaign_id Maps INCR day-month-year-type-location_id -campaign_id: 18-10-2012 adrequest 1 1 18-10-2012-adrequest-1-1 Redis INCR Time Attribute day-month-year type location_id publisher_id Maps INCR day-month-year-type-location_id -publisher_id: 18-10-2012-adrequest-1-1 ... 18-10-2012 adrequest 1 1 Redis INCR Combinations

20. how does it work: keys r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize n = attribute dimensions (5) d = max depth (3) t = time dimensions (3) Total Amount of = Keys per Tracking (5 + 10 + 10) • 3 = 75 Keys

21. how does it work: increment r Computed Keys day-month-year-type:18-10-2012-adrequest month-year-type:10-2012-adrequest year-type:2012-adrequest day-month-year-type-location_id-publisher_id:18-10-2012-adrequest-1-1 ... Digest SHA1 Redis Batch INCR INCR 76f374viq4vi7fti374oo76 INCR o8qf374fvlqizfvil7v2o8v INCR fqwevlrfli74l3fzlizflzf INCR fbliaguzvf736i4cjsztudz ...

22. how does it work: geo cubing r Redis Sorted Set set with a score for each member ZADD key score member # adds a member to key ZINCRBY key 1 member # increases member by 1 Mapping Rounded position specifies precision member = “latitude:longitude” score = # of trackings at that location ZINCRBY sdvouzg 1 53.3666:-1.5 ZRANGE sdvouzg 0 -1 withscores 1) 54.3215:0.3456 2) 5 3) 53.3666:-1.5 4) 2763

23. how does it work: geo increment r Computed Keys INCR 76f374viq4vi7fti374oo76 INCR o8qf374fvlqizfvil7v2o8v INCR fqwevlrfli74l3fzlizflzf INCR fbliaguzvf736i4cjsztudz ... Round Position ZINCRBY 1 Redis Batch ZINCRBY INCR 76f374viq4vi7fti374oo76 ZINCRBY 76f374viq4vi7fti374oo76g 1 53.3666:-1.5 INCR o8qf374fvlqizfvil7v2o8v ZINCRBY o8qf374fvlqizfvil7v2o8vg 1 53.3666:-1.5 INCR fqwevlrfli74l3fzlizflzf ZINCRBY fqwevlrfli74l3fzlizflzfg 1 53.3666:-1.5 INCR fbliaguzvf736i4cjsztudz ZINCRBY fbliaguzvf736i4cjsztudzg 1 53.3666:-1.5 ...

26. r the queryinterface Query converts the attributes to keys multi-fetch to save round-trip time

27. the queryresponse r Query-able Ruby Hash allows to iterate over the responses merged aggregations

28. the geoqueryresponse r Query-able Ruby Hash list of positions scores are trackings at that position

31. the scalingstrategy r Trackings In add worker PR > 0 remove worker PR < 0 monitor value over a time period, e.g. 10 mins Processing Ratio = Trackings Out

34. the speedtest r 2.4 GHz Intel Core i5 with 8 GB 1067 MHz DDR3 5 attribute dimensions 5 max depth geo cube enabled 85 processings / second (5 + 10 + 10 + 5) • 3 • 2 = 180 keys / tracking = 15,300 keys / second bottleneck = cpu; calculation of combinations

37. the lessonslearned r not NoSQL, newSQL! we’re still querying, right? dare to customize know your data you’ll be rewarded dynamics kills performance process on write if you need fast read-access denormalize no CRON in this field, it’s your friend recurring tasks don’t scale use mini-benchmarks when you’re unsure during implementation

38. r now let’sscale! try scales! itscales.org s thomas fankhauser, m.sc. tommy@southdesign.de southdesign.de

Restats

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (12)

Más de Thomas Fankhauser

Más de Thomas Fankhauser (6)

Último

Último (20)

Restats