3. What is Phoenix? “a platform that allows data to be transferred seamlessly to the people who want it, presented in the way they want to see it, regardless to the source, type or location of the inputs”
4. What sort of data? Time-series Sensor readings External data feeds Meta data Adding context Credits: http://commons.wikimedia.org/wiki/File:CPI-plot.png
5. What is Phoenix? “a web basedplatform capable of ingesting, combining, processing and distributing data - in real time”
7. Why develop a new platform? No existing platforms for the use cases Typical solutions use Excel, SQL cannot cope with amount of data no real-time capability
8. Real-time Analysis COMPLEX EVENT PROCESSING STREAM COMPUTING Reporting Processing Big Data Target Customer BUSINESS INTELLIGENCE BATCHPROCESSING Batch Analysis
9. Real-time Analysis COMPLEX EVENT PROCESSING STREAM COMPUTING PHOENIXPLATFORM Reporting Processing Big Data Target Customer BUSINESS INTELLIGENCE BATCHPROCESSING Batch Analysis
10. Project Timelines 2010 Research Market Customers Design Architecture Data model Development 4 developers Rapid! Release Version 1 Phoenix local Q1 Q2 Q3 Q4
24. Data Stores Phoenix Customer APPLICATION STREAM PROCESSOR ALGORITHMS DATASTORE DATA FEEDS TIME SERIES MARKERS
25. Marker Store T I M E FROM FROM TO L I N K E D H I G H W A Y S
26. Marker Store { "fromId" : ”Athlete1", "toId" : ”Ergometer", "startTime" : 1, "endTime" : 50 } { "fromId" : ”Athlete2", "toId" : ”Ergometer", "startTime" : 60, "endTime" : null } L I N K E D H I G H W A Y S
27. Marker Store { ”deviceId" : ”1", ”calibrationType" : ”sensor1", ”validFrom" : 1, ”validTo" : null “calibrations" : [ {”expected”: 10.0, “actual”: 12.9 }, {”expected”: 50.0, “actual”: 61.7 } ] } C A L I B R A T I O N M A R K E R S
30. Marker Store API JSON POST to /linkedhighway/create GET from /linkedhighway/findactive/{fromId} POST to /calibration/create GET from /calibration/{markerId}
31. Phoenix Data Store Ergometer Force Reading (Newtons) {X1, Y1} {Xn, Yn} Distance Travel (Metres) {X1, Y1} {Xn, Yn}
32. Phoenix Data Store { “_id”: “HandleForce”, “xDomainType”: “SINT64”, “xUnits”: “ms”, “yDomainType”: “DOUBLE”, “yUnits”: “N” } L A N E M E T A D A T A
33. Phoenix Data Store { ”_id" : 1, ”y" : 0 } { “_id”: 2, “y”: 0.5 } { “_id”: 3, “y”: 0.707 } { “_id”: 4, “y”: 0.866 } { “_id”: 5, “y”: 1 }… T I M E S E R I E S
35. Phoenix Data Store Stack Tcp JbossNetty Server – NIO Google Protocol Buffers Java
36. Phoenix Data Store API Commit connect() openLane(…) persist(x, y) disconnect() Query connect() submitQuery(…) returns a Cursor disconnect() J A V A C L I E N T
38. Phoenix Local AMD Athlon II 2.8GHz 64-bit 640GB 2.5” Hard Drive SATA 5400 rpm 2 X 2GB DDR3 Ubuntu 10.10 EXT4
39. Issues Faced No Journaling in Mongo 1.6 (Fixed in 1.8) Compression DB repair did not work once – unclean shutdown Assertion failure magic == 0x41424344 db/pdfile.h