Publicidad

Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022

InfluxData
20 de Oct de 2022
Publicidad

Más contenido relacionado

Presentaciones para ti(20)

Similar a Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022(20)

Publicidad

Más de InfluxData(20)

Publicidad

Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022

  1. The Journey of InfluxDB Paul Dix Founder & CTO @pauldix
  2. 1.0 2.0 IOx 2013-2019 2019-2022 2022-
  3. 1.0 2.0 IOx 2013-2019 2019-2022 2022-
  4. Talk themes • What are time series? • What is a time series database? (TSDB)
  5. Time series are a method for interpreting and understanding data
  6. [[23.2, t1], [64.6, t2], [54.1, t3], [77.8, t4], [62.1, t5], [99.6, t6], …]
  7. Two ways to think about TSDBs: • Store values & metadata directly • Compute on the fly from raw data
  8. Back in 2013…
  9. Round Robin Database Structure t0 marker t 1 t2 t20 Length: 20 Delta: 60s Start Epoch: ...
  10. statsD
  11. mysql.bytes_received 1287333217 327810227706 schema=foo host=db1 mysql.bytes_sent 1287333217 6604859181710 schema=foo host=db1 mysql.bytes_received 1287333232 327812421706 schema=foo host=db1 mysql.bytes_sent 1287333232 6604901075387 schema=foo host=db1 mysql.bytes_received 1287333321 340899533915 schema=foo host=db2 mysql.bytes_sent 1287333321 5506469130707 schema=foo host=db2 Metric name Epoch Value Tags
  12. TSDB that store time series • Time series data • Metadata • Hierarchical (Graphite) • Tags (OpenTSDB)
  13. Fall of 2013…
  14. InfluxDB 0.8 data model C1 C2 C3 C4 C5 C6 time measurement 1 measurement2 C1 C2 C3 C4 time
  15. select value from response_times where time > now() - 1h limit 1000;
  16. select value from response_times where time > now() - 1h and server = ‘foo’ limit 1000; Range scan
  17. API changes: tags and line protocol
  18. API changes: tags and line protocol foo,tag1=bar,tag2=asdf field1=23i,field2=2.4,field3=“hello”,field4=true 12344321 measurement name Fields Timestamp Tags
  19. TSM Data Block 1 Series 1 Series 2 Block 2 Block 3 Series 3 Series 4 Block 1 Block 2 Block 3 Block 4
  20. Inverted Index weather,city=Denver,state=CO,zip=80222 temp=62.3,humidity=32 weather,city=Bellevue,state=WA,zip=98007 temp=50.7,humidity=76 weather,city=Brooklyn,state=NY,zip=11249 temp=58.2,humidity=55 weather,city=Denver,state=CO,zip=80222#temp – 1 weather,city=Denver,state=CO,zip=80222#humidity - 2 weather,city=Bellevue,state=WA,zip=98007#temp - 3 weather,city=Bellevue,state=WA,zip=98007#humidity – 4 weather,city=Brooklyn,state=NY,zip=11249#temp – 5 weather,city=Brooklyn,state=NY,zip=11249#humidity - 6 Line Protocol Series
  21. Inverted Index weather,city=Denver,state=CO,zip=80222 temp=62.3,humidity=32 weather,city=Bellevue,state=WA,zip=98007 temp=50.7,humidity=76 weather,city=Brooklyn,state=NY,zip=11249 temp=58.2,humidity=55 weather,city=Denver,state=CO,zip=80222#temp – 1 weather,city=Denver,state=CO,zip=80222#humidity - 2 weather,city=Bellevue,state=WA,zip=98007#temp - 3 weather,city=Bellevue,state=WA,zip=98007#humidity – 4 weather,city=Brooklyn,state=NY,zip=11249#temp – 5 weather,city=Brooklyn,state=NY,zip=11249#humidity - 6 Line Protocol Series
  22. Inverted Index – Posting Lists Measurements Fields Tags weather – [1,2,3,4,5,6] temp – [1,3,5] humidity – [2,4,6] city=Denver – [1,2] city=Bellevue – [3,4] city=Brooklyn – [5,6] state=CO – [1,2] state=NY – [5,6] state=WA – [3,4] zip=11249 – [5,6] zip=80222 – [1,2] zip=98007 – [3,4] select temp from weather where time > now() - 1h and state = “NY”;
  23. Inverted Index – Posting Lists Measurements Fields Tags weather – [1,2,3,4,5,6] temp – [1,3,5] state=NY – [5,6] select avg(temp) from weather where time > now() - 1h and state = “NY”; Intersect these postings lists = 5
  24. Inverted Index – Posting Lists Measurements Fields Tags weather – [1,2,3,4,5,6] temp – [1,3,5] select avg(temp) from weather where time > now(); [1, 3, 5]
  25. Time Series Blocks Lookup - [4, 73, 128, 222, 245, 278, 333, 561]
  26. Design Tradeoffs • Fast Low Cardinality Lookups & Execution • Inverted Index maintenance on Ingest • Sorted time series key space • Expensive and Slow Higher Cardinality
  27. InfluxDB had evolved to become a TSDB that stored time series
  28. 2.0 - Flux, all-in-one, cloud first delivery
  29. The problem with time series: you summarize everything before you get to the database
  30. http_request_duration_seconds_bucket{le=“0.1”} 23 http_request_duration_seconds_bucket{le=“0.5”} 30 http_request_duration_seconds_bucket{le=“2”} 34 http_request_duration_seconds_bucket{le=“+Inf”} 130 http_request_duration_seconds_sum http_request_duration_seconds_count
  31. http_request_duration_seconds_bucket{host=“foo”, le=“0.1”} 23 http_request_duration_seconds_bucket{host=“foo”, le=“0.5”} 30 http_request_duration_seconds_bucket{host=“foo”, le=“2”} 34 http_request_duration_seconds_bucket{host=“foo”, le=“+Inf”} 130 http_request_duration_seconds_sum{host=“foo”} http_request_duration_seconds_count{host=“foo”}
  32. http_request_duration_seconds_bucket{host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{host=“foo”, service=“asdf”} http_request_duration_seconds{count host=“foo”, service=“asdf”}
  33. http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“1423”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“1423”, host=“foo”, service=“asdf”}
  34. http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“1423”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“1423”, host=“foo”, service=“asdf”} http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“52”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“52”, host=“foo”, service=“asdf”}
  35. http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“1423”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“1423”, host=“foo”, service=“asdf”} http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“52”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“52”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“52”, host=“foo”, service=“asdf”} http_request_duration_seconds_bucket{customer=“999”, host=“foo”, le=“0.1”, service=“asdf”} 23 http_request_duration_seconds_bucket{customer=“999”, host=“foo”, le=“0.5” , service=“asdf”} 30 http_request_duration_seconds_bucket{customer=“999”, host=“foo”, le=“2” , service=“asdf”} 34 http_request_duration_seconds_bucket{customer=“999”, host=“foo”, le=“+Inf” , service=“asdf”} 130 http_request_duration_seconds_sum{customer=“999”, host=“foo”, service=“asdf”} http_request_duration_seconds_count{customer=“999”, host=“foo”, service=“asdf”}
  36. Pull represents full state every scrape
  37. Push an event http_request customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”, response_time=23
  38. What about tracing? • Level of detail so high you must sample
  39. Match precision to use case • Query log • Auditing • Usage metering • Analytics of many kinds
  40. Columnar databases solve performance problems for storing events
  41. Back to Tables City State Zip Temp Humidity Time Denver CO 80222 62.3 32 1 Denver CO 80222 61.2 32 2 Bellevue WA 98007 50.7 76 1 Bellevue WA 98007 50.5 76 2 Brooklyn NY 11249 58.2 55 1 Brooklyn NY 11249 57.0 55 2 Tags Timestamp Fields
  42. Back to Tables City State Zip Temp Humidity Time Denver CO 80222 62.3 32 1 Denver CO 80222 61.2 32 2 Bellevue WA 98007 50.7 76 1 Bellevue WA 98007 50.5 76 2 Brooklyn NY 11249 58.2 55 1 Brooklyn NY 11249 57.0 55 2 select avg(temp) from weather where time < 100 and state = “NY”;
  43. Solving the performance problem • Per column compression (dictionary, RLE) • Vectorized execution • Parallelism • Partitioning • Pruning
  44. Per-column Compression City State Zip Temp Humidity Time Denver CO 80222 62.3 32 1 Denver CO 80222 61.2 32 2 Bellevue WA 98007 50.7 76 1 Bellevue WA 98007 50.5 76 2 Brooklyn NY 11249 58.2 55 1 Brooklyn NY 11249 57.0 55 2 Repeated values: Dictionary Encoding, RLE
  45. Partitioned Layout measurement 1 C1 C2 C3 C4 time 2022-11--01 2022-11--02 C1 C2 C3 C4 time
  46. Query Properties • Execution against all rows from included partitions • Accessing a single series through brute force • 1B rows/sec/core
  47. Partitioned Layout measurement2 C1 C2 C3 time 2022-11--01 2022-11--02 region=west region=east C1 C2 C3 time
  48. Time Series Blocks Partitions
  49. Problems to Solve • Wide Tables (many columns) • Sparse Tables • Repartitioning (automatic) • Secondary Indexes
  50. Cloud Databases Solve for Scale
  51. Federation Solves for Scale
  52. InfluxDB IOx • Columnar database • SQL natively supported • Cloud Database • Federated at the Edge
  53. Available in InfluxDB Cloud Today!
  54. InfluxDB IOx is a TSDB for storing AND calculating time series
  55. T H A N K Y O U
Publicidad