How we got to 1 millisecond latency in 99% under repair, compaction, and flushes at ScyllaDB
•
0 recomendaciones•886 vistas
Scylla is an open source reimplementation of Cassandra which performs up to 10X with drop in-replacement compatibility. At ScyllaDB, performance matters but even more importantly, stable performance under any circumstances. A key factor for our consistent performance is our reliance on userspace schedulers. Scheduling in userspace allows the application, the database in our case to have better control on the different priorities each task has and to provide an SLA to selected operations. Scylla used to have an I/O scheduler and recently won a CPU scheduler. At ScyllaDB, we make architectural decisions that provide not only low latencies but consistently low latencies at higher percentiles. This begins with our choice of language and key architectural decisions such as not using the Linux page-cache, and is fulfilled by autonomous database control, a set of algorithms, which guarantees that the system will adapt to changes in the workload. In the last year, we have made changes to Scylla that provide latencies that are consistent in every percentile. In this talk, Dor Laor will recap those changes and discuss what ScyllaDB is doing in the future.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a How we got to 1 millisecond latency in 99% under repair, compaction, and flushes at ScyllaDB
Similar a How we got to 1 millisecond latency in 99% under repair, compaction, and flushes at ScyllaDB (20)
Más de ScyllaDB
Más de ScyllaDB (20)
Último
Último (20)
How we got to 1 millisecond latency in 99% under repair, compaction, and flushes at ScyllaDB
- 2. + What’s ScyllaDB; Why ScyllaDB ; How ScyllaDB + Chase for lowest 99% tail latency Today we will cover:
- 4. Production Databases used? What’s the use-case?
- 5. Production Databases used? What’s the use-case? Your biggest headache?
- 6. No clear winner in NoSQL Challenges: • Cost • Lock-in Challenges: • Scale • Multi DC • Latency Challenges: • Not persistent • Manageability Challenges: • Price/performance • Complexity • JVM..
- 7. What we do: Scylla, towards the best NoSQL Cassandra
- 8. What we do: Scylla, towards the best NoSQL + > 1 million OPS per node + < 1ms 99% latency + Auto tuned + Scale up and out + Open source + Large community (piggyback on Cassandra) + Blends in the ecosystem- Spark, Presto, time series, search, ..
- 9. Where Scylla is deployed?
- 10. Scylla benchmark by Samsung op/s
- 11. 1.6 MHz
- 12. Cassandra Scylla Throughput: Cannot utilize multi-core efficiently Scales linearly - shard-per-core Latency: High due to Java and JVM’s GC Low and consistent - own cache Complexity: Intricate tuning and configuration Auto tuned, dynamic scheduling Admin: Maintenance impacts performance SLA guarantee for admin vs serving
- 14. 14 ▪ ▪ ▪ ▪
- 15. Scylla (w/o cache) vs Cassandra + Memcached Scylla Cassandra Diff% Requests/ Minute 12,000,000 500,000 (memcache handles 11,500,000) 24X AVG Latency 4 ms 8 ms 2X Max Latency 8 ms 35 ms 3.5X Hardware 9 machines 30+9 machines 4.3X 15 Cassandra’s latency Scylla’s latency
- 17. DBAAS Hardware Drivers Presto Elastic Search K/V Large blob store Cache/ in-memory Wide row/ Time series Secondary index Graph Spark Kafka
- 18. Seastar, Scylla’s engine: “All things async”
- 19. Threads Shards Two-level sharding - shard per core
- 20. SCYLLA DB: Network Comparison Kernel Cassandra TCP/IPScheduler queuequeuequeuequeuequeue threads NIC Queues Kernel Traditional stack SeaStar’s sharded stack Memory Lock contention Cache contention NUMA unfriendly Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace No contention Linear scaling NUMA friendly Core Database Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue Userspace
- 21. SCYLLA IS DIFFERENT ❏ DMA ❏ Log structured ❏ merge tree ❏ DBaware cache ❏ Userspace I/O ❏ scheduler ❏ NUMA friendly ❏ Log structured allocator ❏ Zero copy ❏ Thread per core ❏ lock-free ❏ Task scheduler ❏ Reactor programing ❏ C++14 ❏ Multi queue ❏ Poll mode ❏ Userspace TCP/IP
- 22. IBM Power - 20 core, 160 cpus
- 23. ▪ ▪ ▪ ▪ ▪ ▪
- 24. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog Monitor Memory Monitor Adjust priority Adjust priority WAN CPU Design Decision: #7..#10 Autonomous Database
- 28. ▪ o o ▪ latency mean : 51.9 latency median : 9.8 latency 95th percentile : 125.6 latency 99th percentile : 1184.0 latency 99.9th percentile : 1991.2
- 29. ▪ o o ▪ latency mean : 51.9 latency median : 9.8 latency 95th percentile : 125.6 latency 99th percentile : 1184.0 (x 22) latency 99.9th percentile : 1991.2 (x 38)
- 30. ▪ o
- 31. ▪ o ▪ o
- 33. latency mean : 54.9 latency median : 43.5 latency 95th percentile : 126.9 latency 99th percentile : 253.9 latency 99.9th percentile : 364.6
- 34. latency mean : 54.9 latency median : 43.5 latency 95th percentile : 126.9 latency 99th percentile : 253.9 (x 4.6) latency 99.9th percentile : 364.6 (x 6.6)
- 36. Traditional stack Scylla’s stack Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise Task Promise Task Promise Task Promise Task CPU Promise is a pointer to eventually computed value Task is a pointer to a lambda function Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Scheduler CPU Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread Stack Thread is a function pointer Stack is a byte array from 64k to megabytes
- 37. ▪ ▪ ▪
- 38. ▪ ▪ ▪
- 40. Query Commitlog Compaction Queue Queue Queue Userspace I/O Scheduler Disk Max useful disk concurrency I/O queued in FS/deviceNo queues
- 41. ▪ o o ▪ o
- 42. ▪ o ▪ o o
- 45. Memtable Seastar Scheduler Compaction Query Repair Commitlog SSD Compaction Backlog controller Memory controller Adjust priority Adjust priority WAN CPU Design Decision: #7..#10 Autonomous Database
- 48. latency mean : 0.6 latency median : 0.5 latency 95th percentile : 0.8 latency 99th percentile : 3.6 (x 6.0) latency 99.9th percentile : 4.5 (x 7.5) latency mean : 0.4 latency median : 0.4 latency 95th percentile : 0.6 latency 99th percentile : 0.8 (x 2.0) latency 99.9th percentile : 1.9 (x 4.7)
- 49. ▪ ▪ o ▪ o o