This document outlines the process for auditing an implementation of the LDBC Social Network Benchmark (SNB). It describes downloading the necessary driver and data generator software. The main metrics for benchmarking are throughput at scale and query latency constraints during crash recovery testing. It provides guidelines for implementing the driver, loading data, running queries for 2 hours, and verifying crash recovery. Audit results will report throughput, load time, recovery time and system configuration. Rules prohibit hinted query plans, precomputed indexes, and require serializable consistency.

1. LDBC SNB Benchmark Auditing
Moritz Kaufmann
Technische Universität München

2. Process
Rules
+
Metrics
+

3. 2014-03-05 3
Where to start?
Download driver
https://github.com/ldbc/ldbc_driver
Implement driver for SUT
Examples: https://github.com/ldbc/ldbc_snb_implementations
Verify it using validation dataset + validation file
https://github.com/ldbc-dev/ldbc_snb_interactive_validation
Download data generator + generate dataset
https://github.com/ldbc-dev/ldbc_snb_datagen_0.2

4. Base setting
System with 3 years of data will be
benchmarked for 2h of simulation time
Main metric: Throughput @ Scale
Main obstacles: Query latency constraints +
Crash recovery

5. 2014-03-05 5
Query Implementation Dataset @ SF
Clean Execution
Environment
System under test
Serializability Impl. (optional)
Preparation

6. 2014-03-05 6
Install base system
Setup benchmark configuration
Validation run
Audit workflow
1

7. 2014-03-05 7
Reset system
Setup loading configuration
Load data (timed)
Audit workflow
2

8. 2014-03-05 8
Optional system warmup
Setup benchmark configuration
Benchmark >= 2h of
simulation time
Audit workflow
3

9. 2014-03-05 9
Crash System +Recovery testing
Audit workflow
Verify benchmark run4
Continue run

10. 2014-03-05 10
Audit workflow
Additional tooling
Validation of loaded data (potentially requires
implementation by the sponsor)
Query latency checks from driver log
Serializability verificatoin (potentially requires
implementation by the sponsor)
Recovery verification

11. 2014-03-05 11
Audit result
Report
Throughput @ Scale
Load time
Recovery time
System configuration, cost, …

12. 2014-03-05 12
Rules
Inspiration from TPC, but we want to support
systems without a declarative query language.
Prohibit using hinted query plans, e.g. for join
ordering
Disallow precomputed indexes
Force serializable consistency

13. 2014-03-05 13
Rules
Rough guidelines
Datatypes have to conform to specification
Indexes have to be updatable
System has to be generally obtainable within 6
months

14. 2014-03-05 14
Rules
Intricate edges
How to define the weight of each query?
How to determine cost of system?
- Cloud? Open-source?

15. Discussion