Talk from Dremio Subsurface Conference on July 30, 2020

2. •
•
•
•
•

3. Some Partners
● https://ursalabs.org
● Apache Arrow-powered
Data Science Tools
● Funded by corporate
partners
● Built in collaboration with
RStudio

4. Systems that move
structured data often
cause significant waste

5. •
•
•

6. •
• …
•

8. Server 1 Server 2 Server 3
Client 1 Client 2

9. Scalable Blob Storage
System 1 System 2 System 3

10. Executor Executor Executor
Executor /
Coordinator
Client
Result Set
Result Set
Result Set
Result Set
Result Set
Result Set

11. Executor Executor Executor Executor
Client
Result Set
Result Set
Result Set

12. •
•
•
•
•

14. •
•
•
•
•

15. •
•
•
•
•

16. •
•
•
•
•
•

17. •
•
•
•

19. •
•
•
•
SCHEMA DICTIONARY DICTIONARY
RECORD
BATCH
RECORD
BATCH

20. •
metadata body

21. https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

22. https://medium.com/google-cloud/announcing-google-cloud-bigquery-version-1-17-0-1fc428512171

23. •
•
•

24. •
•
•
•
•
•

25. •
•
•

26. Client Planner
GetFlightInfo
FlightInfo
DoGet Data Nodes
FlightData
DoGet
FlightData
...

27. •
•
•
message SQLQuery {
binary database_uri = 1;
binary query = 2;
}
Commands.proto GetFlightInfo RPC
type: CMD
cmd: <serialized command>

28. Client
DoGet
Data Node
FlightData
Row
Batch
Row
Batch
Row
Batch
Row
Batch
Row
Batch
...
Data transported in a Protocol
Buffer, but reads can be made
zero-copy by writing a custom
gRPC “deserializer”

29. Mainlining Databases: Supporting Fast Transactional Workloads on
Universal Columnar Data File Formats
Li, Butrovich, Ngom, Lim,
Pavlo, McKinney
https://arxiv.org/pdf/2004.14471.pdf

31. •
•