This presentation discusses data distribution and ordering in Apache Iceberg's Data Source V2. It explains that proper distribution and ordering of data is important for performance when writing and reading large datasets. The new version introduces an API for connectors to specify their required distribution and ordering, addressing issues in V1 where connectors could apply arbitrary transformations. Supported distribution options include ordered, clustered, and unspecified, and the API supports batch and streaming writes. Future work includes supporting distribution and ordering in table creation and improving partition handling. Proper data distribution and ordering is key to scaling performance in Iceberg.

1. Data Distribution and Ordering for
Efficient Data Source V2
Anton Okolnychyi
This is not a contribution
Data + AI Summit 2021

2. Presenter
• Apache Iceberg PMC member
• Apache Spark contributor
• Data Lakes at Apple
• Open source enthusiast

3. Agenda
• Why V2?
• Data distribution and ordering
• Future work

4. What’s wrong with V1?

5. Reliability
• Behavior of DataFrameWriter is not defined
- Connectors interpret SaveMode differently
- SaveIntoDataSourceCommand vs InsertIntoDataSourceCommand

6. Reliability
• Validation rules are not consistent
- PreprocessTableCreation vs PreprocessTableInsertion
- No schema validation for path-based tables

7. Design choices
• Connectors interact with internal APIs
- SQLContext
- RDD
- DataFrame

8. Extensibility
• Hard to support new features
- No easy way to extend PrunedFilterScan
- Exposing ColumnarBatch instead of Row is challenging

9. Features
• No Structured Streaming support
• No multi-catalog support
• Limited bucketed tables support

10. What’s different in V2?

11. Reliability
• Predictable and reliable behavior
- Clearly defined logical plans for all connectors
- Consistent validation rules
- Less delegation to connectors

12. Design choices
• Proper abstractions
- Connectors interact only with InternalRow and ColumnarBatch
- Mix-in traits for optional functionality

13. Features
• Multi-catalog support
• Structured Streaming
• Vectorization
• Bucketed tables (in progress)

14. Data distribution and ordering

15. Distribution

16. Distribution

17. Ordering

18. Ordering

19. Why should I care?

20. Impact
• Writes
- Control the number of generated files
- Reduce the overall memory consumption
- Reduce the actual writing time

21. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Unspecified distribution

22. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Unspecified distribution

23. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Unspecified distribution

24. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Proper distribution

25. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Proper distribution

26. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Proper distribution

27. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Unspecified ordering

28. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Unspecified ordering

29. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Proper ordering

30. © 2021 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Proper ordering

31. Impact
• Reads
- Cluster data on write for faster reads
- Enable efficient data skipping

32. Impact
• Storage footprint
- Columnar encodings perform better on sorted data (e.g. dictionary encoding)

33. How do connectors control this?

34. Data Source V1
• Connectors can apply arbitrary transformations on DataFrame
• Built-in connectors sort data within tasks using partition columns

35. Data Source V2
• No way to control (SPARK-23889)
• Severe performance issues unless explicitly handled by the user
• Blocks migration to V2
• Fixed in upcoming Spark 3.2

36. Solution

37. Use cases
• Global sort
• Cluster + sort within tasks
• Local sort within tasks
• No distribution and sort

38. API
interface WriteBuilder {
Write build()
}

39. API
interface Write {
BatchWrite toBatch();
StreamingWrite toStreaming();
}

40. API
interface RequiresDistributionAndOrdering extends Write {
Distribution requiredDistribution();
SortOrder[] requiredOrdering();
}

41. Distributions
• OrderedDistribution
• ClusteredDistribution
• UnspecifiedDistribution

42. SortOrder
interface SortOrder extends Expression {
Expression expression();
SortDirection direction();
NullOrdering nullOrdering();
}

43. Current state
• Available and fully functional in master for batch queries
• Structured Streaming support is in progress (SPARK-34183)

44. Future work
• Distribution and ordering in CREATE TABLE
• Ability to control the number of shuffle partitions
• Coalesce partitions during adaptive query execution

45. Key takeaways

46. Summary
• Consider migrating to Data Source V2
• Data distribution and ordering is critical at scale

47. Feedback
• Your feedback is important to us
• Don’t forget to review and rate sessions

48. Thank you!

49. TM and © 2021 Apple Inc. All rights reserved.