Data Engineering Meetup #1

1. Deep Dive Into
Takuya UESHIN
Data Engineering Meetup #1, Mar 2019
1
SQL
with Advanced Performance Tuning

2. 2
About Me
- Software Engineer @databricks
- Apache Spark Committer
- Twitter: @ueshin
- GitHub: github.com/ueshin

3. DATABRICKS WORKSPACE
Databricks Delta ML Frameworks
DATABRICKS CLOUD SERVICE
DATABRICKS RUNTIME
Reliable & Scalable Simple & Integrated
Databricks Unified Analytics Platform
APIs
Jobs
Models
Notebooks
Dashboards End to end ML lifecycle

4. Spark SQL
A highly scalable and efficient relational
processing engine with ease-to-use APIs and
mid-query fault tolerance.
4

5. The not-so-secret truth...
5
is not only SQL.SQL

6. Spark SQL
6

7. Spark SQL
7
A compiler from queries to RDDs.

8. Performance Tuning for Optimal Plans
Run EXPLAIN Plan.
Interpret Plan.
Tune Plan.
8

9. Declarative APIs
9

10. Declarative APIs
何をしたいのか?
• SQL API: ANSI SQL:2003 / HiveQL.
• Dataset/DataFrame APIs: richer,
user-friendly な language-integrated
インターフェイス
10

11. Metadata Catalog
11

12. Metadata Catalog
• Persistent Hive metastore [Hive 0.12 - Hive 2.3.3]
• Session-local temporary view manager
• Cross-session global temporary view manager
• Session-local function registry
12

13. Performance Tips - Catalog
Partition metadata 取得のコスト:
- Hive metastore のアップグレード
- Cardinality の高いパーティションカラムを避ける
- Partition pruning predicates (improved in [SPARK-20331])
13

14. Cache Manager
14

15. Cache Manager
• プランが一致した場合にキャッシュデータと置き
換える
• Cross-session
• 初めて利用する時にキャッシュデータ作成
• 依存しているtable/viewのデータが更新される
とキャッシュデータを無効化
15

16. Performance Tips - Cache
Cache: 必ずしも速いわけではない
- ディスクに書き出される場合がある
- 必要なければキャッシュしないようにする
16

17. Optimizer
17

18. Optimizer
heuristics と cost ベースでプランの書き換え
18
• Outer join elimination
• Constraint propagation
• Join reordering
and many more.
• Column pruning
• Predicate push down
• Constant folding

19. Performance Tips - Optimizer
独自の Optimizer や Planner Rule の組み込み
• SparkSessionExtensions
• ExperimentalMethodsクラス
• var extraOptimizations: Seq[Rule[LogicalPlan]] = Nil
• var extraStrategies: Seq[Strategy] = Nil
• Examples in the Herman’s talk Deep Dive into Catalyst
Optimizer
• Join two intervals: http://dbricks.co/2etjIDY
19

20. Planner
20

21. Planner
• Logical PlanをPhysical Planへ (what to how)
• コストに基づいて最適な Physical Plan を選択
21
table1 table2
Join
broadcast
hash join
sort merge
join
OR
broadcast join has lower cost if
one table can fit in memory
table1 table2 table1 table2

22. Performance Tips - Join Selection
broadcast join vs shuffle join (broadcast の方が速い)
• spark.sql.autoBroadcastJoinThreshold
• 統計情報を最新に保つ
• broadcastJoin ヒント
22

23. Performance Tips - Equal Join
… t1 JOIN t2 ON t1.id = t2.id AND t1.value < t2.value
… t1 JOIN t2 ON t1.value < t2.value
Join 条件に少なくとも1つの equal 条件を入れる
23

24. Performance Tips - Equal Join
… t1 JOIN t2 ON t1.id = t2.id AND t1.value < t2.value
… t1 JOIN t2 ON t1.value < t2.value
24
O(n ^ 2)
O(n)

25. Query Execution
25

26. Query Execution
• Memory Manager: メモリ利用状況を追跡してtaskや
operator間のメモリを調整
• Code Generator: Physical PlanをコンパイルしてJava
コードを生成
• Tungsten Engine: CPUやメモリに対して効率的なバイ
ナリデータフォーマットやデータ構造
26

27. Performance Tips - Memory Manager
spark.executor.memory と spark.memory.fraction を、監視外メ
モリのために余裕を持って設定する. Sparkが監視できないメモ
リ領域がある(netty buffer, parquet writer buffer).
spark.memory.offHeap.enabled と spark.memory.offHeap.size
を設定して off-heapを有効化, それにあわせて
spark.executor.memory を減らす.
27

28. Whole Stage Code Generation

29. Performance Tip - WholeStage codegen
spark.sql.codegen.hugeMethodLimit の設定
バイトコードのサイズが8kバイトを超える大きな
メソッドはJITコンパイラがコンパイルできない
29

30. Data Sources
• computation と storage の分離
• Complete data pipeline:
• External storage がデータを供給
• Spark が処理
• Sparkのデータ処理がとても速い場合、データ
ソースがボトルネックになり得る
30

31. Scan Vectorization
• Vectorizationでより効率的なカラムなデータ読み込み
• JVMがSIMDを利用しやすい
• ……

32. Partitioning and Bucketing
• data skipping と pre-shuffle のためのファイルレイアウト
• 不要なIOとシャッフルを避けてスピードアップ
• The summit talk: http://dbricks.co/2oG6ZBL

33. Performance Tips - DataSource
• vectorized reading が可能なファイルフォーマット
を利用 (Parquet, ORC)
• ファイルベースのデータソースの場合には、
Patitioning や Bucketing を検討
33

34. Apache Spark™
• Use Cases
• Research
• Technical Deep Dives
AI
• Productionizing ML
• Deep Learning
• Cloud Hardware
Fields
• Data Science
• Data Engineering
• Enterprise
5000+ ATTENDEES
Practitioners:
Data Scientists, Data Engineers,
Analysts, Architects
Leaders:
Engineering Management, VPs,
Heads of Analytics & Data, CxOs
TRACKS
databricks.com/sparkaisummit

35. 35
Nike: Enabling Data Scientists to bring their Models to Market
Facebook: Vectorized Query Execution in Apache Spark at Facebook
Tencent: Large-scale Malicious Domain Detection with Spark AI
IBM: In-memory storage Evolution in Apache Spark
Capital One: Apache Spark and Sights at Speed: Streaming, Feature
management and Execution
Apple: Making Nested Columns as First Citizen in Apache Spark SQL
EBay: Managing Apache Spark workload and automatic optimizing.
Google: Validating Spark ML Jobs
HP: Apache Spark for Cyber Security in big company
Microsoft: Apache Spark Serving: Unifying Batch, Streaming and
RESTful Serving
ABSA Group: A Mainframe Data Source for Spark SQL and Streaming
Facebook: an efficient Facebook-scale shuffle service
IBM: Make your PySpark Data Fly with Arrow!
Facebook : Distributed Scheduling Framework for Apache Spark
Zynga: Automating Predictive Modeling at Zynga with PySpark
World Bank: Using Crowdsourced Images to Create Image Recognition
Models and NLP to Augment Global Trade indicator
JD.com: Optimizing Performance and Computing Resource.
Microsoft: Azure Databricks with R: Deep Dive
ICL: Cooperative Task Execution for Apache Spark
Airbnb: Apache Spark at Airbnb
Netflix: Migrating to Apache Spark at Netflix
Microsoft: Infrastructure for Deep Learning in
Apache Spark
Intel: Game playing using AI on Apache Spark
Facebook: Scaling Apache Spark @ Facebook
Lyft: Scaling Apache Spark on K8S at Lyft
Uber: Using Spark Mllib Models in a Production
Training and Serving Platform
Apple: Bridging the gap between Datasets and
DataFrames
Salesforce: The Rule of 10,000 Spark Jobs
Target: Lessons in Linear Algebra at Scale with
Apache Spark
Workday: Lesson Learned Using Apache Spark

36. Thank you
Takuya UESHIN (ueshin@databricks.com)
36