1.
Photo by Priscilla Du Preez on Unsplash

2.
Animation by Mike Mk and lottiefiles: https://lottiefiles.com/user/775169

3.
Failed Tasks in Spark UI - Executers
@adipolak

4.
Client-Request-ID=------ Retry policy did not allow for a retry: , HTTP status
code=Unknown, Exception=HTTPSConnectionPool(host='-----.net', port=443):
Max retries exceeded with url: /xxxxxxx?restype=container&comp=list
(Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
xxxxxxxx>: Failed to establish a new connection: [Errno 8] nodename nor
servname provided, or not known',)).
HTTPSConnectionPool(ho
st='your_account.blob.cor
e.windows.net', port=443):
Read timed out. (read
timeout=[your timeout])
Exceptions in Apache Spark Executers logs
@adipolak

5.
On-prem
Public Cloud

6.
Degrading Performance?
You might be suffering
from the
Small Files Syndrome
Adi Polak
Microsoft
@adipolak

7.
About Me
M.Sc & B.Sc - BGU University
ML Researcher @ DT &BGU Cyber
Security lab
Sr. Big Data Engineer @ Akamai
Sr. Software Developer &
Cloud Advocate @ Microsoft
@adipolak
https://www.linkedin.com/in/adi
-polak-68548365/

8.
Agenda
§ The Problem
§ Why it Happens
§ Detect and Mitigate
§ Delta Lake vs Parquet Demo
@adipolak

9.
Why it happens?
@adipolak

10.
Query Life Cycle Abstraction
storage storage storage
@adipolak

11.
Query Life Cycle Abstraction
storage
@adipolak

12.
How Read and Write works?
@adipolak

13.
File size matters!
@adipolak
1 Million files of 60 bytes ~ 0.06 GB ~ Reading == 1 M RPCs
1 file of 0.06 GB ~ 60Mb ~ Reading == 1 RPCs

14.
Detect and Mitigate?
@adipolak

15.
Where can it happen?
• Event streams
• IoT devices, servers, or applications are being translated into KB-scale JSON files during the ingestion
procedure
• Over Paralleled Apache Spark jobs Sub-bullet
• Over Partitioned Hive tables
@adipolak

16.
What to check?
• Data skew - Hive partitions file sizes
• Spark job writers in the Spark History Server UI
• Ingestion file size
@adipolak

17.
Mitigate
• Use file hierarchy - source/api_type/yyyy/mm/dd/hh/mm
• design partitions w/ usage in mind
• Re-partition vs Coalesce
• Databricks Auto Optimize
• SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true,
delta.autoOptimize.autoCompact = true)
• Delta Lake Optimize performance
• Compaction (bin-packing)
• ZORDER BY
• delta.targetFileSize
• delta.tuneFileSizesForRewrites
@adipolak

18.
Demo – optimizing read queries
@adipolak

19.
@adipolak

20.
Summary
§ The Problem
§ Why it Happens
§ Detect and Mitigate
§ Delta Lake vs Parquet Demo
@adipolak

21.
“Intellectual growth should commence at birth
and cease only at death.”
― Albert Einstein
@adipolak