education

1. Marcin Szymaniuk
DataFrames in Spark
Data Analysts’ perspective

2. www.tantusdata.com
About me
• Data Engineer @TantusData
• Have worked for: Spotify, Apple, telcos, startups
• Cluster installations, application architecture and
development, training, data team support
marcin@tantusdata.com
marcin.szymaniuk@gmail.com
@mszymani

3. www.tantusdata.com
Agenda
• Spark for Data Analysts
• Spark execution model
• Case by case…
• Summary

4. www.tantusdata.com
Data Analysts’/Scientists’ perspective
• Explain what happened and why
• Drive decisions to be made
• Build ML models

5. www.tantusdata.com
Bring analysis to data
DATA
R / PYTHON
/ SAS
Sample
• Sample only (region, latest month…)
• Coarse aggregate eg. month vs hour (1:720)

6. www.tantusdata.com
Bring analysis to data
• - Base station position
- Network technology
- Antenna specs
Photo credit: productcoalition.com

7. www.tantusdata.com
Bring analysis to data
• Analyze all data
• Faster analysis
• No extra data copies (GDPR!)

8. www.tantusdata.com
Data Analysts’/Scientists’ perspective
• Ad-hoc queries and reports
• Data cleanup

9. www.tantusdata.com
Data Analysts’/Scientists’ perspective
• Ad-hoc queries and reports
• Data cleanup
Photo credit: CrowdFlower

10. www.tantusdata.com
Spark API - DataFrame
eventsDf
.groupBy("userId")
.agg(
max("value").alias("maxVal"),
avg("value").alias("avgValue")
)
.join(usersDf, usersDf("id") === eventsDf("userId"))
.select("userId", "maxVal", "avgValue","name")

11. www.tantusdata.com
Spark API - DataFrame
eventsDf
.groupBy("userId")
.agg(
max("value").alias("maxVal"),
avg("value").alias("avgValue")
)
.join(usersDf, usersDf("id") === eventsDf("userId"))
.select("userId", "maxVal", "avgValue","name")

12. www.tantusdata.com
Data Analysts’ perspective
• Simple query fails?
• Simple query never finishes?
Photo credit: thedeliberatemusician.com

13. www.tantusdata.com
Deep dive
Photo credit: amazon.ca

14. www.tantusdata.com
DataFrame
DataFrame
Partition 1
Partition 2
Partition 3
Partition 4
…

15. www.tantusdata.com
Transformation type
• Narrow - single partition at time
• Wide - exchange partition’s data

16. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

17. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

18. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed",sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

19. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

20. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

21. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

22. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)

23. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)
Stage 1

24. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)
read Input Data
Stage 1

25. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)
read Input Data
do the magic
Stage 1

26. www.tantusdata.com
Narrow transformation
dataFrame.
.withColumn("uIdHashed", sha2(col("userId"), 256))
.withColumn("strReversed", reverse(col(“str")))
.withColumn(“v1v2Sum”, col("v1") + col(“v2"))
.write
.save(output)
read Input Data
do the magic
write out the result
Stage 1

27. www.tantusdata.com
Stage 1

28. www.tantusdata.com
Wide transformations
Partition 1
Partition 2
Partition 3
DataFrame
2 | VIEW | … 1 | SEND | …1 | CLICK | …
3 | SEND | … 1 | CALL | …1 | CLICK | …
2 | VIEW | … 3 | CALL | …2 | SEND | …
1 | CALL | …3 | SEND | …
Partition 1
Partition 2
Partition 3
New DataFrame
groupBy

29. www.tantusdata.com
Wide transformations
Partition 1
Partition 2
Partition 3
DataFrame
2 | VIEW | … 1 | SEND | …1 | CLICK | …
3 | SEND | … 1 | CALL | …1 | CLICK | …
2 | VIEW | … 3 | CALL | …2 | SEND | …
1 | CALL | …3 | SEND | …
Partition 1
Partition 2
Partition 3
New DataFrame
1 | SEND | … 1 | CLICK | …
1 | CALL | …
1 | CLICK | …
1 | CALL | …

30. www.tantusdata.com
Wide transformations
Partition 1
Partition 2
Partition 3
DataFrame
2 | VIEW | … 1 | SEND | …1 | CLICK | …
3 | SEND | … 1 | CALL | …1 | CLICK | …
2 | VIEW | … 3 | CALL | …2 | SEND | …
1 | CALL | …3 | SEND | …
Partition 1
Partition 2
Partition 3
New DataFrame
1 | SEND | … 1 | CLICK | …
1 | CALL | …
1 | CLICK | …
1 | CALL | …
2 | VIEW | … 2 | SEND | … 2 | VIEW | …

31. www.tantusdata.com
Wide transformations
Partition 1
Partition 2
Partition 3
DataFrame
2 | VIEW | … 1 | SEND | …1 | CLICK | …
3 | SEND | … 1 | CALL | …1 | CLICK | …
2 | VIEW | … 3 | CALL | …2 | SEND | …
1 | CALL | …3 | SEND | …
Partition 1
Partition 2
Partition 3
New DataFrame
1 | SEND | … 1 | CLICK | …
1 | CALL | …
1 | CLICK | …
1 | CALL | …
2 | VIEW | … 2 | SEND | … 2 | VIEW | …
3 | SEND | … 3 | CALL | … 3 | SEND | …

32. www.tantusdata.com
Wide transformations
• join
• groupBy
• repartition
• coalesce
• distinct
• …

33. www.tantusdata.com
Wide transformations
events
.join(users, users("id") === events("userId"))
//Extra logic
.write
.parquet(output)

34. www.tantusdata.com
Wide transformation
Stage 1 Stage 2 Stage 3

35. www.tantusdata.com
Wide transformation
Shuffle - exchange data between nodes
Stage 1 Stage 2 Stage 3

36. www.tantusdata.com
Simplest scenario ever
val df = spark.read.parquet(“…”)
HDFS
TASK

37. www.tantusdata.com
Simplest scenario ever
HDFS
TASK
ADDCOL ADDCOL
val df = spark.read.parquet(“…”)
df
.withColumn("year", year(col(“timestamp")))
.withColumn("month", month(col(“timestamp”)))
.withColumn("day", dayofmonth(col(“timestamp”)))

38. www.tantusdata.com
Simplest scenario ever
HDFS
TASK
ADDCOL ADDCOL
HDFS
val df = spark.read.parquet(“…”)
df
.withColumn("year", year(col(“timestamp”)))
.withColumn("month", month(col("timestamp")))
.withColumn("day", dayofmonth(col("timestamp")))
.write.save(output)

39. www.tantusdata.com
Simplest scenario ever
1TB of
events
8000 blocks

40. www.tantusdata.com
Simplest scenario ever
1TB of
events
8000 blocks
8000 tasks

41. HDFS
TASK
ADDCOL ADDCOL
HDFS
HDFS
TASK
ADDCOL ADDCOL
HDFS
HDFS
TASK
ADDCOL ADDCOL
HDFS
HDFS
TASK
ADDCOL ADDCOL
HDFS
…
8000

42. www.tantusdata.com
Case #1
• Faster queries with date condition?
• Organize your data by date!

43. www.tantusdata.com
Case #1
/DATA

44. www.tantusdata.com
Case #1
/DATA
.filter(
col("year")==="2018"
)

45. www.tantusdata.com
Case #1
2015
/DATA

46. www.tantusdata.com
Case #1
2016/DATA

47. www.tantusdata.com
Case #1
2017
/DATA

48. www.tantusdata.com
Case #1
2017
2016
2015
/DATA

49. www.tantusdata.com
Case #1
/DATA 2016
2015
01
01
02
032017
…

50. www.tantusdata.com
Case #1
val df = spark.read.parquet(“…”)
df
.withColumn("year", year(col(“timestamp”)))
.withColumn("month", month(col("timestamp")))
.withColumn("day", dayofmonth(col("timestamp")))
.partitionBy("year", "month", "day")
.write.save(output)

51. www.tantusdata.com
Case #1

52. www.tantusdata.com
Case #1

53. www.tantusdata.com
Case #1

54. www.tantusdata.com
Case #1

55. www.tantusdata.com
Case #1

56. www.tantusdata.com
Case #1

57. www.tantusdata.com
Case #1
$ hdfs dfs -ls

58. www.tantusdata.com
Case #1
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1

59. www.tantusdata.com
Case #1
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l

60. www.tantusdata.com
Case #1
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
22

61. www.tantusdata.com
Case #1
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
22
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
25

62. www.tantusdata.com
Case #1
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
22
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
25
$ hdfs dfs -ls /user/marcin/out/year=2019/month=3/day=1 | wc -l
40

63. www.tantusdata.com
val df = spark.read.parquet(“…”)
df
.withColumn("year", year(col(“timestamp”)))
.withColumn("month", month(col("timestamp")))
.withColumn("day", dayofmonth(col("timestamp")))
.partitionBy("year", "month", "day")
.write.save(output)
HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…
Case #1

64. HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…
HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…
HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…HDFS
TASK
READ ADDCOL ADDCOL
HDFS
Day 1
Day 2
Day 90
…
…

65. www.tantusdata.com
Case #1
90 days of
data
8000 blocks

66. www.tantusdata.com
Case #1
90 days of
data
8000 blocks
90*8000
files
produced

67. www.tantusdata.com
Consequences
• Slow!
• Downstream jobs will struggle
• Memory pressure on NameNode
• Might kill the cluster!

68. www.tantusdata.com
Consequences

69. www.tantusdata.com
Case #1
• repartition before partitionBy in order to limit #files

70. www.tantusdata.com
Repartition
df
.withColumn("year", year(col("eventTimestamp")))
.withColumn("month", month(col("eventTimestamp")))
.withColumn("day", dayofmonth(col("eventTimestamp")))
.repartition(col("year"), col("month"), col("day"))
.write
.partitionBy("year", "month", "day")
.save(output)

71. www.tantusdata.com

72. www.tantusdata.com

73. HDFS
TASK
HDFS
TASK 1
STAGE 1

74. HDFS
TASK
HDFS
TASK 1
STAGE 1

75. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
STAGE 1

76. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
STAGE 1
HDFS
HDFS
HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
STAGE 1
HDFS
HDFS

77. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 1
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
STAGE 1 STAGE 2
HDFS
HDFS
HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 1
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
STAGE 1 STAGE 2
HDFS
HDFS

78. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 1
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 2
STAGE 1 STAGE 2
HDFS
HDFS

79. www.tantusdata.com
Case #1 - summary
• each partitionBy task may create many files in HDFS
• repartition to the rescue

80. www.tantusdata.com
Case #2
• It fails after repartitioning…

81. www.tantusdata.com

82. www.tantusdata.com

83. www.tantusdata.com

84. www.tantusdata.com
Case #2

85. www.tantusdata.com
Case #2

86. www.tantusdata.com
Case #2

87. www.tantusdata.com
Case #2
One day of
events is
100G

88. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 1
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 2
STAGE 1 STAGE 2
HDFS
HDFS

89. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
Day 1
Day 2
Day 3
LOCAL
TASK 1
TASK 2
Day 1
Day 2
Day 3
LOCAL
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 2
STAGE 1 STAGE 2
HDFS
HDFS
100G100G
100G

90. www.tantusdata.com
Problems with shuffle
• Spill to disk
• Timeouts
• GC overhead limit exceeded
• OOM
• ExecutorLostFailure

91. www.tantusdata.com
Case #2
df
.withColumn("year", year(col("eventTimestamp")))
.withColumn("month", month(col("eventTimestamp")))
.withColumn("day", dayofmonth(col("eventTimestamp")))
.repartition(col("year"), col("month"), col(“day”), col(”hour”))
.write
.partitionBy("year", "month", "day")
.save(output)

92. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", "month",“day”)

93. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", "month",“day”)

94. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", "month",“day”)

95. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", “month”,“day”,”hour”)

96. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", “month”,“day”,”hour”)

97. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", “month”,“day”,”hour”)

98. www.tantusdata.com
day month year hour event …
1 1 2017 11 CALL
1 1 2017 1 TEXT
1 1 2017 4 CLICK
1 1 2017 7 SEND
2 1 2017 1 CLICK
2 1 2017 3 TEXT
2 1 2017 1 TEXT
2 1 2017 1 CALL
.repartition(“year", “month”,“day”,”hour”)

99. www.tantusdata.com
Case #2

100. www.tantusdata.com
Case #2 - recap
• Know your data distribution when repartitioning
• Keep all your tasks busy - repartition by more columns if
necessary

101. www.tantusdata.com
Case #3
events
.join(users, users("id") === events("userId"))
//Extra logic
.write
.parquet(output)

102. www.tantusdata.com
Case #3
events
.join(users, users("id") === events("userId"))
//Extra logic
.write
.parquet(output)

103. www.tantusdata.com
Case #3
events
.join(users, users("id") === events("userId"))
//Extra logic
.write
.parquet(output)

104. www.tantusdata.com
Case #3
events
.join(users,users(“id")===events("userId"))
//Extra logic
.write
.parquet(output)

105. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 2
users1
users2
users3
LOCAL
STAGE 1
HDFS

106. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2

107. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

108. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

109. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

113. www.tantusdata.com
Case #3
10TB of
events
uniform
distribution
1GB of users

114. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

115. .config(
“spark.sql.shuffle.partitions",
“200”
)
HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

116. .config(
“spark.sql.shuffle.partitions",
“200”
)
HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3
10TB/200 = 50GB/TASK

117. 10TB/200 = 50GB/TASK
HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3
50G
50G
.config(
“spark.sql.shuffle.partitions",
“200”
)

118. www.tantusdata.com
Problems with shuffle
• Spill to disk
• Timeouts
• GC overhead limit exceeded
• OOM
• ExecutorLostFailure

119. www.tantusdata.com
What to do?
• Understand your data!
• Control the level of parallelism
• Choose number of partitions according to your data size
.config("spark.sql.shuffle.partitions", “2000")
.repartition(2000)

120. www.tantusdata.com
Case #3 - result

121. www.tantusdata.com
Case #3 - result

122. www.tantusdata.com
Case #4 - Skewed join

123. www.tantusdata.com
Case #4
Photo credit: hiveminer.com

124. www.tantusdata.com
Case #4
Photo credit: hiveminer.com

125. www.tantusdata.com
Skewed join
Photo credit: hiveminer.com

126. www.tantusdata.com
Skewed join
Photo credit: hiveminer.com

127. www.tantusdata.com
Case #4
10TB of
events
One user with
1 TB of events
others are
uniformly
distributed

128. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3

129. HDFS
TASK
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 2
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
HDFS
TASK 1
users1
users2
users3
LOCAL
TASK 1
TASK 2
users1
users2
users3
LOCAL
TASK 1
users1
users2
users3
LOCAL
TASK 2
STAGE 1
HDFS
HDFS
HDFS
TASK X
Day 1
Day 2
Day 3
LOCAL
TASK 1
users1
users2
users3
LOCAL
HDFS
STAGE 2
STAGE 3
1TB

130. www.tantusdata.com
Skewed join
• Bad data?
• Wrong logic?
• Just ok?
Photo credit: hiveminer.com

131. www.tantusdata.com
Skewed Join
userId …
1
2
3
… …
eventId userId …
af8 1
bf9 1
ff1 1
881 1
91f 2
cc6 1
b22 1
ee4 1

132. www.tantusdata.com
userId …
1
2
3
… …
eventId userId …
af8 1
bf9 1
ff1 1
881 1
91f 2
cc6 1
b22 1
ee4 1
Skewed Join

133. www.tantusdata.com
userId …
1
2
3
… …
eventId userId … salt
af8 1 1
bf9 1 2
ff1 1 1
881 1 3
91f 2 2
cc6 1 3
b22 1 3
ee4 1 1
Skewed Join

134. www.tantusdata.com
userId … salt
1 1
1 2
1 3
2 1
2 2
2 3
3 1
3 2
3 3
… …
Skewed Join
eventId userId … salt
af8 1 1
bf9 1 2
ff1 1 1
881 1 3
91f 2 2
cc6 1 3
b22 1 3
ee4 1 1

135. www.tantusdata.com
userId … salt
1 1
1 2
1 3
2 1
2 2
2 3
3 1
3 2
3 3
… …
eventId userId … salt
af8 1 1
bf9 1 2
ff1 1 1
881 1 3
91f 2 2
cc6 1 3
b22 1 3
ee4 1 1
Skewed Join

136. www.tantusdata.com
eventId userId … salt
af8 1 1
bf9 1 2
ff1 1 1
881 1 3
91f 2 2
cc6 1 3
b22 1 3
ee4 1 1
Skewed Join
userId … salt
1 1
1 2
1 3
2 1
2 2
2 3
3 1
3 2
3 3
… …

137. www.tantusdata.com
userId … salt
1 1
1 2
1 3
2 1
2 2
2 3
3 1
3 2
3 3
… …
Skewed Join
eventId userId … salt
af8 1 1
bf9 1 2
ff1 1 1
881 1 3
91f 2 2
cc6 1 3
b22 1 3
ee4 1 1

138. www.tantusdata.com
Skewed Join
val eventsSalted = events
.withColumn("salt", toInt(rand() * 100))

139. www.tantusdata.com
Skewed Join
val eventsSalted = events
.withColumn("salt", toInt(rand() * 100))

140. www.tantusdata.com
Skewed Join
val eventsSalted = events
.withColumn("salt", toInt(rand() * 100))

141. www.tantusdata.com
Skewed Join
val allSalts = spark.sparkContext
.parallelize((1 to 100).toList)
.toDF("salt")
val usersSalted = users.join(allSalts)

142. www.tantusdata.com
Skewed Join
val allSalts = spark.sparkContext
.parallelize((1 to 100).toList)
.toDF("salt")
val usersSalted = users.join(allSalts)

143. www.tantusdata.com
Skewed Join
val allSalts = spark.sparkContext
.parallelize((1 to 100).toList)
.toDF("salt")
val usersSalted = users.join(allSalts)

144. www.tantusdata.com
Skewed Join
eventsSalted
.join(
usersSalted,
usersSalted("id") === eventsSalted("userId")
&& usersSalted("salt") === eventsSalted("salt")
)

145. www.tantusdata.com
Skewed Join
eventsSalted
.join(
usersSalted,
usersSalted("id") === eventsSalted("userId")
&& usersSalted(“salt") === eventsSalted("salt")
)

146. www.tantusdata.com
Skewed Join
eventsSalted
.join(
usersSalted,
usersSalted("id") === eventsSalted("userId")
&& usersSalted(“salt") === eventsSalted("salt")
)

147. www.tantusdata.com
Skewed Join
eventsSalted
.join(
usersSalted,
usersSalted("id") === eventsSalted("userId")
&&
usersSalted("salt")===eventsSalted("salt")
)

148. www.tantusdata.com
Skewed Join

149. www.tantusdata.com
• Know your data!
• Understand the transformations you are doing
• Keep your tasks busy
Use cases - recap
Photo credit: shamakern.com

150. www.tantusdata.com
Challenges ahead
• Broadcast
• Cache
• Locality
• File formats
• Sizing executors
• …

151. www.tantusdata.com
Challenges ahead
OPS
DEVELOPERS
BUSINESS ANALYTICS
Common tools and knowledge

152. www.tantusdata.com
Q&A
• marcin@tantusdata.com
• marcin.szymaniuk@gmail.com
• @mszymani

153. www.tantusdata.com
Thank you!