This document describes Opaque, a secure distributed data analytics framework that allows complex analytics to run on sensitive data while preserving data privacy and functionality. Opaque utilizes hardware enclaves to protect computation and data within enclaves from a malicious operating system or cloud provider. It implements various oblivious primitives and oblivious operators to execute Spark SQL queries without leaking data access patterns. This allows sensitive data to be analyzed using existing Spark SQL queries while preventing privacy leaks from memory, network or computation access patterns.

1. Opaque: A Secure
Distributed Data Analytics
Framework
Wenting Zheng, Ankur Dave, Jethro Beekman,
Raluca Ada Popa, Joseph Gonzalez, Ion Stoica

2. Complex analytics run on sensitive data
client cloud provider
sensitive data

3. Complex analytics run on sensitive data
client
Spark
SQL
MLLib GraphX
Spark
Streaming
cloud provider
sensitive data

4. Complex analytics run on sensitive data
client
Spark
SQL
MLLib GraphX
Spark
Streaming
cloud provider
sensitive data

10. Threat model
client cloud provider
sensitive data

11. Threat model
client cloud provider
sensitive data

12. Threat model
client cloud provider
sensitive data

13. Threat model
client cloud provider
sensitive data

14. Challenge:
protect data and preserve functionality

15. Spark SQL
Opaque*: secure data analytics
* Oblivious Platform for Analytic QUEries
SQL
Machine
Learning
Graph
Analytics

16. Spark SQL
Opaque
Opaque*: secure data analytics
* Oblivious Platform for Analytic QUEries
SQL
Machine
Learning
Graph
Analytics

17. Prior work

18. Prior work
• Computation on encrypted data

19. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption

20. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption
– Either impractically slow (FHE), or limited functionality (CryptDB)

21. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption
– Either impractically slow (FHE), or limited functionality (CryptDB)
• Hardware-based systems

22. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption
– Either impractically slow (FHE), or limited functionality (CryptDB)
• Hardware-based systems
– Use trusted hardware

23. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption
– Either impractically slow (FHE), or limited functionality (CryptDB)
• Hardware-based systems
– Use trusted hardware
– Only single machine computation (Haven), or weaker security
guarantees (VC3)

24. Prior work
• Computation on encrypted data
– A cryptographic approach using homomorphic encryption
– Either impractically slow (FHE), or limited functionality (CryptDB)
• Hardware-based systems
– Use trusted hardware
– Only single machine computation (Haven), or weaker security
guarantees (VC3)
Opaque utilizes trusted hardware

25. Hardware enclaves

26. Hardware enclaves
• Hardware-protected containers in
presence of malicious OS

27. Hardware enclaves
• Hardware-protected containers in
presence of malicious OS

28. Hardware enclaves
• Hardware-protected containers in
presence of malicious OS

29. Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
Untrusted OS

30. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
Untrusted OS

31. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
Untrusted OS

32. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
Untrusted OS
Secret
data

33. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
Untrusted OS
Secret
data
Code

34. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
• Encrypted enclave memory
Untrusted OS
Secret
data
Code

35. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
• Encrypted enclave memory
• Software attestation
Untrusted OS
Secret
data
Code

36. Enclave
Hardware enclaves
• Hardware-protected containers in
presence of malicious OS
• Shielded execution
• Encrypted enclave memory
• Software attestation
• Example: Intel SGX, AMD
memory encryption Untrusted OS
Secret
data
Code

37. System initialization
Database
Client Server

38. System initialization
• Remote
attestation to
load Opaque
code
Database
Client Server

39. System initialization
• Remote
attestation to
load Opaque
code
Opaque
SQL
operators
Database
Client Server

40. System initialization
• Remote
attestation to
load Opaque
code
Opaque
SQL
operators
Database
Client Server

41. System initialization
• Remote
attestation to
load Opaque
code
Opaque
SQL
operators
hash
Database
Client Server

42. System initialization
• Remote
attestation to
load Opaque
code
Opaque
SQL
operators
hash
Database
Client Server

43. System initialization
• Remote
attestation to
load Opaque
code
Opaque
SQL
operators
hash
Database
Client Server

44. System initialization
• Remote
attestation to
load Opaque
code
Database
Client Server

45. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

46. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

47. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

48. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

49. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

50. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
Database
Client Server

51. System initialization
• Remote
attestation to
load Opaque
code
• Key exchange
protocol
• This is NOT on a
per-query basis Database
Client Server

52. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3

53. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3query = SELECT sum(*)
FROM table

54. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3
Query
query = SELECT sum(*)
FROM table

55. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3
Query
query = SELECT sum(*)
FROM table

56. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3query = SELECT sum(*)
FROM table

57. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3query = SELECT sum(*)
FROM table

58. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3query = SELECT sum(*)
FROM table

59. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3query = SELECT sum(*)
FROM table

60. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
1 2 3
query = SELECT sum(*)
FROM table

61. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
query = SELECT sum(*)
FROM table
10 13 4

62. Spark
Driver
Opaque
Catalyst
Query execution
Client Server
Database
Scheduler
query = SELECT sum(*)
FROM table
10
13
4

63. Spark
Driver
Opaque
Catalyst 27
Query execution
Client Server
Database
Scheduler
query = SELECT sum(*)
FROM table

64. Spark
Driver
Opaque
Catalyst
27
Query execution
Client Server
Database
Scheduler
query = SELECT sum(*)
FROM table

65. Problem: cloud can alter distributed computation

66. Problem: cloud can alter distributed computation
• Drop data

67. Problem: cloud can alter distributed computation
• Drop data
• Modify data

68. Problem: cloud can alter distributed computation
• Drop data
• Modify data
• Skip task

69. Problem: cloud can alter distributed computation
• Drop data
• Modify data
• Skip task
• Replay old state

70. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
1 2 3
Client
query = SELECT sum(*)
FROM table

71. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
1 2 3
Client
query = SELECT sum(*)
FROM table

72. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
10 13 4
Client
query = SELECT sum(*)
FROM table

73. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
10
13
Client
query = SELECT sum(*)
FROM table

74. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
23
Client
query = SELECT sum(*)
FROM table

75. Example: drop data
Spark
Driver
Opaque
Catalyst
Server
Database
Scheduler
23
Client
query = SELECT sum(*)
FROM table

76. Self-verifying computation

77. Self-verifying computation
Invariant: if computation does not abort,
the execution completed so far is correct

78. Self-verifying computation
Invariant: if computation does not abort,
the execution completed so far is correct

79. Self-verifying computation
Invariant: if computation does not abort,
the execution completed so far is correct
If the computation is complete, then the entire
query was executed correctly

80. Self-verifying computation
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

81. Self-verifying computation
20
1413 15
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

82. Self-verifying computation
20
1413 15
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

83. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

84. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

85. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

86. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

87. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

88. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

89. Self-verifying computation
20
1413 15
10
13
4
Task 13
Task 14
Task 15
Task 20
query = SELECT sum(*)
FROM table

90. ID Name Age Disease
12809 Amanda D. Edwards 40 Diabetes
29489 Robert R. McGowan 56 Diabetes
13744 Kimberly R. Seay 51 Cancer
18740 Dennis G. Bates 32 Diabetes
98329 Ronald S. Ogden 53 Cancer
medical table:
Problem: access pattern leakage
32591 Donna R. Bridges 26 Diabetes

91. ID Name Age Disease
SELECT count(*) FROM medical
GROUP BY disease
12809 Amanda D. Edwards 40 Diabetes
29489 Robert R. McGowan 56 Diabetes
13744 Kimberly R. Seay 51 Cancer
18740 Dennis G. Bates 32 Diabetes
98329 Ronald S. Ogden 53 Cancer
medical table:
Problem: access pattern leakage
32591 Donna R. Bridges 26 Diabetes

92. Problem: access pattern leakage
SELECT count(*) FROM medical
GROUP BY disease
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes

93. Problem: access pattern leakage
SELECT count(*) FROM medical
GROUP BY disease
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes

94. Problem: access pattern leakage
SELECT count(*) FROM medical
GROUP BY disease
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes

95. Problem: access pattern leakage
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Attack viable for both
memory and network
access patterns!

96. Oblivious mode

97. Oblivious mode
Oblivious
primitives

98. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives

99. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators

100. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…

101. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
Oblivious
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…

102. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
Oblivious
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…
Oblivious
Filter
Oblivious
Sort
Oblivious
Aggregation
Oblivious
Join

103. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
Oblivious
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…
Oblivious
Filter
Oblivious
Sort
Oblivious
Aggregation
Oblivious
Join
Oblivious Query Plan

104. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
Oblivious
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…
Oblivious
Filter
Oblivious
Sort
Oblivious
Aggregation
Oblivious
Join
Oblivious Query Plan
{

105. Oblivious mode
intra-machine
o-sort
inter-machine
o-sort
random
permutation
Oblivious
primitives
Opaque
operators
Oblivious
operators
project-
filter
low-cardinality
agg.
sort-
merge join
broadcast
join
…
Oblivious
Filter
Oblivious
Sort
Oblivious
Aggregation
Oblivious
Join
Oblivious Query Plan
{Query
optimization

106. Oblivious aggregation
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Oblivious
sort
SELECT count(*) FROM medical GROUP BY disease
Map Sort

107. Oblivious aggregation
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Oblivious
sort
SELECT count(*) FROM medical GROUP BY disease
Map Sort

108. Oblivious aggregation
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Oblivious
sort
SELECT count(*) FROM medical GROUP BY disease
Map Sort

109. Map Sort
Oblivious
sort
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

110. Sort
Oblivious
sort
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

111. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

112. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan
Statistics
Statistics
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

113. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Statistics
Statistics
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

114. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Statistics
Statistics
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

115. 2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

116. 2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Result size
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

117. 2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

118. 2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Offset
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

119. 2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

120. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

121. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
2; 1
2; 2
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

122. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

123. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

124. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Cancer:2
Diabetes:3
Diabetes:1
DUMMY
Final
result
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

125. 12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
12809 … Diabetes
29489 … Diabetes
13744 … Cancer
18740 … Diabetes
98329 … Cancer
32591 … Diabetes
Scan Boundary
processing
Scan
Final
result
Cancer:2
Diabetes:4
Oblivious aggregation
SELECT count(*) FROM medical GROUP BY disease

126. Opaque modes

127. Opaque modes
• Encryption mode

128. Opaque modes
• Encryption mode
– Data encryption and
authentication

129. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected

130. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected

131. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
Snapshot attacker
e.g. external hacker

132. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
• Oblivious mode
Snapshot attacker
e.g. external hacker

133. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
• Oblivious mode
– Additionally hide
access patterns
Snapshot attacker
e.g. external hacker

134. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
• Oblivious mode
– Additionally hide
access patterns
Snapshot attacker
e.g. external hacker

135. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
• Oblivious mode
– Additionally hide
access patterns
Snapshot attacker
e.g. external hacker
Persistent attacker
e.g. insider

136. Opaque modes
• Encryption mode
– Data encryption and
authentication
– Computation is
integrity protected
• Oblivious mode
– Additionally hide
access patterns
Snapshot attacker
e.g. external hacker
Persistent attacker
e.g. insider
Trade off: performance and security

137. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

138. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

139. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

140. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

141. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

142. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

143. Project-filter
Obliv. sort
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

144. Project-filter
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical

145. Project-filter
Filter
Query optimization - oblivious
SELECT count(*)
FROM medical
WHERE age > 30
GROUP BY disease
Low-card. obliv. agg.
Scan
Obliv. sort
Aggregate
medical
Reduced # of
oblivious sorts
by 1

146. D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
Query optimization - mixed sensitivity

147. D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
Query optimization - mixed sensitivity

148. D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
Query optimization - mixed sensitivity

149. Query optimization - mixed sensitivity

150. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)

151. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id

152. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
|P| < |D| < |M|

153. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
|P| < |D| < |M|

154. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
SQL join order
|P| < |D| < |M|

155. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
SQL join order
|P| < |D| < |M|

156. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
Patient
Disease
⨝
Medication
⨝ ᵞ
SQL join order
|P| < |D| < |M|

157. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
Patient
Disease
⨝
Medication
⨝ ᵞ
SQL join order
Opaque join order
|P| < |D| < |M|

158. Query optimization - mixed sensitivity
D_ID
AGE
NAME
P_ID
END_DATE
START_DATE
PID
COMMENT
DOCTOR
T_ID
DOSAGE
END_TIME
START_TIME
M_ID
T_ID
COMMENT
DATE
TR_ID
G_ID
NAME
D_ID
COST
D_ID
NAME
M_ID
COMMENT
NAME
G_ID
Patient (P_)
Treatment Plan
(TP_)
Treatment
Record (TR_)
Disease (D_) Medication (M_)
Gene (G_)
SELECT p_name, d_name, med_cost
FROM patient, disease,
(SELECT d_id, min(cost) AS med_cost
FROM medication
GROUP BY d_id) AS med
WHERE disease.d_id = patient.d_id
AND disease.d_id = med.d_id
Patient Disease
⨝
Medication
⨝
ᵞ
Patient
Disease
⨝
Medication
⨝ ᵞ
SQL join order
Opaque join order
|P| < |D| < |M|

159. Evaluation setup

160. Evaluation setup
• Single machine experiments
– Intel Xeon E3-1280 v5, 4 cores, 64 GB RAM
– Intel SGX: 128 MB of enclave page cache (EPC)
– Hardware mode

161. Evaluation setup
• Single machine experiments
– Intel Xeon E3-1280 v5, 4 cores, 64 GB RAM
– Intel SGX: 128 MB of enclave page cache (EPC)
– Hardware mode
• Distributed experiments
– EC2: five r3.2xlarge instances, 8 cores, 61 GB RAM
– Simulation mode only

162. Evaluation

163. Evaluation
• How does Opaque compare to Spark SQL?

164. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)

165. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)
• Queries 1, 2, 3: filter, aggregation, join

166. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)
• Queries 1, 2, 3: filter, aggregation, join
• 1 million records

167. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)
• Queries 1, 2, 3: filter, aggregation, join
• 1 million records
• How does Opaque compare to state-of-the-art oblivious
systems?

168. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)
• Queries 1, 2, 3: filter, aggregation, join
• 1 million records
• How does Opaque compare to state-of-the-art oblivious
systems?
– GraphSC (graph analytics)

169. Evaluation
• How does Opaque compare to Spark SQL?
– Big Data Benchmark (BDB)
• Queries 1, 2, 3: filter, aggregation, join
• 1 million records
• How does Opaque compare to state-of-the-art oblivious
systems?
– GraphSC (graph analytics)
• PageRank

170. Big Data Benchmark
(encryption mode)
Single machine

171. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine

172. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine

173. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine

174. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
DistributedSingle machine

175. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
DistributedSingle machine

176. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
DistributedSingle machine

177. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
DistributedSingle machine

178. Big Data Benchmark
(encryption mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Distributed
With very little cost, you will have data
encryption, authentication and
computation protection!
Single machine

179. Single machine Distributed
Big Data Benchmark
(oblivious mode)

180. Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

181. Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

182. Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

183. Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

184. Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

185. Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque
Single machine Distributed
Big Data Benchmark
(oblivious mode)Runtime(s)
0.01
0.1
1
10
100
Query number
Query 1 Query 2 Query 3
Spark SQL Opaque

186. PageRank: comparison with GraphSC

187. We have an NSDI 2017 paper!

188. Open source release

189. Open source release
• Available at github.com/ucbrise/opaque

190. Open source release
• Available at github.com/ucbrise/opaque
• Opaque is implemented as a Spark package

191. Open source release
• Available at github.com/ucbrise/opaque
• Opaque is implemented as a Spark package
• Features

192. Open source release
• Available at github.com/ucbrise/opaque
• Opaque is implemented as a Spark package
• Features
– Supports DataFrame select, filter, group by, join

193. Open source release
• Available at github.com/ucbrise/opaque
• Opaque is implemented as a Spark package
• Features
– Supports DataFrame select, filter, group by, join
– Allows users to specify DataFrames in encryption/
oblivious modes

194. Open source release
• Available at github.com/ucbrise/opaque
• Opaque is implemented as a Spark package
• Features
– Supports DataFrame select, filter, group by, join
– Allows users to specify DataFrames in encryption/
oblivious modes
• Automatic sensitivity propagation in mixed
sensitivity

195. Open source release

196. Open source release
• Extension

197. Open source release
• Extension
– More functionality requires rewriting operators in C++

198. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet

199. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions

200. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++

201. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++
• Run JVM in the enclave

202. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++
• Run JVM in the enclave
• Deployment

203. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++
• Run JVM in the enclave
• Deployment
– Master must be trusted

204. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++
• Run JVM in the enclave
• Deployment
– Master must be trusted
– SGX available now on Skylake processors

205. Open source release
• Extension
– More functionality requires rewriting operators in C++
– No UDF support yet
– Possible solutions
• Automatically generate C++
• Run JVM in the enclave
• Deployment
– Master must be trusted
– SGX available now on Skylake processors
• Cloud providers have no support yet

206. Demo

207. Conclusion
Opaque is a secure distributed analytics platform
Opaque
SQL
Machine
Learning
Graph
Analytics
Try it out at github.com/ucbrise/opaque
Wenting Zheng - wzheng@eecs.berkeley.edu