In this talk I demonstrate and explain how I tackled the problem of importing a huge data set with many repetitive contents. The dataset was provided in JSON format.

1. ME AND MY IMPORTERS
DONNY WALS

2. EXPLORING THE FIELD
PROLOGUE:

3. LET’S SYNC ALL DATA FROM THE BACK-
END INSTEAD OF REQUESTING TINY BITS.
IT WILL BE GREAT!
Back-end development
ME AND MY IMPORTERS

4. PROBLEM 1,
API DEPENDENCIES

6. App Settings Remote app config

7. Events
Information about all the
visible events in the app
App Settings Remote app config

8. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
App Settings Remote app config

9. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
App Settings Remote app config

10. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
App Settings Remote app config

11. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config

12. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1

13. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
2

14. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
3
2

15. Events
Information about all the
visible events in the app
Genres
All the possible genres that
are assigned to events
Userdata
Information for the currently
logged in user
Prizes Favorites Profile
Homepage
Which data to display on the
homepage
App Settings Remote app config 1
3
2
4
4

16. PROBLEM 2,
DUPLICATION

17. EVENT
Location
Categories
Artists
Gallery
Sponsor
Country
Texts

18. EVENT
Location
Categories
Artists
Gallery
Sponsor
Country
Texts
A LOT OF DATA IS DUPLICATED
😭

19. PROBLEM 3,
STORAGE

20. THE USER SHOULD ALWAYS BE ABLE TO
USE THE UI, EVEN WHEN A SYNC IS
TAKING PLACE IN THE BACKGROUND.
Me
ME AND MY IMPORTERS

21. ME AND MY IMPORTERS
AGENDA

22. ME AND MY IMPORTERS
AGENDA
▸ Chapter 1: Chaining operations

23. ME AND MY IMPORTERS
AGENDA
▸ Chapter 1: Chaining operations
▸ Chapter 2: Optimizing the data

24. ME AND MY IMPORTERS
AGENDA
▸ Chapter 1: Chaining operations
▸ Chapter 2: Optimizing the data
▸ Chapter 3: Storing in CoreData

25. CHAINING
OPERATIONS
CHAPTER 1:

26. ADVANCED NSOPERATIONS
WWDC 2015

27. Events
Userdata
Homepage
Genres
Information about all the
visible events in the app
All the possible genres that
are assigned to events
Information for the currently
logged in user
Prizes Favorites Profile
Which data to display on the
homepage
App Settings Remote app config 1
3
2
4
4

28. Sync operation

29. App Settings
Sync operation

30. App Settings
Genres
Sync operation

31. App Settings
Genres
Sync operation

32. App Settings
Events
Genres
Sync operation

33. App Settings
Events
Genres
Sync operation

34. App Settings
Events
Genres
Homepage
Sync operation

35. App Settings
Events
Genres
Homepage
Sync operation

36. App Settings
Events
Genres Userdata
Homepage
Sync operation

37. App Settings
Events
Genres
Prizes Favorites Profile
Userdata
Homepage
Sync operation

38. App Settings
Events
Genres
Prizes Favorites Profile
Userdata
Homepage
Sync operation

39. App Settings
Events
Genres
Prizes Favorites Profile
Userdata
Homepage
Completion
Sync operation

40. App Settings
Events
Genres
Prizes Favorites Profile
Userdata
Homepage
Completion
Sync operation

41. App Settings EventsGenres
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

42. App Settings EventsGenres
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

43. EventsGenres
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

44. EventsGenres
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

45. Events
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

46. Events
Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

47. Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

48. Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

49. Prizes Favorites Profile
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

50. Prizes Favorites
Userdata
HomepageCompletion
SYNC OPERATION QUEUE
USERDATA QUEUE

51. Prizes Favorites
Userdata
Completion
SYNC OPERATION QUEUE
USERDATA QUEUE

52. Prizes
Userdata
Completion
SYNC OPERATION QUEUE
USERDATA QUEUE

53. Completion
SYNC OPERATION QUEUE
USERDATA QUEUE

54. Completion
SYNC OPERATION QUEUE
USERDATA QUEUE

55. SYNC OPERATION QUEUE
USERDATA QUEUE

56. EVENT FETCH OPERATION
GENRE FETCH OPERATION

57. EVENT FETCH OPERATION
override func generateDependencies() -> [DWOperation] {
return [GenreFetchOperation()]
}
GENRE FETCH OPERATION

58. EVENT FETCH OPERATION
override func generateDependencies() -> [DWOperation] {
return [GenreFetchOperation()]
}
GENRE FETCH OPERATION
override func generateDependencies() -> [DWOperation] {
return [SyncAppSettingsOperation()]
}

59. DWOPERATION QUEUE

60. DWOPERATION QUEUE
override func addOperation(_ op: Operation) {
// details, details
for dependency in operation.generateDependencies() {
operation.addDependency(dependency)
addOperation(dependency)
}
operation.willEnqueue()
}

61. FETCH USER DATA OPERATION

62. FETCH USER DATA OPERATION
init(operations: [Operation]) {
super.init()
internalQueue.isSuspended = true
internalQueue.delegate = self
internalQueue.addOperation(startingOperation)
for operation in operations {
internalQueue.addOperation(operation)
}
}

63. FETCH USER DATA OPERATION

64. FETCH USER DATA OPERATION
final func operationQueue(_ operationQueue: MyOperationQueue, operationDidFinish operation: Operation, withErro
errors: [Error]) {
aggregatedErrors.append(contentsOf: errors)
if operation === finishingOperation {
internalQueue.isSuspended = true
finish(errors: aggregatedErrors)
} else if operation !== startingOperation {
operationDidFinish(operation: operation, withErrors: errors)
}
}

65. App Settings
Events
Genres Userdata
Prizes Favorites Profile
Homepage
Completion
Sync operation

66. ME AND MY IMPORTERS
PROBLEM 1 SOLVED

67. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies

68. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies
▸ Operations can execute in parallel or serially

69. ME AND MY IMPORTERS
PROBLEM 1 SOLVED
▸ Operations simplify dependencies
▸ Operations can execute in parallel or serially
▸ Operations can contain their own private queues for complex nesting

70. OPTIMIZING THE DATA
CHAPTER 2:

71. EVENT
Location
Categories
Artists
Gallery
Sponsor
Country
Texts
A LOT OF DATA IS DUPLICATED
😭

72. EVENT
Location
Categories
Artists
Gallery
Sponsor
Country
Texts
THE SAME OBJECTS ARE IN THE
JSON MANY, MANY, MANY TIMES

73. PROCESSING IMPLEMENTATION V1

74. PROCESSING IMPLEMENTATION V1
Loop over events
in JSON response

75. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Loop over events
in JSON response

76. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response

77. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist

78. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find gallery in DB /
Create new gallery

79. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery

80. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find sponsor in DB /
Create new sponsor
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery

81. PROCESSING IMPLEMENTATION V1
Find event in DB /
Create new event
Find location in DB /
Create new location
Loop over events
in JSON response
Find artist in DB /
Create new artist
Find sponsor in DB /
Create new sponsor
Find text in DB /
Create new text
Find gallery in DB /
Create new gallery
Persist DB

82. BUT WHAT ABOUT THE
DUPLICATED DATA?

83. THE DEFINITION OF INSANITY IS DOING
THE SAME THING OVER AND OVER AGAIN
AND EXPECTING A DIFFERENT RESULT.
Not Albert Einstein
ME AND MY IMPORTERS

84. PROCESSING IMPLEMENTATION V2

85. PROCESSING IMPLEMENTATION V2
Loop over events
in JSON response

86. PROCESSING IMPLEMENTATION V2
Collect all unique
events & relations
Loop over events
in JSON response

87. PROCESSING IMPLEMENTATION V2
Collect all unique
events & relations
Fetch / create
objects
Loop over events
in JSON response

88. PROCESSING IMPLEMENTATION V2
Collect all unique
events & relations
Fetch / create
objects
Loop over events
in JSON response
Loop over event
objects

89. PROCESSING IMPLEMENTATION V2
Collect all unique
events & relations
Fetch / create
objects
Loop over events
in JSON response
Loop over event
objects
Assign relations
to event object

90. PROCESSING IMPLEMENTATION V2
Collect all unique
events & relations
Fetch / create
objects
Loop over events
in JSON response
Persist DB
Loop over event
objects
Assign relations
to event object

91. COLLECT ALL UNIQUE EVENTS AND
RELATIONS?

92. COLLECT ALL UNIQUE EVENTS AND
RELATIONS?
for event in events {
guard let eventId = event["id"] as? Int
else { continue }
eventIds.append(eventId)
if let location = event["location"] as? DWJSON,
let locationId = location["id"] as? Int {
locationIds.append(locationId)
}
// etc...
}

93. EACH OBJECT IS FETCHED/
CREATED AND UPDATED ONCE

94. ME AND MY IMPORTERS
PROBLEM 2 SOLVED

95. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects

96. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go

97. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go
▸ Each object is only updated once

98. ME AND MY IMPORTERS
PROBLEM 2 SOLVED
▸ Collecting all unique ids for objects
▸ Fetching all objects in one go
▸ Each object is only updated once
▸ Pulling apart the data before processing it can have great benefits

99. STORING IN COREDATA
CHAPTER 3:

100. THE USER SHOULD ALWAYS BE ABLE TO
USE THE UI, EVEN WHEN A SYNC IS
TAKING PLACE IN THE BACKGROUND.
Me
ME AND MY IMPORTERS

101. EASY, JUST USE MULTIPLE CONTEXTS
You
ME AND MY IMPORTERS

103. Read context
MAIN QUEUE

104. Read context
MAIN QUEUE
Import context
PRIVATE QUEUE

105. THE UI IS FLASHING A LOT WHEN I
LAUNCH THE APP
The almighty userbase
ME AND MY IMPORTERS

106. UI UPDATING IMPLEMENTATION V1

107. UI UPDATING IMPLEMENTATION V1
Managed object
context did save

108. UI UPDATING IMPLEMENTATION V1
Update UI
Managed object
context did save

109. EACH OPERATION SAVES THE
CONTEXT, RESULTING IN MANY UI
UPDATES

110. UI UPDATING IMPLEMENTATION V2

111. UI UPDATING IMPLEMENTATION V2
Sync helper did
start notification

112. UI UPDATING IMPLEMENTATION V2
Managed object
context did save
Sync helper did
start notification

113. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification

114. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification
Sync helper did
end notification

115. UI UPDATING IMPLEMENTATION V2
Hold on to
notification
Managed object
context did save
Sync helper did
start notification
Sync helper did
end notification
Fire one notification
for each entity

116. ME AND MY IMPORTERS
PROBLEM 3 SOLVED

117. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing

118. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing
▸ Store notifications while you’re importing

119. ME AND MY IMPORTERS
PROBLEM 3 SOLVED
▸ Use multiple contexts for reading and writing
▸ Store notifications while you’re importing
▸ Filter notifications and fire only the notifications you really need to fire

120. ME AND MY IMPORTERS
TAKEAWAYS

121. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage

122. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage
▸ You might need to pre-process imported data for performance

123. ME AND MY IMPORTERS
TAKEAWAYS
▸ Operations can make complex processes easier to manage
▸ You might need to pre-process imported data for performance
▸ Inserting a layer between your app and CoreData can help with managing an
import pipeline

124. 🍷