Presentation taken from Future of Web Apps Boston (http://futureofwebapps.com/boston-2014) In this talk, Steve will build a system from scratch for cross-device data synchronization in JavaScript. Through demos, he will explore all the things you're probably not thinking about when rolling your own sync engine, like offline caching, change notification, and conflict resolution. Drawing on his experience from Dropbox, Steve will discuss the thorny challenges around sync and how to solve them.

1. Steve Marx
Developer advocate
Dropbox

3. “Data synchronization is the process of
establishing consistency among data from
a source to a target data storage and vice
versa and the continuous harmonization
of the data over time.”
– Wikipedia

4. Cloud storage

5. DEMO
Let’s build cloud storage.

6. Simple cloud storage
if request.method == 'PUT' then
storage.data = request.body
end
return storage.data

7. Concurrency

8. Concurrency anti-pattern
This algorithm doesn’t work:
1. GET latest from the server
2. PUT update
What if things change between 1 and 2?

9. if (file.exists) {
file.open();
}

10. Optimistic concurrency

11. Optimistic concurrency
• Try update “optimistically”
• Conditionally update if nothing changed
• Else change is rejected
• HTTP has Etag and If-Match headers

12. DEMO
Improved cloud storage, now without data loss!

13. local hash =
crypto.sha256(storage.data or '').hexdigest()
if request.method == 'PUT' then
if request.headers['If-Match'] == hash then
storage.data = request.body or ''
return 200
else
return 409
end
end

14. Conflict resolution

15. Transactions
Algorithm for applying an operation:
1. Read current value
2. Apply operation to value
3. Send to server
4. If rejected, GOTO 1

16. function increment() {
superagent.put('https://sync.webscript.io')
.set('If-Match', etag).send(counter + 1)
.done(function (res) {
if (res.status !== 200) {
counter = parseInt(res.text);
etag = res.headers.etag;
increment();
}
});
}

17. Transaction downsides
• Commits end up serialized
• Transactions lost on shutdown?
• Firebase and Dropbox Datastores
(Python)

18. Working offline

19. Diff and merge
• Keep track of a common ancestor
• 3-way merge, git-like:
– Diff server data against ancestor
– Apply patch to client data
– (or something equivalent)

20. Merging counts
Ancestor: 3
Server: 4
Client: 5
Merged: 3 + (4-3) + (5-3) = 6

21. DEMO
Merge all the things.

22. Merging downsides
• Requires tracking common ancestor
• Sometimes ambiguous

23. Summary
• Syncing is hard, so think about:
– Dealing with concurrency
– Thinking about offline
– Resolving conflicts in a consistent way
• Or use someone else’s platform
(Dropbox!)

24. Image credits
Thanks!
https://www.flickr.com/photos/kalyan02/5458325252
https://www.flickr.com/photos/gara/3560355335
https://www.flickr.com/photos/somoslarevolucion/3609196127
http://geekandpoke.typepad.com/geekandpoke/2012/09/simply-explained-
concurrency.html
https://www.flickr.com/photos/onesevenone/6361400909

25. @smarx
smarx@dropbox.com