(Or, building better UX / Apps with distributed databases and data synchronisation techniques). This was my talk at Cocoaheads Berlin 17th February 2016.

1. Data Sync on iOS with
Couchbase Mobile
(Or, building better UX / Apps with distributed
databases and data synchronisation techniques)
http://ti.eng.br @thiago_sjc
Thiago Faria Alencar
February 17, 2016HERE Maps @ Berlin

2. What do you mean with User Experience anyway ?
Combination of technology to delivery the best user experience.
That’s why people switch / opt to services: think about
Blacklane, to Immobile scout, Soundcloud.. What can you offer
at the end of the day? Is it the best possible?

3. -Blacklane :)
-Amazon
-Airbnb
-Facebook
-Spotify
-Etc
Don’t “own" things, but redefine how user experience
music, transport,..
What do you mean with User Experience anyway ?

4. A little (real life) story

5. The coffee sales man
in the middle of nowhere
Back in 2007..

6. Let’s go back in time
< 1935 2007
Instant coffee for internal
Brazilian market.. let’s build
a system for sales …
Product’s website at:
www.cafesaojose.com.br

7. A website & mini ERP

8. A website & mini ERP

9. Basic LAMP Setup
(Linux + Apache + Mysql + Php)
Good old times.. right?

10. Using the system
Pedralva - Minas Gerais

11. Network coverage (as of today)
Oops, no connection? all data in the server miles away..

12. Back to 2016… (last weekend)
same old problems….

13. Transit routes (online-only
feature)

14. Clear found/searched route, hop in the train

15. Ups, inside subway now..
1 min later.. hm.. couldn’t we show the past/found route?

16. Other examples
Spotify - offline advertising!
‘Hobby' projects (telemetry)
IoT
Even the most ‘connected' applications will
experience network issues from time to time -> make
them operate offline as much as possible

17. Network coverage getting better …
but as of today still far from ideal..

18. A data locality/expiration issue

19. Can we do better?

20. hell yes.

21. Recognise the problem..

22. “Git-like” for databases?

23. “A distributed database is a database in which portions of
the database are stored on multiple computers within a
network.”

24. “Users have access to the portion of the database at their
location so that they can access the data relevant to their
tasks without interfering with the work of others.”
Turns out, a similar set of features needed on of distributed
databases (computer clusters) is desired to have at the end
nodes (mobile devices) > Same fundamental design decisions.

25. Analysing DB designs
Document / NoSql databases vs. relational databases
scalability / flexibility

26. CAP Theorem : set of basic requirements that
describe any distributed system (not just
storage/database systems).
Consistency - All the servers in the system will have the same
data so anyone using the system will get the same copy
regardless of which server answers their request.

27. CAP Theorem
Availability - The system will always respond to a request
(even if it's not the latest data or consistent across the system or
just a message saying the system isn't working).

28. CAP Theorem
Partition Tolerance - The system continues to operate as
a whole even if individual servers fail or can't be reached.

29. Document / NoSql databases vs. relational databases

30. Document / NoSql databases vs. relational databases
RDMS are more or less ‘all or nothing’, transaction-based, atomic
operations oriented.
Distributed databases relaxes this ‘atomic' requirement in order to provide
better scalability. It turns out this feature is useful for offline mobile apps as
well as we’ll see when compared to relation traditional databases.

31. Document / NoSql databases vs. relational databases
‘NoSQL databases' improves scaling by relaxing
consistency in favor of availability and partition tolerance.

32. A bit or market research..
Find the right tool to do the job

33. Intro to Couchbase
• Open source
• Cross-platform (really)
• Open protocols
• Backed by company, but small ‘lock-in’
• Easy to get started
• Small foot-print on client
• Built-in sync handling
• No ‘migration-hell’ (JSON docs)

34. How to solve the data sync problem?

35. Database per-user doesn’t scale

36. Instead, share a single database
Compare this to a “repository” in git: each subset of data
would be like a “repo”.. not all users will have access to all
data (e.g. ACL with gitolite)

37. The ‘traditional' way of exchanging data..
Encapsulate all of it in a SSL connection and everything will
be fine (that is, while your connection works).

38. What is happening here?
Notice how in the request – response model, data is “routed” by the different
API end-points: URI path will trigger a specific request handler in the server.

39. How such problems are solved during sync?
• Authentication
• Authorization
• Data validation / routing

40. How such problems are solved during sync?
• Data routing:
When another user connects to the same backend, we want only the
documents belonging to him to be synchronised. The term used for this in
Couchbase Mobile is data replication.

41. How such issues are resolved during sync?
That’s when Couchbase data orchestration features kicks in:
Not to depend on an active connection, what each data represents and
the respective routing are described in the document’s content itself.
-Optimal solution sits neither at one or the other end of the spectrum -
normally you need a mix of things : so normal http reqs still useful!

42. How such issues are resolved during sync?

43. For iOS, called
‘Couchbase Lite’
-Available for many other
platforms.
-Advantages of being a
JSON database (no
migration hell)
-Sync gateway ‘like the git
merge'

44. Document: the primary entity stored in a database
is called a document instead of a “row” or “record”.
Views: ‘denormalised tables/indexes ’; dynamically
generated from JSON objects
Queries: the action of looking up results from a
view’s index
Some terminology in Couchbase Lite land..
(Roots in Map/Reduce programming model)

45. 1
2
3
4
5
6
CBLManager *manager = [CBLManager sharedInstance];
NSError *error;
CBLDatabase* _database = [manager databaseNamed: @"conektapp_db" error: &error];
if (error) { NSLog(@"error getting database %@",error); }
NSDictionary* properties = @{@"user_id": @"joe",
@"type": @"contactInfo",
@"contactData": @“+49610234192"};
CBLDocument* document = [database documentWithID:
[NSString stringWithFormat: @"%@:%@", type, user_id]];
NSError* error;
if (![document putProperties: properties error: &error]) {
[self handleError: error];
}
Saving an object in Couchbase Lite
( For questions on this see more detailed articles at http://ti.eng.br )

46. 1
2
3
4
5
6
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user
{
//1- createView
CBLView * contactInfoView = [self.database viewNamed:
@"contactDataByUsername"];
[contactInfoView setMapBlock: MAPBLOCK({
if ([doc[@"type"] isEqualToString: @"contactInfo"]) {
if (doc[@"user_id"])
emit(doc[@"user_id"], doc[@"contactData"]);
}
}) version: @"2"];
}
Retrieve an object in Couchbase Lite

47. 1
2
3
4
5
6
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user
{
//1- createView
CBLView * contactInfoView = [self.database viewNamed:
@"contactDataByUsername"];
[contactInfoView setMapBlock: MAPBLOCK({
if ([doc[@"type"] isEqualToString: @"contactInfo"]) {
if (doc[@"user_id"])
emit(doc[@"user_id"], doc[@"contactData"]);
}
}) version: @"2"];
//2 - make the query
CBLQuery* query = [contactInfoView createQuery];
NSLog(@"Querying username: %@", user);
query.startKey = user;
query.endKey = user;
return query;
}
Retrieve an object in Couchbase Lite

48. 1
2
3
4
5
6
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user
{
//1- createView
CBLView * contactInfoView = [self.database viewNamed:
@"contactDataByUsername"];
[contactInfoView setMapBlock: MAPBLOCK({
if ([doc[@"type"] isEqualToString: @"contactInfo"]) {
if (doc[@"user_id"])
emit(doc[@"user_id"], doc[@"contactData"]);
}
}) version: @"2"];
//2 - make the query
CBLQuery* query = [contactInfoView createQuery];
NSLog(@"Querying username: %@", user);
query.startKey = user;
query.endKey = user;
return query;
}
//run, enumerate
CBLQuery *contactQuery = [myObject queryContactInfoFromUsername:
@"thiago"];
CBLQueryEnumerator* result = [contactQuery run: &error];
for(CBLQueryRow* row in result)
{
NSLog(@"Found document: %@", row.document);
}

49. 1
2
3
4
5
6
NSURL* url = [NSURL URLWithString:
@“http://yourapp.com/sync_gateway/"];
CBLReplication *push = [database createPushReplication: url];
CBLReplication *pull = [database createPullReplication: url];
push.continuous = pull.continuous = YES;
[pull setCookieNamed:@"SyncGatewaySession" withValue:sessionValue
path:nil expirationDate:nil secure:NO];
[push setCookieNamed:@"SyncGatewaySession" withValue:sessionValue
path:nil expirationDate:nil secure:NO];
NSLog(@"startSync");
[pull start];
[push start];
Getting replication running.. client side

50. The server-side
Sync gateway configuration
Located in the sync gateway, the 'sync function' is responsible for
document authorisation, validation, routing.

51. The server-side
Sync gateway configuration
1
2
3
4
5
6
7
8
9
10
11
12
{
"interface":":4984",
"adminInterface":":4985",
"log":["REST"],
"databases":{
"sync_gateway":{
"server":"http://localhost:8091",
"bucket":"app_bucket",
"sync":`function(doc) {channel(doc.channels);}`
}
}
}
sync function, not much defined for now.. will
grow according to your application’s needs

52. When the sync function kicks in?

53. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Validation

54. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Routing

55. More on the sync function..
function(doc, oldDoc)
{
requireUser(oldDoc.owner);
channel(doc.channel);
access(doc.members, doc.roomID);
}
Access
Control

56. About the concept of “Channels”
App server tags Documents on entry; Docs are indexed by channel for
replication.

57. Summary
Data scalability: Replicate only docs relevante to
respective user.
Access Control: Replicate only documents visible to
this user
Data Validation: Replicate only documents that
contains valid information

58. Syncing events.. (change notifications)

59. Web Bonus

60. Web
Others
Bonus

61. Web
Others
Bonus

62. Thank you :)
Finally we have a way to automate data
synchronisation!! Please use it! :)
And let’s shift our designs / apps away from the old "LAMP style”
into something that actually works in real life scenarios also when
'on the go'..

63. Q&A

64. One more slide to show how a more ‘elaborated'
sync function may look like..

65. function (doc, oldDoc) {
if (doc._deleted) {
// Only editors with write access can delete documents:
requireRole("role:editor");
requireUser(oldDoc.writers);
// Skip other validation because a deletion has no other properties:
return;
}
// Required properties:
if (!doc.title || !doc.creator || !doc.channels || !doc.writers) {
throw({forbidden: "Missing required properties"});
} else if (doc.writers.length == 0) {
throw({forbidden: "No writers"});
}
if (oldDoc == null) {
// Only editors can create documents:
requireRole("role:editor");
// The 'creator' property must match the user creating the document:
requireUser(doc.creator)
} else {
// Only users in the existing doc's writers list can change a document:
requireUser(oldDoc.writers);
// The "creator" property is immutable:
if (doc.creator != oldDoc.creator) {
throw({forbidden: "Can't change creator"});
}
}
// Finally, assign the document to the channels in the list:
channel(doc.channels);
}