Introduction to Azure DocumentDB: overview, use cases, consistency models, indexing.

2. Alexander Zyl
.NET Developer
azyl@scnsoft.com

3. A new child

4. NoSQL solutions in Azure
HBase
TS
• Column Family Store • Key/Value Store
Redis
TS

5. Features:
• Fully managed
• Schema agnostic
• Scalable
• Tunable consistency levels
• Tunable indexing policies
• Familiar SQL syntax for querying
• JavaScript execution
DocumentDB

6. DocumentDB resource model
RESTAPI
DocumentDB Infrastructure

7. Databases
/dbs/{id}
DocumentDB
Account

8. Databases
/dbs/{id}
DocumentDB
Account

9. Collections
/colls/{id}
Databases
/dbs/{id}
Users
/users/{id}
Permissions
/permissions/{id}

10. Collections
/colls/{id}
Databases
/dbs/{id}
Users
/users/{id}
Permissions
/permissions/{id}

11. Triggers
/triggers/{id}
Functions
/functions/{id}
Stored Procedures
/sprocs/{id}
Attachments
/attachments/{id}
Documents
/docs/{id}
Collections
/colls/{id}

12. DocumentDB
Account
Users
/users/{id}
Databases
/dbs/{id}
Permissions
/permissions/{id}
Collections
/colls/{id}
Attachments
/attachments/{id}
Documents
/docs/{id}
Triggers
/triggers/{id}
Functions
/functions/{id}
Stored Procedures
/sprocs/{id}

13. Server ZServer CServer BServer A
Logical containers
Physical containers
DocumentDB
Account
CollectionsCollections Collections Collections

14. What about cost?

15. = 𝑓 𝑀𝑒𝑚𝑜𝑟𝑦, 𝐶𝑃𝑈, 𝐼𝑂
Request Unit

16. Performance levels
RUs per second RUs per second20 20 20RUs per second RUs per second250 RUs per second 1k RUs per second 2.5k RUs per second

17. How to model data?

18. {
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"GPSLocation": {
"Latitude": 44.6516185,
"Longitude": -63.5820275
},
"StatusData": […]
}
Vehicle Dealer
StatusData
VehicleIdPK PK
PK
ReleaseYear
Make
DealerId
Name
StatusId
VehicleId
EngineOn
TimeStamp
Address
Vin
Model
Latitude
Longitude

19. Approaches to document modeling
 Reference data
 Embed data

20. Modeling relations

21. Vehicle document:
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"GPSLocation": {
"Latitude": 44.6516185,
"Longitude": -63.5820275
},
"StatusData": [
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23 }
]
}
Bad design

22. When to embed:
 One-to-few relations
 Infrequent changes
 Embedded data has
bounds
Modeling relations

23. Vehicle document:
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"GPSLocation": {
"Latitude": 44.6516185,
"Longitude": -63.5820275
},
"StatusData": [
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23 }
]
}

24. When to embed:
 One-to-few relations
 Infrequent changes
 Embedded data has
bounds
When to reference:
 One-to-many relations
 Many-to-many relations
 Data changes frequently
 Unbounded reference
Modeling relations

25. Vehicle document:
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"GPSLocation": {
"Latitude": 44.6516185,
"Longitude": -63.5820275
},
"StatusData": [
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23 }
]
}
Vehicle document:
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"StatusData": [
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33 },
{ "Id": 3, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 23 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23 }
]
}
VehicleLocation document:
{
"VehicleId": 44,
"GPSLocation": {
"Latitude": 44.651617,
"Longitude": -63.582027
}
}

26. Vehicle
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
"StatusData"95,: [
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33 },
{ "Id": 3, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 23 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23 }
]
}
Vehicle document:
{
"Id": 44,
"ReleaseYear": "2014",
"Make": "Aston Martin",
"Vin": "2G1WT58KX79250102",
"Model": "DBS",
"Dealer": {
"Name": "Atlant-M",
"Address": "Some st. 9"
},
}
VehicleStatus documents:
{ "Id": 1, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 40, "VehicleId": 44 },
{ "Id": 2, "TimeStamp": "2014-07-04", "EngineOn": false, "FuelLevel": 33, "VehicleId": 44 },
{ "Id": 3, "TimeStamp": "2014-07-04", "EngineOn": true, "FuelLevel": 23, "VehicleId": 44 },
{ "Id": 4, "TimeStamp": "2014-07-05", "EngineOn": true, "FuelLevel": 10, "VehicleId": 44 },
{ "Id": 5, "TimeStamp": "2014-07-06", "EngineOn": false, "FuelLevel": 55, "VehicleId": 44 },
{ "Id": 999, "TimeStamp": "2014-08-12", "EngineOn": true, "FuelLevel": 23, "VehicleId": 44 }

27. Indexing

28. Documents in a collection
{
"id": 16,
"text": "Bonjour",
"user": {
"name": "Francois",
"nickname": "@franky"
},
"entities": {
"hashtags": [
{
"text": "#heof"
}
]
}
}
{
"id": 4,
"text": "Hello",
"user": {
"name": "Jerome",
"nickname": "@juim"
},
"entities": {
"hashtags": [
{
"text": "#rutib",
"indices": [ 10, 26 ]
}
]
}
}

29. Index tree
textid userentities
4 16 Hello Bonjourname nickname
Jerome Francois @juim @franky
hashtags
text indices
0
10 26
0
#rutib #heof

30. What we can controlAutomatic indexing
 Manual include
 Manual exclude

31. What we can control
Automatic indexing
 Manual include
 Manual exclude

32. What we can control
Indexing modes
 Consistent
 Lazy
 None

33. What we can control
Indexing modes
 Consistent
 Lazy
 None

34. Building paths
{
"id": 16,
"text": "Bonjour",
"user": {
"name": "Francois",
"nickname": "@franky"
},
"entities": {
"hashtags": [
{ "text": "#heof" }
]
}
}
_.text
_.user.name
_.entities.hashtags
_.entities.hashtags[0].text

35. Building paths
Applicable wildcards:
? – single selection
/text/?
/user/nickname/?
/entities/hashtags/[]/text/?

36. {
"id": 4,
"text": "Hello",
"user": {
"name": "Jerome",
"nickname": "@juim"
},
"entities": {
"hashtags": [
{
"text": "#rutib",
"indices": [ 10, 26 ]
}
]
}
}
/text/?
Building paths: examples

37. {
"id": 4,
"text": "Hello",
"user": {
"name": "Jerome",
"nickname": "@juim"
},
"entities": {
"hashtags": [
{
"text": "#rutib",
"indices": [ 10, 26 ]
}
]
}
}
/user/nickname/?
Building paths: examples

38. Building paths
Applicable wildcards:
? – single selection
* – recursive selection/text/?
/user/nickname/?
/entities/hashtags/[]/text/?
/user/*
/entities/*
/entities/hashtags/[]/*

39. {
"id": 4,
"text": "Hello",
"user": {
"name": "Jerome",
"nickname": "@juim"
},
"entities": {
"hashtags": [
{
"text": "#rutib",
"indices": [ 10, 26 ]
}
]
}
}
/user/*
Building paths: examples

40. {
"id": 4,
"text": "Hello",
"user": {
"name": "Jerome",
"nickname": "@juim"
},
"entities": {
"hashtags": [
{
"text": "#rutib",
"indices": [ 10, 26 ]
}
]
}
}
/entities/hashtags/*
Building paths: examples

41. Indexing options:
Include to index
Exclude from index
Applying rules

42. Index kinds:
Hash – equality queries
Applying rules
SELECT *
FROM collection c
WHERE c.prop = 'value'

43. Index kinds:
Hash – equality queries
Range – range + OrderBy queries
Applying rules
SELECT *
FROM collection c
WHERE c.prop = 'value'
SELECT *
FROM collection c
WHERE c.prop >= 15
ORDER BY c.prop

44. Index kinds:
Hash – equality queries
Range – range + OrderBy queries
Spatial – ST_DISTANCE, ST_WITHIN
Applying rules
SELECT *
FROM collection c
WHERE c.prop >= 15
ORDER BY c.prop
SELECT *
FROM collection c
WHERE ST_DISTANCE(c.Location, {
"type": "Point",
"coordinates": [-122.19, 47.36]
}) < 100 * 1000

45. Index precision:
For numbers: 1-8 bytes
For strings: 1-100 bytes
Applying rules

46. Data consistency

47. •Strong consistency
•Eventual consistency
DocumentDB
•Strong consistency
•Eventual consistency
•Session
•Bounded staleness
Offered consistency models

48. Strong consistency

49. Strong consistency: Write operation
Version2
Version1
Version2
User
Replica A
Replica B
Replica CGateway
Async
Infra
Version2

50. Strong consistency: Read operation
Version2
Version1
Version2
User
Replica A
Replica B
Replica CGateway
Infra
Version2

51. Eventual consistency

52. Eventual consistency: Write operation
Version2
Version1
Version1
User
Replica A
Replica B
Replica CGateway
Async
Infra
Version2

53. Eventual consistency: Read operation
Version2
Version1
Version1
User
Replica A
Replica B
Replica CGateway
Infra
Version1

54. Session consistency

55. Session: Write operation
Version2
Version1
Version1
User
Replica A
Replica B
Replica CGateway
Async
InfraSession Id
Version2

56. Session: Read operation
Version2
Version1
Version1
Replica A
Replica B
Replica CGateway
Infra
Version2
User A
User B
Session Id
Version1

57. Bounded staleness

58. Bounded staleness: Write operation
Version2
Version1
Version1
User
Replica A
Replica B
Replica CGateway
Async
Infra
Version2

59. Bounded staleness: Read operation
Version3
Version1
Version2
User
Replica A
Replica B
Replica CGateway
Sync
Infra
Version2

60. Scalability issues
Out of space
>Data Collection

61. Scalability issues
Too many requests
Out of space
>Data Collection

62. Solutions?
Vertical scaling

63. Horizontal scaling
Vertical scaling
Solutions?

64. Collection = Partition

65. Collection
RequestPartitioning our data

66. Partition 1
Request
Request
Partition 2
Logical grouping
Partitioning our data

67. Partitioning strategies
Hash partitioning
Partition 1
Partition 3
Partition 2
UserId: 14 Hash(14)=>
P3
Infra
Logical grouping

68. Partitioning strategies
Range partitioning
Partition 1
Partition 3
Partition 2
Name: David
Infra
K > David >= A
A-I
K-Q
R-Z
Logical grouping

69. Partitioning strategies
Lookup partitioning
Region name Partition Id
Asia Partition1
Version3Europe Partition2
United States Partition3
Partition 1
Partition 3
Partition 2
Region: Europe
Infra
Logical grouping

70. Use Cases

71. Use Cases: user-defined data

72. Use Cases: storing and analyzing logs
Logs

73. Use Cases: storing materialized views