Exploring four Datomic superpowers

1. Exploring Four Datomic Superpowers Lucas Cavalcanti & Edward Wible

2. 2 Why build a bank from scratch in Brazil using Clojure and Datomic?

5. 4 1 Audit Trail

6. 5 event stream 01 NOV 10:00 Customer joins waiting list for a card

7. 5 event stream 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card

8. 5 event stream 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card

9. 5 event stream 15 NOV 15:00 Support agent increases limit to R$5K 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card

10. 5 event stream 15 NOV 17:05 Customer blocks card 15 NOV 15:00 Support agent increases limit to R$5K 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card

11. facts over time 6 15 NOV 17:05 15 NOV 15:00 09 NOV 08:00 01 NOV 11:00 01 NOV 10:00 [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] [<account> :account/limit 5000] [<account> :account/limit 3000] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100] [<purchase> :purchase/merchant “Starbucks”] [<account> :account/customer <customer>] [<account> :account/limit 3000] [<card> :card/account <account>] [<card> :card/status :card.status/active] [<customer> :customer/id #uuid “b2c90…”]

12. facts over time 6 15 NOV 17:05 15 NOV 15:00 09 NOV 08:00 01 NOV 11:00 01 NOV 10:00 [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] [<account> :account/limit 5000] [<account> :account/limit 3000] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100] [<purchase> :purchase/merchant “Starbucks”] [<account> :account/customer <customer>] [<account> :account/limit 3000] [<card> :card/account <account>] [<card> :card/status :card.status/active] [<customer> :customer/id #uuid “b2c90…”] entity attribute value

14. 7 don’t lose data

15. 7 don’t lose data what changed when

16. 7 don’t lose data what changed when as-of since

17. 7 don’t lose data what changed when as-of since fork merge

18. 8 What was the initial limit for the card?

19. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated?

20. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated? How long did the customer spend on each stage of the acquisition funnel?

21. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated? How long did the customer spend on each stage of the acquisition funnel? How frequently do we see amount changes on Starbucks transactions?

22. 9 What sequence of events resulted in the Starbucks transaction being persisted?

23. 9 What sequence of events resulted in the Starbucks transaction being persisted? iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A

24. 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

25. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

26. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3

27. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3

28. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3

29. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A :fraud-preventative :recurring-scheduled :late-payment :data-migration user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3

30. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3 :fraud-preventative reason :recurring-scheduled :late-payment :data-migration

31. (defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked] ])) 10 storing transaction metadata

32. (defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked] {:db/id (d/tempid :db.part/tx) :audit/user "lucas@nubank.com.br" :audit/cid "iZb.jnA.9Cd.l9A" :audit/tags :fraud-preventative} ])) 10 storing transaction metadata

33. 11 transactions over time with metadata [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] 15 NOV 17:05 [<account> :account/limit 5000M] 15 NOV 15:00 [<account> :account/limit 3000M] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100M] [<purchase> :purchase/merchant “Starbucks”] 09 NOV 08:00 [<account> :account/customer <customer>] [<account> :account/limit 3000M] [<card> :card/account <account>] [<card> :card/status :card.status/active] 01 NOV 11:00 01 NOV 10:00 [<customer> :customer/id #uuid “b2c90…”]

34. 11 transactions over time with metadata [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] 15 NOV 17:05 [<account> :account/limit 5000M] 15 NOV 15:00 [<account> :account/limit 3000M] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100M] [<purchase> :purchase/merchant “Starbucks”] 09 NOV 08:00 [<account> :account/customer <customer>] [<account> :account/limit 3000M] [<card> :card/account <account>] [<card> :card/status :card.status/active] 01 NOV 11:00 01 NOV 10:00 [<customer> :customer/id #uuid “b2c90…”]

35. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

36. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?entity attribute value txn [?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

40. 13 2 Authorization

41. should we return data? GET /purchases/1337/comments

42. should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized

43. customer account payment card purchase comment should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized

46. 15 explicit relationship traversal (defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))

47. 15 explicit relationship traversal (defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))

48. generic relationship traversal 16 recursive rule (logical OR)

49. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

55. generic relationship traversal 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

56. generic relationship traversal :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules)) 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]]) (defn owns? [customer-id entity db] (d/q '{:find [?e .] (owns? customer-id [:purchase/id id] db)

57. generic relationship traversal :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules)) 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]]) (defn owns? [customer-id entity db] (d/q '{:find [?e .] (owns? customer-id [:purchase/id id] db)

58. 17 Recursion rules! But can we go further? Can we prevent queries from returning entities owned by other customers?

59. all owned entities 18 query binding attribute

60. all owned entities 18 query binding attribute (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

61. all owned entities 18 query binding attribute (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

62. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

63. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules)) (defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))

66. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

71. filtered db (as value) (let [owned (set (owned-entities customer-id db))] (d/filter db 19 (defn owned-db [customer-id db] (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom))))) filtered db will not contain datoms from other owners!

72. passing a filtered db is transparent for queries 20 (defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id])) (all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))

73. passing a filtered db is transparent for queries 20 (defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id])) (all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))

74. 21 We’re Mobile! Bandwidth is costly

75. 22 3 HTTP Cache

76. 23 http last modified header GET /accounts/1234/purchases 200 OK Last-Modified: Fri, 14 Nov 2014 14:28:50 UTC {"purchases": [...]}

77. http last modified header If-Modified-Since: Fri, 14 Nov 2014 14:28:50 UTC GET /accounts/1234/purchases 304 Not Modified 23 GET /accounts/1234/purchases 200 OK Last-Modified: Fri, 14 Nov 2014 14:28:50 UTC {"purchases": [...]}

78. 24 How to keep track of a good last-modified date for a URL?

79. 24 How to keep track of a good last-modified date for a URL? The last time any owned entity changed!

80. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

89. 26 Cache hits are awesome, but what about cache misses?

90. 27 4 Mobile Sync

91. desired hypermedia from the API 28 GET /accounts/1223/purchases 200 OK {"purchases": [...] "_links": { "updates": "https://...?since=2014-11-10T11:12:13Z" } }

92. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))

99. 30 Bonus

100. 31 5 Future DBs

101. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32

106. 33 6 Testing

107. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

113. db testing - vector of datoms 35 (fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))

114. db testing - vector of datoms 35 (fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))

115. 36 7 Schema Extension

116. schema is data! 37 (d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])

117. :nubank/transform :pii schema is data! 37 (d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])

118. 38 8 Sharding Reads

119. 39 sharding to the transactor customers accounts notification acquisition processor auth

120. 39 sharding to the transactor customers accounts notification acquisition processor auth

121. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth

122. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth shard-specific peer cache

123. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth shard-specific peer cache inter-shard ACID transactions

124. 40 9 Db Aggregation

125. 41 data science aggregation customers accounts notification acquisition processor auth data science

126. 42 querying multiple databases (defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))

127. 42 querying multiple databases (defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))

128. 43 1 Audit Trail 2 Authorization 3 HTTP Cache 4 Mobile Sync 5 Future DBs 6 Testing 7 Schema Extension 8 Sharding Reads 9 Db Aggregation

129. 44 We’re hiring

130. 45 We’re hiring

131. Thanks! Lucas Cavalcanti & Edward Wible @lucascs lucas@nubank.com.br edward@nubank.com.br

Exploring four Datomic superpowers

Recomendados

Recomendados

Más contenido relacionado

Último

Último (20)

Destacado

Destacado (20)

Exploring four Datomic superpowers