Can My Inventory Survive Eventual Consistency?
•Descargar como PPTX, PDF•
2 recomendaciones•477 vistas
Let’s explore an inventory use case and discover how it’s possible to use an eventually consistent data store like Cassandra / Datastax Enterprise to support scalability, consistency, continuous availability. Combine that with analytics and I’ll show you how to build an inventory system for the future.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Can My Inventory Survive Eventual Consistency?
Similar a Can My Inventory Survive Eventual Consistency? (20)
Último
Último (20)
Can My Inventory Survive Eventual Consistency?
- 1. Jeff Carpenter Can My Inventory Survive Eventual Consistency?
- 2. Who am I? • Developer • Architect • Author • Evangelist • Defense • Hospitality • R&D • Distributed Systems • Large Scale • Cassandra © DataStax, All Rights Reserved. 2
- 3. DataStax Enterprise / Cassandra Use Cases • Customer 360 • Personalization • IoT/Sensors • Security • Inventory Management! © DataStax, All Rights Reserved. 3
- 4. What Is Inventory? A tangible object that can be sold… Shirt, Clif Bar, Used Car CREATE TABLE physical_inventory (product_id uuid PRIMARY KEY, int amount_available, int amount_consumed ); A space that can be used for some time Hotel Room, My house, Restaurant table CREATE TABLE temporal_inventory (product_id uuid, date date, int amount_available, int amount_consumed, PRIMARY KEY ((product_id), date) ); © DataStax, All Rights Reserved. 4
- 5. Inventory in Context © DataStax, All Rights Reserved. 5 rates temporal inventory products reservations physical inventory purchase orders prices Retail Reservation and rental
- 6. Why use DSE/Cassandra for inventory? • Effortless scale • Geographic distribution • Real-time access / Low latency • Continuous availability • Consistency © DataStax, All Rights Reserved. 6
- 7. Design Considerations • Systems of Record • Microservices • Tuneable Consistency • Compensating Transactions • Data Maintenance via TTL (for Temporal Inventory) © DataStax, All Rights Reserved. 7
- 8. Systems of Record • Are there other systems that track this inventory data? • What is the level of uncertainty in my system vs. external systems? © DataStax, All Rights Reserved. 8 Property Management System Reservation Systeminventory synchronization Walk-in customer Online customerProperty manager
- 9. Notional Microservice Architecture – Temporal Inventory © DataStax, All Rights Reserved. 9 Product Service Booking Service Rates Service Shopping Service Data Maintenance Apps Inventory Service Reservation Service Inventory keyspace Rates keyspace Product keyspace Reservations keyspace
- 10. Consistency Error Cases • Undercounting sold inventory – Reservations – Overbooking • Bumping, walking – Retail • Back order, cancel order • Overcounting sold inventory – Lost opportunity to sell, wasted resource • Inconsistencies across data types – Booked reservation without decrementing inventory © DataStax, All Rights Reserved. 10
- 11. Inventory Service Instance 1 Strong Consistency via Quorum on Read/Write © DataStax, All Rights Reserved. 11 C* node node node node Inventory keyspace Replication Factor = 3 Inventory Service Instance 2 Write @ QUORUM (2 nodes) Read @ QUORUM (2 nodes)
- 12. Distributed Transaction? Consistencies Across Data Types © DataStax, All Rights Reserved. 12 Commit the contract Reserve the inventory Booking Service Inventory Service Reservation Service inventoryreservations Monolithic Reservation System
- 13. Tuneable Consistency Revisited © DataStax, All Rights Reserved. 13 Approach Example Scope Lightweight Transaction Updating inventory counts Data Tier Logged Batch Writing to multiple denormalized tables Data Tier Retrying failed calls Data synchronization, reservation processing Service Compensating processes Verifying reservation processing System Eventual consistency Strong consistency
- 14. Compensating Transactions • Technical – Run repairs more frequently – Consistency checking apps • Business Processes – Reservation: reschedule, “walk” – Retail: back order, refund It’s all about the tradeoffs… © DataStax, All Rights Reserved. 14 Development and operations cost Customer satisfaction and retention cost
- 15. TTL for Temporal Inventory Cleanup © DataStax, All Rights Reserved. 15 Now Time Yesterday’s data is ancient history Rate + Inventory Data
- 16. Tomorrow’s Inventory Management System • Leverage DSE Search / Analytics / Graph for value added real-time use cases © DataStax, All Rights Reserved. 16 Identify undersold product Personalize shopping experience Upsell popular products Offer discount
- 17. Plug © DataStax, All Rights Reserved. 17 Cassandra: The Definitive Guide, 2nd Edition Completely reworked for Cassandra 3.X: • Data modeling in CQL • SASI indexes • Materialized views • Lightweight Transactions • DataStax drivers • New chapters on security, deployment, and integration
- 18. Contact © DataStax, All Rights Reserved. 18 jeff.carpenter@datastax.com @jscarp jeffreyscarpenter
Notas del editor
- (Andrew) Hotels - descriptive data about the hotels and their products, and policies. Quite static Rates - prices that are charged for the products. these can change many times a day, and could include an automated pricing system Inventory - constantly changes as rooms are booked, cancelled, etc. Data quality and currency is extremely important here so we don’t oversell our hotels Reservations - contract with the customer. Generally only changed when initiated by the customer, infrequent changes (The Marriott may disagree after dealing with me)
- (Andrew) After we identified our key data types, they seemed like a good way to divide the work and the system landscape As work commenced, we actually divided things a little further, and kept the keyspace per service idea going, approaching a share-nothing architecture style. We stayed with a single cluster for now to ease operations and reduce cost. On top of this we added services to encapsulate the complexities shopping and booking We used rules at this level to define business logic likely to change We also built data maintenance applications to: synchronize of data from other systems – our legacy system as well as some other systems that will stay in operation, such as property management systems Verify data accuracy across systems and across service boundaries Correct data issues caused by defects
- (Jeff) We’ve made use of Netflix’s Simian Army in order to build reliability into the system Part of this was allowing Chaos Monkey to kill Cassandra nodes to make sure our clusters could survive losing nodes abruptly This helped us mature our cluster monitoring and test automated cluster management capabilities. It also helped us uncover an unexpected behavior of the Java driver, which has since been fixed in the 3.0 driver. Our configuration was using a DNS name to locate the nodes in the cluster. Calling the “addContactPoint()” operation with the well known name initially bound to one record. If this happened to be the node that was killed by Chaos Monkey, before the record was cleared from DNS, the driver would fail to connect to that node, and would be unable to bootstrap. We worked around this by calling addContactPoints() instead, which binds to multiple IP addresses so that it can make multiple connection attempts. This has been fixed in the 3.0 driver – addContactPoint() now binds to multiple Ips if you use a DNS name. The moral of the story - to help mitigate against common connection issues, we created a common library to manage connections across services and connectivity. It loads connection information from the environment, including the cluster name, security credentials
- (Jeff) One of the challenges of a microservices architecture is keeping changes in sync across service boundaries. One example situation is in booking a reservation. Since the reservation represents our contract with the customer to reserve a specific room at a specific price and with certain conditions, we need to mark a reservation as committed at the same time as we reserve the inventory. This is important so that we don’t accidentally overbook our hotel. Making the situation more complicated, there could be simultaneous bookings and data maintenance activities also trying to access the same inventory Since these types are split across microservice boundaries, there is no transaction mechanism. In fact, since the data is in different rows (and different tables), Cassandra’s lightweight transactions are of no use to us here. We solved this by a layered approach – LWTs to protect inventory counts, retries within the booking service, and compensating processes to detect and cleanup failures
- (Jeff) Thankfully we have a variety of tools in our toolbox for guaranteeing consistency. Some of these are provided by Cassandra but some of them are architecture approaches.
- (Andrew) We have separated the shopping and booking concerns from our analysis and history uses, which means that in the shopping and booking systems, data relevant to the past is not much use. As we insert our data, we set the TTL for when it will no longer be needed, which saves us from developing our own cleanup process and reduces our storage footprint. We still need the historic data for analysis and customer service purposes, though, so we store it in a separate data platform which we feed from the reservation system using asynchronous event processing Our colleague Narasimhan Sampath is talking at Strata NYC later this month about our data and analytics platform, which is based on Spark and Hadoop. Make sure to check out his talk if you’re attending Strata.
- (Jeff) after religiously following the mantra of designing tables for each access pattern, we soon ran into cases where adding a table per unique access pattern proved to be too much Take for example hotels and the number of ways by which various clients could search for hotels Since the hotel records are quite large, imagine the impact of all of these tables on our cluster size and storage requirements for 6000+ hotels. We reined this in by designing tables to support multiple queries and doing some filtering at the service layer, which helped us rein in our computing costs. We’re also looking to move to Cassandra 3.X in order to take advantage of materialized views and SASI indexes, which will allow us to shift some of the processing burden back to the database