Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
1 de 51

#GeodeSummit - Design Tradeoffs in Distributed Systems

2

Compartir

Descargar para leer sin conexión

How Southwest Airlines Uses Geode

Distributed systems and fast data require new software patterns and implementation skills. Learn how Southwest Airlines uses Apache Geode, organizes team responsibilities, and approaches design tradeoffs. Drawing inspiration from real whiteboard conversations, we’ll explore: common development pitfalls, environment capacity planning, streaming data patterns like consumer checkpointing, support roles, and production lessons learned.

Every day, Apache Geode improves how Southwest Airlines schedules nearly 4,000 flights and serves over 500,000 passengers. It’s an essential component of Southwest’s ability to reduce flight delays and support future growth.

Libros relacionados

Gratis con una prueba de 30 días de Scribd

Ver todo

Audiolibros relacionados

Gratis con una prueba de 30 días de Scribd

Ver todo

#GeodeSummit - Design Tradeoffs in Distributed Systems

  1. 1. Design Tradeoffs In Distributed Systems How Southwest Uses Apache Geode
  2. 2. Brian Dunlap @brianwdunlap Technical Lead - Aircraft Systems March 8th, 2016
  3. 3. APACHE GEODE AT SOUTHWEST OPS SUITE CARGO CREW SOUTHWEST.COM
  4. 4. Optimize decisions with integrated schedule information.
  5. 5. Grow over the next 10 years.
  6. 6. Show a real-time view of our operational day… to 1,000s of web users.
  7. 7. Scale across data centers and ensure consistency.
  8. 8. Support our most critical operational systems.
  9. 9. Southwest’s Network Operations Control integrates decision makers. BOISE ALBANY OKLAHOMA CITY AUSTIN PANAMA CITY BEACH CHARLESTON GREENVILLE-SPARTANBURG TUCSON LUBBOCK AMARILLO MIDLAND/ODESSA EL PASO LITTLE ROCK NASHVILLE DALLAS (LOVE FIELD) SACRAMENTO OAKLAND SAN JOSE BURBANK LOS ANGELES (LAX) ORANGE COUNTY ONTARIO SAN DIEGO SAN FRANCISCO (SFO) BIRMINGHAM LOUISVILLE CLEVELAND OMAHA TULSA RENO/TAHOE HARLINGEN/SOUTH PADRE ISLAND PUERTO VALLARTA CORPUS CHRISTI ALBUQUERQUE DES MOINES MEMPHIS CABO SAN LUCAS/LOS CABOS ROCHESTER AKRON/ CANTON WICHITA PENSACOLA MEXICO CITY NASSAU PUNTA CANA SAN JUAN MONTEGO BAY ARUBA CANCÚN FLINT GRAND RAPIDS CHARLOTTE DAYTON MINNEAPOLIS/ ST. PAUL PHOENIX DENVER INDIANAPOLIS COLUMBUS RALEIGH/DURHAM CHICAGO (MIDWAY) FT. LAUDERDALE (MIAMI AREA) DETROIT HOUSTON (HOBBY) SEATTLE/TACOMA LAS VEGAS NEW ORLEANS ST. LOUIS MILWAUKEE BUFFALO/ NIAGARA FALLS ATLANTA ORLANDO FT. MYERS/NAPLES JACKSONVILLE TAMPA WEST PALM BEACH SAN ANTONIO KANSAS CITY BELIZE CITY SAN JOSÉ LIBERIA PORTLAND WASHINGTON, D.C. (REAGAN NATIONAL) RICHMOND MANCHESTER PROVIDENCE HARTFORD/SPRINGFIELD NORFOLK/VIRGINIA BEACH BOSTON LOGAN PHILADELPHIA BALTIMORE/WASHINGTON (BWI) WASHINGTON, D.C. (DULLES) PITTSBURGH NEW YORK (LAGUARDIA) LONG ISLAND/ISLIP NEW YORK (NEWARK) SALT LAKE CITY SPOKANE PORTLAND
  10. 10. NOC
  11. 11. CREW PASSENGER MAINTENANCE FLIGHT GATE CARGO AIRCRAFT FACILITY OPS SUITE CONSUMES 10MJMS MESSAGES DAILY
  12. 12. RECOVERY OPTIMIZATION USES OVER 1,000,000 SCHEDULES 4,000 FLIGHTS 700 AIRCRAFT 500K PASSENGERS / DAY
  13. 13. a response time measured in seconds RECOVERY OPTIMIZATION DRIVES
  14. 14. Luv!
  15. 15. Thinking about boundaries TEAMS SOFTWARE FOCUS ORG FOCUS DOMAINS CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
  16. 16. Domain tradeoffs TEAMS SOFTWARE FOCUS ORG FOCUS DOMAIN CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
  17. 17. What do you own? What do you need? How long can you keep it? Domain tradeoffs
  18. 18. Books by: @ericevans0 @VaughnVernon Get Organized! Domain Driven Design (DDD) CORE DOMAIN SUPPORTING DOMAIN UBIQUITOUS LANGUAGE AGGREGATES DOMAIN EVENTS
  19. 19. What do you own? (core) <invest> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
  20. 20. Crew Maint Pax Cargo Flight Gate Existing domain silos… OVER 15 YEARS OF COMPLEXITY
  21. 21. Crew Maint Pax Cargo Flight Gate 100% 100% CORE DOMAINSSUPPORTING DOMAINS INCREMENTAL STEPS SELECTED INTEGRATION
  22. 22. What do you own? (core) <focus> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
  23. 23. Adding is very easy. Watch out for data that’s around for too long. Do all of these data need to be in-memory? Data at rest for a long time? (>365 days) GEODE REGION SIZES
  24. 24. Determine if each subdomain should use Geode. Don’t make an automatic decision. Domain tradeoffs
  25. 25. Maybe it needs an entirely different home? Domain tradeoffs
  26. 26. Pattern tradeoffs TEAMS SOFTWARE FOCUS ORG FOCUS DOMAIN CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
  27. 27. How far? How fast? Pattern trade-offs
  28. 28. The chart of scalability!
  29. 29. OLD NEW NORMALIZED JOINS REGIONS FOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN ServiceManagerHandlerImpl
  30. 30. We’re learning!
  31. 31. OLD NEW NORMALIZED JOINS REGIONS FOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN
  32. 32. We write immutable domain events into event regions. Clients receive events using Geode CQs. Clients checkpoint their position into separate regions. Event regions expire messages. checkpointing
  33. 33. Akka Cluster manages Actor Singletons which coordinate parallel processing based on a logical groupId. Backpressure is implemented through a competing consumer pattern. Take a look at Akka Streams! All Geode replicate regions use distributed ack. We don’t want to converge. (some write wins) coordination (*important concept)
  34. 34. JMS adapter Command adapters Command handlers - to CQ clients View model builders - to CQ clients JMS publishers data flow
  35. 35. PUSH or PULL How do we scale expensive read I/O? Contain expensive reads With CQRS view model builders, perform heavy state enriching “select *” once. Push read updates vs. polling (Geode CQs) Conflate triggering view model rebuild events
  36. 36. Be careful with timeouts! Be careful with alerts! Be careful with joins! Be careful with large values! Be careful with old habits! safety tips
  37. 37. Teams TEAMS SOFTWARE FOCUS ORG FOCUS DOMAIN CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
  38. 38. Distributed systems are created by distributed teams. Communication coordination is a thing.
  39. 39. Integrate Geode security with a directory Tune JVM size and GC Deploy and upgrade environments Size and configure VMs Support production events Enable WAN Gateway Sender / Receivers Load snapshots between environments Automate starting and stopping clusters Teaching distributed concepts - like CAP How do we share new distributed system responsibilities? DBAs UNIX DEVs Middleware Release Management Offshore Support New Geode Team DevOps EARLIER IS BETTER
  40. 40. Learn to luv conversation tension. When there’s tension, you’re on the right track!
  41. 41. opssuite-all schedule-core.jar Use separate repos to help with boundaries. Align Teams with repo ownership. Minimize jar dependencies across teams.
  42. 42. EMBRACE a 100X MENTALITY! What does 100x mean? (msg/sec) Normal rate: 50 Busy rate: 500 Recover rate: 5,000 HOW FAST CAN WE RECOVER?
  43. 43. Create great learning resources Watch out for old habits!
  44. 44. Geode TEAMS SOFTWARE FOCUS ORG FOCUS DOMAIN CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
  45. 45. Prefer less-shared disk I/O. (local to a VM rack, or dedicated) Prefer larger + fewer Geode nodes. (4 larger nodes vs. 8 smaller ones) Take advantage of availability zones (AZs). CONVERSATION LEADERSHIP ACROSS TEAMS SHARED or SHARED LESS What infrastructure supports Geode?
  46. 46. Know your memory (and GC) limits. Watch out for slow heap growth that triggers continuous GC. -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -Xloggc:/your/path/node-name.GC.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=5M Check out GCViewer for GC log analysis.
  47. 47. Essential tool for real-time decision optimization testing! Helpful for QA performance and functional testing. Wonderful Geode feature! WAN Gateway
  48. 48. Optimization binary consumes PDX via C++ Native Client Moving > 200 MB per optimization request Be careful with refactoring PDX data types! C++ Native Client
  49. 49. Questions TEAMS SOFTWARE FOCUS ORG FOCUS DOMAIN CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSS AZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR QUESTIONS
  50. 50. 51 Join the Apache Geode Community! • Check out http://geode.incubator.apache.org • Subscribe: user-subscribe@geode.incubator.apache.org • Download: http://geode.incubator.apache.org/releases/ `

×