5 Years Of Building SaaS On AWS

2. $ whoami Co-Founder & CTO, Sumo Logic Cloud-based Machine Data Analytics Service Applications, Operations, Security Chief Architect, ArcSight Major SIEM player in the enterprise space Log Management for security and compliance

3. From Data to Decisions DEVOPS Streamline continuous delivery Monitor KPI’s and Metrics Accelerate Troubleshooting IT INFRASTRUCTURE AND OPERATIONS Monitor all workloads Troubleshoot and increase uptime Simplify, Modernize, and save costs COMPLIANCE AND SECURITY COMPLIANCE AND SECURITY Automate and demonstrate compliance Audit all systems Think beyond rules Cloud Analytics Platform DEVOPS IT INFRASTRUCTURE AND OPERATIONS COMPLIANCE AND SECURITY

4. Cloud Analytics Platform From Data to Decisions DEVOPS IT INFRASTRUCTURE AND OPERATIONS COMPLIANCE AND SECURITY Customer A Cloud COLLECTOR COLLECTOR Customer A Data Center Customer B Data Center COLLECTOR Customer B Cloud COLLECTOR

5. Why SaaS? Because enterprise software sucks™

6. Why SaaS? Because enterprise software sucks™ Too much pain for the customer Time spent running the system is not spent using the system Expensive when done adding hardware and people

7. Why SaaS? Because enterprise software sucks™ Too much pain for the customer Time spent running the system, not spent using the system Expensive when done adding hardware and people Disastrous for the vendor No control over the runtime, hard to diagnose problems Kills innovation because each release lives forever

8. Why AWS? We are developers, not data center people AWS has turned the data center into an API As developers, we understand reuse (libraries, OSs, …) Today’s systems require reuse on a higher level Do you really want to care for 4,000 machines? HA? DR?

9. Anti-monolithic In previous gigs, we dealt with monolithic systems With Sumo, we knew what we needed to build, no MVP required Get data into the system, index it, provide query function So we had a logical breakdown immediately And we knew it had to scale… …not just to the biggest customer, but to all customers!

10. Ingestion Path Receiver Bus Index Raw CQ S3

11. Analytics Path Query Service CQ S3

12. Scale Today 50 TB of new incoming data per day Double-digit PB of data under management >2,000,000 queries/day Thousands of instances in 4 regions globally

13. Divide & Conquer

14. Divide & Conquer 3 to 1000s of instances!

15. Divide & Conquer Each box in the previous images is an application

16. Divide & Conquer Blast radius, bulk-heading, concern separation

17. Divide & Conquer Not everything will break all the time – repair engines, not plane

18. Divide & Conquer Not everybody will need to work on everything all the time

19. What We Actually Did Compose applications from layers of modules Whole system is Scala on top of the JVM One Maven POM per module, one main() per application Initially one GitHub repository per module, today just one project Right size AWS instance for each application cluster Each application exposes a façade Avro over HTTP, or Avro over HornetQ, or Avro over Kafka

20. How I Actually Visualize Microservices

21. Deployment wide services Ingest Search Internal tools receiver hornetq- forge forge cqsplitter search cloud collector service api con- cierge stream katta glass, ganglia bill mix meta conﬁg zoo- keeper appvault org raw hornetq- inbound cocoa bloom ﬁlter analyticscsi cqmerger rework view autoview depman hornetq- internal hornetq- metadata nrt 2 to the power of 5 services (“32”), 170+ modules Don’t even ask about the # of dependencies At least 3 of each – everything is a separately scalable cluster

22. Service Discovery Loose coupling in the large… A deployment is made up of many things Some of these things need to talk to each other Some of these things come and go Don’t pass in a huge list of static dependencies Start each application with one parameter $ bin/receiver prod.service-registry.sumologic.com

23. Anti-singletenant Multi-dimensional scaling predicates multitenancy This is a data processing platform – cost matters! Autoscaling single tenants is too fine-grained for us Also, efficiency… one code line “master” in deployment Customers aren’t pets, they are cattle 

24. Anti-singletenant Multi-dimensional scaling predicates multitenancy This is a data processing platform – cost matters! Autoscaling single tenants is too fine-grained for us Also, efficiency… one code line “master” in deployment Customers aren’t pets, they are cattle 

25. Anti-singletenant Multi-dimensional scaling predicates multitenancy This is a data processing platform – cost matters! Autoscaling single tenants is too fine-grained for us Also, efficiency… one code line “master” in deployment Customers aren’t pets, they are cattle  Yum yum yum… FEATURE FLAGS!!!

26. Just one typical Sumo Logic customer - 8x Variance!

27. Just one typical Sumo Logic customer - 8x Variance! Money flushed down the toilet

28. Just one typical Sumo Logic customer - 8x Variance! Money flushed down the toilet Load per tenant fluctuates wildly, but aggregated system load just goes up slowly

29. Anti-manual We use Jenkins, of course We still build system versions as cross-cuts and QA them We are busy moving toward true continuous delivery Application Groups for things that evolve together… …and that can be deployed together

30. ProdLongStagNite

31. dsh: Another AWS Deployment Tool Model-driven, describe desired state, run to make it so High performance due to parallelization Covers all layers of the stack – AWS, OS, Sumo Logic Easy to use and extend, scriptable CLI Developer-friendly, Scala-based, high-level APIs

32. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management EC2 EC2 Route53 S3 Glacier CloudFront DynamoDB RDSElastiCache DynamoDB DynamoDB RedShift WorkSpaces CloudWatch CloudTrail IAM CodeDeploy BeanstalkCloudFormationOpsWorksSWF SWF EMR EMR Kinesis SNS Mobile Analytics Kinesis SNS CognitoDirectory Service CloudSearch AppStream SES SQS SWF XCode Data Pipeline

33. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management 3 ELBs (service, api, receiver)

34. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management EC2, obviously RIs, dabbling with Spot

35. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management SES for alert emails to our customers

36. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management SQS for user registration from corporate website

37. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Petabytes of S3

38. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management ElastiCache Memcache for client object caches

39. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management DynamoDB for feature flags and configuration

40. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management RDS MySQL for configuration and content objects

41. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management SimpleDB for deployment location

42. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Sumo Logic 

43. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Sumo Logic  CloudWatch, CloudTrail

44. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Sumo Logic!

45. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Zuora for billing

46. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Jenkins, GitHub

47. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management Our own automation framework – “dsh”

48. Data Access Layer Delivery Authentication & Authorization MeteringMonitoring Ordering Provisioning Billing Analytics Resource Management SaaS Application(s) Business Services Core Platform Services Interaction Application AdditionalApplications Application Lifecycle Management CloudFormation for Mesos cluster setup

49. Integrations Generic S3 Collection Amazon S3 Audit Elastic Load Balancing Amazon CloudFront AWS CloudTrail Amazon VPC Flow Logs AWS Config

50. What Does the Future Hold? Super happy to see Amazon EFS introduced Borderline unnaturally excited about AWS KMS Planning on using AWS Lambda as a “plugin system” Implementing Mesos for new services Very excited about Docker to enable better utilization

51. Thank You! @raychaser