Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Journey through the Cloud:      Application Services    Ryan Shuttleworth – Technical Evangelist                 @ryanAWS
Journey through the cloudCommon use cases & stepping stones into the AWS cloud                     Learning from customer ...
Application Services                         Build upon services built for the cloud      Address common pain points in ap...
AgendaWhy AWS for application servicesServices overviewDive into select servicesWhere to go next
Why AWS for application services?
Application ServicesServices that wrap software you’d  commonly install and manage             yourself
Application ServicesDatabases     AnalyticsMiddleware   Frameworks
Application ServicesScalability   Reliability   Availability
Application ServicesScalability          Reliability           Availability  Do you focus on        Does core            I...
Application Services Operational Management
Application Services Operational Management     Do you spend more time managing      application services than you do     ...
30%                    70%  On-Premise       Your           Managing All of theInfrastructure   Business   “Undifferentiat...
30%                                     70%  On-Premise       Your                            Managing All of theInfrastru...
Infrastructure &                       Compute                                                  ScalingApplication Service...
Storage & Archive                        AWS is used in a variety of ways… By managing time-consuming                   Br...
Business & technical drivers       You might be able to:
Business & technical drivers                     You might be able to:       Reduce costsRunning software on EC2 can bemor...
Business & technical drivers                     You might be able to:       Reduce costs                      Improve rel...
Business & technical drivers                     You might be able to:       Reduce costs                      Improve rel...
Business & technical drivers                     You might be able to:       Reduce costs                      Improve rel...
Q                     Instance                  Availability Zone  Decoupled                           Reliableapplication...
Q                   Q                    Q   Instance            Instance              InstanceAvailability Zone   Availab...
Q                   Q                   Q                Instance            Instance           Instance             Avail...
Simplified                                          operations                     Availability Zone      Availability Zon...
Queuing on AWS  Setup & manage instances  Install & configure queuing           middlewareSet up persistent message storeI...
Queuing on AWS                        Amazon SQS  Setup & manage instances               Create a queue  Install & configu...
AWS application services
Databases        Middleware          Frameworks         Analytics    Relational                      ElastiCache          ...
Databases        Middleware          Frameworks         Analytics    Relational                      ElastiCache          ...
Amazon Relational Database         Service
Source: Forrester                    Security planning                                    License training Backup, recover...
Migration                                Backup and   Schema design                recovery       Patching                ...
Migration           Focus on                                Backup and                                    these   Schema d...
Migration                                Backup and   Schema design                recovery       Patching                ...
Near zero administration   Painless patching, automatic upgrades   Cloudwatch monitoring, metric alarms   One click. High ...
One click.High availability with Multi-AZ  Automated deployment across multiple AZs  Synchronous replication from master t...
Push-button scale, high performance    Scale storage from 5Gb to 1Tb of storage    Scale instance from small to 4XL (bette...
Relational Database Service                       Database-as-a-Service                       No need to install or manage...
RDBMS on AWS  Setup & manage instances Install & configure database            platform     Configure backupsImplement mas...
RDBMS on AWS                             RDS  Setup & manage instances          Create RDS instance Install & configure da...
Amazon Relational Database Service (Amazon RDS) databases stores forum threads, site content, and project configuration   ...
Amazon DynamoDB
Database services                 DynamoDB & NoSQL                                Requirement: predictable, consistent    ...
Database services                      DynamoDB & NoSQL                                     Requirement: predictable, cons...
Database services                      DynamoDB & NoSQL                                     Requirement: predictable, cons...
Dial it up                          DynamoDB              Provisioned read/write performance per table             Predict...
Low provisioned throughput                                              Table                                             ...
Low provisioned throughput                                 Replica               Table             Replica                ...
Low provisioned throughput                                              Table                                             ...
Increased provisioned throughput                                 Table       Table        Table      Table       Table    ...
High provisioned throughput                               Table       Table       Table       Table       Table       Tabl...
DynamoDB                                              Feature    DetailsProvisioned throughput NoSQL                      ...
NoSQL on AWSPerform capacity planning and  determine nodes required  Define PIOPS EBS volumes &         instance types  Se...
NoSQL on AWS                      DynamoDBPerform capacity planning and          Define tables  determine nodes required  ...
“AWS gave us the flexibility to bring a massive                               amount of capacity online in a short period ...
Amazon Simple Workflow
Simple Workflow                                                  1                                                        ...
A typicalbusinessworkflow…Multiple steps                   Update                Update €   Ship    SendMultiple decision ...
A typicalbusinessworkflow…                                System A                  System C   System   System EMultiple s...
A typicalbusiness                           Process logic                        Middlewareworkflow…State managed in      ...
Implemented in                          Workflow metadata & stateSimpleWorkflow…                                          ...
Amazon SimpleWorkflow Service     (SWF)
DeciderGet Decision Task                          Return Result     Amazon Simple     Workflow Service          (SWF)
Decider                    Get Decision Task                                              Return Result                   ...
Workflows on AWS Setup & manage instances Install & configure workflow          middlewareArchitect for high availability ...
Workflows on AWS                    Simple Workflow Setup & manage instances         Implement ‘decider’ in language Insta...
Amazon CloudSearch
select   *from     articleswhere    contentlike     ‘%cloud%’
1 million hits a day?  5 million items?
Finding things is trickier   than first appears
+
Anatomy of a search         Results & Ranking               Tokenization  FieldedFacets
Endpointsdoc-mydomain.us-east-1.cloudsearch.amazonaws.com            search-mydomain.us-east-1.            cloudsearch.ama...
Testing   Query String                         FacetsResults
Integration                                            Domain          http://search-toilets-in-oz-m6gko2xujhti47jv5w62eld...
Integration              Search                                               Results        {              "rank":"-text_...
Scalable
Time: 1800hRequests            <80% CPU       Small Instance            Partition 1             Copy 1                    ...
Time: 2000hRequests                             >80% CPU                          Small Instance                          ...
Time: 2200hRequests                                              >80% CPU                                           Small ...
Time: 0000hRequests                                           Small Instance                                              ...
Cost savingsRequests                                                 Traditional required capacity                        ...
Cost savingsRequests                                                 Traditional required capacity                        ...
Cost savingsRequests                                                       Traditional required capacity                  ...
Cloud       Search                Search Instance                   Partition 1                     Copy 1 Traffic        ...
Data                           Document quantity & sizeCloudSearch         Search Instance     Search Instance      Search...
Data                                  Document quantity & size       Cloud       Search                Search Instance    ...
Search on AWSPerform capacity planning on size           of indexes   Setup & manage required      number of instances   I...
Search on AWS                         Cloud SearchPerform capacity planning on size   Create search domain in console     ...
Amazon Elastic MapReduce
1 instance for 100 hours            =100 instances for 1 hour
Small instance = $8
Elastic MapReduce       Feature     Details       Scalable    Use as many or as few compute                   instances ru...
But what is it?
A frameworkSplits data into piecesLets processing occur   Gathers the results
Very large  dataset (e.g TBs)
Lots of             instances of ‘X’Very large  dataset (e.g TBs)
Lots of             instances of ‘X’Very large  dataset (e.g TBs)    Split the               log into             many sma...
Process in an             Lots of            EMR cluster             instances of ‘X’Very large  dataset (e.g TBs)    Spli...
Process in an             Lots of            EMR cluster             instances of ‘X’Very large  dataset (e.g TBs)    Spli...
Process in an             Lots of            EMR cluster             instances of ‘X’Very large  dataset                  ...
Very large  dataset                                        Aggregate (e.g TBs)   Insight in a fraction of the time   view ...
1 instance for 100 hours            =100 instances for 1 hour
Small instance = $8
1 instance for 1,000 hours             =1,000 instances for 1 hour
Small instance = $80
S3Input data
S3        Input dataCode    Elastic       MapReduce
S3        Input dataCode    Elastic      Name       MapReduce     node
S3        Input dataCode    Elastic      Name       MapReduce     node                            Elastic                 ...
S3        Input dataCode    Elastic      Name       MapReduce     node                                      HDFS          ...
S3        Input dataCode    Elastic               Name       MapReduce              node                        Queries   ...
S3        Input dataCode    Elastic               Name                           Output       MapReduce              node ...
S3Input data                   Output                  S3 + SimpleDB
Hadoop on AWSConfigure & manage EC2 instances        Setup Namenode        Setup Datanodes     Create HDFS file system   S...
Hadoop on AWS                  Elastic MapReduceConfigure & manage EC2 instances   Specify desired cluster size        Set...
Features powered by Amazon Elastic           MapReduce:     People Who Viewed this Also Viewed             Review highligh...
Elastic MapReduce                     "With Amazon Elastic MapReduce, there                      was no upfront investment...
Where to go next
http://aws.typepad.comhttp://aws.amazon.com/whitepapers
Powerof60.com$100 of creditsThat’s enough to run anAmazon Elastic ComputeCloud (Amazon EC2)Standard Small Instancefor appr...
Summary
AWS is more than compute & storageDatabases, middleware and complex frameworks as servicesRemove the operational burden fr...
aws.amazon.com
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Journey Through the AWS Cloud; Application Services
Próxima SlideShare
Cargando en…5
×

Journey Through the AWS Cloud; Application Services

5.634 visualizaciones

Publicado el

The fourth in our series of webinars, 'Journey Through the AWS Cloud'. This complimentary presentation discusses the use of services offered by AWS that alleviate the need for you to install and manage software on EC2 instances. We introduce the key services customers employ to keep them focused on developing their applications, whilst AWS takes care of running the scalable and reliable building blocks upon which they are built.

Publicado en: Tecnología

Journey Through the AWS Cloud; Application Services

  1. 1. Journey through the Cloud: Application Services Ryan Shuttleworth – Technical Evangelist @ryanAWS
  2. 2. Journey through the cloudCommon use cases & stepping stones into the AWS cloud Learning from customer journeys Best practices to bootstrap your projects
  3. 3. Application Services Build upon services built for the cloud Address common pain points in application architectures Reduce operational management of software componentsFocus on application function, not undifferentiated heavy lifting
  4. 4. AgendaWhy AWS for application servicesServices overviewDive into select servicesWhere to go next
  5. 5. Why AWS for application services?
  6. 6. Application ServicesServices that wrap software you’d commonly install and manage yourself
  7. 7. Application ServicesDatabases AnalyticsMiddleware Frameworks
  8. 8. Application ServicesScalability Reliability Availability
  9. 9. Application ServicesScalability Reliability Availability Do you focus on Does core Is middlewarescaling frameworks component ‘glue’ difficult to rather than reliability affect make as optimizing your your application available as you code? track record? need it?
  10. 10. Application Services Operational Management
  11. 11. Application Services Operational Management Do you spend more time managing application services than you do building and managing assets core to your business?
  12. 12. 30% 70% On-Premise Your Managing All of theInfrastructure Business “Undifferentiated Heavy Lifting”
  13. 13. 30% 70% On-Premise Your Managing All of theInfrastructure Business “Undifferentiated Heavy Lifting” AWS More Time to Focus on Configuring Your Cloud-Based Your Business Cloud AssetsInfrastructure 70% 30%
  14. 14. Infrastructure & Compute ScalingApplication Services Security CDN Backup DNS Database Building blocks for applications Storage Load BalancingDesigned and built for the cloud Workflow MonitoringAvailable at end of a web service call Networking Messaging
  15. 15. Storage & Archive AWS is used in a variety of ways… By managing time-consuming Brought on massive DynamoDB database administration tasks, capacity in an operationally short RDS allows SEGA to focus on period of time to handle demand business critical applications peaks during SuperBowlSaved months of development & Relies on Simple Workflow toarchitecture time and focused on orchestrate complex, heterogeneousapplication development instead scientific workflows by using Cloud Search
  16. 16. Business & technical drivers You might be able to:
  17. 17. Business & technical drivers You might be able to: Reduce costsRunning software on EC2 can bemore expensive than consuming functionality as a service
  18. 18. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability
  19. 19. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability Scale easilyServices scale as you grow and need them, without large investments in infrastructure
  20. 20. Business & technical drivers You might be able to: Reduce costs Improve reliabilityRunning software on EC2 can be Services built with inherent multi-more expensive than consuming AZ functionality improve functionality as a service application reliability Scale easily Re-focus energiesServices scale as you grow and Spend less time doing need them, without large undifferentiated heavy lifting and investments in infrastructure more time on your business
  21. 21. Q Instance Availability Zone Decoupled Reliableapplications on messageEC2 instances queue on instance Region
  22. 22. Q Q Q Instance Instance InstanceAvailability Zone Availability Zone Availability Zone Multiple Technical instances implementation for reliability Region
  23. 23. Q Q Q Instance Instance Instance Availability Zone Availability Zone Availability Zone Remove software Regionrunning on EC2
  24. 24. Simplified operations Availability Zone Availability Zone Availability ZoneReduced costs Amazon Simple Queue Service (SQS) Region Replace with a regional service
  25. 25. Queuing on AWS Setup & manage instances Install & configure queuing middlewareSet up persistent message storeImplement clustering across AZs for HA Implement queue monitoring systems Implement queuing in applications
  26. 26. Queuing on AWS Amazon SQS Setup & manage instances Create a queue Install & configure queuing HTTP PUT to place messages middleware HTTP GET (long polling) to pullSet up persistent message store messagesImplement clustering across AZs for HA Implement queue monitoring systems Implement queuing in applications
  27. 27. AWS application services
  28. 28. Databases Middleware Frameworks Analytics Relational ElastiCache Elastic Database CloudSearch MapReduce Service Simple Simple Queuing Email SimpleDB Service Service Simple Notification CloudWatch DynamoDB Service Simple Workflow
  29. 29. Databases Middleware Frameworks Analytics Relational ElastiCache Elastic Database CloudSearch MapReduce Service Simple Simple Queuing Email SimpleDB Service Service Simple Notification CloudWatch DynamoDB Service Simple Workflow
  30. 30. Amazon Relational Database Service
  31. 31. Source: Forrester Security planning License training Backup, recovery load and unload Script automationPerformance Installation, upgrade,and tuning patching, migration
  32. 32. Migration Backup and Schema design recovery Patching Query construction Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  33. 33. Migration Focus on Backup and these Schema design recovery things Patching Query construction Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  34. 34. Migration Backup and Schema design recovery Patching Instead of Query construction these Configuration Software upgradesFrequent server upgrades Storage upgrades Query optimization Hardware crash
  35. 35. Near zero administration Painless patching, automatic upgrades Cloudwatch monitoring, metric alarms One click. High Availability.
  36. 36. One click.High availability with Multi-AZ Automated deployment across multiple AZs Synchronous replication from master to replica Automatic fail-over; replica promoted to master Test fail-over
  37. 37. Push-button scale, high performance Scale storage from 5Gb to 1Tb of storage Scale instance from small to 4XL (better I/O) Add Read Replicas with asynchronous replication Add ElastiCache for performance
  38. 38. Relational Database Service Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations Feature Details Platform support Create MySQL, SQL Server and Oracle RDBMS Preconfigured Get started instantly with sensible default settings Automated patching Keep your database platform up to date automatically Backups Automatic backups and point in time recovery and full DB backups Backups Volumes can be snapshotted for point in time restore Failover Automated failover to slave hosts in event of a failure Replication Easily create read-replicas of your data and seamlessly replicate data across availability zones
  39. 39. RDBMS on AWS Setup & manage instances Install & configure database platform Configure backupsImplement master-slave for HA Implement read-replicas for performanceManage maintenance updates
  40. 40. RDBMS on AWS RDS Setup & manage instances Create RDS instance Install & configure database Select Multi-AZ platform Choose backup period Configure backups Choose maintenance windowsImplement master-slave for HA Implement read-replicas for performanceManage maintenance updates
  41. 41. Amazon Relational Database Service (Amazon RDS) databases stores forum threads, site content, and project configuration data.High availability Multi-AZ database deployment to handle live game metadata and user-generated content.Enterprise-grade fault tolerance for protecting customer data.By managing time-consuming database administration tasks, Amazon RDS allows SEGA to focus on business critical applications.
  42. 42. Amazon DynamoDB
  43. 43. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Scalability
  44. 44. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Reality: performance degrades with scale Scalability
  45. 45. Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Hardware provisioning Data sharding Data caching Reality: performance Cluster management degrades with scale Fault management Scalability
  46. 46. Dial it up DynamoDB Provisioned read/write performance per table Predictable high performance scaled via console or API
  47. 47. Low provisioned throughput Table Partition SSD RegionIllustrative diagram only
  48. 48. Low provisioned throughput Replica Table Replica Partition Partition Partition SSD SSD SSD Availability Zone Availability Zone Availability Zone RegionIllustrative diagram only
  49. 49. Low provisioned throughput Table Partition SSD RegionIllustrative diagram only
  50. 50. Increased provisioned throughput Table Table Table Table Table Partition Partition Partition Partition Partition SS SS SS SS SS D D D D D Table Table Table Table Table Partition Partition Partition Partition Partition SS SS SS SS SS D D D D D RegionIllustrative diagram only
  51. 51. High provisioned throughput Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition Table Table Table Table Table Table Table Table Table Table Partition Partition Partition Partition Partition Partition Partition Partition Partition Partition RegionIllustrative diagram only
  52. 52. DynamoDB Feature DetailsProvisioned throughput NoSQL Provisioned Dial up or down provisioned read/writedatabase throughput capacityFast, predictable performance Predictable Average single digit millisecond latenciesFully distributed, fault tolerant performance from SSD backed infrastructurearchitecture Strong consistency Be sure you are reading the most up to date values Fault tolerant Data replicated across availability zones Monitoring Integrated to Cloud Watch Secure Integrates with AWS Identity and Access Management (IAM) Elastic Integrates with Elastic MapReduce for MapReduce complex analytics on large datasets
  53. 53. NoSQL on AWSPerform capacity planning and determine nodes required Define PIOPS EBS volumes & instance types Setup & manage instancesInstall NoSQL database software Configure replica sets Implement auto-scaling Define tables Define key strategy
  54. 54. NoSQL on AWS DynamoDBPerform capacity planning and Define tables determine nodes required Define key strategy Define PIOPS EBS volumes & Choose performance level instance types Setup & manage instancesInstall NoSQL database software Configure replica sets Implement auto-scaling Define tables Define key strategy
  55. 55. “AWS gave us the flexibility to bring a massive amount of capacity online in a short period of DynamoDB: time and allowed us to do so in an operationally over 500,000 writes per straightforward way. second AWS is now Shazam’s cloud provider of choice,” Amazon EMR:more than 1 million writes Jason Titus, per second CTO
  56. 56. Amazon Simple Workflow
  57. 57. Simple Workflow 1 2 Task A 3 Task B (Auto-scaling)Flow framework for simplification ofcross system task coordination Task CLong running transaction state and taskdistribution
  58. 58. A typicalbusinessworkflow…Multiple steps Update Update € Ship SendMultiple decision Inventory $ or €? account order Emailpoints New Order Update $ account
  59. 59. A typicalbusinessworkflow… System A System C System System EMultiple steps Update Update € D Ship SendMultiple decision Inventory $ or €? account order EmailpointsHeterogeneous New Ordersystems Update $ account System B
  60. 60. A typicalbusiness Process logic Middlewareworkflow…State managed in System A System C System D System Eend systems Update Update € Ship Send Inventory $ or €? account order EmailProcess logicembedded in New OrderapplicationsComplex queuing,message ordering,de-duplication anddependencies State Update $Scaling and failover accountof tasks troublesome System B
  61. 61. Implemented in Workflow metadata & stateSimpleWorkflow… DeciderImplement a‘Decider’ with simple, Update Update € Ship Send Inventory $ or €? accountlinear decision logic order EmailProcesses/tasks in a New Orderworkflow become‘Workers’All state & metadatais handled by highly Workeravailable and durable Update $AWS ‘Workflow’ account
  62. 62. Amazon SimpleWorkflow Service (SWF)
  63. 63. DeciderGet Decision Task Return Result Amazon Simple Workflow Service (SWF)
  64. 64. Decider Get Decision Task Return Result Amazon Simple Workflow Service (SWF)Get Activity Task Return Result Worker
  65. 65. Workflows on AWS Setup & manage instances Install & configure workflow middlewareArchitect for high availability Implement workflows in proprietary/complicated languages Implement process audit
  66. 66. Workflows on AWS Simple Workflow Setup & manage instances Implement ‘decider’ in language Install & configure workflow of choice middleware Implement task ‘workers’ toArchitect for high availability consume work Implement workflows in proprietary/complicated languages Implement process audit
  67. 67. Amazon CloudSearch
  68. 68. select *from articleswhere contentlike ‘%cloud%’
  69. 69. 1 million hits a day? 5 million items?
  70. 70. Finding things is trickier than first appears
  71. 71. +
  72. 72. Anatomy of a search Results & Ranking Tokenization FieldedFacets
  73. 73. Endpointsdoc-mydomain.us-east-1.cloudsearch.amazonaws.com search-mydomain.us-east-1. cloudsearch.amazonaws.com
  74. 74. Testing Query String FacetsResults
  75. 75. Integration Domain http://search-toilets-in-oz-m6gko2xujhti47jv5w62eldyoy.us- east-1.cloudsearch.amazonaws.com/2011-02-01/search ?q=True+Blue &return-fields= address1 %2Caddressnote %2Cf__name %2Clastupdatedate %2Clatitude %2Clongitude Query String %2Cnotes %2Cpostcode %2Ctoiletdetails_id %2Ctoileturl Search Options %2Ctext_relevance
  76. 76. Integration Search Results { "rank":"-text_relevance", "match-expr":"(label True Blue)", "hits":{"found":1, "start":0, "hit":[{"id":"toiletdetails_csv_41", "data":{"address1":["51-59 Samwell Street"], "addressnote":["3 male and 3 female toilets"], "f__name":["Croydon True Blue Visitor"], "lastupdatedate":["2010-03-04Z"], "latitude":["-18.204249"], "longitude":["142.244534"], "notes":[], "postcode":["4871"], "text_relevance":["309"], "toiletdetails_id":["40”] }] }, "info”{ "rid":"ccd66a5219f938d295e4326391ce31e1359fb7bc306564", "time-ms":3, "cpu-time-ms":0} }
  77. 77. Scalable
  78. 78. Time: 1800hRequests <80% CPU Small Instance Partition 1 Copy 1 Time Elastic Search
  79. 79. Time: 2000hRequests >80% CPU Small Instance Partition 1 Copy 2 Small Instance Small Instance Partition 1 Partition 1 Copy 1 Copy 1 Time Elastic Search
  80. 80. Time: 2200hRequests >80% CPU Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  81. 81. Time: 0000hRequests Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 <30% CPU Copy 2 Copy 2 Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  82. 82. Cost savingsRequests Traditional required capacity Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  83. 83. Cost savingsRequests Traditional required capacity Small Instance Partition 1 Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Elastic Capacity Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  84. 84. Cost savingsRequests Traditional required capacity Small Instance Savings Partition 1 Savings Copy 3 Small Instance Small Instance Partition 1 Partition 1 Copy 2 Copy 2 Elastic Capacity Small Instance Small Instance Small Instance Small Instance Partition 1 Partition 1 Partition 1 Partition 1 Copy 1 Copy 1 Copy 1 Copy 1 Time Elastic Search
  85. 85. Cloud Search Search Instance Partition 1 Copy 1 Traffic Search Instance Partition 1 Request Copy 2 volume &complexity Search Instance Partition 1 Copy n
  86. 86. Data Document quantity & sizeCloudSearch Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy 1 Copy 1 Copy 1
  87. 87. Data Document quantity & size Cloud Search Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy 1 Copy 1 Copy 1 Traffic Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Request Copy 2 Copy 2 Copy 2 volume &complexity Search Instance Search Instance Search Instance Partition 1 Partition 2 Partition n Copy n Copy n Copy n
  88. 88. Search on AWSPerform capacity planning on size of indexes Setup & manage required number of instances Install & configure search software such as Solr across clusterManage cluster partitioning and size over time Implement monitoring
  89. 89. Search on AWS Cloud SearchPerform capacity planning on size Create search domain in console of indexes Upload documents Setup & manage required Retrieve results number of instances Install & configure search software such as Solr across clusterManage cluster partitioning and size over time Implement monitoring
  90. 90. Amazon Elastic MapReduce
  91. 91. 1 instance for 100 hours =100 instances for 1 hour
  92. 92. Small instance = $8
  93. 93. Elastic MapReduce Feature Details Scalable Use as many or as few compute instances running Hadoop as you want. Modify the number of instances while your job flow is runningIntegrated with Works seamlessly with S3 as origin other services and output. Integrates with DynamoDB Managed, elastic Hadoop clusterComprehensive Supports languages such as Hive and Pig for defining analytics, and allows Integrates with S3 & DynamoDB complex definitions in Cascading, Leverage Hive & Pig analytics scripts Java, Ruby, Perl, Python, PHP, R, or Integrates with instance types such as spot C++ Cost effective Works with Spot instance types Monitoring Monitor job flows from with the management console
  94. 94. But what is it?
  95. 95. A frameworkSplits data into piecesLets processing occur Gathers the results
  96. 96. Very large dataset (e.g TBs)
  97. 97. Lots of instances of ‘X’Very large dataset (e.g TBs)
  98. 98. Lots of instances of ‘X’Very large dataset (e.g TBs) Split the log into many small pieces
  99. 99. Process in an Lots of EMR cluster instances of ‘X’Very large dataset (e.g TBs) Split the log into many small pieces
  100. 100. Process in an Lots of EMR cluster instances of ‘X’Very large dataset (e.g TBs) Split the Aggregate log into the results many small from all pieces the nodes
  101. 101. Process in an Lots of EMR cluster instances of ‘X’Very large dataset Aggregate (e.g TBs) Split the Aggregate view of ‘X’ log into the results many small from all pieces the nodes
  102. 102. Very large dataset Aggregate (e.g TBs) Insight in a fraction of the time view of ‘X’
  103. 103. 1 instance for 100 hours =100 instances for 1 hour
  104. 104. Small instance = $8
  105. 105. 1 instance for 1,000 hours =1,000 instances for 1 hour
  106. 106. Small instance = $80
  107. 107. S3Input data
  108. 108. S3 Input dataCode Elastic MapReduce
  109. 109. S3 Input dataCode Elastic Name MapReduce node
  110. 110. S3 Input dataCode Elastic Name MapReduce node Elastic cluster
  111. 111. S3 Input dataCode Elastic Name MapReduce node HDFS Elastic cluster
  112. 112. S3 Input dataCode Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  113. 113. S3 Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  114. 114. S3Input data Output S3 + SimpleDB
  115. 115. Hadoop on AWSConfigure & manage EC2 instances Setup Namenode Setup Datanodes Create HDFS file system Setup extras (Hive, Pig etc) Prepare data Execute job
  116. 116. Hadoop on AWS Elastic MapReduceConfigure & manage EC2 instances Specify desired cluster size Setup Namenode Execute job Setup Datanodes Create HDFS file system Setup extras (Hive, Pig etc) Prepare data Execute job
  117. 117. Features powered by Amazon Elastic MapReduce: People Who Viewed this Also Viewed Review highlights Auto complete as you type on search Search spelling suggestions Top searches Ads200 Elastic MapReduce jobs per day Processing 3TB of data
  118. 118. Elastic MapReduce "With Amazon Elastic MapReduce, there was no upfront investment in hardware, no hardware procurement delay, and no need to hire additional operations staff. Because of the flexibility of the platform, our first new online advertising campaign experienced a 500% increase in return on ad spend from a similar campaign a year before.” Mark Taylor, Program Director
  119. 119. Where to go next
  120. 120. http://aws.typepad.comhttp://aws.amazon.com/whitepapers
  121. 121. Powerof60.com$100 of creditsThat’s enough to run anAmazon Elastic ComputeCloud (Amazon EC2)Standard Small Instancefor approximately 600hoursOr ~600 EC2 StandardSmall Instances for 1 hour
  122. 122. Summary
  123. 123. AWS is more than compute & storageDatabases, middleware and complex frameworks as servicesRemove the operational burden from running common software servicesApplication services gives access scalable, sophisticated software without the overheads
  124. 124. aws.amazon.com

×