Más contenido relacionado La actualidad más candente (20) Similar a AWS re:Invent 2016: Relational and NoSQL Databases on AWS: NBC, MarkLogic, and FileMaker Perspectives on Data Management for Enterprise & Consumer Apps (DAT205) (20) Más de Amazon Web Services (20) AWS re:Invent 2016: Relational and NoSQL Databases on AWS: NBC, MarkLogic, and FileMaker Perspectives on Data Management for Enterprise & Consumer Apps (DAT205)1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
11/29/16
Jorge Garzon, Software Development Manager, AWS Marketplace
Aaron Rosenbaum, Vice President, Product Strategy, MarkLogic
Anand Vaghela, Senior Product Manager, FileMaker
November 2016
DAT205
Relational & NoSQL Databases on AWS
NBC, MarkLogic, and FileMaker Perspectives
on Data Management for Enterprise & Consumer Apps
2. Architecture and
configuration decisions
that drove success with
a large automotive
company
What to Expect from the Session
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
01
Database and data
management from
AWS and Marketplace
Scaling, architecture and
operational decisions that
drove NBC success
02
03 0
3. AWS Marketplace
Discover, procure, deploy, and manage software in the cloud
• Easily find +300 data management solutions
• Ready-to-run on AWS
• Quick deployment
• AWS Marketplace cluster & AWS resources*
• Pay for what you use
• Software costs integrated on AWS bill
• AWS Cloud compliance, shared security
responsibilities with software vendors
https://aws.amazon.com/mp/database/
*Selected vendors
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. Operate your own Database on AWS
IF YOU NEED CONSIDER USING
A relational database services
with minimal administration
Amazon RDS, a managed relational database service that offers a choice of
MySQL, Oracle, SQL Server, PostgresSQL, or Amazon Aurora database
engines, scale compute & storage. Multi-AZ availability, Read, Replace, and
more.
A fast highly scalable
nonrelational database services
Amazon DynamoDB, a managed NoSQL database that offers extremely fast
performance, seamless scalability and reliability, low cost and more.
An easy-to-operate
in-memory cache
Amazon ElastiCache, a managed in-memory cache cloud service that offers a
choice of two in-memory caching engines, Memcached and Redis.
A fast, petabyte-scale
data warehouse
Amazon Redshift, a managed data warehousing service that makes it simple
and cost-effective to efficiently analyze all of your data using your existing
business intelligence tools.
A database you can
manage on your own
Your choice of relational AMIs on Amazon EC2 and EBS that provide scale
compute & storage, complete control over instances, and more.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5. AWS Marketplace
Database sub-categories and data related categories
A1
Operational
Data Stories
A2
Data
Integration
A3
Data
Warehousing
A4
Database
performance
optimization
tools
A6
Data lake and
analytics
environment
B1
Business Intelligence
& Big Data
• Data Analysis & Visualization
• Advanced analytics and machine
learning
B2
Application
Development
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A5
Application and
reporting data
stories
9. • Semantic search to
power an intelligent
recommendation engine
• Massive scalability and
elasticity to support 40th
anniversary of SNL
• Lower cost and ease-of-
delivery via MarkLogic
and AWS Marketplace
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NBCUniversal – SNL40 App
40 Years of Comedy Delivered on Saturday Night Live App
10. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NBCUniversal – SNL40 App
Smart Content: Enriched and targeted
content constantly tuned recommendations
Season
34
Episode 4
Anne Hathaway
& Killers
Date
10/4/08
Character
Maharelle Sister
Era
Talent
Kristen
Wiig
Segment
The Lawrence
Welk Show
Characteristic
Tiny hands
Includes
Part of
Part of
Aired
on
Acted
in
Played
Has
Shows
Acted
in
Segment
The Lawrence Welk
Show
Talent
Kristen Wiig
12. MarkLogic on AWS Marketplace
LICENCE TYPE SINGLE-INSTANCE CLUSTER
Developer HVM Yes Yes
Essential
Enterprise
HVM Yes Yes
• http://developer.marklogic.com/products/aws#clusters
• http://www.marklogic.com/resources/marklogic-aws/
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
All
Regions
Annual
or Per
Hour
13. Traditional Modern
DATABASE, SEARCH, APP SERVICES
HTML / CSS / JAVASCRIPT
VIEW
GENERATION
WEB
SERVICES
DATABASE SEARCH
HTML / CSS / JAVASCRIPT
VIEW
GENERATION
WEB
SERVICES
BUSINESS LOGIC
OBJECT-RELATIONAL MAPPING
Database
Middle Tier
Front End
14. MarkLogic Scale Out Architecture
• Non-disruptive scale operations
• Rebalancing of content to keep
performance optimal
• REST APIs to expand and
contract clusters
• On-premises or in the cloud
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADD NODES TO ACCOMMODATE GROWTH
Rebalances data
15. Scale vertically
• Use different instance
types (more CPUs,
more memory)
Scale horizontally
• Add evaluator nodes
• Add data nodes
• Add replica clusters
Scaling Patterns for MarkLogic Clusters
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
16. • If your application uses CPU cycles
for “eval” activities
• Snippetting/highlighting
• Semantic queries
• They are simpler to add
• There is no data
• Join the cluster, ready to go
• Consider a separate MarkLogic group
• Suitable for dynamic scaling
Adding E-nodes
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
17. • Add “new rings”
• Example: 3 nodes at a time
• Spread replicas within them
• Allow re-balancer to populate new
forests
• Consider tiered storage APIs
• Retire forests when scaling back
Adding D-nodes
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
18. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adding
Foreign
Clusters
19. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HA
Proxy
MarkLogic SNL App Architecture
r3.2xlarge
GP2 EBS
Elastic
Load
Balancer
AWS Availability Zone 1 AWS Availability Zone 2 AWS Availability Zone 3
Elastic
Load
Balancer
VPC
Router
Node.js
Region-wide
Cluster
DNS CDN
Rights Management
ABR Steams
User
Devices
HA
ProxyNode.js
HA
ProxyNode.js
20. NBC Universal Scaling Strategy
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Latency > 4 seconds? Add E-Nodes
Writes > 10 seconds? Route to SQS
CPU over 70%? Add E-Nodes
CPU at 100%? Add D-Nodes
AZ Unavailable, MarkLogic Failover
21. Results
We've done a
number of popular
products in the
past, but this one is
without any doubt
and by any metric
a breakaway
success.”
Michael Martin, SVP,
NBC Entertainment
Digital -
More than
2 million
downloads
2M+
200M
200 million
streamed
videos
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“
22. Focus of Planning
Architecture:
• Fenced scope – internal contracts
• Scalable services
• Metrics + monitoring
Run Book:
• Team roles + responsibilities
• Scaling procedures
• Failure responses
• Escalation paths
Best Practices
Lessons Learned
• Scale services, don’t flatten experience
• No failure “cliffs”, degrade smoothly
• Contain unknown user behavior traffic
risks through scope control.
• Scale-out much easier to orchestrate than
scale-up.
• Single region sufficient resiliency/latency
for US-only app.
• 3X your upper traffic prediction – a
successful launch will generate more
traffic
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
24. Automaker Journey to the Cloud
• Participated in FileMaker’s “Early Testing” program
• 2-weeks after launch of FileMaker Cloud, successfully deployed 1st
mission critical custom app running on EC2 instance
• Majority of current install-base is on-premises server, now opportunity to
use more FileMaker Cloud based on business needs
• New project underway utilizing FileMaker Cloud running on AWS, based
on success of previous project
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
25. Challenge: FileMaker Server (Windows) on AWS
FLMK
Database
Engine
xDBC
ESS
SASE
Admin Server
WebDirect
FileMaker
Server
(Windows)
Server
Configuration
FLMK database & files &
remote container data
FLMK WebDirect
xDBC Client
FLMK Pro
FLMK Go
Universal Admin
Console
FileMaker Server
Running Windows FileMaker Server on AWS:
Deployment:
• Windows EC2 compute
• Set up security group settings
• Install & configure FileMaker Server
Manageability:
• Software updates
• EC2 upgrade/downgrade
• EBS upgrade
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
26. Solution: FileMaker Cloud
Deployment:
• Simplified using AWS
CloudFormation
• Few clicks and fully
deployed and configured
• Simple Email Service (SES)
Separate OS, FileMaker,
Config, & data
• Allows easier scalability of
EC2 and storage
FileMaker Cloud
FLMK WebDirect
xDBC Client
FLMK Pro
FLMK Go
Cloud Admin
Console
Admin Server
FileMaker
Cloud Server
(Linux)
FLMK Cloud
Robot
Customer database, files &
remote container data
OS & FileMaker
Configuration
AWS Cloud StorageAWS Cloud Storage
xDBC
ESS
SASE
WebDirect
FLMK
Database
Engine
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
27. Overcoming Challenges
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Problem: 90-days to deploy a
Server
• Problem: FileMaker WebDirect
is resource hungry, requires
upgrading existing hardware
• Problem: integration with
FileMaker platform
• Now: deployed in 20-mins
running on AWS resulting to
30000% productivity gain
• Now: ability with single-click to
upgrade/downgrade EC2 to
meet business needs
• Now: BYOL allowed use of
existing FileMaker platform and
FileMaker Cloud
• Using AWS CloudFormation • Using Amazon EC2 scalability • Using AWS Marketplace
29. • Store data in volumes of a provisioned size,
similar to a local disk drive on your physical
server
• Leverage Amazon SES instead of system
administrator setting up independent email
service
• Future to use Amazon S3 for performance,
price, and flexibility
Lessons Learned & Best Practices
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon SESAmazon Elastic
Block Storage (EBS)
30. Summary
• Use AWS Marketplace to accelerate the
evaluation, deployment and launch of
data management and analytics
projects.
• Understand your storage, read/write
throughput requirements
• Follow AWS best practices to architect
and develop your services
• Test your application at scale
• Review and implement a Disaster
Recovery plan
• Enable monitoring using Amazon
CloudWatch
https://aws.amazon.com/mp/database/
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.