This session gives an in-depth look at the current state of big data at AWS. Learn about the latest big data trends and industry use cases. We’ll focus on how other organizations are using the AWS big data platform to innovate and remain competitive. Met Office also joins us to offer an inside look on how they are using AWS to enable citizens, business, and governments to consume its data on demand. Speaker: Ben Snively, Principal Solutions Architect, Amazon Web Services & Jacob Tomlinson, Lead Engineer, Met Office Informatics Lab.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THE POWER OF BIG DATA
Ben Snively, Solutions Architect, Amazon Web Services
Jacob Tomlinson, Lead Engineer, Met Office Informatics Lab

2. Why Big Data?
Rapid Data Growth
1.2 ZB in 2015
44 ZB by 2020
180 ZB by 2025
Data Generation
• Speed of the data.
G
B
T
B
PB
ZB
E
B
Source:
IDC,
2015
IoT
Social Media
Enterprise
Systems
Data Variety
• 80 billion devices by 2025
• 500 million tweets daily
Volume Velocity Variety

3. There will be more words written on Twitter
than contained in all books ever printed.
Source: The Huffington Post, Is Twitter Bad For Language? Statistical Analysis Says No
(New Book from Christian Rudder, Dataclysm: Who We Are)In 2014…

4. Artificial
Intelligence
What is Big Data Powering on AWS?

5. The Modern Data Architecture
… in Public Sector

20. Storage & Streams
Catalogue & Search
Entitlements
API & UI
Attributes of a Modern
Data Architecture
Key Pillars of a
Data Lake
Key Components of a Successful Data Strategy

22. Putting it all together

23. One tool to
rule them all

24. Different Types of Services…
Virtualized Managed Serverless
You can easily
provision servers and
focus on OS and
above.
You focus higher in the
stack but still need to
consider servers, how
much CPU is needed, and
how much RAM.
AWS manages based the
customer configuration.
Build applications and services
without thinking of servers.
Don’t be concerned about
provisioning, scaling, and
maintaining servers for fault
tolerance and availability.
AWS does all of this for you.

25. Real-time Analytics
Collect, process, analyze, and visualize insights in real-
time with AWS.

26. Easy add On-Demand Analytics
Real-time
On-
Demand

27. Building a Data Strategy on AWS
Kinesis Firehose
1
2
3
4
5
6
Athena
Query Service
7
8
Glue
Batch
9
10
AWS
IoT
Glue ETL

28. Petabytes to Exabytes…
§ AWS Snowball
§ AWS Snowball Edge
§ AWS Snowmobile

29. Processing Data for Analytics
on your data lake

30. Processing & Analytics
Transactional &
RDBMS
DynamoDB
NoSQL DB Relational Database
Aurora
BI & Data Visualization
Kinesis Streams
& Firehose
Batch
EMR
Hadoop, Spark,
Presto
Redshift
Data Warehouse
Athena
Query Service
AWS Batch
Predictive
Real-time
AWS Lambda
Apache Storm
on EMR
Apache Flink
on EMR
Spark Streaming
on EMR
Elasticsearch
Service
Kinesis Analytics,
Kinesis Streams
EastiCache DAX

31. Amazon AI
“Inside AWS, we’re excited to lower the costs
and barriers to machine learning and AI so
organizations of all sizes can take advantage of
these advanced techniques”

32. Amazon AI

33. What does the customer say?
https://aws.amazon.com/solutions/case-studies/analytics/
https://aws.amazon.com/solutions/case-studies/big-data/

34. Empowering Smart Cities and IoT

35. Royal National Institute of Blind People creates and
distributes accessible information in the form of
synthesized content
Amazon Polly delivers
incredibly lifelike voices
which captivate and engage
our readers.
John Worsfold
Solutions Implementation Manager, RNIB
”
“ • RNIB delivers largest library of
audiobooks in the UK for nearly 2
million people with sight loss
• Naturalness of generated speech is
critical to captivate and engage readers
• No restrictions on speech
redistributions enables RNIB to create
and distribute accessible information in
a form of synthesized content
RNIB provides the largest library in the UK for people with sight loss

36. JustGiving Supports 24 Million Users on Charity Site Using AWS
• Needed a new platform to support general
operations and new analytics service
• Moved to AWS, using a wide range of services
• Can scale system faster in response to
unanticipated spikes in traffic
• Receives query results in seconds compared
to 30 minutes under old system
• Obtains deeper insights into billions of data
points, using information to deliver better
services
Using the new AWS tools, we can
extract much finer-grained data
points based on millions of
donations and billions of visits, and
then use that information to provide
a better platform for our visitors.
Richard Atkinson
Chief Information Officer, JustGiving
”
“
JustGiving is a major online platform that supports
charitable giving. The organization is based in London.

37. We have an analysis kit we run
every day, looking at data,
comparing patterns over
previous years’ information,
and in a matter of seconds, we
can tell if a student is likely to
succeed or fail. The results
have been phenomenal
Lige Hensley
CTO
Ivy Tech
”
“
Ivy Tech – Improving the classroom and beyond…
Predicting Student Success:
Identify behaviors of successful and unsuccessful students.
We can now predict, with over 80% accuracy, which students are
likely to fail a course within the first two weeks of a 16-week term.
Improvements across the University:
Struggling faculty can be identifies and addressed well before they
have an impact on students
Using the same behavior analysis techniques, we can flag
fraudulent activity far earlier than ever before.
Using Natural Language Understanding tools to analyze course
evaluations.

38. FINRA: Monitor and enforce trading regulations
FINRA handles approximately
75 billion market events every
day to build a holistic picture of
trading in the U.S. Hundreds of
surveillance algorithms against
massive amounts of data.
FINRA mission
§ Deter misconduct by enforcing the rules.
§ Detect and prevent wrongdoing in US markets
§ Discipline those who break the rules
Scale brings unique challenges
§ Market volumes are volatile and increasing
§ Exchanges are dynamically evolving
§ Regulatory rules are created and enhanced
§ New securities products are introduced
§ Market manipulators innovate

39. Fraud Detection
FINRA uses Amazon EMR and Amazon S3 to process up to 75 billion
trading events per day and securely store over 5 petabytes of data,
attaining savings of $10-20MM per year.

40. Met Office Uses AWS to Deliver Tailored Meteorological Data
The Met Office has been a widely respected
national weather service in the United Kingdom
for 160 years.
“We are using the AWS
Cloud to drive the mass-
market availability of
customizable weather
information.
• Needed the means to send weather data to device
users and third-party customers.
• Deployed Amazon ElastiCache to respond to peak
demands.
• Attracted more than half a million users with its app.
• Scaled data storage tenfold and reduced solution
costs by 50 percent.
• Enabled innovation of big data services in a
competitive landscape.
James Tomkins
Head of Enterprise IT Architecture
Met Office
”
“

41. Please welcome
Jacob Tomlinson
Met Office

42. Big Data in the
Met Office
Informatics Lab
Jacob Tomlinson
Met Office Informatics Lab
@_jacobtomlinson

45. Laziness

49. Iris

55. What does the Met Office do with this?

56. What can we do together?

57. Value for money

58. Spot pricing

59. Storytime

61. Packing workloads

62. Vertical scaling

63. Horizontal scaling

64. Spot fleet scaling

65. Statelessness

66. CC-BY-2.0 johnjack https://flic.kr/p/6fNsTY

67. External storage

68. Distributed queues

69. Conclusions

72. • Prescriptive guidance and rapidly deployable solutions to help
you store, analyze, and process big data on the AWS Cloud
• Derive Insights from IoT in Minutes using AWS IoT, Amazon
Kinesis Firehose, Amazon Athena, and Amazon QuickSight
• Deploying a Data Lake on AWS - March 2017 AWS Online Tech
Talks
• Harmonize, Search, and Analyze Loosely Coupled Datasets on
AWS with Glue, Athena and QuickSight
• From Data Lake to Data Warehouse: Enhancing Customer 360
with Amazon Redshift Spectrum
• Implement Continuous Integration and Delivery of Apache
Spark Applications using AWS
http://amzn.to/2vHIwBq
http://amzn.to/2i9gqZn
http://bit.ly/2qipA8h
http://amzn.to/2qpiFaK
http://amzn.to/2lpbc8p
Takeaways
https://aws.amazon.com/blogs/big-data/
https://aws.amazon.com/answers/big-data/
http://amzn.to/2gIJcj8

73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!