(Presented by Basho) This session will discuss the transformation of the most widely distributed cable TV network in the United States, building on one of the world's most visited digital properties, to create a world class Big Data platform. Architects, CTOs, CIOs, IT Director, and development managers will learn how to run highly scalable analytics workloads on Amazon EC2 and Amazon EMR for complex, real-time analysis of large data sets. All while decreasing time to results and increasing business agility. Bryson Koehler, EVP & CIO of The Weather Company, will discuss architecture, technology choices, performance results and business benefits realized as part of their use of AWS services to host an exciting set of weather.com solutions and generate new revenue streams. Weather impacts over 30% of the global GDP daily and is the source of vast amounts of data collection. The Weather Company is the leader in weather forecasting and is bringing the world's most accurate forecasting capabilities alive in a full suite of data APIs built fully on Infrastructure as a Service platforms, including AWS and next generation products like Basho Riak, Hadoop, and Dasein. This session will discuss how the application of these technologies help keep people safe and helps businesses plan and become more profitable, thanks to the latest intersection of consumer behavior and weather forecasting and reporting.

1. How the Weather Company Monetizes
Weather, the Original Big Data Problem
Sathish Gaddipati, VP - Data Management, The Weather Company
Raja Selvaraj, Manager - Data and System Engineering, The Weather Company
November 14, 2013

2. B2B2C
DIRECT
CONSUMER
ONLINE & APPS
B2B
TV
GOV’T
PRODUCTS
WHERE THE WORLD GETS ITS WEATHER
61.6M
#1 MOST
DISTRIBUTED Cable
Network*
*Nielsen, October 2012.
Unduplicated Monthly Uniques
(weather.com + wunderground)*
72% visit 2x or more Daily**
*comScore, Oct 2012 **Omniture, July 2012
100M+ App Downloads*
#2 All Time Most Downloaded iPad App**
47.2M Unduplicated Monthly Uniques***
*Various app stores, TWC only,
**App Store Rewind, 2011
***Omniture Site Catalyst, August 2012. TWC Mobile & wunderground

3. Did You Know? TWC Weather Services for
Businesses
World’s Leading Weather Media Company
•
- Power 500+ broadcast channels world-wide
World’s Leading Aviation Weather Service
•
- Support 50,000+ flights daily with embedded
forecasters, systems and tools
•
- 85% of major US airlines. 30% of Top 100
global.
World’s Leading Energy Forecaster
•
- Over 300 power trading clients worldwide
•
- Hourly, daily, and seasonal trading
Fastest Growing Insurance Weather Service
•
- 7 of the top 10 P&C insurance customers
•
- Connecting Marketing, Underwriting, & Claims
with systems & tools

4. WEATHER IS CHANGING
• 2012 was the hottest in recorded
history for the US
• Last time January was colder than
normal was 1983
• In July 2012 61% of the country was in
drought conditions
• 9.2MM acres charred in 2012; third
highest in 13 years
Source: Time.com; Climate Central

5. • In the next 12 months we will:
•
•
•
•
•
•
Increase our data points by 1,400%
Increase our data velocity 300%
Increase our number of deployments
by 2,000%
Increase our number of TV feeds
from 1 to 7
Increase the amount of video content
by 100%
Increase the amount of social
content by 500%
Huge projects required to support these
business needs
Weather changes…
So must IT
•
•
•
•
•
•
•
•
•
13 data centers to 4
75% of workloads to IaaS by
building a cloud abstraction
toolkit
Move to agile development
teams
New TV Transmission System
(retire 2, build 1)
575 APIs and 20 data
platforms to 1 platform
2500+ new localization servers
New HR, Email, Finance,
Inventory, Expense, etc.
New Ad Stack
New Video Asset
Management
5

6. The Weather Company’s Core
Science
Technology
Data

7. Weather Data Services - Use Cases
Max. Premium Rate
Min. Claims
Insurance
WDS
Wind
Energy
Retail
 Tornado and flood forecasts
 Weather warnings
 Historical weather trends
PPC
Maintenance
 Wind speed forecast
 Historical wind speeds
Inventory management
Distribution
 Precipitation forecast
 Temperature forecast
 Extreme weather forecast

8. Weather Data Service - Use Cases
Weather data
Weather content
Media
WDS
Energy
Exchange
Ad
Targeting
 Hourly forecasts
 Daily forecast
 Current conditions
Commodity bidding
Demand forecasting
 Temperature forecast
 Historical trends
Impression serving
Improved Ad. Exchange
 Real-time conditions
 Forecasts
 Customer location

9. Weather Data Service - Use Cases
Weather data
Weather content
Mobile
Apps
WDS
Gov
Hospitality
 Hourly forecasts
 Daily forecast
 Current conditions
Local weather
National weather
 Weather forecast
 Historical forecast
 Watches and warnings
Room rates
Revenue optimization
 Current conditions
 Forecasts
 Airline delays/
cancellations

10. Weather Data Service - Use Cases
O&O
Products
WDS
Weather
Analytics
Airlines
Weather data
Weather content
 Hourly forecasts
 Daily forecast
 Current conditions
 Historical trends
Business impact
Consumer behavior
 Historical data
 Consumer spend vs.
weather conditions
Optimal Routes
Flight Schedules
 Air turbulence and
wind speeds
 Weather forecasts
 Current conditions

11. ACCURACY THROUGH SUPERIOR DATA
Industry Best
Forecast Modeling
Proprietary
Radar
Algorithms
Largest collection of
worldwide forecast data
PhD
State-of-theScience Forecast
Technologies
220+ Full-time
Meteorologists
Proprietary
weather data
analytics
Global Lightning
Detection Network
Network of 30,000+
Weather Sensors
APIs
Robust
weather data
APIs
Number 1 in forecasting accuracy per ForecastWatch in the US and Europe for
the last 6 months… 5 points better than NWS Forecast
11

12. We defined an
architecture that
could support the
needs of all of our
divisions.
•
•
•
•
API interfaces
IaaS abstraction
Shardable data
Global scale

13. Top 10 Objectives of Weather Data Services Platform
1. Reduce time to deploy and market new data sets
2. Reduce operating cost of data services
3. Centralize data services across the company
4. Provide visibility of data access
5. Provides data governance process and framework
6. Serves world’s best weather forecast across all products
7. Low latency, highly scalable APIs
8. Secured access to data
9. Centralized and scalable architecture
10. Consistent “rich” content across platforms

14. Weather Data Services Platform

15. Top 10 Computing Challenges
1. Distribute thousands of gridded binary files to multiple locations
across globe within 5 minutes
2. Serve more than billion data services API requests/day
3. Metering and authentication of API calls with low latency
4. Process multiple TBs of data every day
5. Ensure business continuity
6. Leverage data caching
7. Store petabytes of historical data
8. Meshing weather data with consumer behavior and derive analytics
9. Build flexible data ingestion platform to manage 100s of data feeds
from external parties
10.Maintain above systems within OPEX budget

16. DATA STORES – WHAT WE WANTED
Performance
Availability
Manageability
Replication
Data
Store
Capacity

17. DATA STORES – WHAT WE USE
Raster/Image data
Historical weather data
Analytics platform
Severe weather data

18. RIAK REPLICATION @AWS
Amazon – Region 1
Amazon – Region 2
VPC
Subnet
Public
Subnet
AZ2
AZ1
AZ2
Private Subnet
AZ1
Public
Private Subnet
VPC

19. NEXT GENERATION – WEATHER FORECASTING SYSTEMS

20. Weather’s Big Data Analytics Platform for Enterprises and Government
TWC’s Redshift Cluster now
contains:
WeatherFX
CreativeFX
Partner
EDW
Platform
Amazon S3
AudienceFX
…
App N
TWC
Private
Data
Center
EDW
Amazon
Amazon
Redshift
Redshift
600+ million records of weather
forecast data since March 2005
700+ million records of current
conditions observation data since
April 2004
11+ billion records of Int. Civil
Aviation Org. (ICAO) observation
data Since 1970
TBs of storm event data from
from 1950 - 2012
Weather sentiment survey data
~1 Billion records
Bulk loaded in 36
minutes!

21. Contact Us
Sathish Gaddipati –
sathish.gaddipati@weather.com
Raja Selvaraj –
raja.selvaraj@weather.com

22. Please give us your feedback on this
presentation
BDT208
As a thank you, we will select prize
winners daily for completed surveys!