In this conversation, AWS and government thought leaders will discuss ways to encourage public private partnership to solve societal problems thru open data. Ariel Gold, Program Manager, AWS and Tsengdar Lee of NASA will shares insight on NASA NEX.
Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014
1. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Big Open Data:
Transformation through Public Data Sets
Ariel Gold
arigold@amazon.com
2. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Agenda
• Open Data Overview (Ariel Gold, AWS)
• NASA NEX (Dr. Tsengdar Lee, NASA)
• Data Distribution Model (Ariel Gold, AWS)
• Q&A
4. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
What is Open Data?
Data released to the public in ways that make it easy to
discover, access, and use.
5. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Why are governments making their data open?
Five main goals seen across different policies and
programs.
6. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Reduced backlog in manual submissions while improving
accuracy and reducing costs
Enabled ingestion of new open data sources to improve
mission delivery
Shifted from quarterly reports to developer-focused APIs, raw
data, and documentation
7. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Release Definition & Policy Launch Open Data Catalog
Do Things w/ Open Data
Release New (Big) Open Data
Repeat w/ Increased Focus
Impact
Gather Customer Feedback
Open Data Adoption Curve
8. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
How do you drive impact with (big) (open) data?
Build communities and tools around sustainable open
data sets.
9. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
OpenNEX as an Open
Data Analytical Service
Tsengdar Lee, Ph.D.
tsengdar.j.lee@nasa.gov
10. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
NASA EARTH EXCHANGE (NEX)
Virtual collaborative-for-global-change science
COLLABORATION
Web & System
COMPUTING
NASA HECC
Pleiades 180k+ cores
Endeavour 1.5k+ cores, 6TB
D-wave Quantum 512 Qbit
DATA REPOSITORY
800 TB, 126 PB
network links
11. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
NEX Provides a Complete Work Environment
“Science as a Service”
COLLABORATION
Over 400 Members
CENTRALIZED DATA
REPOSITORY
Over 1300 TB of Data
COMPUTING
Scalable
Diverse
Secure/Reliable
KNOWLEDGE
Workflows
Machine Images
Re-useable software
12. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
• 33 CMIP5 models available
• All 4 RCPs (2.6, 4.5, 6.0, 8.5) and
historical runs
• 30 arc-second (800m) spatial resolution,
monthly time-step, 1950-2099
• Max/min temperature and precipitation
• Statistical downscaling (bias-corrected
spatial disaggregation; Maurer et al.,
2007)
Thrasher et al., 2013, Eos
Creating NEX-DCP30 Downscaled Climate
Projections at 30 Arc-Second Resolution for the
National Climate Assessment
13. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
• Sam Goward, PI
• Detecting 30m
resolution forest
disturbance
annually
• National runs
complete
• Data distribution
being worked
North American Forest Dynamics (NAFD)
13
14. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
April 2010
October 2010
• David Roy, PI
• Completed processing 140,000 scenes for first year of global Landsat
(monthly and annual)
• Public release later in June
Web-Enabled Landsat Data (WELD)
14
15. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
California Drought Monitoring
at 250m Scale
Changes in High Latitude
Ecosystems - Greening
16. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
A tale of two droughts/Amazon 2005 & 2010
TRMM MODIS
Workflows Improving Research Productivity
Faster (24 months vs. 6 months), consistent
(same analytical methods, quality flags) and
reproducibleSamantha et al., GRL, 2011
Xu et al., GRL, 2011
17. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
NEX and AWS
Sandbox on Public Cloud
- Prototyping
- Development
NAS HECC
- Large-Scale processing
- NCA Example
COLLABORATION
(more public members)
COMPUTING
HECC + AWS1
DATA REPOSITORY
(AWS OpenData, with
“Market driven” Cache)
AWS
18. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps Challenge
46 countries participated
735 virtual participants
69 virtual team projects
Space Apps kicked off Thursday night in Doha, Qatar at midnight and ended in
Seattle, WA at 6 p.m. Pacific—76 hours of around-the-world hacking on NASA
data.
19. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps:
Most Popular Challenges
Earth Watch: Where on Earth (44 projects), Climate & Neighborhood
(22), Cool It (21)
Robotics: Exomars Rover is My Robot (42)
Asteroids: Asteroid Prospector (38)
Space Tech: Space Wearables (25), Alert-Alert (24)
Space Flight: Growing Food for A Martian Table (22), SpaceT (20)
20. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps:
OpenNEX Data Challenges
Climate and My Neighborhood: 22 projects
Climate Alert: People’s Choice Local Award Winner
Climatehood: Best Use of Data Local Award Winner
Ways to Die: Most Inspiring Local Award Winner
NASA-NEX-Climate-Changes: People’s Choice Local Award Winner
21. 21
OpenNEX
Workshops and Challenges
7/1 – 11/15/2014
• Virtual Workshop teaches how to use OpenNEX
AMIs and Public Data sets on AWS.
• Collaborate within arms length
- Users work with vm templates including
prepackaged tools, they instantiate, extend
and own accordingly.
- Core data is read only public data sets on S3
• Bring citizen scientists together around data,
services and knowledge
• Extract more value out of assets, i.e. data and
knowledge, via the group
• http://nex.nasa.gov/OpenNEX
22. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Thank You
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Thank You
Tsengdar Lee, Ph.D.
tsengdar.j.lee@nasa.gov
23. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
How do you drive impact with (big) (open) data?
Build communities and tools around sustainable open
data sets.
24. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
National Agriculture Imagery Program
25. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Before
• Pay $5k
• Wait weeks to get data on disks
• Technical and GIS expertise to
make data usable
After
• No or minimal cost
• Get access immediately
• No technical or GIS expertise to
start using
Significantly increase openness of a 48TB aerial imagery data set at a low
and sustainable cost ($1,600/month)
National Agriculture Imagery Program
26. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Managing the Cost of Data Distribution
• Amazon Simple Storage
Service (S3) “requestor pays”
• Eliminate variable costs
• Multiple configuration options
depending on your use case
27. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Insert NAIP Demo Slides (or link to live pages)
28. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Q & A
For more information on AWS and open data, visit:
http://aws.amazon.com/government-education/open-data/
29. AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium
Washington, DC | June 24, 2014 - June 26, 2014
Thank You
Ariel Gold
arigold@amazon.com