Video can be found here: https://vimeo.com/63300016

1. DATA AS A SERVICE
How Web APIs and Data-Centric
Tools Power the Materials Project
Shreyas Cholia (scholia@lbl.gov)
Dan Gunter (dkgunter@lbl.gov)
Lawrence Berkeley National Laboratory
PyData 2013

2. Outline
• Data driven science
• Materials Project Overview
• Open data and APIs
• Dropping APIs on your data
• Things to think about in your API
• Writing libraries for code AND data (pymatgen REST
interface)
• Science stories to back this up
• Ipython notebook demo

3. About Us
• Dan and Shreyas are Computer Scientists/Engineers at
Berkeley Lab.
• We work with science teams to help build software and
computing infrastructure that facilitates awesome
SCIENCE

4. Science
• Science is now a collaborative effort
• Large teams of people
• Lots of computational power

5. The Fourth Paradigm

6. Big Data
Science is
increasingly data-
driven
Computational
cycles are cheap
Take an –omics
approach to
science
Compute all
interesting things
first, ask questions
later

7. The –omics approach
• Instead of trying to derive a solution and compute the
results, just compute the space of all possibilities and look
for the optimal result in there.
• OK – so we are generating more data than we know what
to do with but that is ok
• (and might be a topic for another talk …)

8. An open science initiative that makes available
a huge database of computed materials
properties for all materials researchers.
The Materials Project
Wordcloud showing
frequencies of elements
in Materials Project's
database
..except Oxygen, which appears
12,751 times (3.5x as much as the
next most frequent, Phosphorus)

9. The Materials Project http://materialsproject.org/

10. 18 years
from creation
to commercial
manufacture!
Teflon
Titanium
Velcro
Polycarbonate
GaAs
Diamond-like Thin
Films
Materials Data from: Eagar, T.; King, M. Technology Review (00401692) 1995, 98, 42.
invented
1960 19701950
"Need for speed" in new materials
Lithium ion
S. Whittingham
Sony
1980 1990 2000

11. Materials have strategic importance
Sept 7, 2010
Japan arrests
Chinese boat captain
after collision in
disputed waters
China blocks
shipments of Rare
Earth Metals to
Japan
Sept 22, 2010
Japan releases
captain
Sept 24, 2010
Japan invests in induction motors… coincidence?
“Toyota Readying Motors That Don’t Use Rare Earths…”
Jan 14, 2011 1:50 PM PT
Content for this slide courtesy Gerbrand Ceder, MIT & Kristin Persson, LBNL
2010 "Senkaku Boat Collision Incident"

12. Solution: Computation
Many materials properties can be computed
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Voltage
(V)
computed experimental
literature
stage I+II
Stage
II
Stage I
stage III
+II
+ =
ΔH = [ E (X) + E (Y) ] –
E(XY)
Photovoltaics, Thermoelectrics,
Energy Storage, Hydrogen,
Catalysts, Magnets….

13. Infrastructure
Submitted
Materials
Materials
Data
Materials Properties
Supercomputers
• Over 10 million CPU hours of
calculations in < 6 months
• Over 40,000 successful VASP
runs
(30,000+ materials)
• Generalizable to other high-
throughput codes/analyses
Calculation
Workflows
Supercomputers Codes to run
(in sequence)
Atomic positions

14. 0
10000
20000
30000
40000
50000
60000
Jul 2011 Oct 2011 Jan 2012 Apr 2012 Jul 2012 Oct 2012
Date
Numberofruns
state
Failed
Successful
Computation
• Run VASP on NERSC
supercomputing
resources
• Use Fireworks to
manage large groups
of runs
• Results in … data for
MP
Just sit back and
enjoy the
automation..
Total

15. Data Islands
• Data is still heavily silo-ed and inaccessible.
• Data sits on a machine somewhere, and you give people
local ssh/DB accounts to access it.
• Good luck combining multiple datasets
• Does not scale!
• This is 2013 – we can do better!

16. Sharing (your data) is important!
• The Most Important Scientific Result Published in the
Last Year
J.M. Wicherts, M. Bakker, and D. Molenaar:
Willingness to Share Research Data Is Related to the
Strength of the Evidence and the Quality of Reporting of
Statistical Results
PLoS ONE, 6(11): e26828, 2011, doi:10.1371/journal.pone.0026828.
Content for this slide courtesy Greg Wilson, Software Carpentry

17. Data Sharing
• Open access to data through programmatic interfaces
• Sub-select the data on demand rather than pulling down
the entire dataset
• Use your own local tools with centrally managed data
• Everyone sees the same data – better collaboration

18. Web portal
• Materials data stored in a Mongo DB
• http://materialsproject.org web portal makes materials data
easily accessible
• Materials Explorer
• Phase Diagrams
• Crystal Toolkit
• Battery Explorer
• Reaction Calculator
• Structure Predictor
• Focus on a highly functional and usable website to query
materials data. (We heart Django!)
• Additionally we distribute the tools used to compute and
analyze the data as an open source library – pymatgen

19. API access
• But we quickly found that scientists wanted programmatic
access to data
• eg. Give me property X for all materials with Li and O so
that I can pass it through my own codes
• Lesson – make your data available through an API and
people will start to do amazing things

20. Why Web APIs?
• Big push towards HTTP APIs across the web.
• Web APIs provide programmatic access to data and
resources to developers over the web
• Access to data as well-defined objects allows users to
develop their own custom applications and code
Enables a thriving COMMUNITY built around data.

21. What is The Materials API?
An open platform for
accessing Materials
Project data over the
web.
Flexible and scalable to
cater to large number
of collaborators, with
different access
privileges.
Simple to use and code
agnostic.

22. HTTP API design
https://www.materialsproject.org/rest/v1/materials/Fe2O3/vasp/energy
Preamble URL
Unique Identifier.
Eg. a formula
(Fe2O3), id (1234)
or chemical
system (Li-Fe-O)
Data type
(vasp,
exp, etc.)
Propert
y
Materials API maps URLs to data
objects

23. Access via an API key
• To maintain privileged access, each user has an
associated API key (with certain defined access
privileges).
• To get your key, login to materialsproject.org and go to
www.materialsproject.org/profile
• All MP https requests must supply API key as:
• A x-api-key header, e.g., {‘X-API-KEY’: ‘MYKEY’}, or
• As a GET or POST variable, e.g., {‘API_KEY’: ‘MYKEY’}

24. Sample JSON output
GET https://www.materialsproject.org/rest/v1/materials/Fe2O3/vasp/energy
{
"created_at": "2013-03-17T09:14:58.158081",
"valid_response": true,
"version": {
"pymatgen": "2.5.4",
"db": "2013.02.25",
"rest": "1.0"
},
"response": [{
"energy": -132.33005625,
"material_id": 542309
}, {
"energy": -66.62512425,
"material_id": 24972
}],
"copyright": "Copyright 2012, The Materials Project"
}
Just the energy and the
id of the material

25. Getting started – Hello World API
> pip install flask

26. Our dirty little secret
• It involves a certain language that ends with “uby” that we
don’t like to talk about in these parts
• Version 0.0.0 was of the Materials Project was coded in
Sinatra
• Sinatra is a microframework much like Flask
• But it proves that this approach is viable and can be the
onramp to more amazing things.

27. Un-considerations
• Don’t worry too much about pure REST
• Initially just think of how URLs and verbs can map to functions
• Don’t worry too much about data formats
• JSON is easy and a great place to start
• Feel free to avoid XML unless you really need it

28. Our Stack
• Apache + mod_wsgi
• Django
• pymatgen
• pymongo + Mongo DB

29. pymatgen
• The open source python library that powers the Materials
Project.
• Defines core Python objects for materials data representation.
• Provides a well-tested set of structure and thermodynamic analysis
tools relevant to many applications.
• Establishes an open platform for researchers to collaboratively
develop sophisticated analyses of materials data obtained both
from first principles calculations and experiments.

30. Integration with pymatgen
The Materials API
Powerful Materials
Analytics Tool

31. Where we’re going with this
• Libraries that integrate data with computation!
• The scientific python ecosystem has a ton of data analysis
tools and libraries
• Just starting to think about baking in datasets directly into
these tools
• Pymatgen allows you to access core MP data directly
from the library

32. Compute + data
pymatgen has hooks into the materials data so you can do
stuff like this:
entries = api.get_entries_in_chemsys([’Li', ‘Fe', 'O'])
But it also has computational tools that you can then use to
act on the data
pd = PhaseDiagram(entries)

33. Blurring the lines
• Yes – we are blurring the lines between compute and data
• But this is not a new idea
• Think of all the tools built around commercial APIs
• Twitter, Netflix etc. - python clients built around the API

34. Write First Class Science Functions
• Web APIs are extremely useful, but ultimately you want to
encapsulate core science functionality as python functions
so that scientists aren’t worrying about things like
How do I set the
X-API-KEY header?

35. Sample use cases
• Screening for CO2 sorbents (with Clare Grey)
• Using the Materials API (MAPI) + pymatgen to calculate reaction
energies of thousands of oxides with CO2.
• Calculation of XAFS, XANES and other spectra for
clusters of atoms (with Alan Dozier)
• Alan wrote a io add-on to pymatgen for FEFF input/output.
• Uses MAPI + pymatgen to extract structures.
• Defects (with Maciej Haranczyk)
• Uses MAPI + pymatgen to pull structures to perform Voronoi
analysis to find possible interstitial sites.

36. Ipython Notebook Examples
• http://nbviewer.ipython.org/5199610
• http://nbviewer.ipython.org/5022735

37. from pymatgen.matproj.rest import MPRester
#This initializes the REST adaptor. Put your own API key in.
a = MPRester("YOUR_API_KEY")
#This gives you the Structure corresponding to material id 2254 in
the Materials Project.
structure = a.get_structure_by_material_id(2254)
#Entries are the basic unit for thermodynamic and other analyses
in pymatgen.
#This gets all entries belonging to the Ca-C-O system.
entries = a.get_entries_in_chemsys(['Ca', 'C', 'O'])
#With entries, you can do many sophisticated analyses,
#like creating phase diagrams.
pd = PhaseDiagram(entries)
plotter = PDPlotter(pd)
plotter.show()
Materials API + pymatgen example

38. Sandboxes
• A virtual private dataset
• Useful for
• Everyone as a sort of "scratch"
space
• Industry partners who want to use
the tools but not share their data

39. Import format: Structure Notation
Language (SNL)
• Contains structure/molecule object, and provenance
about
created_at
authors
projects
references
remarks
data
history
Another way to remember the acronym..

40. Fireworks
• FireWorks is a code for defining, managing, and executing
scientific workflows
• It can be used to automate most types of calculations over
arbitrary computing resources, including those that have a
queueing system
• It is very dynamic: Fireworks can begat other fireworks at
runtime
http://pythonhosted.org/FireWorks/

41. Pymatgen-db
• Sick of MongoHub et al.? We were. So we wrote a simple
Web UI using prettytable, pymatgen, and Django
• https://github.com/materialsproject/pymatgen-db
Which we
proceeded to
use for deep
scientific inquiry

42. We’re not the only ones …
• Bioinformatics
• KBase (http://kbase.us) - DOE predictive and systems biology.
• Astronomy
• Sloan Digital Sky Survey (http://skyserver.sdss.org)
• Spectroscopy
• Advanced Light Source (ALS), Advanced Photon Source (APS)
• According to ProgrammableWeb, ~130 others
http://www.programmableweb.com/apis/directory/1?apicat=Science&protocol=REST
..though probably many of these are

43. More information
• Materials API + pymatgen examples
• https://gist.github.com/gists/search?q=materials+api+pymatgen
• The Materials API wiki
• https://materialsproject.org/wiki/index.php/The_Materials_API
• Python Materials Genomics
• http://packages.python.org/pymatgen/
• Shyue Ping Ong, William Davidson Richard, Anubhav Jain, Geoffroy
Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent
Chevrier, Kristin A. Persson, Gerbrand Ceder. Python Materials
Genomics (pymatgen) : A Robust, Open-Source Python Library for
Materials Analysis. (submitted)
• These slides:
• https://speakerdeck.com/shreddd/data-as-a-service-pydata-2013

44. Takeaways
• Make scientific data easily available to end-users
• Friendly, powerful Web UI is a great way to engage, but then..
• Build APIs around your data to make it easily accessible
• Write scientific libraries with *both* analysis and data, by
hooking them up to APIs.

45. We’re hiring
• Talented, science-loving, web-savvy, math-anything
Python programming code-slingers who would rather pass
a Nobel prize winner on the way to lunch than get free
dry-cleaning
• downside: or even free coffee (groan)
• upside: some of your tax dollars go towards your own salary!
• http://jobs.materialsproject.org/

46. Contact Us
• Shreyas Cholia – scholia@lbl.gov
• Dan Gunter – dkgunter@lbl.gov
• Materials Project Team – feedback@materialsproject.org