The DuraMat Data Hub and Analytics Capability provides a centralized resource for sharing solar PV data. It collects performance, materials properties, meteorological, and other data through a central Data Hub. A data analytics thrust works with partners to provide software, visualization, and data mining capabilities. The goal is to enhance efficiency, reproducibility, and new analyses by combining multiple data sources in one location. Examples of ongoing projects using the hub include clear sky detection modeling to automatically classify sky conditions from irradiance data.

1. The DuraMat Data Hub and
Analytics Capability
A Resource for Solar PV Data
Robert White1 and Anubhav Jain2
1National Renewable Energy Laboratory 2Lawrence Berkeley National Laboratory

2. Data and Analy*cs overview - 1
Field deployment
Data Hub Data analytics
Materials
Forensics &
Characterization
Predictive
simulationModule testing
Techno-economic
analysis
DuraMAT capability areas
and partners will share PV
performance, materials
properties, meteorological
data, & other data through
the Data Hub

3. Data and Analy*cs overview - 2
Field deployment
Data Hub Data analytics
Materials
Forensics &
Characterization
Predictive
simulationModule testing
Techno-economic
analysis
Data analytics is a
cross-cutting thrust
that will work with
DuraMat partners to
provide software,
visualization, and data
mining capabilities

4. Why a capability dedicated to data?
Data relevant to solar manufacturing and deployment
is largely sca5ered and cannot be easily exploited.
A centralized data hub enhances:
• efficiency – reduce produc=on of duplicate data
• reproducibility – collect together results from
mul=ple sources for quality assurance purposes
• context – bring together mul=ple data sources
(materials proper=es, accelerated tes=ng, field
data, meteorological data) into an analysis
• new analysis types – machine learning algorithms
require data as much as they require CPUs or
memory! More data means be5er models.

5. Some ways in which data is enabling
innova*on in solar
Google Project Sunroof and PowerScout
partner to iden=fy target consumers for
solar to innovate in sales / marke=ng
kWh Analy=cs uses data to develop risk
assessment models for solar investments
IBM’s “Wa5-Sun” uses data and machine
learning to provide more reliable solar
forecas=ng and power management

6. Some ways in which data is enabling
innova*on in solar
Google Project Sunroof and PowerScout
partner to iden=fy target consumers for
solar to innovate in sales / marke=ng
kWh Analy=cs uses data to develop risk
assessment models for solar investments
IBM’s “Wa5-Sun” uses data and machine
learning to provide more reliable solar
forecas=ng and power management
These are all
“data-driven”
technologies!

7. The value of a common “data hub”
• Much data has li5le value when used alone and becomes
much more impacVul when combined
• If you need to collect every data point yourself, data-driven
technologies become inaccessible to all
• Thus, it is in the best interest if some data is put in a
“commons”, available to all
Roger Chen, “Data liquidity in the age of inference”, h5p://bit.ly/2yt521W

8. Examples of data for “the commons”
Footer 4
4 4
Time series data Meteorological data Test chamber data
Materials simulation Materials properties Degradation analysis
The Data Hub will host
and connect to data
from multiple sources
and specializations
These research areas
and data sources are a
small sample of what
the hub plans to host

9. How to structure this resource?
Roger Chen, “Data liquidity in the age of inference”, h5p://bit.ly/2yt521W
Data heterogeneity /
incompa=bility and lack of
coordina=on are some
problem with typical open
data ini=a=ves
For-profit data sharing
brokerages are incen=vized
to collect/standardize data,
but may shy away from data
with high acquisi=on cost
Data coopera=ves
enhance standardiza=on
and work well for
members, but are difficult
to get off the ground

10. How to structure this resource?
Roger Chen, “Data liquidity in the age of inference”, h5p://bit.ly/2yt521W
Data heterogeneity /
incompa=bility and lack of
coordina=on are some
problem with typical open
data ini=a=ves
For-profit data sharing
brokerages are incen=vized
to collect/standardize data,
but may shy away from data
with high acquisi=on cost
Data coopera=ves
enhance standardiza=on
and work well for
members, but are difficult
to get off the ground
The DuraMat data hub combines elements of the “open data” and “data
cooperative” models, providing support for both open and private data. The
hub will aid in data standardization while providing flexibility to data providers,
and also contribute data with high acquisition cost as generated by DuraMat.

11. Data Hub Infrastructure

12. What is the Data Hub
“A collec6on of so8ware components and
databases all working in concert to support secure
data sharing and distribu6on”
Criteria for our Data Hub
• Manage research data.
• Provide access to historical and =me-series databases.
• Ability to search data resources and metadata.
• Provided hierarchical data security.
• Provide access to analysis tools.
• Easy (but secure) access to consor=um members and partners.
• Provide access to publically released data.

13. Elements of the Data Hub
Modeling and
Simula=on
Forensics,
Measurement
and Synthesis
Techno-Economic
Analysis
Documenta=on
and Publica=ons

14. Data Management Structure
Project
Sub-Project
Dataset
Files and Resources
(Optional)

15. Example: of Data Management Structure
Project
Sub-Project
Dataset
Files and Resources
Predic've Simula'on
Thermomechanical
Cell
Demonstra'on
Ø Input
Ø Geometry
Ø Mesh
Ø Output
Ø 5191b5Ab5c9d.pdf
Ø cell.SLDPRT
Ø cells.SLDASM
Ø Complete.SAT
Ø EVAandStack.SLDPRT
Ø ribbon.SLDPRT
Ø ribbondead.SLDPRT

16. “Dynamic, sequen6al data, indexed by 6me and loca6on,
that is con6nuously being acquired from fielded PV
systems.”
Time Series Data
FTP Site Transla=on
Scripts
API
Data
Hub
API
Public Website
Programma*c Access
Field Sites Time-Series
Database
• Slice and examine performance in *me scales from minutes to years.
• Cross compare system performance.
• System event informa*on.
• Loca*on and climate metadata.
• Weather condi*ons.

17. The DuraMAT Data Hub
h5ps://datahub.duramat.org
A_er registering, email: robert.white@nrel.gov
include your ins=tu=on or company and which
projects you will need access to.

18. Analy*cs

19. Data Analy*cs capability
Visualize, model, and predict
Release open
source software
• The data analy=cs thrust is cross-
cuang, i.e., we can work with any
thrust that needs help
• Help visualize, model, and predict
based on your data
• Seeking collabora*ons!

20. Example project: clear sky detec*on
Visualize, model, and predict
Release open
source software
rdtools
Data from
partners

21. Clear sky detec*on problem
Persistently
cloudy, noisy
Persistently
cloudy, noisy
Borderline
cases
Generally clear
Given measured solar irradiance and expected clear sky irradiance, can we
automa=cally detect clear vs. cloudy sky periods?
• Important for downstream data analyses – e.g., correctly filtering out
cloudy / noisy points can significantly change degrada=on rate calcula=on
• Prior algorithms require tuning based on site and on data resolu=on
• Can we infer a “universal” clear sky model based on past data?
• NSRDB has labeled historical data across en=re U.S. (2 million pixels) every
30 minutes for >15 years! Plenty of data to train a model…

22. Clear sky detec*on analy*cs
x1
x2
x3
…
x1
x2
x3
…
x1
x2
x3
…
x1
x2
x3
…
x1
x2
x3
…
x1
x2
x3
…
+ NSRDB clear sky labels
ML classifier,
e.g., random
forest
Features used (similar to
pvlib)
Single point features:
• GHI - GHICS
• GHI’ - GHI’CS
• Time from solar
noon
Window based metrics
calculated every 1 hour
• GHI - GHICS average
and standard
devia=on
• GHI’ - GHI’CS
average and
standard devia=on
• GHILL - GHILL
CS
(difference of line
length)
Step 2: describe each
irradia*on measurement
point by a set of
numerical features
Step 3: train a
machine learning
algorithm to make
predic*ons
Step 1: clean
data set of
unambiguously
mislabeled data
clear points labeled as
“cloudy” in NSRDB – filter
from analysis

23. ML model achieves ~98% accuracy!
No data cleaning Data cleaning applied
Next steps:
• Evaluate sensi=vity of model to different irradiance measurement intervals
• Run analysis across en=re NSRDB data set and test applicability of a single
model across mul=ple geographical loca=ons
• Comparisons of satellite vs ground-based measurements
• Benchmark against exis=ng algorithms as well as expert labeling
• Publish model and accompanying so_ware implementa=on

24. Currently seeking collabora*ons
• The Data Analy=cs thrust is also looking to extend
our experience with =me series and weather data
to new applica=ons
• Irradiance forecas=ng
• PV genera=on forecas=ng
• Degrada=on analysis
We are also interested in other areas of research in need of
data mining, machine learning, and sta=s=cal analysis! Come
talk to us at the poster session or during the workshop.

25. NREL EMN
Development Team
• Kris=n Munch
• Nick Wunder
• Chris Webber
• Dave Evenson
• Courtney
Pailing
DuraMat Data
Team
• Ben Ellis
• Birk Jones
• Pedro Perez
• Stephanie Moffi5
• Kevin Leung
• Jonathan Trinas=c
Acknowledgements
A Coordinated Team Effort
Data Analy@cs
• Ben Ellis
• Mike Deceglie
• NSRDB / PVDAQ
teams