Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Presentation to the UM Library Emergent Research Series
1. SEAD: Sustainable Environment
through Actionable Data
Margaret Hedstrom
Professor of Information
Faculty Associate Institute for Social Research (ICPSR)
PI, SEAD
June 23, 2014
2. Overview
• What is SEAD?
• Vision and Rationale
• Target Audience and User Communities
• Current Status
• SEAD, Universities, and Libraries
• Some Lessons Learned (so far)
• Plans and Future Engagement
2
3. What is SEAD?
• A Cooperative Agreement funded by NSF to
develop sustainable cyberinfrastructure for
preservation and access to scientific data ($8
million/5 years)
• A partnership between the universities of
Michigan, Indiana and Illinois
• An emerging set of services for data management,
sharing, curation, discovery and preservation for
researchers in the “long tail”
• A case study of data needs in sustainability
science
3
4. SEAD Vision and Rationale
• Small teams, researchers with short-term
projects, and individual scientists (the long tail)
are under served by today’s data preservation
and access infrastructure
• These communities will take advantage of
evolving data preservation and access
infrastructure if:
– it supports science objectives and enables new kinds
of science
– it is easy to use
– collaborators and peers are also using it
• Sustainability science is a good test case
8. Target Audience / User Communities
Sustainability Scientists
• Focused on problems that require data, methods,
tools, and expertise from multiple disciplines
• Requires many different types of data about physical,
natural, and social phenomena in order to understand
interactions between natural and human systems
• Uses a combination of observational (field) data,
experimental data, simulations, and models
• Conducts research in small to medium-sized labs or
centers under the direction of a single PI or a Center
Director.
8
9. Target Audience / User Communities
the “Long Tail” of Scientific Research
• Data discovery is via targeted foraging and word-of-mouth
• Almost all data are stored locally
• Minimal local IT support
• Metadata standards and ontologies, where they do exist, are based
on disciplinary norms or local practices
• Data formats and metadata standards are often controlled by
multiple independent third-parties (e.g. instrument and application
providers
• Data are vulnerable to interruptions in organizational arrangements
(graduate students finish PhD’s and move on – lab or center funding
sunsets)
• No single data set is likely to have great value standing alone, but
when aggregated, combined and integrated data become valuable
resources of discovery and innovation.
9
10. Overview
Project Start 10/01/11
User Requirements Report 5/12
NCED Repository Ingest 8/12
Prototype Review 4/22/13
SEAD 1.0 Released 10/13
DataOne Member Node 11/13
End User Workshop 4/11/14
10th User Group 5/11/14
36-Month Review 10/14/14
Renewal (?) for Years 6-10
10
11. Summary of Current Status
• Working Platform
– SEAD Active Content Repository (ACR)
• Collaboration / File Sharing Space for Research Projects
• Staging Area for Data Prior to Publishing or Archiving
– SEAD Virtual Archive
• Capability to push data from ACR and/or local research
environments to preservation and discovery services
(Institutional Repositories/DataONE)
– SEAD Research Network
• Researcher initiated profiles with harvesting of citations,
linkage of data-people-publications, reporting
11
13. SEAD, Universities, and Libraries
• From the researcher’s perspective
– SEAD is an project work space that enables data
sharing, commenting, secure storage, extraction
of metadata, and active/social curation
• From the university research infrastructure
perspective
– SEAD is a staging area for data curation prior to
publication, submission, and preservation
13
14. Data Set Publishing Workflow
•Data content used
within ACR
•Researcher Profile
Established in VIVO
NCED Data Set
Ingested to ACR
•Data Set ready to
publish
NCED Data Set
Ingested to VA •DataCite minted
DOI attached to
finalized Data Set
NCED Data Set
Deposited with IR
•DOI Resolution to
designated IR
NCED Data Set
Published to
VIVO
19. SEAD Virtual Archive
• Purpose: Long-term preservation and discovery
– Thin virtualization layer on top of multiple university
Institutional Repositories (IRs)
– Enhances IRs by being sustainability science-aware
• Team: IU Libraries, UIUC Libraries, and Data To
Insight Center at IU
• Starting point: Data Conservancy code (Johns
Hopkins U.)
– Extended for sustainability science long tail use cases
20. Making Data Sustainable: Use Case
Active Curation
Repository
(ACR)
SEAD Virtual
Archive
IUScholarwork
s
UIUC Ideals
Packaged
object
Preserve data
Keep private for 5 years
Index data, metadata
and relationships
• Collected data about Lower
Mississippi flood
• Stored in Active Repository
• Organized as a collection
• Marked “Ready for
publication”
• Collections visible to team only
for 5 years
• Deposited to repository based
on dataset creator affiliation
• Find by author, location,
keywords or repository
24. Some Lessons Learned
• Some researchers and projects in the “long tail”
have sophisticated demands for active data
services
• Supporting analysis of data in SEAD adds
complexity and cost
• Users want some degree of customization of
bare-bones file storage and active project space
• A big gap remains between data producers and
the campus/library/archive infrastructures for
long-term access and preservation.
24
25. SEAD Priorities and Future Plans
• Make SEAD more stable and more usable
• Attract a larger, broader, and more diverse
user community
– Network effects in the long tail
– Self service
• Expand repository options
• Resolve Governance and Sustainability
25
MH: Revise Slide and Clarify message.
One might say the everyone is under-served by today’s DPAI but Interdisciplinary researchers have particular barriers / requirements
Multiple Sources
extracts from reference databases
observations
experimental results and model outputs
images
derived data products
Multiple file types, data types, data structures, data models
Multiple resolutions (spatial, temporal, granularity)
Multiple metadata standards and ontologies
Local standards and data practices developed on the fly
Data are vulnerable to interruptions in organizational arrangements
graduate students finish PhD’s and move on
project funding lapses
lab or center funding sunsets
One might also say that the long tail under utilizes existing DPAI (which is true) but for good reasons.
Build from Praveen’s life cycles.
Mention some of the steps that occur in curation.
Mention time lag
Build from Praveen’s life cycles.
Mention some of the steps that occur in curation.
Mention time lag
Support inter-disciplinary research and data driven research by:
Enabling access to:
Publications
Data
People (Expertise / Potential Collaborators
in novel innovative ways
that continuously anticipate and adapt to changes in technologies and in user needs and expectations;
Specifically,
Accelerate data discovery
Support new types of analyses with heterogeneous data
Reduce overall costs of curation [rather than shift costs between researchers and repositories]
Accelerate the movement of data from researchers into preservation, discovery and access environments
Increase the quantity, improve the quality, and enhance the utility of scientific data for reuse.
Start 2:01 Stop 4:00 ACR
Start 4:00 Stop: 4:53 Vivo
Start 9:57 Stop: 11:21
11:55 – end Vivo
Might move this section on Ingest workflow?
Reporting (Extra win for SEAD) and responsive to the community
- less emphasis on features and functionality
- remove "context" slides (done)
matchmaker workflow slide – simplify
make multiple dimensions of decision-making process of matchmaker more clear
- record a demo of how ingest and matchmaking works
deposit to ideals; make decision-making process points clear through example of Praveen, and demonstrate visually the embargo in ideals
- move DataNet slide to other decks (done)
VA - ACR interactions - user or science side of the story
A researcher at U of Illinois led the data collection effort related to Lower Mississippi flood.
The data have been collected and uploaded to ACR. In ACR the data have been organized into collections, processed for easy previewing and described (tagged and annotated). One subcollection has been marked as “Ready to publish”, i.e., ready for long-term preservation. Praveen wants to preserve the subcollection, but keep it private for 5 years.
SEAD Virtual Archive queries ACR and finds this subcollection. It packages the subcollection using its BagIT protocol and invokes its matchmaker algorithm to decide where to ingest the subcollection. The Matchmaker queries VIVO and finds that Praveen is from the University of Illinois. VA automatically creates a collection in IDEALS and marks it “embargoed” for 5 years.
After the collection is ingested, it appears in Virtual Archive and in IDEALS. In Virtual Archive this collection can be found by searching by author, location, keywords and repository. In the future, it will also provide search by data types (e.g., images, geo, video, etc.), instruments (e.g., Lidar, Aviris) and methods (e.g., data models, experiments, etc.)
- VA - IR communication - bring out details about solutions for large files (SDA), explain why numbers of files are so different for SDA, Scholarworks and Ideals (slide 10)