1.
Valuing Software and Other
Research Outputs
Daniel S. Katz
dkatz@nsf.gov & d.katz@ieee.org
@danielskatz
Program Director, Division of
Advanced Cyberinfrastructure
1am:London – The Altmetrics Conference
26 September 2014

2.
National Science Foundation
• Federal agency created in 1950 "to promote the
progress of science; to advance the national
health, prosperity, and welfare; to secure the
national defense…”
• Annual budget of $7.2 billion (FY 2014)
• Funds 24 percent of all federally supported
basic research at US colleges and universities
• In many fields such as mathematics, computer
science and the social sciences, NSF is the
major source of federal funds

3.
NSF
NATIONAL SCIENCE FOUNDATION
DIRECTORATE FOR
BIOLOGICAL
SCIENCES
(BIO)
John C.l Wingfied,
Assistant Director
Jane Silverthorne,
Deputy AD
703.292.8400
DIRECTORATE FOR
EDUCATION & HUMAN
RESOURCES
(EHR)
Joan Ferrini-Mundy,
Assistant Director
Jermelina L. Tupas,
Acting Deputy AD
703.292.8600
DIVISION OF BIOLOGICAL
INFRASTRUCTURE (DBI)
Scott Edwards,
Division Director
703.292.8470
DIVISION OF ENVIRONMENTAL
BIOLOGY (DEB)
Penelope Firth,
Division Director
703.292.8480
DIVISION OF INTEGRATIVE
ORGANISMAL SYSTEMS (IOS)
Michelle Elekonich,
Acting Division Director
703.292.8420
DIVISION OF MOLECULAR &
CELLULAR BIOSCIENCES (MCB)
Gregory Warr,
Acting Division Director
703.292.8440
OFFICE OF EMERGING
FRONTIERS (EF)
Charles Liarakos,
Acting Division Director
703.292.8508
DIRECTORATE FOR
COMPUTER &
INFORMATION SCIENCE &
ENGINEERING (CISE)
Suzane Iacono,
Acting Assistant Director
Vacant,
Deputy AD
703.292.8900
DIRECTORATE FOR
ENGINEERING
(ENG)
Pramod P.
Khargonekar,
Assistant Director
Grace Wang,
Deputy AD
703.292.8300
DIVISION OF CHEMICAL,
BIOENGINEERING, ENVIRONMENTAL &
TRANSPORT SYSTEMS (CBET)
JoAnn Lighty,
Division Director
703.292.8320
DIVISION OF CIVIL,
MECHANICAL & MANUFACTURING
INNOVATION (CMMI)
George Hazelrigg,
Acting Division Director
703.292.8360
DIVISION OF ELECTRICAL,
COMMUNICATIONS & CYBER
SYSTEMS (ECCS)
Samir El-Ghazaly,
Division Director
703.292.8339
DIVISION OF ENGINEERING
EDUCATION & CENTERS (EEC)
Theresa Maldonado,
Division Director
703.292.8380
DIVISION OF INDUSTRIAL
INNOVATION & PARTNERSHIPS (IIP)
Vacant,
Division Director
703.292.8050
OFFICE OF EMERGING
FRONTIERS IN RESEARCH &
INNOVATION (EFRI)
Sohi Rastegar,
Senior Advisor
703.292.8301
France A. Córdova
Director
DIRECTORATE FOR
GEOSCIENCES
(GEO)
Roger Wakimoto,
Assistant Director
Margaret Cavanaugh,
Deputy AD
703.292.8500
DIRECTORATE FOR
MATHEMATICAL &
PHYSICAL SCIENCES
(MPS)
Fleming Crim,
Assistant Director
g
Celeste M. Rohlfin
,
Deputy AD
703.292.8800
DIVISION OF ASTRONOMICAL
SCIENCES (AST)
James Ulvestad,
Division Director
703.292.8820
DIVISION OF CHEMISTRY (CHE)
Jacquelyn Gervay-Hague,
Division Director
703.292.8840
DIVISION OF MATERIALS
RESEARCH (DMR)
Mary Galvin-Donoghue,
Division Director
703.292.8810
DIVISION OF MATHEMATICAL
SCIENCES (DMS)
Michael Vogelius,
Division Director
703.292.8870
DIVISION OF PHYSICS (PHY)
Denise Caldwell,
Division Director
703.292.8890
OFFICE OF MULTIDISCIPLINARY
ACTIVITIES (OMA)
Clark e
Cooper,
Offic
He ad
703.292.8800
OFFICE OF INTERNATIONAL &
INTEGRATIVE ACTIVITIES (OIIA)
DIRECTORATE FOR
SOCIAL, BEHAVIORAL, &
ECONOMIC SCIENCES
(SBE)
Fay Cook,
Assistant Director
Deborah Olster,
Acting Deputy AD
703.292.8700
DIVISION OF BEHAVIORAL &
COGNITIVE SCIENCES (BCS)
Mark Weiss,
Division Director
703.292.8740
DIVISION OF SOCIAL &
ECONOMIC SCIENCES (SES)
Jeryl Mumpower,
Division Director
703.292.8760
NATIONAL CENTER FOR
SCIENCE AND ENGINEERING
STATISTICS (NCSES)
John Gawalt,
Division Director
703.292.8780
DIVISION OF GRADUATE
EDUCATION (DGE)
Valerie Wilson,
Acting Division Director
703.292.8630
DIVISION OF HUMAN RESOURCE
DEVELOPMENT (HRD)
Sylvia James,
Division Director
703.292.8640
DIVISION OF RESEARCH ON
LEARNING IN FORMAL &
INFORMAL SETTINGS (DRL)
Sarah McDonald,
Acting Division Director
703.292.8620
DIVISION OF UNDERGRADUATE
EDUCATION (DUE)
Susan Singer,
Division Director
703.292.8670
DIVISION OF ATMOSPHERIC &
GEOSPACE SCIENCES (AGS)
Scott Borg,
Division Director
703.292.8520
DIVISION OF EARTH
SCIENCES (EAR)
Sonia Esperança
Acting Division Director
703.292.8550
DIVISION OF OCEAN
SCIENCES (OCE)
Deborah Bronk,
Acting Division Director
703.292.8580
DIVISION OF
POLAR PROGRAMS (PLR)
Kelly Falkner,
Division Director
703.292.8030
DIVISION OF COMPUTER &
NETWORK SYSTEMS (CNS)
Keith Marzullo,
Division Director
703.292.8950
OFFICE OF INFORMATION
& RESOURCE
MANAGEMENT
(OIRM)
Clifford Gabriel,
Acting Head
Amy Northcutt,
Chief Information Offic
e
r
703.292.8100
OFFICE OF DIVERSITY &
INCLUSION (ODI)
Claudia J. Postell, Head
703.292.8020
Wanda Ward, Head
703.292.8040
OFFICE OF BUDGET,
FINANCE, & AWARD
MANAGEMENT
(BFA)
Martha A. Rubenstein,
Head / e
Chief Financial
Offic
r
Joanna E. Rom,
Deputy Head
703.292.8200
BUDGET DIVISION (BUD)
Michael Sieverts,
Division Director
703.292.8260
DIVISION OF ACQUISITION AND
COOPERATIVE SUPPORT (DACS)
Jeffery Lupis,
Division Director
703.292.8240
DIVISION OF FINANCIAL
MANAGEMENT (DFM)
Shirl Ruffin
,
Division Director / Deputy CFO
703.292.8280
DIVISION OF ADMINISTRATIVE
SERVICES (DAS)
Mercedes Eugenia,
Division Director
703.292.8190
DIVISION OF INFORMATION
SYSTEMS (DIS)
Dorothy Aronson,
Division Director
703.292.8150
DIVISION OF HUMAN RESOURCE
MANAGEMENT (HRM)
Judy Sunley,
Division Director /
Acting Chief Human Capital Offic
National Science Foundation
4201 Wilson Boulevard
Arlington, Virginia 22230
TEL: 703.292.5111 | FIRS: 800.877.8339 | TDD: 800.281.8749 September 2014
e
r
703.292.8180
DIVISION OF GRANTS &
AGREEMENTS (DGA)
Karen Tiplady,
Division Director
703.292.8210
DIVISION OF INSTITUTION &
AWARD SUPPORT (DIAS)
Mary Santonastasso,
Division Director
703.292.8230
LARGE FACILITIES OFFICE
Matthew Hawkins,
Acting Deputy Director
703.292.4416
DIVISION OF COMPUTING &
COMMUNICATION
FOUNDATIONS (CCF)
Rao Kosaraju,
Division Director
703.292.8910
DIVISION OF ADVANCED
CYBERINFRASTRUCTURE (ACI)
Irene Qualters,
Division Director
703.292.8970
DIVISION OF INFORMATION &
INTELLIGENT SYSTEMS (IIS)
Deborah Lockhart,
Acting Division Director
703.292.8930
OFFICE OF THE GENERAL
COUNSEL (OGC)
Lawrence Rudolph,
General Counsel
Peggy Hoyle, Deputy GC
703.292.8060
OFFICE OF LEGISLATIVE & PUBLIC
AFFAIRS (OLPA)
Judy Gan, Head
703.292.8070
OFFICE OF INSPECTOR
GENERAL (OIG)
Allison C. Lerner, Inspector General
703.292.7100
NATIONAL SCIENCE BOARD
OFFICE
Michael Van Woert
Executive Offic
e
r
703.292.7000
OFFICE OF THE DIRECTOR
703.292.8000
NATIONAL SCIENCE BOARD (NSB)
Dan E. Arvizu
Chair
Kelvin K. Droegemeier
Vice Chair
703.292.7000
Vacant
Deputy Director

4.
Advanced Cyberinfrastructure
(ACI) Division
• Supports and coordinates the development,
acquisition, and provision of state-of-the-art
cyberinfrastructure resources, tools, and
services
• Supports forward-looking research and
education to expand the future capabilities of
cyberinfrastructure
• Serves the growing community of scientists and
engineers, across all disciplines, whose work
relies on the power of advanced computation,
data-handling, and networking

5.
Cyberinfrastructure
“Cyberinfrastructure consists of
computing systems,
data storage systems,
advanced instruments and
data repositories,
visualization environments, and
people,
all linked together by
software and
high performance networks,
to improve research productivity and
enable breakthroughs not otherwise possible.”
-- Craig Stewart

6.
Software as Infrastructure
Science
Software
Computing
Infrastructure
• Software (including services) essential for
the bulk of science
- About half the papers in recent issues of
Science were software-intensive projects
- Research becoming dependent upon
advances in software
- Significant software development being
conducted across NSF: NEON, OOI,
NEES, NCN, iPlant, etc
• Wide range of software types: system, applications, modeling,
gateways, analysis, algorithms, middleware, libraries
• Software is not a one-time effort, it must be sustained
• Development, production, and maintenance are people intensive
• Software life-times are long vs hardware
• Software has under-appreciated value
For software to be sustainable,
it must become infrastructure

7.
NSF Software Infrastructure Projects
See http://bit.ly/sw-ci for current projects
5 rounds of funding,
65 SSEs
4 rounds of funding,
35 SSIs
2 rounds of funding,
14 S2I2
conceptualizations
SSE & SSI – NSF 14-520: Cross-NSF, all Directorates participating
Next SSEs due Feb 2015; Next SSIs due June 2015

8.
SI2 Solicitation and Decision Process
• Proposal reviews well -> my role becomes
matchmaking
– I want to find program officers with funds, and convince them
that they should spend their funds on the proposal
• Unidisciplinary project (e.g. bioinformatics app)
– Work with single program officer, either likes the proposal or
not
• Multidisciplinary project (e.g., molecular
dynamics)
– Work with multiple program officers, ...
• Omnidisciplinary project (e.g. http, math library)
– Try to work with all program officers, often am told “it’s your
responsibility”
To judge software, need to
understand/forecast impact

9.
Measuring Impact – Scenarios
1. Developer of open source physics simulation
– Possible metrics
• How many downloads? (easiest to measure, least value)
• How many contributors?
• How many uses?
• How many papers cite it?
• How many papers that cite it are cited? (hardest to measure,
most value)
2. Developer of open source math library
– Possible metrics are similar, but citations are less
likely
– What if users don’t download it?
• It’s part of a distro
• It’s pre-installed (and optimized) on an HPC system
• It’s part of a cloud image
• It’s a service
• Future impacts – let proposers suggest

10.
ACI Software Cluster Programs
• In these programs, ACI works with other NSF
units to support projects that lead to software
as an element of infrastructure
• Issue: amount of software that is
infrastructure grows over time, and grows
faster than NSF funding
Q: How can NSF ensure that software as
infrastructure continues to appear, without
funding all of it?
A: Incentives
• The devil is in the details

11.
Other Software Discussions
• Working Towards Sustainable Software for
Science: Practice and Experience (WSSSPE)
– Google: WSSSPE
– 2 previous workshops, one upcoming
– WSSSPE2 @ SC14 (Nov 16, New Orleans)
• Lessons:
Many of the issues in developing
sustainable software are social, not
technical
Software work is inadequately visible in
ways that “count” within the reputation
system underlying science

12.
Where We Are
• To judge software, need to understand/forecast impact
• Q: How can NSF ensure that software as infrastructure
continues to appear, without funding all of it?
• A: Incentives
• Many of the issues in developing sustainable software are
social, not technical
• Software work is inadequately visible in ways that “count”
within the reputation system underlying science
Hypothesis: better measurement of
contributions can lead to rewards
(incentives), leading to career paths,
willingness to join communities, leading to
more sustainable software

13.
Moving Forward - NSF
• Recent CISE/ACI & SBE/SES Dear Colleague
Letter: Supporting Scientific Discovery through
Norms and Practices for Software and Data
Citation and Attribution (NSF 14-059,
http://www.nsf.gov/pubs/2014/nsf14059/nsf14059
.jsp)
– Need well-developed metrics to assess the
impact and quality of scientific software and
data
– Explore new norms and practices for software
and data citation and attribution, so that data
producers, software and tool developers, and
data curators are credited
• 6 projects and 3 collaborative workshops funded

14.
Moving Forward - Dan
• Products (software, paper, data set) are
registered
– Credit map (weighted list of contributors—
people, products, etc.) is an input
– DOI is an output
– Leads to transitive credit1
• E.g., paper 1 provides 25% credit to software A, and
software A provides 10% credit to library X -> library
X gets 2.5% credit for paper 1
• Helps developer show: “my tools are important”
– Issues:
• Social: Trust in person who registers a product
• Technological: How2, Registration system
1D. S. Katz, "Transitive Credit as a Means to Address Social and Technological Concerns Stemming from Citation
and Attribution of Digital Products," Journal of Open Research Software, v.2(1): e20, 2014. DOI: 10.5334/jors.be
2D. S. Katz, A. M. Smith, "Implementing Transitive Credit with JSON-LD," 2nd Workshop on Sustainable Software
for Science: Practice and Experiences (WSSSPE2), 2014. URL: http://arxiv.org/abs/1407.5117

15.
Moving Forward - Community
• Career paths – Is there a role for non-tenure-track
researchers who produce software, data,
etc. in universities?
– Assuming yes, do universities recognize and support
this? If not, how to get them to?
• What is needed to support reproducibility of
science, in terms of data and software?
• Lots of entities with similar interests in both
software and data, e.g. JISC, RCUK, NIH, DOE,
Sloan & Moore, Mozilla, Apache, etc.
• Participate in WSSSPE
• Other ideas and questions are welcome, now or
later
– dkatz@nsf.gov or d.katz@ieee.org

16.
Resources
• NSF Software as Infrastructure Vision:
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12113
• Implementation of NSF Software Vision:
http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504817
• Software Infrastructure for Sustained Innovation (SI2) Program
– Scientific Software Elements (SSE) & Scientific Software Integration (SSI) solicitation:
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf14520
– 2013 PI meeting: https://sites.google.com/site/si2pimeeting/
– 2014 PI meeting: https://sites.google.com/site/si2pimeeting2014/
– Awards: http://bit.ly/sw-ci
• Working towards Sustainable Software for Science: Practice and Experiences
(WSSSPE)
– Home: http://wssspe.researchcomputing.org.uk (includes links to all slides & papers)
– 1st workshop paper: http://arxiv.org/abs/1404.7414
– 2nd workshop site: http://wssspe.researchcomputing.org.uk/wssspe2/
• NSF 14-059: “Dear Colleague Letter - Supporting Scientific Discovery through
Norms and Practices for Software and Data Citation and Attribution”
– http://www.nsf.gov/pubs/2014/nsf14059/nsf14059.jsp
• Transitive Credit Papers
– http://dx.doi.org/10.5334/jors.be
– http://arxiv.org/abs/1407.5117

17.
Credits:
• SI2 Program:
– Current program officers: Daniel S. Katz, Rudolf Eigenmann, William
Y. B. Chang, John C. Cherniavsky, Almadena Y. Chtchelkanova,
Cheryl L. Eavey, Evelyn Goldfield, Sol Greenspan, Daryl W. Hess,
Peter H. McCartney, Bogdan Mihaila, Dimitrios V. Papavassiliou,
Andrew D. Pollington, Barbara Ransom, Thomas Russell, Massimo
Ruzzene, Nigel A. Sharp, Paul Werbos, Eva Zanzerkia
– Formerly-involved program officers: Manish Parashar, Gabrielle
Allen, Sumanta Acharya, Eduardo Misawa, Jean Cottam-Allen,
Thomas Siegmund
• WSSSPE:
– Organizers: Daniel S. Katz, Gabrielle Allen, Neil Chue Hong, Karen
Cranston, Manish Parashar, David Proctor, Matthew Turk, Colin C.
Venters, Nancy Wilkins-Diehr
– WSSSPE1 Summary paper authors: Daniel S. Katz, Sou-Cheng T.
Choi, Hilmar Lapp, Ketan Maheshwari, Frank Löffler, Matthew Turk,
Marcus D. Hanwell, Nancy Wilkins-Diehr, James Hetherington,
James Howison, Shel Swenson, Gabrielle D. Allen, Anne C. Elster,
Bruce Berriman, Colin Venters
– WSSSPE1 Keynote speakers: Phil Bourne, Arfon Smith