NASA CoECI Presentaion on Crowdsourcing and Challenges
1. Harnessing the
Power of the Crowd
Using Crowd-Based Challenges as
Tools for Engineering Projects
NASA’s Center of Excellence for
Collaborative Innovation (CoECI)
Steve Rader steven.n.rader@nasa.gov
@NASA_NTL
5. 5
We are used to the model of “the expert”, that keeps up with
all of the tech advancements that we need for a given field.
In the past, our biggest struggle has been to find and keep
these experts!
But now, the
expert you
need today is
different than
the expert you
need
tomorrow….
The pace of technology development is just too much.
Individuals
cannot possibly
keep up with
the technology
advancements
6. Even if you could find the right expert for your problem, could you afford them?
Would you be able to work through the red tape to get them to work with you?
What would you do with him after you have your answer and you
need to move to the next expert area?
What about all of the technology advancement that
is going on out there that doesn’t
NEED an expert…
it just needs you to find it!
It is all like looking for
a
needle
in a
haystack!
8. 8
Internet User Growth from 2000-2012 and Potential
Asia
1.1B
28%
Europe
519M
63%
Africa
167M
16%
North
America
274M
79%
Latin
America
255M
43%
Middle
East
90M
40%
Oceana/A
ustrialia
24M
68%
The
World
2.4B
34%
Users in
2000
Additional
users by 2012
Population
Not on the
Internet YET
Current #
of Users
% of Population
that is Online
566% Growth
from 2000-2012
LegendBased on table from: http://www.internetworldstats.com/stats.htm
9. 9
With more
and more of
the world’s
population on
the Internet,
the ability to
access
people with
unique skills,
expertise, and
experience is
rapidly
increasing.
Academic students
researchers
faculty
Retired scientists,
engineers, &
programmers
Current
“under-employed”
scientists, engineers,
& programmers
The Tip of the Iceberg! Not only are the Billions of people around the world
rapidly getting online. Right now, the 2.8B people that ARE online are just
barely aware of these platforms and what they mean to them…
11. 11
Cognitive
Surplus
This “interest” in using their skills to make a difference
or win a prize outside of normal employment (at
night or on the weekends) can be described as….
People seem genuinely interested in engaging in
problem solving and being “the one” to make a
difference!
12. 12
If two heads
are better
than one….
Now consider how this cognitive surplus begins to become a powerful
force as more of the current 2.8 billion connected individuals join in the
efforts and the remaining 4.2 billion become connected on the Internet.
The possibilities to harness the creativity of the human mind are enormous!
13. Image: Illumination Entertainment, Universal Studios
But just because 2.8 Billion people
are connected does NOT mean
that they are organized ….
AT ALL…
In fact it can be chaos.
14. 14
In fact… it’s fascinating what global
connectivity has meant to many.
15. 15
Networks & Communities
Curated Communities
Inventors
Software Coders
Film-Makers
Photographers
Graphic Artists
Engineers & Designers
800,000*
84,000*
1,081,000
745,000
50,000
1,860,000
*estimate
Well formulated crowd-based platforms actively work to
build a community of users that are passionate enthusiasts.
16. 16
Curated Communities
Resources and
Tools for
Members
Incentives for
Members to
Participate
User Agreements
for Privacy and
Payment
Mechanism for
Handling IP
Licensing and/or
Transfer
Community
Building &
Communication
Curated communities
are built around
enabling people to
pursue their passion
and create a win-win
for the company and its
community members.
They provide structure
and incentives.
Communities do NOT
like to be exploited!
17. GoldGutsGoodGlory
Why Does The Crowd Contribute?
Earn Money (real or virtual)
Have Fun (or pass the time)
Socialize with Others
Obtain Recognition or Prestige
(leaderboards, badges)
Do Good (altruism)
Learn Something New
Obtain Something Else
Create Self-Serving Resource
17
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
by Matthew Lease, Assistant Professor at University of Texas at Austin on May 03, 2013
http://www.slideshare.net/mattlease/crowdsourcing-human-computation-labeling-data-building-hybrid-systems
Multiple
Incentives
can often
operate in
parallel
18. Effectively Using Communities
Create an
Innovative New
Solution
Apply an
Existing
Technology
(in an innovative way)
Access Best
Possible
Product or
Service
(competition winner)
Find an
Existing
Solution
(you didn’t know existed)
Access Very
Specific
Expertise
(found through
competition)
18
Solve a
Problem
Develop a
Product
Provide a
Service
Innovation from Diversity found via Challenges
(Experience, Context/Perspective, Expertise)
Expert or Domain Focused MembershipDiverse Membership
High Quality Products/Services
(via Competition to get Best in Domain)
22. 22Executing a Crowd-Sourced Challenge
Formulate the
Problem Statement
Design the
Challenge
Execute the
Challenge
Pick the Winner(s)
Judging
Get Your Solution
IP licensing and/or transfer
Solution Filtering (optional)
Knowing how
to do all of
these steps
really helps to
mitigate the issues
associated with this
“too many solutions” problem.
ALL of these steps can help to
minimize the number of solutions
you end up needing to judge.
A well formulated problem statement
(with good success criteria)
A well designed challenge
(including setting the right prize amount)
Solution filtering
mechanisms are
offered
by some
platforms
23. We Can Use the Crowd as
a Force Multiplier…
23
…To Do MORE with LESS
28. 28
“One Stop Shop” Website which Lists ALL NASA Challenges
http://www.nasa.gov/solve
29. Centennial Challenges
NTL Curated Community Challenges
Citizen Science Challenges
Space Apps Challenge
Software
Apps
Scientific
Observations/C
ontributions
Tech Dev &
Demo
Years
Months
Months
Days/We
eks
Duration Prizes Product Participation
$M+
$1K-
$100Ks
Recognition
Recognition
Ideas,
Designs,
Software
29
30. 30
First Things First!
NASA has some of
the most amazing
employees on the
PLANET!
Tap into that
crowd FIRST!
31. 31
Over 15,000 Registered Members
(25% of NASA’s 60,000 CS & Contractor Workforce)
15-20 Challenges per Year
Growth of the NASA@work
Community since October 2012
2-4 Active challenges
posted at any one time
New challenge posts every
~2-3 weeks
People that work at
NASA want to make
a difference!
32. Innovation & Problem Solving
Using Challenges with Diverse Communities to
develop unique and innovative approaches
to unsolved problems
33. NASA Innovation Pavilion on
33
Global Community
of 350,000+ Solvers
Diverse Member Base
High Success Rate in
Solving Hard Problems
with Innovative
Solutions
Map showing participation of 2900 solvers
from 80 countries in NASA’s pilot challenges
34. “Diverse perspectives and tools
enable collections of people to
find more and better solutions.”
34
Diversity Trumps Ability Theorem:
“The people you’d least expect to
solve a problem were exactly the
ones most likely to crack it.”
Scott Page
Author of Difference:
How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies
35. 35
Diversity is the Key to Innovation
One MIT study into InnoCentive
revealed that solvers were more
successful when they had less
experience in the relevant discipline.
Some data suggests that as much as
70% of successful InnoCentive
challenge solutions are solved by
individuals outside of the challenge’s
specific technical domain.
36. 36
“In 60 days, Roche was able to solve a problem that it
and its partner have been tinkering with and optimizing
for the last 15 years. The solutions provided actually
mirrored the entire history of Roche’s R&D programme.
All of the solutions Roche had tried came in. “
Swiss company with 80,000
employees, Roche operates in 150
countries and has R&D operations in
Europe, North America and Asia-
Pacific
Roche is a world leader in in vitro
diagnostics.
Julian Birkinshaw, MLabnotes, University of London Business School
Roche ran an InnoCentive challenge
37. 37
$20,000 in prizes
Over 2800 registrants
219 Entries
Winning Submission: Barium tracers for atmospheric analysis
Winner was Ted
Ground from
Rising Star, Texas
(population 799)
38. Galactic Cosmic Ray Mitigation
Finding ways to protect humans from the
effects of cosmic galactic rays in deep
space.
Non-Invasive Intra-Cranial Pressure Monitoring
Finding methods to monitor the pressure in
the human brain that are non-invasive.
Strain Measurement of Vectran and Kevlar
Webbing
Seeking improved methods for strain
measurement under very specific
conditions for inflatable habitat testing.
Additional
InnoCentive Challenges
38
40. Case Study
CODERS SUBMITTED SOLUTIONS DIFFERENT APPROACHES TO
SOLVE PROBLEM IDENTIFIED
WINNING COUNTRIES
RUSSIA, FRANCE, EGYPT, BELGIUM & US
122 654 89 5
Improve on NIH MegaBlast algorithm
for nucleotide sequence alignment
Winning solution performs 120x faster
ANTIBODY SEQUENCE ANNOTATION
Source:
47 min. 16 sec.4.3 hours
$2M+
Multi-year
Development
41. 15% Improvement!
Over current method of
identifying asteroids in
the main belt of Asteroids
that orbit between Mars
& Jupiter
1241 Registrants
625 Solutions Submitted
$74,124 in Prizes Awarded
Algorithm AND App
42. Asteroid Tracker
Optimized algorithm for tracking multiple
asteroids with an array of antennas.
Earth Science
Developing ideas for tools that use NASA
earth science data to help local
communities with climate resilience.
Planetary Data System (PDS) - Cassini
Image processing of Saturn rings data to
detect possible new satellites or moons.
Additional Recent Harvard/
TopCoder Algorithm Challenges
42
43. Using Competitions for Software Development
Conceptualization
Specification
Wireframes
Storyboards
Architecture
Assembly
Component Dev
Bug Hunt
Bug Race
Idea Generation
Image Credit: Wikipedia, Systems development life-cycle, http://en.wikipedia.org/wiki/Systems_development_life-cycle (as of Mar. 27, 2013, 05:48 GMT).
Source:
44. Lunar Mapping and Monitoring Project (LMMP)
Has reduced image processing times from
18 hours to 3 hours.
Disruption/Delay Tolerant Network (DTN)
Attempting to solve a problem with
distributing security keys in a disrupted/
delayed network. Building out the protocol
stack.
ISS Food Intake Tracker
An iPAD app designed for the ISS to better
track their food intake (featured at the
Apple Developers Conference in 2014).
Recent Harvard/TopCoder
Software Development Challenges
44
53. Yet2.com
Provides a “matching” service that finds
technologies and solutions from industry, academia,
and/or individuals for a given need/challenge.
Includes a 130,000 member community and links to
over 16,000 commercial entities.
Very effective (and cost effective) in searching for
existing products or development efforts.
54. Yet2.com Challenges
Bone Density Measurement
Monitoring of Water and
Biocides
Non-Invasive Intra-Cranial Pressure Measurement
“Much more than we expected! Very pleasantly surprised
that this process exposed so many potential solutions with
such wide breadth and depth”
“Learned that we should have revisited technologies that
we rejected earlier
81 Leads Identified
63 Rejected
3 High Interest Solutions
5 Other Interesting Solutions
6 Potential Complementary
Technologies
2 Potential Solutions
55. Non-
129 Challenges
4 Tech Surveys
2 Videos
1 Design
14 Algorithms
15 Software
2 Ideation
2 Graphics
76 Challenges
NASA Innovation Pavilion
9 Theoretical
2 Ideation
2 Reduction to
Practice
NASA Challenge Experience
with Curated Communities
Includes NASA and other U.S. Gov’t Agency Challenges executed under NASA’s CoECI
56. Crowd-Based Challenges are valuable tools that
should be considered for future engineering
projects
Effective Method to Find Truly Innovative Solutions
Key Method to Find the Right Existing Technologies
(out of the sea of R&D work across a variety of domains)
Efficient Method to Augment Project Teams
with Specialized Skills as Needed
Using Crowds as Engineering Tools
58. Acknowledgements
References:
Crowdsourcing Landscape: www.crowdsourcingresults.com, www.infographically.com
Global Internet Use: Based on table from: http://www.internetworldstats.com/stats.htm
Howe, Jeff (2008-08-18). Crowdsourcing: Why the Power of the Crowd Is Driving the Future
of Business, Doubleday Religious Publishing Group.
“Diverse perspectives and tools enable collections of people to find more and better
solutions.”
Scott Page. Difference: How the Power of Diversity Creates Better Groups, Firms, Schools,
and Societies.
Karim R. Lakhani et al., “The Value of Openness in Scientific Problem Solving.”
Graphics:
experts - http://directoryofexcelexperts.com/
needle in haystack - http://siliconbeachclearly.com/
global network - http://www.emirates.com/
iceberg - http://www.weirdoptics.com/iceberg-illusion/
minions - Illumination Entertainment, Universal Studios
lolcats – http://background-pictures.picphotos.net/
farmville – http://farmville.startpagina.nl
tinder – http://apptiinder.com
thumbsdown – http://commons.wikimedia.org
community – http://www.blogworld.com
minivan – http://vzyalslidingdoor.blogspot.com
milking machine - greenfusestock.photoshelter.com
rocket scientists – LIFE magazine
innovation – http://abovethecrowd.com
problem solving – http://creativerealities.com
equations – http://gallaryhip.com, http://imaginenvision.com
designs – http://www.grabcad.com
Editor's Notes
VERSION: 2.19.2015 8:24 AM
We have BIG CHALLENGES to Tackle:
Global Warming
Keeping asteroids from ending humanity
Food and Water Supplies
VERSION for Non-NASA
In fact… the number of people connected to the Internet is growing…. FAST!!!!
And as the number of people on the internet grows… so does the power of crowdsourcing… and the ability to find the expert or the existing solution!
And many these people are NOT who you think they are… they are YOU! Highly qualified, skilled, motivated people WITH DAY JOBS.
Winners of the one of our recent challenge were engineers at GE.
Cognitive Surplus
Even if a fraction of these new users have the knowledge and experience to provide value… it is still a very large number.
AND…. Of those online… increasingly larger numbers are learning that they can engage in their passion to learn, contribute, and earn through communities.
Example: Currently, there are an estimated 16 million software development professionals worldwide. TopCoder is 650K…. Or 4% of all software developers.
How about 2.8 Billion Heads and growing….
But just because 2.8 Billion people are connected does NOT mean that they are organized …. AT ALL… In fact it can be chaos.
In fact… it’s a bit scary what global connectivity has meant to many
Initially people started forming communities using things like bulletin boards… remember those…
But then with the web, companies started understanding various value propositions in helping these communities.
So …. About now, you might be thinking… Isn’t this just going to be a much bigger job of sorting though the haystack to find my winning needle?
(CLICK)
And Now I’m Back... I’ve traded looking for an expert to looking for a winning solution!
But with the right tools and processes is actually more like methodical mining than picking through garbage where you have to process through tons of ore just to extract out a small fraction of value in the form of gold or diamonds.
This is a GROWING INDUSTRY… with some firms that have been around for 10-15 years….
Now moving from “new” and “pilot” phases…. To “operational” usage.
Being used more across the government.
Now moving from “new” and “pilot” phases…. To “operational” usage.
Being used more across the government.
Back in 2011, the White House Office of Science and Technology Policy, saw what Jeff Davis and Jason Crusan had been doing with crowdsourcing pilots and asked NASA if they would stand up a Center of Excellence to help get this infused across all of the federal agencies.
So in 2011, NASA stood up CoECI to do just that.
This included the NASA Tournament Lab or NTL using a Contract with Harvard and their subcontract with TopCoder
It also included NASA’s contracted with InnoCentive to provide both the platform for NASA@work, but also another contract for Challenges using the InnoCentive crowd
It also included a contract with a company called Yet2.com
And more recently, we’ve started working on challenges with GrabCAD using a Government Purchase Card.
-----------------------
We currently have 4 tools (and are working to get more) that are available to EVERY NASA project
NASA@WORK – 2 opportunities… 1 to be a solver and help find solutions (and win prizes) and 2, use this tool on your project to tap into the rest of NASA’s brainpower
NASA Tournament Lab (NTL) – our contract with Harvard and TopCoder – If your problem is getting the right software development expertise & horsepower or if the key to your problem is a complex algorithm, TopCoder’s community of over 650,000 software geeks from around the world may be the answer.
InnoCentive’s NASA Innovation Pavilion – Post your problem as a competitive challenge to a worldwide community of solvers with over 300,000 people. (an amazing track record of solving previously unsolved problems….. 70% of the solutions come from OUTSIDE the discipline of the challenge owner.
GrabCAD
If you are trying to find a particular technology that you think might be out in industry, but you just can’t access that information via Google…. Our Yet2.com platform might be what you need… this is like a Match.com for technology needs. You pose a technology need and they search the globe’s academic and industry base via special agreements with companies.
These tools are not new… some have been around 10-15 years…. However, they are relatively new to NASA, and so CoECI is here to help you tap into them and use them.
But, in addition to what our office is doing with the NASA Tournament Lab ..
(CLICK)
NASA is va;spery active with crowdsourcing & challenges via Centennial Challenges, Citizen Science, and Space Apps (Hackathons)
(CLICK)
To find what NASA is doing with Crowdsourcing and Challenges… go to the NASA Solve website…. It is a one stop shop list of all of the programs and all of the challenges.
Wait… before you roll your eyes!!! Innovation is real…. You’ve seen it…. You’ve experienced it… it is that idea that seems obvious in retrospect… “why didn’t I think of that”…. Its that “different” approach that works out spectacularly… unexpectedly…
We are usually looking for ideas…. Maybe “out of the box” thinking…. Maybe its ideas on how to solve a really hard problem… the kind that may not even have a solution.
For our pilot challenges, NASA challenges got 2900 solvers from 80 countries.
NOTE: InnoCentive membership includes 300,000 solvers from over 200 countries.
Back when community was just 160,000.
Swiss company Roche spent 8.8 billion Swiss francs on R&D in 2008 alone – nearly 20 per cent of its sales.
This includes products used to test blood and tissues to obtain information for early detection, diagnosis, prevention and treatment monitoring of diseases.
Inside the company….people could see enormous potential in increased collaboration, but they didn’t know how to do it”. The number of hierarchical levels, a lack of resources, and pressure to deliver were all mentioned as major obstacles.
(Mars 2020 mission is considering a late change to include this in the design)
Solver is from Rising Star, TX (population 800)…. How would we have found this idea if we were not using challenges?
For about 5 years now, I have been self-employed as a technical consultant for a wide range of science and technology interests and enterprises.
Master’s degree in Aquatic Biology from Texas State University.
“it sort of came naturally, as a creative process, and I appreciate opportunities to be creative. “
Spent 80-100 hours on his submission
“Participating in this NASA Challenge has encouraged me to participate in 11 other InnoCentive Challenges, in very diverse areas where I’ve really “pushed the envelope” out of my confidence or comfort zone. “
Launched in March of 2014 and in 10.5 months got an algorithm that is 15% better at detecting asteroids (than the standard: catalina sky survey) AND buidlng an app that citizen scientists, hobbyist astronomers and even professional organizations/institutions can download and use.
The can feed their images into the algorithm and if an asteroid is detected, they can automatically upload their findings to the pros that are officially tracking asteroids for the planet! All for less than $200K… a bit less than the cost of a single full time engineer for 10.5 months.
<Go through the list>
Additionally in other agencies:
EPA – Mobile app for Cyanobacteria, Toxcast
DOE – Wave Energy
CMS – A number of apps
OPM – Payment app
DoD – Possible Exoskeleton
DOJ – Possible crime pattern algorithm
<Go through the list>
Additionally in other agencies:
EPA – Mobile app for Cyanobacteria, Toxcast
DOE – Wave Energy
CMS – A number of apps
OPM – Payment app
DoD – Possible Exoskeleton
DOJ – Possible crime pattern algorithm
There is no validated tool to quantify intracranial pressure non-invasively.
Clinical Gold Standard methods are invasive
Lumbar puncture
Cranial implant
Accurate pre-, in-, and postflight ICP measurements are required to prove or disprove the intracranial pressure hypothesis and quantify the extent and time course of ICP changes.
Techwatch and market surveys indicated that the current state of technology was insufficient to meet NASA’s research needs.