Big Data: Beyond the hype, Delivering value explains Big Data technology and how it is transforming industry and society to members of the IDEAL-IST project. IDEAL-IST is an international ICT (Information and Communication Technologies) network, with more than 65 ICT national partners from EU and Non-EU Countries. It assists ICT companies and research organizations worldwide wishing to find project partners for a participation in the Horizon 2020 program of the European Commission.

1. Big Data:
Beyond the hype, Delivering value
Edward Curry
Insight @ NUI Galway
ed.curry@insight-centre.org
www.edwardcurry.org

2. About Me
Vice President

3. New Horizons for a Data-Driven Economy
A Roadmap for Usage and Exploita>on of Big Data in Europe
Jose Maria Cavanillas (Atos)
Prof. Wolfgang Wahlster (DFKI)
Co-Editors:
3
Open Access PDF
hJp://>ny.cc/NewHorizons
• Provides big picture on how to exploit
big data, including technological,
economic, poliEcal and societal issues
• Details complete lifecycle of big data
value chain, ranging from data
acquisiEon, analysis, curaEon and
storage, to data usage and exploitaEon
• Illustrates potenEal of big data value
within different sectors, including
industry, healthcare, finance, energy,
media and public services
• Summarizes more than two years of
research with wide stakeholder
consultaEon
Overview
Many of the slides today
are based on the work of
the chapter authors

4. Overview
n Part I: What is “Big Data”?
n Part II: Data Driven Innovation:
Big Data is Transforming
Sectors by Breaking Silos and
Driving Ecosystems
n Part III: The Data Value Chain:
Tools and Techniques
n Part IV: The Next Wave of Big
Data Research and Innovation
n Part V: Data Science and Skills
Agenda
n Understand of what is
Big Data and its use
n High-level overview of
key technologies
¨ No formulas or complex
examples
¨ Lots of keywords (Sorry!)
n Feel for the key trends
and issues
Learning Objectives

5. PART I: WHAT IS “BIG DATA”?

7. Definitions of Big Data

8. 09/02/16 8www.bdva.eu
The “V’s” of Big Data
Volume Velocity Veracity Variety Value
Data at Rest
Terabytes to
exabytes of exis>ng
data to process
Data in
Mo>on
Streaming data,
requiring mseconds to
respond
Data in Many
Forms
Structured,
unstructured, text,
mul>media,…
Data in Doubt
Uncertainty due to
data inconsistency &
incompleteness,
ambigui>es, latency,
decep>on
€
€
€
€
€
€ €
€
Data into
Money
Business models can
be associated to the
data
Adapted by a post of Michael Walker on 28 November 2012

9. Isn’t Big Data Just Hype?

10. Big Data

11. Emerging Technologies Hype Cycle 2015
“I would not consider big data to be an
emerging technology…”
- Betsy Burton, Gartner

13. 09/02/16 13www.bdva.eu
PART II: DATA DRIVEN
INNOVATION: BIG DATA IS
TRANSFORMING SECTORS BY
BREAKING SILOS AND DRIVING
ECOSYSTEMS

15. Big Data is transforming Business models

16. Key Enablers
Internet of Things
Availability of Data

17. Key Enablers
Ecosystems Approaches
Open Innovation

18. Technology
Providers
Data Value Chain
Core Value Chain
Extended Value Chain
Big Data Ecosystem
Suppliers of Complementary
Data Products and Services
End-Users of
my End-Users
Direct Data
End-Users
Direct Data
Suppliers
Data Value
Distribution
Channels
Suppliers of
my Data
Suppliers
Co-opetitors
(Competitors and cooperation)
Other Stakeholders and Peripheral Actors
Government Organisations
Regulators
Investors, Venture Capitalist & Incubators
Industry Associations
Data
Marketplace
Standardisation
Bodies
Start-ups and
Entrepreneurs
Researchers
& Academics
Stakeholders in a Big Data Value Ecosystem

19. Legal
Social
EconomicTechnology
Application
Data &
Skills
Big Data Value Ecosystem
Ownership
Copyright
Liability
Insolvency
Privacy
User Behaviour
Societal Impact
Collaboration
Business Models
Benchmarking
Open Source
Deployment Models
Information Pricing
Data-Driven Decision Making
Risk Management
Competitive Intelligence
Digital Humanities
Internet of Things
Verticals
Industry 4.0
Scalable Data Processing
Real-Time
Statistics/ML
Linguistics
HCI/Visualisation
The Dimensions of a Big Data Value Ecosystem
[adapted from Cavanillas et al. (2014)]

20. HEALTH
HEALTH AND
WELLBEING

21. Macro trends driving healthcare needs
Increase in life
expectancy
100% increase 1 out of 2
Shortage of
+ 18 years
0.2 doctors and
2 healthcare
workers per 1,000
people in sub-Saharan
African
German hospitals
makes a loss
90,000+
physicians
in the US by 2020
From 11% in 2000 to 22%
in 2050
of # of people >60
Between 1950 and 2050,
globally
246 million people
with diabetes, increasing to 380 m in 2025
Private Healthcare
growing fastest in
Emerging
Markets

22. Macro trends driving healthcare needs
Increase in life
expectancy
100% increase 1 out of 2
Shortage of
+ 18 years
0.2 doctors and
2 healthcare
workers per 1,000
people in sub-Saharan
African
German hospitals
makes a loss
90,000+
physicians
in the US by 2020
From 11% in 2000 to 22%
in 2050
of # of people >60
Between 1950 and 2050,
globally
246 million people
with diabetes, increasing to 380 m in 2025
Private Healthcare
growing fastest in
Emerging
Markets
Big data needed
to op>mize the
triangle of
healthcare
Cost
Quality Access

23. August, 2014 Philips Research 23 23
Making a difference across the health con>nuum
1,000,000
paEents monitored in their
homes every day
18 petabytes
of imaging study data managed
for healthcare providers
250 million appliances sold each year making homes healthier
Hundreds of thousands
of people tracking their
health with AcEveLink®
Last year 6.5 million
people improved their oral health
with our oral healthcare products
101 million lives
improved globally through
access to diagnosEc X-Ray
275 million
pa>ents
tracked with our
paEent monitors
in 2014
Healthy living Preven>on Diagnosis Treatment Home care

24. August, 2014 Philips Research 24 24
6,000,000
paEents monitored in their
homes every day
18 petabytes
of imaging study data managed
for healthcare providers
250 million appliances sold each year making homes healthier
Hundreds of thousands
of people tracking their
health with AcEveLink®
Last year 6.5 million
people improved their oral health
with our oral healthcare products
101 million lives
improved globally through
access to diagnosEc X-Ray
275 million
pa>ents
tracked with our
paEent monitors
in 2014
Philips Big Health Data

25. August, 2014 Philips Research 25 25
6,000,000
paEents monitored in their
homes every day
18 petabytes
of imaging study data managed
for healthcare providers
250 million appliances sold each year making homes healthier
Hundreds of thousands
of people tracking their
health with AcEveLink®
Last year 6.5 million
people improved their oral health
with our oral healthcare products
101 million lives
improved globally through
access to diagnosEc X-Ray
275 million
pa>ents
tracked with our
paEent monitors
in 2014
Philips Big Health Data

26. August, 2014 Philips Research 26 26
6,000,000
paEents monitored in their
homes every day
18 petabytes
of imaging study data managed
for healthcare providers
250 million appliances sold each year making homes healthier
Hundreds of thousands
of people tracking their
health with AcEveLink®
Last year 6.5 million
people improved their oral health
with our oral healthcare products
101 million lives
improved globally through
access to diagnosEc X-Ray
275 million
pa>ents
tracked with our
paEent monitors
in 2014
Philips Big Health Data

27. BIG
Big Data Public Private Forum
27
DATA POOLS IN HEALTHCARE
MAIN IMPACT BY INTEGRATING VARIOUS AND
HETEROGENEOUS DATA SOURCES
Clinical Data
§ Owned by providers (such as
hospitals, care centers, physicians,
etc.)
§ Encompass any information stored
within the classical hospital
information systems or EHR, such as
medical records, medical images, lab
results, genetic data, etc.
Claims, Cost &
Administrative Data
§ Owned by providers and payors
§ Encompass any data sets relevant for
reimbursement issues, such as
utilization of care, cost estimates,
claims, etc.
Pharmaceutical &
R&D Data
§ Owned by the pharmaceutical
companies, research labs/
academia, government
§ Encompass clinical trials,
clinical studies, population and
disease data, etc.
Patient Behaviour &
Sentiment Data
§ Owned by consumers
or monitoring device
producer
§ Encompass any
information related to
the patient behaviours
and preferences
Health data on the
web
§ Mainly open source
§ Examples are
websites such as
PatientLikeMe,
Linked Open Data,
etc.
Highest Impact
on integrated data sets

28. Big Data is Impacting in All Sectors
Economy Energy Environment Education
Health &
Wellbeing
Tourism Mobility Grovenance

30. Ci>zen Sensors
“…humans as ci,zens on the ubiquitous Web, ac,ng as
sensors and sharing their observa,ons and views…”
¨ Sheth, A. (2009). CiEzen sensing, social signals, and enriching human
experience. Internet Compu,ng, IEEE, 13(4), 87-92.
Air Pollution

31. Crisis Response

33. PART III: THE DATA VALUE CHAIN:
TOOLS AND TECHNIQUES

34. The Big Data Landscape is Complex

35. 35 BIG 318062
BIG
Big Data Public Private Forum
THE DATA VALUE CHAIN
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage
• Structured data
• Unstructured
data
• Event
processing
• Sensor
networks
• Protocols
• Real-time
• Data streams
• Multimodality
• Stream mining
• Semantic
analysis
• Machine
learning
• Information
extraction
• Linked Data
• Data discovery
• ‘Whole world’
semantics
• Ecosystems
• Community data
analysis
• Cross-sectorial
data analysis
• Data Quality
• Trust / Provenance
• Annotation
• Data validation
• Human-Data
Interaction
• Top-down/Bottom-
up
• Community /
Crowd
• Human
Computation
• Curation at scale
• Incentivisation
• Automation
• Interoperability
• In-Memory DBs
• NoSQL DBs
• NewSQL DBs
• Cloud storage
• Query Interfaces
• Scalability and
Performance
• Data Models
• Consistency,
Availability,
Partition-tolerance
• Security and
Privacy
• Standardization
• Decision support
• Predictions
• In-use analytics
• Simulation
• Exploration
• Modeling
• Control
• Domain-specific
usage
Big Data Value Chain

36. 36 BIG 318062
BIG
Big Data Public Private Forum
36 BIG 318062
DATA ACQUISITION OVERVIEW
▶ Process of gathering, filtering and cleaning data before the
data is put in a data warehouse or any other storage solution
on which data analysis can be carried out
Definition
▶ Mainly driven by 4 of 9 Vs
• Volume
• Velocity
• Variety
• Value
Scope
▶ Most data acquisition scenarios
assume high-volume, high-
velocity, high-variety but low-
value data
Key Technology
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage

37. 37 BIG 318062
BIG
Big Data Public Private Forum
37 BIG 318062
END-TO-END ARCHITECTURES
Architectures
▶ Design end-to-end architectures for full data lifecycle
▶ Support for both “Data-at-Rest” and “Data-in-Motion”
▶ Data Hubs and Markets: Hadoop-based solutions tend to
become central integration point for all enterprise data

38. 38 BIG 318062
BIG
Big Data Public Private Forum
38 BIG 318062
DATA ANALYSIS OVERVIEW
Core Techniques
The techniques associated with Big Data Analysis will encompass those
related to data mining and machine learning, to information
extraction and new forms of data processing and reasoning including
for example, stream data processing and large-scale reasoning.
▶ Big Data Analysis is concerned with making raw data which has
been acquired amenable to use
▶ Supports decision making as well as domain specific usage.
Big Data Analysis
▶ Entity summarisation
▶ Data abstraction based on ontologies and communication workflow patters
▶ Recommendations and personal data
▶ Stream data processing
▶ Large scale reasoning & Large scale machine learning
State of the art areas
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage

39. 39 BIG 318062
BIG
Big Data Public Private Forum
39 BIG 318062
THE ROLE OF COMMUNITY IN ANALYSIS
Community Analysis and Collection
§ Number of data collection points can be dramatically increased;
§ Communities are creating bespoke tools for the particular situation and to
handle any problems in data collection (Developer Ecosystem)
§ Citizen engagement is increased significantly
Real-time radiation monitoringCity Noise Levels

40. 40 BIG 318062
BIG
Big Data Public Private Forum
DATA CURATION OVERVIEW
▶ Digital Curation “Selection, preservation, maintenance,
collection, and archiving of digital assets”
▶ Data Curation “Active management of data over its life-cycle”
Definition
▶ Individual Curators
▶ Curation Departments
▶ Community-based (Emerging trend)
Who?
▶ (Semi-)Automated
▶ Crowdsourced Data Management
How?
▶ Accessible
▶ Authenticity
▶ Collaboration
▶ Discoverability
▶ Fitness for Use
Why?
▶ Integrity
▶ Reusability
▶ Security
▶ Sustainability
▶ Trustworthy
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage

41. 41 BIG 318062
BIG
Big Data Public Private Forum
41 BIG 318062
Internal Community
- Domain Knowledge
- High Quality Responses
- Trustable
BLENDING HUMAN AND ALGORITHM
Blended Approaches
▶ Blended human and algorithmic data processing
approaches for coping with data acquisition, transformation,
curation, access, and analysis challenges for Big Data
Analytics &
Algorithms
Entity Linking
Data Fusion
Relation Extraction
Human
Computation
Relevance Judgment
Data Verification
Disambiguation
Better Data
Web Data
Databases
Sensor Data
Programmers Managers
External Crowd
- High Availability
- Large Scale
- Expertise Variety

42. 42 BIG 318062
BIG
Big Data Public Private Forum
RECAPTCHA
n OCR
¨ ~ 1% error rate
¨ 20%-30% for 18th and 19th
century books

43. 43 BIG 318062
BIG
Big Data Public Private Forum
A CROSS-SECTOR TREND…
Telco, Media, & Entertainment
Manufacturing, Retail, Energy & Transport
Public Sector Life Sciences

44. 44 BIG 318062
BIG
Big Data Public Private Forum
44 BIG 318062
DATA STORAGE OVERVIEW
▶ Is responsible for analysing different aspects of storing,
organizing and manipulating of information on electronic
data storage devices
Definition
▶ Data organization and modelling
▶ Basic data manipulations (Create,
Read, Update, Delete - CRUD)
▶ Data compression
▶ Data recovery, concurrency,
consistency, integrity and security
▶ Database systems architecture,
availability and partition tolerance
Key Topics
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage

45. 45 BIG 318062
BIG
Big Data Public Private Forum
BIG DATA STORAGE AS A COMMODITY

46. 46 BIG 318062
BIG
Big Data Public Private Forum
46 BIG 318062
Mathworks
Analytical
Databases
ANALYSIS OF BIG DATA VOLUMES
Towards Integrated Analy>cs
• Integrated Systems
• Single data model
• Potentially higher
performance
• Lower development
complexity
• Separate Systems
• Different data models
• May negatively impact
performances
• Higher development
complexity
DBMS
Data
Management
Analytics
Rasdaman
SciDB Revolution
Analytics
ClouderaRDBMS

47. 47 BIG 318062
BIG
Big Data Public Private Forum
47 BIG 318062
TRADEOFF: SIZE VS. COMPLEXITY

48. 48 BIG 318062
BIG
Big Data Public Private Forum
48 BIG 318062
§ Decision support
§ Descriptive
§ Predictive
§ Prescriptive analysis
§ Data exploration
§ Extends Visualisation to
§ Visual Analytics
§ Key areas include:
§ Industry 4.0 (industrial
internet)
§ Predictive maintenance
§ Smart data and service
integration
DATA USAGE OVERVIEW
▶ Key task of Data Usage is to
support business decisions
▶ Lookup, Learn, Investigate
▶ Exploratory browsing
▶ Search
▶ Analytics
▶ Closely related to Business
Intelligence and Data Mining
technologies, but extending them
▶ Off-line vs. real-time support
▶ Automated decisions
Definition Decision Making
Data
Acquisition
Data
Analysis
Data
Curation
Data
Storage
Data
Usage

49. 49 BIG 318062
BIG
Big Data Public Private Forum
49 BIG 318062
IMPROVING USABILITY
Usability
▶ Lowering the usability barrier for data tools: Users should
be able to directly manipulate the data
▶ Improvement of Human-Data interaction: Enabling experts
& casual users to query, explore, transform, & curate data
▶ Interactive exploration: Big Data generates insights beyond
existing models, new analysis interfaces must support browsing
and modeling (visual analytics)
▶ Convergence within
analytical frameworks
Analytical databases for better
performance and lower
development complexity
(Mahout, Spark, Hadoop/R,
rasdaman, SciDB)

50. PART IV: THE NEXT WAVE OF BIG
DATA RESEARCH AND INNOVATION

51. 09/02/16 51www.bdva.eu
The Big Data Value Strategic Research and
Innovation Agenda (BDV SRIA) defines the
overall goals, main technical and non-technical
priorities, and a research and innovation
roadmap for the European contractual Public
Private Partnership (cPPP) on Big Data Value.
What is the SRIA?
Strategic Research and Innovation Agenda
What is the SRIA?
Version 1.0 was published by BDVA in January 2015
Version 2.0 due this month.
Latest Version
• Built upon inputs and analysis from SMEs and Large
Enterprises, public organisaEons, and research and
academic insEtuEons.
• Mul>ple workshops and consulta>ons took place to ensure
the widest representaEon of views and posiEons
• Approximately 200 organisa>ons and other relevant
stakeholders physically par>cipa>ng and contribuEng.
SRIA is based on strong community involvement

52. 09/02/16 52www.bdva.eu
BDV SRIA Technical Priorities
Data Management
Engineering the management of data
Data Processing Architectures
Optimized architectures for analytics both data at rest and in motion with low latency delivering real-time analytics
Deep Analytics
Deep analytics to improve data understanding, deep learning, meaningfulness of data
Data Protection and Preservation Mechanism
To make data owners comfortable about sharing data in an experimental setting
Data Visualization and User Experience
Enable intelligent visualization of complex information relying on enhanced user experience and usability

53. 09/02/16 53www.bdva.eu
How do semantically annotated unstructured and semi-structured
data without imposing extra-effort to data producers.
How to unlock data silos by creating interoperability standards and
technologies for storing and exchanging of data?
How to improve and assess the data quality from the various
domains?
How to ensure consistent data provenance along the data value
chain?
How do handle the sheer unbound size of data as well as enforcing
consistent quality as the data scales in volume, velocity and
variability?
How to integrate analytics results from two different worlds: the
data and the business processes?
How to bundle and provision data, software and data analytics results
to ensure reuse of intermediate results?
Data Management
Challenges

54. 09/02/16 54www.bdva.eu
How to integrate the processing of data in motion and data at rest,
e.g.
• Real-time Analytics & Stream Processing
• New Big Data-specific parallelization techniques
How to parallelize and distribute analytics tasks in order to cope
efficiently with data in motion? The challenge is to develop complex
analytics techniques at scale and for data in motion in order to extract
knowledge out of the data and develop decision support applications
How to analyze data generated by IoT applications? I.e. how to
develop algorithms for IoT dataflows analytics
How to ensure performance and scalability of the algorithms? I.e.
the performance has to scale by orders of magnitude while reducing
energy consumption with the best effort integration between hardware
and software.
Data Processing Architectures
Challenges

55. 09/02/16 55www.bdva.eu
How to produce predictive and prescriptive analytics results?
i.e.by deep learning techniques and graph mining techniques applied
on extremely large graphs. Contextualization that combines
heterogeneous data and data streams via graphs to improve the
quality of mining processes, classifiers, and event discovery
How to foster the semantic analysis of data? I.e. How to improve
data analysis to provide a near-real-time interpretation of the data
How t o validate content? I.e. How to implement veracity models for
validating content
How to develop new and open analytics frameworks?
How to improve the scalability and processing speed for the
aforementioned algorithms
How to develop advanced business analytics and Intelligence
techniques?
Deep Analytics
Challenges

56. 09/02/16 56www.bdva.eu
How to ensure privacy and data anonymisation as key
requirements for data sharing and exchange?
How to foster differential privacy, private information retrieval,
homomorphic encryption?
How to provide technical means that allow data owners to control
the access and usage of their data?
How to ensuring irreversibility of the anonymisation?
How to develop scalable solutions?
How to preserve data anonymity while ensuring high data quality ?
Data Protection and Anonymisation
Challenges

57. 09/02/16 57www.bdva.eu
Data Visualization
How to present data analytics reports that encompass complex
documents containing a variety of data sources?
How to address the various design challenges in representing
complex information?
Interfaces need to be humane
just-in-time delivery of relevant information
Filtering versus hiding of information
How to enable advanced data visualisation incorporating data
variety?
How to align the user-driven vs. data-driven data access paradigm?
How to develop intuitive interfaces while exploiting the advanced
discovery aspects of Big Data analytics?
Challenges

58. 09/02/16 58www.bdva.eu
Non-Technical Challenges
Skills development
Business Models and
Ecosystems
Policy, Regulation and
Standardization
Social perceptions and
societal implications

59. PART V: Data
Science and Skills

61. The Skills GAP

62. CONCLUSION

63. The Data Landscape
▶ Much of Big Data technology is evolutionary
▶ Old technologies applied in a new context
▶ Volume, Variety, Velocity, Value …
Technology Evolution
Process Revolution
▶ Business process change must be revolutionary
to enable new opportunities
▶ Industry 4.0 (Smart Manufacturing)
▶ Predictive maintenance
▶ Opportunities for data-driven improvements
▶ integration with customer and supplier data
▶ Moving from infrastructure services (IaaS) to
software (SaaS) to business processes (BPaaS) to
knowledge (KaaS)

64. The Data Landscape
▶ The long tail of data variety is a major shift in
the data landscape
▶ Coping with data variety and verifiability are
central challenges and opportunities for Big Data
▶ Cross-sectorial uses of Big Data will open up
new business opportunities
▶ Need for scalable approaches to cope with data
under different format and semantic assumptions
Variety and Reuse

65. Resources on Big-Data

66. QuesEons?

67. Credits
• Members of the Big Project. In parEcular the
leaders and members of the Technical
Working groups and Sectorial Forums.
• Reused Images are credited on each slide
• SRIA Group from the BDVA