This document summarizes the DICE Horizon 2020 project, which aims to develop methods and tools for quality-aware development of data-intensive cloud applications. The project has a budget of 4 million euros over 3 years with 9 academic and industry partners across Europe. It addresses challenges in ensuring software quality for big data applications involving technologies like Hadoop, Spark, and cloud infrastructure. The DICE project will develop a UML profile and quality-aware modeling approach, as well as analysis, simulation, and verification tools to help reason about quality aspects during development. It will also produce an integrated development environment and deployment/delivery tools to support the overall methodology.

1. DICE Horizon 2020 Project
Grant Agreement no. 644869
http://www.dice-h2020.eu Funded by the Horizon 2020
Framework Programme of the European Union
DICE: Developing Data-Intensive
Cloud Applications with Iterative
Quality Enhancements
Giuliano Casale
Imperial College London
Project Coordinator

2. DICE RIA - Overview
DICE Project
o Horizon 2020 Research & Innovation Action
 Quality-Aware Development for Big Data applications
 Feb 2015 - Jan 2018, 4M Euros budget
 9 partners (Academia & SMEs), 7 EU countries
2©DICE 11/9/2015

3. o Software market rapidly shifting to Big Data
 32% compound annual growth rate in EU through 2016
 35% Big data projects are successful [CapGemini 2015]
o European call for software quality assurance (QA)
 ISTAG: call to define environments “for understanding the
consequences of different implementation alternatives (e.g.
quality, robustness, performance, maintenance, evolvability, ...)”
o QA evolving too slowly compared to the trends in
software development (Big data, Cloud, DevOps ...)
 Still crucial for competiveness!
DICE RIA - Overview
Motivation
3©DICE 11/9/2015

4. Platform-Indep.
Model
Domain
Models
DICE RIA - Overview
Quality-Aware MDE Today
4©DICE 11/9/2015
QA
Models
Architecture
Model
Platform-Specific
Model
Code
generation
C#JavaC++
Platform
Description
MARTE
Analytical Models
Cost-Quality Models

5. DICE RIA - Overview
Challenge 1: QA for Big Data
o 5Vs:
o Volume,
o Velocity,
o Variety,
o Veracity,
o Value
o Problem: today no QA toolchain can reason on the
quality of complex Big Data applications
o Heteregeous Big Data Technologies
o NoSQL, Spark, Hadoop/MapReduce, Storm, CEP, ...
o Cloud infrastructure adds complexity
o Cloud storage, auto-scaling, private/public/hybrid, ...
5©DICE 11/9/2015

6. DICE RIA - Overview
Challenge 2: Embracing DevOps
6©DICE 11/9/2015
o QA must become lean as well
 Continuous quality checks and model versioning
o Modelling of the operations
 Dev needs awareness of infrastructure and costs
o Continuous feedback
 Forward and backward model synchronisation
 Tracking of self-adaptation events (e.g. auto-scaling)
o Big data coming from continuous monitoring
 QA has its own Big data, use machine learning?

7. Platform-Indep.
Model
Domain
Models
DICE RIA - Overview
An Holistic Approach: DICE
7©DICE 11/9/2015
Continuous
Validation
Continuous
Monitoring
Data
Awareness
Architecture
Model
Platform-Specific
Model
Platform
Description
DICE MARTE
Deployment &
Continuous
Integration
DICE IDE
Big Data
QA
Models

8. DICE RIA - Overview
Benefits
o Tackling skill shortage and steep learning curves
 Data-aware methods, models, and OSS tools
o Shorter time to market for Big Data applications
 Cost reduction, without sacrificing product quality
o Decrease development and testing costs
 Select optimal architectures that can meet SLAs
o Reduce number and severity of quality incidents
 Iterative refinement of application design
8©DICE 11/9/2015

9. DICE RIA - Overview
DICE QA: Quality Dimensions
o Reliability
o Efficiency
o Safety &
Privacy
9©DICE 11/9/2015
 Risk of harm
 Privacy & data protection
 Performance
 Time behaviour
 Costs
 Availability
 Fault-tolerance

10. Footer
DICE Platform Independent Model (DPIM)
10©DICE 11/9/2015

11. DICE RIA - Overview
DICE Profile: PIM Level
o Functional approach to data to be expanded
o Data dependencies
 graph relationships between data, archives and streams
o QA focuses on quantitative aspects of data
o Static characteristics of data
 volumes, value, storage location, replication pattern,
consistency policies, data access costs, known schedules of
data transfers, data access control / privacy, ...
o Dynamic characteristics of data
 cache hit/miss probabilities, read/write/update rates,
burstiness, ...
11©DICE

12. Footer
DICE Platform and Technology Specific Model (DTSM)
12©DICE 11/9/2015

13. Footer
DICE Platform, Technology and Deployment Specific
Model (DDSM)
13©DICE 11/9/2015

14. DICE RIA - Overview
DICE Profile: PSM Level
o Need for technology-specific abstractions
 Hadoop: Number of mappers and reducers , ...
 In-memory DBs: Peak memory and variable threading
 Streaming: merge/split/operators, networking, ...
 Storage: Supported operations, cost/byte , ...
 NoSQL: Consistency policies , ...
o Generation of deployment plan
 Proposed Chef + TOSCA extension
o Interest is both on private and public clouds
 Private clouds more relevant for batch processing
 Public clouds more relevant for streaming
14©DICE

15. DICE RIA - Overview
Demonstrators
15©DICE 11/9/2015
Case study Domain Features & Challenges
Distributed data-
intensive media
system (ATC)
• News & Media
• Social media
• Large-scale software
• Data velocities
• Data volumes
• Data granularity
• Multiple data sources and channels
• Privacy
Big Data for e-
Government
(Netfective)
• E-Gov
application
• Data volumes
• Legacy data
• Data consolidation
• Data stores
• Privacy
• Forecasting and data analysis
Geo-fencing
(Prodevelop)
• Maritime
sector
• Vessels movements
• Safety requirements
• Streaming & CEP
• Geographical information

16. DICE RIA - Overview
Thanks!
www.dice-h2020.eu
16©DICE

17. DICE RIA - Overview
Challenge 2: Embracing DevOps
o Software development process is evolving
 Developer: “I want to change my code”
 Operator: “I want systems to be stable”
o...but code changes are the cause of most instabilities!
o DevOps closes the gap between Dev and Ops
 Lean release cycles with automated tests and tools
 Deep modelling of systems is the key to automation
17©DICE 11/9/2015
Agile
Development
DevOps
Business Dev Ops

18. DICE RIA - Overview
Main Technical Outputs
1. DICE Profile (WP2)
 New UML profile to characterize data location, processing,
transformation, and usage
 Data-aware quality annotations
 Deployment models (output to TOSCA)
2. QA Tools (WP3/WP4)
 OSS tools (analysis, simulation, verification, feedback)
3. Integrated Development Environment (WP1)
 Guides through the DICE methodology
4. Delivery Tools (WP5)
 Deployment, continuous integration, testing
18©DICE 11/9/2015

19. DICE RIA - Overview
DICE QA: Possible Baselines
 UML MARTE
 Performance
 Timing
 Verification
 MODACloudML
 Cloud/PMI
 Not UML
 UML DAM
 Dependability/ZAR, covers our quality dimensions
19©DICE
UML DAM core package

20. DICE RIA - Overview
Year 1 - Expected Achievements
20©DICE 11/9/2015
Milestone Deliverables
Baseline and
Requirements -
July 2015
• State of the art analysis
• Requirement specification
• Dissemination, communication,
collaboration and standardisation report
• Data management plan
Architecture
Definition -
January 2016
• Design and quality abstractions
• DICE simulation tools
• DICE verification tools
• Monitoring and data warehousing tools
• DICE delivery tools
• Architecture definition and integration plan
• Exploitation plan