Combining software-based and human-based services is crucial for several complex problems that cannot be solved using software-based services alone. In this paper, we present novel methods for modeling and developing
hybrid compute units of software-based and human-based services. We discuss high-level programming elements for different types of software- and human-based service units and their relationships. In particular, we focus on novel pro-
gramming elements reflecting hybridity, collectiveness and adaptiveness properties, such as elasticity and social connection dependencies, and on-demand and pay-per-use economic properties, such as cost, quality and benefits, for complex problem solving. Based on these programming elements, we present programming constructs and patterns for building complex applications using hybrid ser-
vices.
Augmenting Complex Problem Solving with Hybrid Compute Units
1. Augmenting Complex Problem Solving
with Hybrid Compute Units
Hong-Linh Truong, Hoa Khanh Dam, Aditya Ghose, Schahram
Dustdar
Distributed Systems Group, Vienna University of Technology
University of Wollongong
truong@dsg.tuwien.ac.at
http://dsg.tuwien.ac.at/research/viecom
WESOA@ICSOC 2013, 2.12.2013
1
2. Outline
Motivation
Programming hybrid, collective adaptive systems in
the cloud
Hybrid compute units
Fundamental programming elements and constructs
Framework
Illustrating examples
WESOA@ICSOC 2013,
2.12.2013
2
3. Hybridity and
Diversity of
Computing Models
Human-based
Computing
Things-based
computing
Architecture
Processing
Unit
Machine-based
Computing
S. Dustdar, H. Truong, “Virtualizing Software and Humans
for Elastic Processes in Multiple Clouds – a Service
Management Perspective”, in International Journal of Next
Generation Computing, 2012
Today’s complex problems need
Ad hoc networks
Web of things
hybridity and diversity-aware “collective
SMP
adaptive systems” (HDA-CAS)
Comm.
Grid
WESOA@ICSOC 2013,
2.12.2013
3
4. Motivation -- Dynamic hybridity in a
collective
Mixture of different types of resources working in
concert in the same collective
Mixture of different roles performed in the same
collective
Different functions based on different capabilities
Mixture of different quality from a single collective
Functions are performed under different quality
Resources possible offer different quality wrt their
role
Mixture of cost/benefit models
A collective might perform a function with different
costs/qualities
WESOA@ICSOC 2013,
2.12.2013
4
5. Motivation – HAD-CAS
Challenges/difficulties in programming HDACAS in the cloud
Human-in-the-loop
Elasticity of costs, quality/benefits and resources
Unified framework for humans and software
Cross-platform with complex underlying APIs
Open problem: there is a lack of suitable programming
frameworks
WESOA@ICSOC 2013,
2.12.2013
5
6. Our goals
Construct hybrid collective adaptive systems
Consist of service units abstracting software, people
and things
With different service units roles (computation, data,
management)
Be elastic in multi-cloud environments
Provide common programming features/APIs that
enable different coordination/composition ways for
CASs
Our approach: hybrid compute units for collective adaptive
systems (hCAS)
WESOA@ICSOC 2013,
2.12.2013
6
7. Unified service unit model for
capturing hybridity in a collective
Consumption,
ownership, provisioning,
price, etc.
Service
model
Elastic
Service
Unit
Function
The functional
capability of the unit
and interface to
access the function
Unit
Dependency
Modeling type of
units (e.g.,
computation, data,
monitor,) and their
dependencies
Elastic
Capability
Capabilities to be elastic
under different
requirements
WESOA@ICSOC 2013,
2.12.2013
7
Software
VolunteersProfessionals
Thing
People
Resources
8. Hybrid compute units for CAS
Hybrid compute unit (HCU): a set of service units
includes software-based services, human-based
services and things-based services that can be
provisioned, deployed and utilized as a collective
on-demand based on different quality, pricing and
incentive models.
This paper focuses on abstracting service units and
relationships to provide fundamental programming elements
for HCU
WESOA@ICSOC 2013,
2.12.2013
8
9. Hybrid compute unit design –
fundamental elements
WESOA@ICSOC 2013,
2.12.2013
9
10. Hybrid compute unit design -Relationships
Relationship Type
HBS
SBS
TBS
HCU
Similarity
Yes
Yes
Yes
Yes
Composition
Yes
Yes
Yes
Yes
Data Dependency
Yes
Yes
Yes
Yes
Control Dependency
Yes
Yes
Yes
Yes
Location Dependency
Yes
Yes
Yes
Yes
Forwarding
Yes
Yes
No
Yes
Delegation
Yes
Yes
No
Yes
Social Relation
Yes
No
No
Yes
Elasticity
Yes
Yes
No
Yes
WESOA@ICSOC 2013,
2.12.2013
10
11. Hybrid compute unit design – highlevel programming constructs
Current prototype: Fundamental elements can
be implemented as classes in high-level
programming languages
Service units and their properties
Relationship constructs
But complex runtime is still open
Relationships require complex code libraries for
different templates
Complex integration with existing clouds
WESOA@ICSOC 2013,
2.12.2013
11
14. Illustrating Example
Evaluating quality of data (QoD) in
simulations/data analytics processes
Typical components in simulation workflows/data
analytics
preprocessing
solving
postprocessing
data
analysis
Goal using hCAS to support quality of data
control of the simulations/data analytics
Reasons: reducing wasting time and computational
cost, dealing with tacit quality evaluation, etc.
WESOA@ICSOC 2013,
2.12.2013
14
15. hCAS and simulations
Only SBS is needed,
for example, in the QoD evaluation step
before pre-processing
SBS or HBS is used interchangeably or in
combination
for example, in the QoD evaluation after preprocessing,
Only HBS is used
e.g., in QoD evaluation after solving.
WESOA@ICSOC 2013,
2.12.2013
15
17. Excerpt -- Programming elasticity
and collectiveness in solving steps
SBS preprocessingUnit =new SBS() ; //create an instance of software unit
File data =new File(fileName);
//...
//create another unit for QoD evaluation
SBS qodEvalUnit = new SBS();
ArrayList paraType = new ArrayList();
paraType.add(File.class.getName());
//the unit must support qodEvaluate function which returns a value in [0,1]
qodEvalUnit.setFunction("qodEvaluate",paraType, Double.class.getName());
//...
//call qod evaluation unit
Double result =(Double)qodEvalUnit.execute("qodEvaluate",params);
…
//call preprocessing activity if QoD is satisfied
if (result > 0.9) {
preProcessedData=preprocessingUnit.execute("preprocessing",params);
}
WESOA@ICSOC 2013,
2.12.2013
17
18. Excerpt -- Programming elasticity
and collectiveness in solving steps
if (qodPreProcessedData < 0.5) {
//initiate a new unit
ICU dataScientist = new ICU();
//create a dropbox place for sharing data
DropboxAPI<WebAuthSession> scuDropbox = null;
// ....
DropboxAPI.DropboxLink link = scuDropbox.share("/hbscloud");
//ask the cloud of HBS to invoke the ICU
VieCOMHBS vieCOMHBS = new VieCOMHBSImpl();
vieCOMHBS.startHBS(dataScientist);
HBSMessage msg = new HBSMessage();
msg.setMsg("pls. use shared dropbox for communication " +link.url);
vieCOMHBS.sendMessageToHBS(dataScientist, msg);
}
WESOA@ICSOC 2013,
2.12.2013
18
19. Excerpt – Programming elasticity
and collectiveness in solving steps
if (qodPreProcessedData < 0.9) {
//specify some static properties of the solver
SBS solverUnit2 = new SBS("solver");
solverUnit2.function.put("DIRTY_DATA", Boolean.valueOf(true));
//specify expected cost and accuracy support
CostModel costModel = new CostModel();
costModel.price = 100; //max in EUR
costModel.usageTime = 1000 * 60 * 60; //1 hour
Quality quality = new Quality();
quality.name = Quality.ACCURACY;
quality.value = 0.95; // minimum value
ArrayList nfps = new ArrayList();
nfps.add(quality); nfps.add(costModel);
//find solvers met quality and cost needs
SBS elasticSolverUnit = (SBS)Relationship.elasticity(solverUnit2,nfps);
Object solverResult2 = elasticSolverUnit.execute("solving", params1);
}
WESOA@ICSOC 2013,
2.12.2013
19
20. Excerpt – Programming location
dependency
//create a new software unit for simulation solvers
SBS solverUnit = new SBS("solver");
//make sure the solver unit and the preprocessing unit are in the same
data center
Relationship.locationDependency(preprocessingUnit,solverUnit,"Amaz
onEC2:Europe");
ArrayList params1 = new ArrayList();
params1.add(preProcessedData);
//execute solver unit with input data from preprocessing unit
Object solverResult= solverUnit.execute("solving",params1);
WESOA@ICSOC 2013,
2.12.2013
20
21. Excerpt – Forwarding and
delegating analysis request
ICU dataScientist = new ICU();
//....
ICU fUnit = new ICU();
Relationship.socialrelation(dataScientist,
fUnit,1,"Linkedin:DataScienceGroup/TUWien");
Relationship.forward(data, fUnit);
//...
SCU studentSCU = new SCU();
//..
Relationship.delegate(data, studentSCU);
WESOA@ICSOC 2013,
2.12.2013
21
22. Conclusions and Future Work
Hybrid compute units for implementing hybridity
and diversity-aware collective adaptive systems
(HDA-CAS)
Design fundamental programming elements and
constructs
We are still just at the abstract programming
elements/constructs
Need to work more on mapping from language
construct abstractions to runtime APIs
Runtime system working with several underlying
frameworks for supporting different composition and
coordination techniques
WESOA@ICSOC 2013,
2.12.2013
22