This presentation is part of the course "184.742 Advanced Services Engineering" at The Vienna University of Technology, in Winter Semester 2012. Check the course at: http://www.infosys.tuwien.ac.at/teaching/courses/ase/

1. Advanced Services Engineering,
WS 2012
Emerging Dynamic Distributed Systems
and Challenges for Advanced Services
Engineering
Hong-Linh Truong
Distributed Systems Group,
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://www.infosys.tuwien.ac.at/staff/truong
ASE WS 2012 1

2. Outline
 Today‘s Internet Computing
 Some emerging models – properties and issues
 Data provisioning models
 Computational infrastructures/frameworks
provisioning
 Human computation provisioning
 Internet-scale service engineering
 Single service/platform engineering
ASE WS 2012 2

3. Today‘s Internet Computing
 Internet infrastructure and software connect
contents, things, and people, each has different
roles (computation, sensing, analytics, etc.)
Size does Any * access Economic
matter behaviour does matter factors do matter
Things Large-scale Unpredictable On-demand, pay-
interactions workload as-you-go
Big data Scalability Complex
Software People generated contract
Internet infrastructure and Big quantities to
be managed
software
Hard to control
quality
ASE WS 2012 3

4. Today‘s Internet Computing
Social Technologies and
computing computing models
Cloud Service
Computing Computing
Peer-to-
Distributed
Peer
Computing
Computing
converge
Things Emerging forms of
computing
introduces models, systems
Software People and applications
ASE WS 2012 4

5. Discussion time:
WHICH EMERGING FORMS OF
COMPUTING MODELS,
SYSTEMS AND APPLICATIONS
DO YOU SEE?
ASE WS 2012 5

6. Some emerging data provisioning
models (1)
Large (near- • Satellites and environmental/city sensor networks
(e.g., from specific orgs/countries)
) realtime • Machine-to-machine (e.g., from companies)
• Social media (e.g., from people + platform providers)
data
• Open science and engineering data sets
Open data • Open government data
Marketable • Statistics and business data
• Commercial data in general
data
Data are assets
ASE WS 2012 6

7. Some emerging data provisioning
models (2)
A lot A few A lot
Things
Social
Data/Service Platforms
Platforms
Data Profiling Data
and Enrichment Storage APPs
Environtments ...
Data Data
Analytics Query
Infrastructures
....
ASE WS 2012 7

8. Examples of large-scale (near-)
realtime data
ASE WS 2012 8

9. Large-scale (near-)realtime data:
properties and issues
Some properties Some issues
 Having massive data  Timely analytics
 Requiring large-scale, big  Performance and
(near-) real time scalability
processing and storing  Quality control
capabilities  Handle of unknown data
 Enabling predictive and patterns
realtime data analytics  Benefit/cost versus
quality tradeoffs
ASE WS 2012 9

10. Example of open data
ASE WS 2012 10

11. Open data: properties and issues
Some properties Some issues
 Having large, multiple  Fine-grained content
data sources but mainly search
static data  Balance between
 Having good quality processing cost and
control in many cases performance
 Usually providing the
data as a whole set
ASE WS 2012 11

12. Marketable data examples
ASE WS 2012 12

13. Marketable data: properties and
issues
Some properties Some issues
 Can be large, multiple  Multiple levels of
data sources but mainly service/data contracts
static data  Compatible with other
 Having good quality data sources w.r.t.
control contract
 Have strong data contract  Cost w.r.t. up-to-date
terms data
 Some do not offer the
whole dataset
ASE WS 2012 13

14. Emerging computational
infrastructure/platform provisioning
models
 Infrastructure-as-a-Service
 Machine-as-a service
 Storage as a Service
 Database as a Service
 Platform-as-a-Service
 Middleware
 Computational frameworks
 Software Defined Networking
ASE WS 2012 14

15. Examples of Infrastructure-as-a-
Service
Source: Hong Linh Truong, Schahram Dustdar: Cloud computing for small research groups in computational science and engineering: current
status and outlook. Computing 91(1): 75-91 (2011)
Amazon S3 Microsoft Aruze
And more
MongoLab
OKEANOS
ASE WS 2012 15

16. Examples of Platform-as-a-Service
Source: Hong Linh Truong, Schahram Dustdar: Cloud computing for small research groups in computational science and engineering:
current status and outlook. Computing 91(1): 75-91 (2011)
Amazon Elastic MapReduce
And more
StormMQ Globus Online (GO)
ASE WS 2012 16

17. Examples of multiple clouds
Source: Katarzyna Keahey, Mauricio Tsugawa, Andrea Matsunaga, and Jose Fortes. 2009. Sky Computing. IEEE Internet
Computing 13, 5 (September 2009), 43-51. DOI=10.1109/MIC.2009.94 http://dx.doi.org/10.1109/MIC.2009.94
 aaa
SOCloud WS 2011 17

18. Emerging computational
infrastructure/platform provisioning
models– properties and issues
Some properties Some issues
 Rich types of services  On-demand information
from multiple providers management from
 Better choices in terms of multiple sources
functions and costs  APIs complexity
 Concepts are similar but  Cross-vendor integration
diverse APIs
 Data locality
 Strong
dependencies/tight
ecosystems
ASE WS 2012 18

19. Emerging human computation
models
 Crowdsourcing platforms
 (Anonymous) people computing capabilities exploited
via task bids
 Individual Compute Unit
 An individual is treated like „a processor“ or “functional
unit“. A service can wrap human capabilities to support
the communication and coordination of tasks
 Social Compute Unit
 A set of people and software that are initiated and
provisioned as a service for solving tasks
The main point: humans are a computing element
ASE WS 2012 19

20. Examples of human computation
(1)
Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured
social computing. UIST 2011: 53-64
ASE WS 2012 20

21. Examples of human computation
(2)
Source: Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, Andrew McGregor: AutoMan: a platform for integrating human-based
and digital computation. OOPSLA 2012: 639-654
ASE WS 2012 21

22. Examples of human computation
(3)
Source: Muhammad Z.C. Candra, Rostyslav Zabolotnyi, Hong-Linh Truong, and Schahram Dustdar, Virtualizing Software and Human for
Elastic Hybrid Services, Web Services Handbook, (c)Springer-Verlag, 2012.
ASE WS 2012 22

23. Human computation models –
properties and issues
Some properties Some issues
 Huge number of people  Quality control
 Capabilities might not  Reliability assurance
know in advance  Proactive, on-demand
 Simple coordination acquisition
models  Incentive strategies
ASE WS 2012 23

24. Summary of emerging models wrt
advanced service-based systems
Engineering advanced service-
based systems
utilize/consist of
Emerging data
provisioning models
Things
Emerging data
provisioning models
Software People
Emerging computational Emerging human Emerging data
infrastructure/platform computation provisioning
provisioning models models models
ASE WS 2012 24

25. Discussion time:
WHERE ARE
OPPORTUNITIES?
DO I NEED TO TAKE
OPPORTUNITIES? WHY?
ASE WS 2012 25

26. Recall our motivating example (1)
Infrastructure/Internet of Things Internet/public cloud Organization-specific
boundary boundary
Equipment Operation
and Maintenance Emergency
Management
Near Enterprise
Civil protection realtime
analytics Resource
Planning
Building Operation Predictive
data
Optimization analytics
Tracking/Log
istics
Visual
Analytics
Infrastructure
Monitoring
...
Cities, e.g. including:
10000+ buildings
1000000+ sensors Can we combine open government data
with building monitoring data?
ASE WS 2012 26

27. Recall our motivating Can we combine them
with open government
example (2) data?
Soil
moisture
analysis for
Sentinel-1
A lot of input data (L0):
~2.7 TB per day
A lot of results (L1, L2):
e.g., L1 has ~140 MB per
day for a grid of
1kmx1km
Michael Hornacek,Wolfgang Wagner, Daniel Sabel, Hong-Linh Truong, Paul Snoeij, Thomas Hahmann, Erhard Diedrich, Marcela Doubkova,
Potential for High Resolution Systematic Global Surface Soil Moisture Retrieval Via Change Detection Using Sentinel-1, IEEE
Journal of Selected Topics in Applied Earth Observations and Remote Sensing, April, 2012
ASE WS 2012 27

28. Recall our motivating example (3)
Source: http://www.undata-api.org/
Source:
http://www.strikeiron.com/Catalog/StrikeIronServices.aspx
Source: http://docs.gnip.com/w/page/23722723/Introduction-
to-Gnip
ASE WS 2012 28

29. Discussion time:
WHICH OPPORTUNITIES DO
YOU SEE?
ASE WS 2012 29

30. Internet-scale service engineering -
- the elasticity
ASE WS 2012 30

31. Internet-scale service engineering -
- big/near-real time data impact
 Which are data concerns that impact the data
processing?
 How to use data concerns to optimize data
analytics and service provisioning?
 How to use available data assets for advanced
services in an elastic manner?
 What are the role of human-based servies in
dealing with complex data analytics?
ASE WS 2012 31

32. Internet-scale service engineering -
- Steps
Single service/platform engineering
Service units for representing
Provisioning of fundamental Engineering with single
fundamental things, people
service units service units
and software
Understanding Properties/Concerns
Data /Service/Application Monitoring, evaluation and Utilization of data/service
concerns; their dependencies provisioning of concerns concerns
Large-scale, multi-platform services engineering
Identify design units, selection development and
Identify the scale,
platform/application of existing service Integration,
complexity and *city
problems units; Optimization
ASE WS 2012 32

33. Discussion time:
WHAT ARE MISSING?
ASE WS 2012 33

34. Single service/platform engineering
– service unit (1)
 The service model and the unit concept can be applied
to things, people and software
Consumption,
ownership, Service
provisioning, price, etc. model
Service
unit
„basic
component“/“basic
function“ modeling Unit
Concept
and description
ASE WS 2012 34

35. Single service/platform engineering
– service units (2)
Source: Stefan Tai, Philipp Leitner, Schahram Dustdar: Design by Units: Abstractions for Human and Compute Resources for Elastic Systems.
IEEE Internet Computing 16(4): 84-88 (2012)
ASE WS 2012 35

36. Single service/platform engineering
– service unit provisioning
 Provisioning software under services
 Provisioning things under services
 Provisioning human under services
 Crowd platforms of massive numbers of individuals
 Individual Compute Unit (ICU)
 Social Compute Unit (SCU)
1. Mark Turner, David Budgen, and Pearl Brereton. 2003. Turning Software into a Service. Computer 36, 10 (October 2003), 38-44.
DOI=10.1109/MC.2003.1236470 http://dx.doi.org/10.1109/MC.2003.1236470
2. Luigi Atzori, Antonio Iera, and Giacomo Morabito. 2010. The Internet of Things: A survey. Comput. Netw. 54, 15 (October 2010), 2787-2805.
DOI=10.1016/j.comnet.2010.05.010 http://dx.doi.org/10.1016/j.comnet.2010.05.010
3. Dominique Guinard, Vlad Trifa, Stamatis Karnouskos, Patrik Spiess, Domnic Savio: Interacting with the SOA-Based Internet of Things:
Discovery, Query, Selection, and On-Demand Provisioning of Web Services. IEEE T. Services Computing 3(3): 223-235 (2010)
4. Schahram Dustdar, Kamal Bhattacharya: The Social Compute Unit. IEEE Internet Computing 15(3): 64-69 (2011)
5. Hong-Linh Truong, Schahram Dustdar, Kamal Bhattacharya "Programming Hybrid Services in the Cloud", Springer-Verlag, 10th
International Conference on Service-oriented Computing (ICSOC 2012), November 12-16, 2012, Shanghai, China
ASE WS 2012 36

37. Single service/platform engineering
– examples (1)
 Service engineering with a single
system/platform
 Using Excel to access Azure datamarket places
 Using Boto to access data in Amazon S3
 Using Hadoop within a cluster to process local data
 Using workflows to process data (e.g.,
Trident/Taverna/ASKALON)
 Using StormMQ to store messages
ASE WS 2012 37

38. Single service/platform engineering
– examples (2)
ASE WS 2012 38

39. Internet-scale multi-platform
services engineering – required
technologies
Middleware (e.g., Workflows (e.g.,
StormMQ) Trident)
Crowd platforms,
Data services (e.g., human-based service
Azure, S3) platforms(e.g.,
Mturks, VieCOM)
Internet-scale,
Data Multi-platform
Services Billing/Monitoring
analysis/Computation
(e.g.,
services in cluster Engineering for thecurrencycloud)
(e.g., Hadoop)
Software, Things
and People
ASE WS 2012 39

40. Discussion time
WHAT ARE MISSING?
ASE WS 2012 40

41. Exercises
 Read papers mentioned in slides
 Get their main ideas
 Check services mentioned in examples
 Examine capabilities of the mentioned services
 Including price models and underlying technologies
 Examine their size and scale
 Examine their ecosystems and dependencies
 Work on possible categories of single service
units that are useful for your work
 Some common service units with capabilities and
providers
ASE WS 2012 41

42. Thanks for
your attention
Hong-Linh Truong
Distributed Systems Group
Vienna University of Technology
truong@dsg.tuwien.ac.at
http://www.infosys.tuwien.ac.at/staff/truong
ASE WS 2012 42