Some of the challenges and oppurtunities

1. SOA & Big data
Arnon
Rotem-­‐Gal-­‐Oz

2. Sept
2012
–
iOS6
launched
with
new
maps
applica>on

3. hEp://theamazingios6maps.tumblr.com/
But
something
went
terribly
wrong….

4. • It
isn’t
just
about
geKng
all
the
data
there
• Algorithms
are
cool
but
we
need
humans
in
the
loop
• Hire
the
right
people
• Test
!
Test
!
Test!
hEp://theamazingios6maps.tumblr.com/

5. hEp://theamazingios6maps.tumblr.com/
It
isn’t
just
one
pile
of
data

6. Integra>ng
Big
data
&
SOA
Yoel
Ben
Avraham
-­‐
hEp://www.flickr.com/photos/epublicist/3546059144/

8. Data
Refinery
Ofer
Berger
hEp://www.haifacity.com/allsites/allpic/a/A1738/A1738Pic3326.jpg

10. ETL
integra>on
Department
DB
integra>on
Server
File-­‐based
integra>on
Online
integra>on
DB

11.
Object
soup
ASB BLT
AFT TGI FRY
ECP
HDL
SWG
DRW MFP
WCP
QYD DLY SKD
DLY
XPS
WIU
KYF
XOI ZIS CUI WKD
WHR
ASB GEX
RMO JIA
HCO
BST VUH KFC AJT FQA DKE

12.
Services
ASB BLT
AFT TGI FRY
Customer
HDL
SWG
Orders
ECP
DRW MFP
WCP
QYD DLY SKD
DLY
XPS
WIU
KYF
XOI ZIS CUI WKD
Promotions
WHR
ASB
Invoices
RMO
HCO
JIA GEX
BST VUH KFC AJT FQA DKE

13. Adheres
to
Policy
Governed
by
Binds
to
Endpoint
Exposes
Serves
Service
consumer
Understands
Contracts
Implements
Service
Describes
Key
Component
Sends/receives Messages Sends/receives Rela>on

14. Customer
Interac>ons
Agents
Categories

15. Integra>ng
Big
data
&
SOA
Yoel
Ben
Avraham
-­‐
hEp://www.flickr.com/photos/epublicist/3546059144/

16. Saga
Key
SOA
component
PaEern
component
Rela>on
Concern/aEribute
Prepare/commit/undo
Protocol
Perform
acDvity
Register
RegistraDon
Compensate
Prepare
/
Coordinator*
commit
/
Perform
undo
acDvity
AcDviDes
and
replies
Par>cipator
Compensate
Create
AcDviDes
and
replies
context
Ini>ator
Service
consumer
Service

17. HCatalog HBase
Data
Management
HBase Interactions
HBase
Customer HBase
Interaction ETL
Recordings
HBase
NIM Raw Categories
(HDFS)
Resolved
Interactions(H
DFS)
Resolved
Interactions(H
DFS)
Hadoop Cluster

18. So,
what’s
the
problem
?

19. &
Big
data
can’t
move

20. Performance
of
joins
in
distributed
system
sucks!
Interactions Interactions
Interactions 0-99 200-299
100-199
customers A-H customers I-M customers N-Z
Node 1 Node 2 Node 3
{”Interac>on":
{
"id":
”5",
”par>cipants":
{
”customer":
[
{”surname":
”McDonalds",
”name":
”Old"},]
}
}}

21. Cookie
cuEer
scalability

22. Cell
architecture
Node
2
Node
3
Node
1
Node
N

23. Cell
Architecture
HBase HBase HBase
Categories Customers ORCA
BUS
Interactions Reference …
Data
HDFS HBase HBase

24. Orchestra>on
Initiate business process
Endpoint
Manage
process
Service
Schedule
Route
request Invoke
servic
Workflow instance
es
Workflow Host
Manage
Monitor
Endpoint
engine workflows
workflows
workflows
Service

25. Map
Reduce
processing
pipeline
Customers
Local cache
Retrive segment
Categorize Update Segment InteractionID, Segment Row
Segment Row data - create Resolve
segment Customer IDs Segment document
document (Custoemr) (Categorization) (Interaction)
(Interaction)
Map pipeline
Write
Write
Categories
Interaction
Results
(interaction)
(Categorization)
Map
Interaction & Update
Categorize Prepare data
Segments Interaction
Interaction mart Export
document
(Categorization) (Datamart)
(Interaction)
Reduce pipeline
Write
Write Write
Categories
Interaction Interaction
Results
(interaction) (interaction)
(Categorization)
Reduce
Hadoop Map/Reduce

26. Map
Reduce
processing
pipeline
Customers
Local cache
Retrive segment
Categorize Update Segment InteractionID, Segment Row
Segment Row data - create Resolve
segment Customer IDs Segment document
document (Custoemr) (Categorization) (Interaction)
(Interaction)
Map pipeline
Write
Write
Categories
Interaction
Results
(interaction)
(Categorization)
Map

27. Data
Facets

28. Memcached
GigaSpaces
Redis
GridGain
Caching Data grid
Oracle
Coherence
Columnar
WebSphere
eXtreme
Scale
Hama
HBase
Cassandra
In-memory
Pregel
Accumulo
Hypertable
Key-value store Neo4j
Graph
Hadoop
GlusterFS
Distributed file systems
RavenDB
ScaleBase
MongoDB
Relational
Document
CouchDB
NewSQL
IndexTank
Amazon
RDS
Analytics/MPP
VoltDB
Apache
Solr
AKvio
Aster
Data
Microsoo
PDW
Indexing Columnar
ParAccel
SAP
HANA
Oracle
Exadata
HP
Ver>ca
IBM
Netezza
EMC
Greenplum

29. Data
is
mul>-­‐>ered
Datamart(s)
Cube
Real-­‐>me
Datawarehouse
(RDBMS)
(MOLAP)
(in
memory)
(Hadoop/Hbase)
1-­‐7
days
detailed
6-­‐12
months
6-­‐?
Months
Detailed
aggregated
20
years
detailed
1-­‐3
years
aggregated
aggregated

30. Data
is
mul>-­‐>ered
Datamart(s)
Real-­‐>me
Data
warehouse
(Columnar)
(Hadoop/Hbase)
1-­‐7
days
6-­‐12
months
detailed
Detailed
20
years
detailed
aggregated

31. SOA
leaves
us
with
a
lot
of
isolated
data

32. Aggregated
Repor>ng
SQL endpoint
Endpoint
Produce
Request
reports
ODS/DM
Report Report
Raw
data
Out
Transpose
Endpoint
Pull data
Ingest
Join
Subscribed/ Load
Landing
area
Clean
pulled data
Transform
Service Data backend
SQL endpoint

33. 5
Report
service
Views
2 Raw
data
Load
DW/ODS
1 service
4
2
3
Transforma>on
service
Landing
1

34. 3
Report tool
Drill through
7
REST API 5
9 6
8 10
HBase 2 Aggregation 4
map/reduce
2
1
ETL Details Aggregates
Raw data (map/reduce
(HDFS) +ETL) Data mart

35. Take
aways
SOA
&
Big
data
are
beEer
together

36. Arnon
Rotem-­‐Gal-­‐Oz
arnonr@nice.com
hEp://www.nice.com
hEp://arnon.me/soa-­‐paEerns
arnon@rgoarchitects.com
@arnonrgo
hEp://arnon.me