Más contenido relacionado La actualidad más candente (18) Similar a Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop (20) Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop1. Next-Generation BPM –
How to create intelligent Business Processes
thanks to Big Data
Talend, Global Leader in Open Source Integration Solutions
Kai Wähner
kontakt@kai-waehner.de
@KaiWaehner
Xing / LinkedIn
www.kai-waehner.de
2. Kai Wähner
Main Tasks
Requirements Engineering
Enterprise Architecture Management
Business Process Management
Architecture and Development of Applications
Service-oriented Architecture
Integration of Legacy Applications
Cloud Computing
Big Data
Contact
Consulting
Developing
Coaching
Speaking
Writing
© Talend 2013
Email: kontakt@kai-waehner.de
Blog: www.kai-waehner.de/blog
Twitter: @KaiWaehner
Social Networks: Xing, LinkedIn
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
3. Key messages
BPM should be used (just) for optimizing business processes!
Intelligent business processes need big data and integration!
Big data will reduce human interactions in BPM further!
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
4. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
5. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
6. Why should you care about big data?
“If you can't measure it,
you can't manage it.”
William Edwards Deming
(1900 –1993)
American statistician, professor,
author, lecturer and consultant
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
7. Why should you care about big data?
„Silence the HiPPOs“ (highest-paid person‘s opinion)
Being able to interpret unimaginable large data
stream, the gut feeling is no longer justified!
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
8. Why does big data exist?
Changing Scale
Sensors
Changing Expectations
Cloud
Changing Interactions
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
9. Three shifts in the way we analyze information
• Messiness: Using ALL data, not just samples
• Also bad data (e.g. Word spell checker, Google autocomplete and „did you mean...“ recommendation
• Correlations: Instead of causalities
• May not tell us WHY something is happening, but THAT it
is happening
• In many situations, this is good enough
• What drug substance cures cancer? When should I buy an
airplane ticket?
• Datafication: Store, process, combine, reuse,
enhance all data!
• Digitalisation (Amazon Kindle Read) vs. Datafication
(Google Books Read, Search, Process, ...)
• Words becomes data: Google books: not just read, but
also search, analyse, etc.
• Locations becomes data: GPS: not just navigation, but
also insurance costs, economic routes, etc.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
10. The Vs of big data
Volume
(terabytes,
petabytes)
Velocity
(realtime or nearrealtime)
Variety
(social networks,
blog posts, logs,
sensors, etc.)
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Value
11. Big data tasks to solve - before analysis
Big Data Integration
– Land data in a Big Data cluster
– Implement or generate parallel processes
Big Data Manipulation
– Simplify manipulation, such as sort and filter
– Computational expensive functions
Big Data Quality & Governance
– Identify linkages and duplicates, validate big data
– Match component, execute basic quality features
Big Data Project Management
– Place frameworks around big data projects
– Common Repository, scheduling, monitoring
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
12. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
13. Storage Reduce costs
Global Parcel Service
A lot of data must be stored „forever“
➜ Numbers increase exponentially
➜ Goal: As cheap as possible
➜ Problem: (Fast) queries must still be possible
➜ Solution: Commodity servers and „Hadoop querying“
➜
http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
14. Replace ETL Improve performance
“The advantage of their new system is that they can now look at their data
[from their log processing system] in anyway they want:
➜ Nightly MapReduce jobs collect statistics about their mail system such as
spam counts by domain, bytes transferred and number of logins.
➜ When they wanted to find out which part of the world their customers
logged in from, a quick [ad hoc] MapReduce job was created and they had
the answer within a few hours. Not really possible in your typical ETL
system.”
http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
15. Risk management Customer success
Deduce
Customer
Defections
http://hkotadia.com/archives/5021
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
16. Flexible pricing Increase revenue
➜
➜
➜
➜
With revenue of almost USD 30 billion and a network of
800 locations, Macy's is considered the largest store operator in the
USA
Daily price check analysis of its 10,000 articles in less than two hours
Whenever a neighboring competitor anywhere between New York
and Los Angeles goes for aggressive price reductions, Macy's follows
its example
If there is no market competitor, the prices remain unchanged
http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
17. Great big data use cases, but ...
➜
➜
➜
➜
How do you put this big data easily in the hands of the people that
need it?
Making the data “actionable” is the real challenge.
Seeing the information that helps make a decision on a composite
dashboard is just the first step and where too many companies stop.
A business must be able to fire off the business process to execute the
decision made regarding the data.
http://smartdatacollective.com/matt-davies/104576/data-driven-bpm-making-big-data-actionable
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
18. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
19. Intelligent business processes
Humans have to interpret large data to make decision.
Using gut feeling is nothing but gambling.
➜ Just doing big data analytics is not enough. Systematic
and monitored human interactions are as important to
get best outcomes.
➜ An intelligent business process
combines big data and BPM. This
enables humans to make data-driven
decisions based on big data analytics.
➜
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
20. Intelligent business processes
➜
Process starts action (PULL Big Data)
• Manual or automated
• Faster responses (e.g. „spam by domain“)
• Better outcomes (e.g. „recommendation engine“)
➜
Data starts action (Big Data PUSH)
• (Usually) automated
• Predictive processes (e.g. „preventing flu epidemic“)
• Handle before it happens (e.g. „customer deduction“)
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
21. Combination of big data and BPM
How are they related?
➜ How to combine?
➜ How to realize this technically?
➜
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
24. Building Blocks for „Intelligent Business Processes“
Integration
• ETL
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing
Big Data
• Processing
• Analytics
BPM
• Do queries to make decisions
• Human or machine
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
25. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
26. Building Blocks for „Intelligent Business Processes“
Integration
• Extract Transform Load (ETL)
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing
Big Data
• Processing
• Analytics
BPM
• Do queries to make decisions
• Human or machine
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
28. Do not write all that “glue code”!
AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStream("AwsCredentials.properties")));
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
try {
s3.createBucket(bucketName);
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix("My"));
s3.deleteObject(bucketName, key);
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
// error handling...
} catch (AmazonClientException ace) {
// error handling...
}
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
29. Integration framework (e.g. Apache Camel)
// Producer
from(“ftp:toS3")
.setHeader(S3Constants.KEY, simple(“order.txt"))
.to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s)
// Consumer
from(„salesforce://orders__c?user=dummy1“)
.filter(„customer == ${dummyCustomer})
.to(“ibm-database:orderData")
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
30. Enterprise Service Bus (e.g. Talend ESB)
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
31. Alternatives for integration
Integration Suite
Enterprise
Service Bus
Integration
Framework
Low
Connectivity
Routing
Transformation
© Talend 2013
High
+
INTEGRATION
Tooling
Monitoring
Support
+
BUSINESS PROCESS MGT.
BIG DATA / MDM
REGISTRY / REPOSITORY
RULES ENGINE
„YOU NAME IT“
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Complexity
of Integration
32. Building Blocks for „Intelligent Business Processes“
Integration
• ETL
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing
Big Data
• Processing
• Analytics
BPM
• Do queries to make decisions
• Human or machine
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
33. Technology perspective
How to process big data?
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
34. How to process big data?
The defacto standard for big data processing
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
35. How to process big data?
“A big part of [the
company’s strategy]
includes wiring SQL Server
2012 (formerly known by
the codename “Denali”) to
the Hadoop distributed
computing platform, and
bringing Hadoop to
Windows Server and Azure”
Even Microsoft (the .NET house) relies on Hadoop since 2011
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
36. What is Hadoop?
Apache Hadoop, an open-source software library, is a
framework that allows for the distributed processing of
large data sets across clusters of commodity hardware
using simple programming models. It is designed to scale
up from single servers to thousands of machines, each
offering local computation and storage.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
37. How to process big data?
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
39. (Near) Realtime?
!
Hadoop cannot solve every big data problem.
Complex event processing and real-time analytics have
to be solved in another way (at least today).
In-memory computing and streaming platforms are good
alternatives or complements to Hadoop for processing
and analyzing big data.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
40. Building Blocks for „Intelligent Business Processes“
Integration
• ETL
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing
Big Data
• Processing
• Analytics
BPM
• Do queries to make decisions
• Human or machine
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
42. BPMN
„Business Process Model and Notation (BPMN) is a graphical
representation for specifying business processes in a business process
model.“ BPMN 2.0 is also executable!
Wikipedia
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
43. Alternatives for BPM
Integration Suite
BPM Suite
BPM
Framework
Low
High
Coding
Service Tasks
Human Interaction
GUI
© Talend 2013
+
BPM
Tooling
Monitoring
Support
+
ESB
BIG DATA / MDM
REGISTRY / REPOSITORY
RULES ENGINE
„YOU NAME IT“
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
Complexity of
Orchestration
44. Building Blocks for „Intelligent Business Processes“
Let‘s
realize
it !!!
Integration
• ETL
• Connectivity / adaptors to connect to external
systems using a variety of different protocols
• Predefined EIP for message routing
Big Data
• Processing
• Analytics
BPM
• Do queries to make decisions
• Human or machine
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
45. Frameworks vs. Tools
Suite of Tools
Specific Tools
Frameworks
Low
High
e.g.
Camel (Integration)
Hadoop (Big Data)
Activiti (BPM)
© Talend 2013
e.g.
Mule ESB (Integration)
MapR (Big Data)
Camunda (BPM)
e.g.
Talend Unified Platform
i.e ALL-IN-ONE
(Integration, Big Data, BPM)
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
Complexity of
Orchestration
46. Custom combination of integration, big data and BPM?
• A lot of glue code
• Testing
• Bugfixing
• No support
Some other people already had
the problems you would have!
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
Kai Wähner
47. Agenda
• Big data paradigm shift
• Use cases for big data
• Intelligent business processes
• Technology and product perspective
• Implementation of an use case
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
48. Flexible pricing Increase revenue
➜
➜
➜
➜
With revenue of almost USD 30 billion and a network of
800 locations, Macy's is considered the largest store operator in the
USA
Daily price check analysis of its 10,000 articles in less than two hours
Whenever a neighboring competitor anywhere between New York
and Los Angeles goes for aggressive price reductions, Macy's follows
its example
If there is no market competitor, the prices remain unchanged
http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
49. Implementation of an use case
Suite of Tools
Specific Tools
Frameworks
Low
High
e.g.
Camel (Integration)
Hadoop (Big Data)
Activiti (BPM)
© Talend 2013
e.g.
Mule ESB (Integration)
MapR (Big Data)
Camunda (BPM)
Complexity of
Orchestration
e.g.
Talend Unified Platform
i.e ALL-IN-ONE
(Integration, Big Data, BPM)
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
50. Talend Unified Platform
Big Data
Data
Quality
Data
Integration
MDM
ESB
BPM
Commercial license
Subscription model
Support included
Open source license
Free of charge
Big
Data
Data
Quality
Data
Integration
MDM
ESB
Optional support
Based on open source
projects such as Eclipse
or Apache Camel, CXF,
Hadoop
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
51. Example (Talend): Integration
Connect to data sources from competitors, for example via REST service,
Twitter API, or custom scripts.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
52. Example (Talend): Big Data Processing
Move data to HDFS for processing, as your classic servers and data warehouses are
not able to process this semi-structured data fast enough (and cheap), probably.
Manipulate the data, in other words, filter relevant information, sort it, and
compare it to prices of your products.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
53. Example (Talend): Business Process
Start a new instance of a business process to review the result and continue
with further tasks, such as calling a web service which does the price reduction
in selected locations. Reviews can be done by human interaction or via
automated tasks depending on the proposed price reduction.
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
54. Implementation of an use case
„Talend Unified Platform“ in action...
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
55. Did you get the key message?
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
56. Key messages
BPM should be used (just) for optimizing business processes!
Intelligent business processes need big data and integration!
Big data will reduce human interactions in BPM further!
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
57. Did you get the key message?
© Talend 2013
“How to create intelligent Business Processes thanks to Big Data” by Kai Wähner
58. Thank you for your attention. Questions?
KAI WÄHNER
kontakt@kai-waehner.de
www.kai-waehner.de
LinkedIn / Xing
@KaiWaehner