More Related Content Similar to Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c (20) More from Guido Schmutz (20) Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c1. 2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
2013 © Trivadis
Echtzeitverarbeitung von Twitter Streams
mit Oracle Event Processing (OEP) 12c
Guido Schmutz
DOAG Konferenz 2014
19.11.2014 – 15:00 Raum Kiev
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
1
2. 2013 © Trivadis
Guido Schmutz
• Working for Trivadis for more than 17 years
• Oracle ACE Director for Fusion Middleware and SOA
• Co-Author of different books
• Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
• Member of Trivadis Architecture Board
• Technology Manager @ Trivadis
• More than 25 years of software development
experience
• Contact: guido.schmutz@trivadis.com
• Blog: http://guidoschmutz.wordpress.com
• Twitter: gschmutz
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
2
3. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
3
4. 2013 © Trivadis
Big Data Definition (4 Vs)
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4
+ Time to action ? – Big Data + Event
Processing = Fast Data
Characteristics of Big Data: Its Volume,
Velocity and Variety in combination
5. 2013 © Trivadis
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are generating data, all others are consuming
data
New Model: all of use are generating data, and all of us are consuming
data
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
5
7. 2013 © Trivadis
Internet Of Things – Sensors
are/will be everywhere
There are more devices tapping into the
internet than people on earth
How do we prepare our
systems/architecture for the future?
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
7
Source: CiscoSource: The Economist
8. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
8
9. 2013 © Trivadis
Retrieve Tweets and Visualize
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
9
10. 2013 © Trivadis
Access to Tweets
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
10
Quelle
Source Limitations Cost
Twitter’s Search API 3200 / user
5000 / keyword
180 requests / 15 minutes
free
Twitter’s Streaming API 1%-40% of total volume free
DataSift
none
0.15 -0.20$ /
unit
Gnip none On request
11. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
11
12. 2013 © Trivadis
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
• Full environment for running Java applications
• Module Framework - OSGi
High Throughput
• Hundreds of thousands of
events/second
Event Processing Infrastructure
Easy-to-use development
environment
• Service Framework – Spring DM,
POJO
Multiple-choice VM
• JRockit or WebLogic RealTime
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
12
13. 2013 © Trivadis
Oracle Event Processing Application
Adapter
• Provide a way for EPN to exchange data with components that are external to network
• Either an Event Source (inbound) or an Event Sink (outbound)
Processor
• Contain query code written in CQL and consumes event data from a preceding
channel
• May also generate new events to an output channel
• CQL is an extension of SQL
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
13
14. 2013 © Trivadis
Oracle Event Processing Application
Bean
Bean
• Provide a place for logic written in Java
• Java code can receive events (Event Sink) and/or send events (Event Source)
• Are implemented using the Spring Framework model or the OEP event bean model
Channel
• Two types of channel exist: Stream and Relation
CQL Processor
• Executes Continous Query Language (CQL) queries
• Used to declaratively describe logic (queries) to be applied on the event stream
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
14
15. 2013 © Trivadis
Oracle Event Processing - CQL
CQL is an extension to Structured Query Language (SQL)
• Same keywords and syntax rules, but additional support for streaming data
event conceptually corresponds to a row in a database table, but
• one event is always before or after another in time
• stream is potentially infinite and ever-changing
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
15
Data Query
Result Tuples Result Tuples
Traditional Database CEP Engine
Query
Data
16. 2013 © Trivadis
Oracle Event Processing – Visual Development Tools
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
16
17. 2013 © Trivadis
Oracle Event Processing – Operation & Management
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
17
18. 2013 © Trivadis
Implementation – complete picture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
18
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
#bam12c,2
gschmutz,1
@doag2014,5
#oep12c,1
JMS
JMSTwitter
range 30 seconds
slide 30 seconds
Sentiment
Enricher
Working on my OEP & BAM demo for
the session ”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
19. 2013 © Trivadis
1) Creating a Twitter Adapter
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
19
Twitter
Adapter
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
Twitter
20. 2013 © Trivadis
2) Send Tweets to BAM
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
20
Twitter
Adapter
BAM
Tweet
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
JMSTwitter Sentiment
Enricher
Working on my OEP & BAM demo for
the session ”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
21. 2013 © Trivadis
3) Extract interesting information from Tweet
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
21
Mention
Extractor
Twitter
Adapter
Hashtag
Extractor
Author
Extractor
BAM
Tweet
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
JMSTwitter Sentiment
Enricher
Working on my OEP & BAM demo for
the session ”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
22. 2013 © Trivadis
4) Count occurrences within period
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
22
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
Working on my OEP & BAM
demo for the session
”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014
#oep12c #bam12c http://
bit.ly/1xafUZJ
gschmutz
#doag2014
#oep12
#bam12c,2
gschmutz,1
@doag2014,5
#oep12c,1
JMS
JMSTwitter
range 30 seconds
slide 30 seconds
Sentiment
Enricher
Working on my OEP & BAM demo for
the session ”Echtzeitverarbeitung von
Twitter Feeds …” #doag2014 #oep12c
#bam12c http://bit.ly/1xafUZJ
1
#bam12c
23. 2013 © Trivadis
Implementing in Oracle Event Processing
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
23
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
JMS
JMSTwitter
range 30 seconds
slide 30 seconds
Sentiment
Enricher
24. 2013 © Trivadis
1) Creating Twitter Adapter –
Connecting to Twitter Stream
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
24
25. 2013 © Trivadis
1) Creating Twitter Adapter –
Tweet Event
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
25
26. 2013 © Trivadis
1) Creating Twitter Adapter –
Adapter Factory
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
26
27. 2013 © Trivadis
1) Creating Twitter Adapter –
Assembly
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
27
28. 2013 © Trivadis
1) Creating Twitter Adapter –
Export Adapter to server
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
28
29. 2013 © Trivadis
1) Creating Twitter Adapter –
Using Twitter Adapter
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
29
30. 2013 © Trivadis
2) Sentiment Enrichment
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
30
31. 2013 © Trivadis
3) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
31
33. 2013 © Trivadis
4) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
33
34. 2013 © Trivadis
4) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
34
35. 2013 © Trivadis
5) Count hashtag occurrences
within period - Using CQL
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
35
37. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
37
38. 2013 © Trivadis
Oracle BAM: Architected for Integration and
Visualization
Processing Twitter Stream with Oracle Event Processing (OEP)
19.11.2014
38
39. 2013 © Trivadis
Oracle BAM – Create a
Data Object
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
39
40. 2013 © Trivadis
Oracle BAM Enterprise Message
Source Configuration
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
40
41. 2013 © Trivadis
Oracle BAM Composer
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
41
42. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (OEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Event Processing in Information Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
42
43. 2013 © Trivadis
Architectural Pattern: Standalone Event Stream
Processing
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4343
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
Enterprise
EventBus
43
Analytical
Applications
DB
Enterprise
ServiceBus
Business Rule
Management
SystemRules
Event Processing
Result
Store
44. 2013 © Trivadis
Hadoop Big Data
Infrastructure
Architectural Pattern: Event Stream Processing as part
of Lambda Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4444
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
Enterprise
EventBus
44
Analytical
Applications
DB
Enterprise
ServiceBus
Event Processing
Map/
Reduce
HDFS Result
Store
Result
Store
45. 2013 © Trivadis
Hadoop Big Data
Infrastructure
Architectural Pattern: Event Stream Processing as part
of “Kappa” Architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4545
Event Processing
(ESP / CEP)
State Store /
Event Store
EnterpriseEventBus
(Ingress)
Event
Cloud
Internet of
Things
Social Media
Streams
45
Analytical
Applications
DB
Enterprise
ServiceBus
Event Processing
ReplayHDFS
Result
Store
46. 2013 © Trivadis
Event
Cloud
Event Processing in modern architecture
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
46
Enterprise
Applications
WS
REST
JMS
RDMBS
LocalESB
External Cloud
Service Providers
EnterpriseServiceBus(ESB)
EJB
Event Processing
(ESP / CEP)
State Store /
Event Store
BPM and SOA Platform
Event
REST
Business
Logic/Rules
NoSQL
Analytical Applications
DataAnalytics
Internet of
Things
Event Processing
46
Mobile Apps
DB
Rich (Web)
Client Apps
DB
Social Media
Streams
EnterpriseEventBus
(Ingress)
EnterpriseEventBus
Visualization
Biz Logic Rules
WS
Event
Business Rule
Management
SystemRules
Processes ACM
HumanWF
Oracle Event Processing
for Java Embedded
Coherence
Oracle NoSQL
Business Activity Monitoring (BAM)
Oracle RDBMS
ServiceBus
SOA Suite
BPM Suite
WebLogicJMS
WebLogicJMSEDN
Oracle Rules
Oracle Event Processing (OEP)
Oracle NoSQLOracle RDBMS
Business Activity
Monitoring (BAM)
APIGateway
EDN
Oracle Real-Time
Decisions (RTD)
47. 2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com
19.11.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
47