2. Who am I?
Takeshi NAKANO
Senior Researcher / Architect.
Co-authered Getting started with Solr in Japanese.
Co-authered Hadoop Hacks in Japanese.
Writing Getting started with Kafka in Japanese.
http://genn.ai/
2
3. Agenda
What is genn.ai?
Let s use genn.ai
Realtime processing
Download & get your key
Usecases
Define a schema and a filter
Make it real!
http://genn.ai/
3
4. A usecase of genn.ai
Customer
A
Web Sites
じゃらん
Customer
B
Customer
C
Marketer
Analyst
Realtime Analysis
Platform
Client
(Hotels)
4
8. Preparing for starting.
http://dev.genn.ai/
> cd gungnir-client
> bin/gungnir -u tf0054 -p 2e2232073109
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
2013/10/14 22:34:58 com.twitter.finagle.Init$ apply
情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057
Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290
Welcome tf0054
gungnir>
9. Define your schema!
CREATE TUPLE userAction (
userId STRING,
hotelId STRING
);
CREATE TUPLE commitAction (
userId STRING,
hotelId STRING,
checkin TIMESTAMP('yyyy-MM-dd'),
nights INT
);
STRING Java type string
TINYINT Java type byte
SMALLINT Java type short
INT Java type int
BIGINT Java type long
FLOAT Java type float
DOUBLE Java type double
TIMESTAMP long value
LIST
MAP
STRUCT
10. Define your filter! (1/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
EMIT
11. Define your filter! (2/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
JOIN
(user)
JOIN
(hotel)
EMIT
12. Define your filter! (3/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
INTO stream_0
;
FROM stream_0
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
JOIN
(user)
JOIN
(hotel)
stream_0
EMIT
(user)
13. Define your filter! (4/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
INTO stream_0
;
FROM stream_0
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
FROM stream_0
BEGIN GROUP BY hotelId
EACH count() AS count, *
EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel')
END GROUP
;
EMIT
(user)
From
JOIN
JOIN
EACH sum(nights)
EACH avg(nights)
stream_0
GROUP
BY
EMIT
(hotel)
14. Register your filter! and ...
SUBMIT TOPOLOGY;
DESC TOPOLOGY;
SHOW TOPOLOGIES;
STOP TOPOLOGY xxx;
gungnir> SUBMIT TOPOLOGY;
OK
gungnir> DESC TOPOLOGY;
{
"id":"52373363e4b0a525adadb4f2",
"explain":" ... ",
"status":"RUNNING",
"owner":"tf0054@gmail.com",
"createTime":"2013-09-16T16:35:47.432Z",
"summary":{
...
}
}
gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ;
DROP TOPOLOGY xxx;
gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;
15. Testing it!
TRACK userAction {"userId":"siniida","hotelId":"226979"};
TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1};
*** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.)
*** The function of this command is to make and post a event to gennai.
gungnir> DESC TOPOLOGY;
{
"id":"52373363e4b0a525adadb4f2",
"explain":" ... ",
"status":"RUNNING",
"owner":"tf0054@gmail.com",
...
"hotelId" : "128767"
"hotelId" : "356564"
"hotelId" : "226979"
"hotelId" : "325073"
"hotelId" : "357090"
"hotelId" : "126942"
"hotelId" : "310294"
... USING kafka_emit('${TOPOLOGY_ID}_user')
"hotelId" : "367746"
"hotelId" : "232896"
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ;
"hotelId" : "220231"
or
"hotelId" : "163175"
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel;
"hotelId" : "367856"
"hotelId" : "360863"
"hotelId" : "119747"
17. Design of genn.ai
Built on Storm
Converts
The
..
properly (e.g. building AST)
Good Parts (e.g. Supporting parallelism setting)
18. The future of genn.ai
genn.ai wants friends!
genn.ai
codes will be open-sourced in the future.
We
are looking for co-challenging partners for
improving genn.ai.
You
can use genn.ai on your web sites completely for
free (infrastructure etc..) if we can share the dream
together.
Please feel free to contact us!
http://genn.ai/
tf0054@gmail.com