筑波大学 集中講義資料 2014/01/31

1. Norikra in Action
TAGOMORI Satoshi (@tagomoris)
LINE Corp.
2014/01/31 (Fri) at University of Tsukuba
the 2nd half
14年1月31日金曜日

2. Norikra:
Schema-less Stream Processing with SQL
Open source software (GPLv2)
http://norikra.github.io/
https://github.com/norikra/norikra
14年1月31日金曜日

3. Norikra:
Schema-less event stream:
Add/Remove data fields whenever you want
SQL:
No more restarts to add/remove queries
w/ JOINs, w/ SubQueries
w/ UDF
Truly Complex events:
Nested Hash/Array, accessible directly from SQL
14年1月31日金曜日

4. Norikra Queries: (1)
SELECT name, age
FROM events
target
14年1月31日金曜日

5. Norikra Queries: (1)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}
SELECT name, age
FROM events
{“name”:”tagomoris”,”age”:34}
14年1月31日金曜日

6. Norikra Queries: (1)
{“name”:”tagomoris”,
“address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}
SELECT name, age
FROM events
nothing
14年1月31日金曜日

7. Norikra Queries: (2)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}
SELECT name, age
FROM events
WHERE current=”Tsukuba”
{“name”:”tagomoris”,”age”:34}
14年1月31日金曜日

8. Norikra Queries: (2)
{“name”:”kawashima”,
“age”:99, “address”:”Tsukuba”,
“corp”:”Univ”, “current”:”Dream”}
SELECT name, age
FROM events
WHERE current=”Tsukuba”
nothing
14年1月31日金曜日

9. Norikra Queries: (3)
SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age
14年1月31日金曜日

10. Norikra Queries: (3)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}
SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age
every 5 mins
{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...
14年1月31日金曜日

11. Norikra Queries: (4)
{“name”:”tagomoris”,
“age”:34, “address”:”Tokyo”,
“corp”:”LINE”, “current”:”Tsukuba”}
SELECT age, COUNT(*) as cnt
FROM
events.win:time_batch(5 mins)
GROUP BY age
SELECT max(age) as max
FROM
events.win:time_batch(5 mins)
{”age”:34,”cnt”:3}, {“age”:33,”cnt”:1}, ...
{“max”:51}
14年1月31日金曜日
every 5 mins

12. Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Tsukuba”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY age
14年1月31日金曜日

13. Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Tsukuba”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
GROUP BY user.age
14年1月31日金曜日

14. Norikra Queries: (5)
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend.$0 AND attend.$1
GROUP BY user.age
14年1月31日金曜日

15. Norikra and Esper
Esper:
CEP engine library, Java, GPLv2
EPL: Event Processing Language (SQL + window)
Streams: schema-full flat field set
Norikra:
Using Esper internally
Schema-less stream -> schema-full stream
conversion
Rewriting compiled queries
14年1月31日金曜日

16. Norikra query execution
1.accept query
2.parse -> find target / field set
3.(if target is not opened) wait for first event
4.compile query
5.rewrite target name into stream name
6.rewrite field names
7.register query
8.input events
14年1月31日金曜日

17. Target mapping
Ignore unused fields
Field set matching between streams and queries
Generate field set inheritance tree
14年1月31日金曜日

18. automated stream inheritance
of norikra's target
Base fieldset
Query fieldset
Data fieldset
14年1月31日金曜日
b_xxxxxxxxx
minimal fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'

19. automated stream inheritance
of norikra's target
Base fieldset
b_xxxxxxxxx
Query fieldset
Data fieldset
14年1月31日金曜日
e_xxxxxxxx1
event data fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
product_code: 'string'
charge: 'integer'
shop_code: 'long'

20. automated stream inheritance
of norikra's target
Base fieldset
event data fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
product_code: 'string'
Query fieldset
charge: 'integer'
shop_code: 'long'
affiliate: 'string'
Data fieldset
14年1月31日金曜日
e_xxxxxxxx1
b_xxxxxxxxx
e_xxxxxxxx2

21. automated stream inheritance
of norikra's target
Base fieldset
b_xxxxxxxxx
new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0
Query fieldset
Data fieldset
14年1月31日金曜日
e_xxxxxxxx1
e_xxxxxxxx2

22. new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0
automated stream inheritance
of norikra's target
Base fieldset
b_xxxxxxxxx
Query fieldset
Data fieldset
14年1月31日金曜日
event data fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
affiliate: 'string'
q_xxxxxxxx0
e_xxxxxxxx1
e_xxxxxxxx2'

23. Registered
automated stream inheritance EPL:
SELECT count(*)
of norikra's targetFROM q_xxxxxxxx0.win:time_batch(1min)
WHERE affiliate.length() > 0
Base fieldset
b_xxxxxxxxx
Query fieldset
Data fieldset
14年1月31日金曜日
q_xxxxxxxx0
e_xxxxxxxx1
e_xxxxxxxx2'

24. automated stream inheritance
of norikra's target
Base fieldset
b_xxxxxxxxx
Query fieldset
q_xxxxxxxx0
q_xxxxxxxx1
Data fieldset e_xxxxxxxx1'
e_xxxxxxxx2'
e_xxxxxxxx3'
14年1月31日金曜日

25. Query rewriting
{“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend.$0 AND attend.$1
GROUP BY user.age
14年1月31日金曜日

26. {“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend.$0 AND attend.$1
GROUP BY user.age
14年1月31日金曜日

27. {“name”:”tagomoris”,
“user:{“age”:34, “corp”:”LINE”,
“address”:”Tokyo”},
“current”:”Kyoto”,
“speaker”:true,
“attend”:[true,true,false, ...]
}
{“name”:”tagomoris”,
“user.age”:34,
“user.corp”:”LINE”,
“user.address”:”Tokyo”,
“current”:”Kyoto”,
“speaker”:true,
“attend.$0”:true,
“attend.$1”:true,
“attend.$2”:false,
...
}
14年1月31日金曜日

28. {“name”:”tagomoris”,
“user.age”:34,
“user.corp”:”LINE”,
“user.address”:”Tokyo”,
“current”:”Kyoto”,
“speaker”:true,
“attend.$0”:true,
“attend.$1”:true,
“attend.$2”:false,
...
}
{“user$age”:34,
“current”:”Kyoto”,
“attend$$0”:true,
“attend$$1”:true,
}
14年1月31日金曜日

29. {“user$age”:34,
“current”:”Kyoto”,
“attend$$0”:true,
“attend$$1”:true,
}
SELECT user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend.$0 AND attend.$1
GROUP BY user.age
14年1月31日金曜日

30. {“user$age”:34,
“current”:”Kyoto”,
“attend$$0”:true,
“attend$$1”:true,
}
SELECT
user.age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend.$0 AND attend.$1
GROUP BY user.age
Conversions for compiled query object in fact.
SELECT user$age, COUNT(*) as cnt
FROM events.win:time_batch(5 mins)
WHERE current=”Tsukuba”
AND attend$$0 AND attend$$1
GROUP BY user$age
14年1月31日金曜日

31. Norikra internal:
Jump
from schema-full world
to schema-less world.
14年1月31日金曜日