Performing analysis and prediction in big scale (aka Big Data) is very popular in IT world now. But how to deal with this amount of data in real time? How to get answers on questions asked by business runners in seconds rather than in hours? How to forget about MapReduce jobs and still deal with BigData? In this talk I will try to answer these questions and introduce you to real time "recipies" and algoritms. Talk covers also basics of Apache Storm Project which is my preferred tool for doing such type of analysis. Presentation also contains facts and lessons learned from designing and developing real time topologies at INTERIA.PL. Marcin Stanislawski - Marcin Stanislawski works as software architect/engineer at INTERIA.PL, 4th biggest portal of the Polish internet. Currently he is working on a system that analyzes and predicts user's behaviour on the portal websites. Moreover he is a huge fan of Open Source, Continous Delivery, TDD, BDD and functional programming languages like Scala. On a daily basis husband, father and urbanomics enthusiast.

1. WHEN STORM HITS DATA.
DATA STREAMS PROCESSING IN REAL TIME.
MARCIN STANISLAWSKI

2. WHO AM I?
Architect/Developer atInteria.pl
Storm and Hadoop user
Github: webik
Twitter: @unilama

3. BIG DATA
HADOOP

4. WELCOME IN ZOO

6. RUN JOB
COFFEE BREAK*
RESULTS
*-there are some solutions

7. IMPALA
implemented in C++
non Map Reduce solution

8. KIJI
KijiREST
HDFS/HBase/Cassandra

9. BATCH PROCESSING VS. STREAMING

12. STREAMING SOLUTIONS
Yahoo S4
Akka
Spark Streaming
Storm

13. STORM WHAT IS THAT?

14. README.MD
Stormis adistributed realtime computation system.
Stormis simple, can be used with any programming
language, and is alotof funto use!

15. CURRENT STATUS
Apache Incubation
Included in HortonWorks DataPlatform
Contributed byYahoo
Easydeployto Amazon EC2

16. WHO USES

17. BASIC IDEA

18. SPOUTS
TAKES EVENTS FROM:
Kafka
Kestrel
RabitMQ
...
AND PASS THEM TO...

19. BOLTS
TUPLES ARE PROCESSED, IN WAY THAT YOU IMPLEMENT IT

20. EVENTS ARE TUPLES
( 1, "TEST", "ATMOSPHERE", "2014-05-20 10:00:40", ... )
OBJECTS ARE SERIALIZED USING KYRO

21. WRITTEN IN JAVA&CLOJURE
TOPOLOGIES ARE DAGS

22. ARCHITECTURE
Nimbus
Nodes(Supervisors)
UI
DRPC

23. EVENT PROCESSED ONE OR MORE TIMES.

24. ACKING FRAMEWORK
Each tuple mustbe acked or failed

25. TUPLES TRACKING
tuple has random 64 bitid
xor of alltuple ids, thathave been created
and/or acked in the tree
if tuple id equals 0, tuple is fullyprocessed

26. COMMUNICATION
Between:
Tasks: Disruptor LMAX
Workers: ⦰MQ -> Netty

27. TRIDENT
high-levelabstraction
same as Cascading/Scaldingin Hadoop World

28. SPOUT
Keydifference -producingStream(s)

29. STREAM
Batches chain with multiplication ability

30. STREAM OPERATIONS
Functions
Filters
Projections
Joins
Merges

31. SATE
Operations:
Grouping
Aggregate
Query

32. STATE TYPES
non-transactional
transactional
opaque transactional

33. STATE
In memorystate
NoSQL databases
Externalsystems viaAPIs

34. DRPC

35. DRPC TOPOLOGY
NAMED DRPC SPOUT
USES MAIN TOPOLOGY STATES
GENERATES ONE TUPLE OUTPUT

36. DRPC ELEMENTS
THRIFT SERVER(S)
WITH PREDEFINED SPOUT
AND BOLT

37. ARE YOU PROGRAMMING IN NON-JVM
LANGUAGE?
NO PROBLEM :)
Ruby
Python
Perl
PHP
...

38. STREAMING API
API defined as Thrift
JSONbased communication

39. RED STORM
Writingtopologies in Ruby

40. REAL TIME ALGORITHMS

41. SIMPLE OPERATIONS
Sum
Count
Multiplication

42. MAXIMUM AND MINIMUM
don'tlose currentvalue

43. USUALLY TWO TOPOLOGIES

44. LEARNING
Classification
Clustering

45. MODEL
Evaluator
Visualiser

46. BASIC ELEMENT TABLE

47. SIMPLE EXAMPLE

48. ALGORITHM EXAMPLES
k-means clustering
statisticaltest(T, F, Z, Chi2)
Hidden MarkovModels

49. ADVERT TIME :)

50. STORMUNIT
http://github.com/webik/StormUnit
MAVEN MOJO - COMMING SOON :)
http://github.com/webik/storm-maven

51. WHAT NEXT...

52. SUMMINGBIRD
Write once, run on:
Storm
Hadoop(Scalding)
Amazon Kinesis

53. MAYBE BACK INTO ZOO
STORM YARN

54. THANK YOU.

55. QUESTIONS?