Riemann (riemann.io) is an event stream processor, usually deployed as a router for events generated by software systems. As Kyle Kingsbury, the author states: “Riemann’s job is to process, analyze and react to the event stream”. In effect, event stream processing is functional transformation writ large; events from various sources (e.g., hardware stats, application metrics, network data etc) are received then grouped, filtered, mapped, averaged, coalesced, forwarded, etc. Riemann is written in Clojure. Riemann is also configured via Clojure. It’s “parentheses all the way down”. But does a tool for functional transformation of data require a functional language? To what extent does the implementation language impact the capabilities Riemann exposes to the end user? This presentation will take a deep dive into both the capabilities of the configuration system and the core Clojure implementation to see why a functional language is such a good choice to implement this type of application.

2. FUNCTIONAL
DEVOPSEVENT PROCESSING IN CLOJURE
Andy Marks
@andee_marks
amarks@thoughtworks.com

3. Background

4. we all code!

5. (prn “Hello World”)

6. (ns hello.core)
(defn -main []
(prn “Hello World”))

7. Jetty
(ns hello.core)
(defn -main []
(prn “Hello World”))
✓access requests
✓queues
✓sessions

8. AWS EC2
Jetty
(ns hello.core)
(defn -main []
(prn “Hello World”))
✓disk
✓network interfaces
✓memory
✓cores/cpus

9. AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS
(Master)
DNS
(Slave)
✓records
✓replication
✓resolution

10. WS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS
(Master)
DNS
DB
✓performance
✓indexing
✓locking
✓connections

11. WS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS
(Master)
DNS
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
✓connections
✓DoS
✓Clients

12. WS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS
(Master)
DNS
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
MessageQueue
✓queues
✓timeouts

13. that is a lot of
information…

14. from a lot of
different places

15. this is a problem

16. this is the problem
Riemann helps
solve

17. Riemann is
written in
Clojure
52 source files
7256 lines
377 functions
30 macros

18. Riemann is written
and configured
in Clojure

19. And so… to Riemann

20. kylekingsbury(@aphyr)

21. OVERVIEW

22. Correlate events from disparate
sources
What is the latency of the web servers when the DB
servers are generating timeouts?
Filter events based on key criteria Ignore all network events with a normal status
Aggregate events to perform
statistical analysis (e.g., percentiles)
What is the average response rate for 95% of requests?
Monitor status of sources by
availability of events
Which services haven’t sent any events for the last 5
minutes?
Route event streams to multiple
consumers
Send all error events from DB servers to the DBA team
as well as the main operational dashboards
SAMPLE RIEMANN USE CASES

23. i = f(e)
e: thundering herd of events
i: information ready for downstream processing

24. Hypothesis
Event processing is function
application at large scale… using
Clojure to build/configure Riemann is a
logical choice.

25. (defrecord Event
[
service
state
description
metric
tags
time
ttl
KEY CONCEPTS: EVENTS
“reg1.foo.bar”
“reg-service http”
“ok”
“apache-log”
201
[“reg1” “http”]
104534892
60
riemannsource

26. KEY CONCEPTS: STREAMS
(logging/init {:file "riemann.log" :console true})
(instrumentation {:interval 5}) ;; self monitor every 5 seconds
(periodically-expire 1) ;; check for expired events every second
(tcp-server)
(repl-server)
(streams
;; insert magic here!!!
prn)
riemannconfig

27. KEY CONCEPTS: INDEX
flow of events
index
riemann
process

28. General FP goodness

29. HOMOICONICITY
“[It] is where a program's source code is written
as a basic data structure that the
programming language knows how to access.”
Wikipedia
(defn validate-config
[file]
(try
(read-strings (slurp file))
(catch clojure.lang.LispReader$ReaderException e
(throw (logging/nice-syntax-error e file)))))
riemannsource

30. HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable
number of child streams and returns a function
that takes an event and responds to the event it is passed
when it is invoked.”
http://riemann.io/quickstart.html
“a higher-order function… does at least one of the
following:
[1] takes one or more functions as an input,
[2] outputs a function”
Wikipedia

31. HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable
number of child streams and returns a function
that takes an event and responds to the event it is passed
when it is invoked.”
http://riemann.io/quickstart.html
(streams
(rate 5 prn)
)
riemannconfig

32. IMMUTABILITY
“Streams ‘change’ events by sending altered,
shared-structure *copies* of events
downstream.”
http://riemann.io/howto.html#split-streams
(streams
(where (host "db04")
(with :service "foo" prn)
(with :service "bar" prn))
)
riemannconfig

33. Specific Clojure
goodness

34. “clojure's remarkably fast for
what it is, sexprs make the
tree structure of the streams
visually apparent, it makes
writing the concurrent algos
much simpler, and macros
allow us to express things
like 'by and 'pipe in ways
that would be awkward in
other languages without
building our own AST &
interpreter or transpiler,
etc.”
“clojure's remarkably fast for
what it is, sexprs make the
tree structure of the streams
visually apparent, it makes
writing the concurrent algos
much simpler, and macros
allow us to express things
like 'by and 'pipe in ways
that would be awkward in
other languages without
building our own AST &
interpreter or transpiler,
etc.”
@APHYR SAID…

35. S-EXPRESSIONS
“a notation for nested list (tree-structured)
data… popularised by the programming
language Lisp”
Wikipedia

36. S-EXPRESSIONS
(streams
(where (and (service #"^riak")
(state “critical”))
(email “delacroix@vonbraun.com"))
)
1 Filter certain events…
2 Let people know…
1
2
riemannconfig
“make the tree structure of the
streams visually apparent”

37. 3
1
2
S-EXPRESSIONS
1 Split on event fields…
2 Detect state changes…
3 Collate and email
(streams
(by [:host :service]
(changed :state
(rollup 5 3600
(email “delacroix@vonbraun.com"))))
)
riemannconfig
“make the tree structure of the
streams visually apparent”

38. MACROS
(defmacro pipe
[marker & stages]
`(let [~@(->> stages
reverse
(interpose marker)
(cons marker))]
~marker))
(streams
(pipe -
(splitp = service
"http req rate" -
"0mq req rate" (scale 2 -))
(rate 5
(graphite {:host “127.0.0.1” :port 2003})))
)
riemannconfig
riemannsource

39. MACROS: PIPE
(streams
(let [aggregate
(rate 5
(graphite {:host “127.0.0.1” :port 2003}))]
(splitp = service
"http req rate" aggregate
"0mq req rate" (scale 2 aggregate)))
)
or
riemannconfigriemannconfig
(streams
(pipe -
(splitp = service
"http req rate" -
"0mq req rate" (scale 2 -))
(rate 5
(graphite {:host “127.0.0.1” :port 2003})))
)

40. Conclusion

41. Hypothesis
Event processing is function
application at large scale: using Clojure
to build/configure Riemann is a logical
choice.
homoiconicity
higherorderfunctions
s-expressions
macros
immutability

42. CLOJURE AD ABSURDUM
(defn do-you-trust-me? [event]
(prn (load-string (:description event))))
(streams
(where (service "event-gen")
do-you-trust-me?)
)
(ns event-gen.core
(:require [riemann.client :refer :all])
(:gen-class :main true))
(defn -main [& args]
(let [c (tcp-client {:host "127.0.0.1"})]
(send-event c
{:service "event-gen" :description "(+ 1 1)"})
(close-client c)))
riemannclientriemannconfig

43. CLOJURE AD ABSURDUM
(defrecord Event
[service
description
])
“event-gen”
“(+ 1 1)”
Riemann
(Clojure)
riemann.config
(Clojure)
Client
(Clojure)
Clojure
=> “2”

44. THANKS
Andy Marks
@andee_marks
amarks@thoughtworks.com