Clojure Reducers / clj-syd Aug 2012

Reducers
A library and model for collection processing in Clojure

Leonardo Borges
@leonardo_borges
http://www.leonardoborges.com
http://www.thoughtworks.com
Thursday, 30 August 12

Reducers
A library and model for collection processing in Clojure

less
or
m i ns
in 20
... Leonardo Borges
@leonardo_borges

Reducers huh? Here’s the gist



You get parallel versions of reduce, map and filter




Ta-da! I’m done!




Ta-da! I’m done!

and well under my 20 min limit :)


Alright, alright I’m kidding


How do reducers make parallelism possible?


How do reducers make parallelism possible?

• JVM’s Fork/Join framework
• Reduction Transformers


Before we start - this is bleeding edge stuff
Java requirements

• Fork/Join framework
• Java 7 [1] or
• Java 6 + the JSR166 jar [2]
Clojure requirements

• 1.5.0-* (this is still MASTER on github [3] as of 30/08/2012)

[1] - http://jdk7.java.net/
[2] - http://gee.cs.oswego.edu/dl/jsr166/dist/jsr166.jar
[3] - https://github.com/clojure/clojure

The Fork/Join Framework



•Based on divide and conquer



•Work stealing algorithm



•Uses deques - double ended queues.



•Progressively divides the workload into tasks, up to a threshold



•Once it ﬁnished one task, it pops another one form its deque



•After at least two tasks have ﬁnished, results can be combined/joined



•After at least two tasks have ﬁnished, results can be combined/joined
•Idle workers can pop tasks from the deques of workers which fall behind


Text is boring


Fork/Join algorithm - simpliﬁed view



Workload is put in “deques”



...and progressively halved



...up to a conﬁgured threshold



Worker 1 Worker 2



Combine

Worker 1 Worker 2



Combine Combine

Worker 1 Worker 2



Worker 1 Worker 2

Idle workers can “steal” items from other workers


Final result

Worker 1 Worker 2


Let’s talk about Reducers



Motivations

• Performance
• via less allocation
• via parallelism (leverage Fork/Join)



Motivations Issues

• Performance • Lists and Seqs are sequential
• via less allocation • map / filter implies order
• via parallelism (leverage Fork/Join)


A closer look at what map does
;; a naive map implementation
(defn map [f coll]
(if (seq coll)
(cons (f (first coll)) (map f (rest coll)))
'()))


(defn map [f coll]
(if (seq coll)
'()))

• Recursion


(defn map [f coll]
(if (seq coll)
'()))

• Recursion
• Order


(defn map [f coll]
(if (seq coll)
'()))

• Recursion
• Order
• Laziness (not shown)


(defn map [f coll]
(if (seq coll)
'()))

• Recursion
• Order
• Consumes List


(defn map [f coll]
(if (seq coll)
'()))

• Recursion
• Order
• Consumes List
• Builds List


(defn map [f coll]
(if (seq coll)
'()))

• Recursion
• Order Oh, and it also applies the function
• Laziness (not shown) to each item before putting the result
• Consumes List into the new list
• Builds List


(defn map [f coll]
(if (seq coll)
'()))
This is what mapping means!

• Recursion
• Order Oh, and it also applies the function
• Laziness (not shown) to each item before putting the result
• Consumes List into the new list
• Builds List


Reduction Transformers



• Idea is to build map / filter on top of reduce to break from sequentiality



• map / filter then builds nothing and consumes nothing



• map / filter then builds nothing and consumes nothing
• It changes what reduce means to the collection by transforming the reducing
functions


What map is really all about
(defn mapping [f]
(fn [f1]
(fn [result input]
(f1 result (f input)))))


But wait!
If map doesn’t consume the list any longer, who does?

• reduce does!
• Since Clojure 1.4 reduce lets the collection reduce itself
(through the CollReduce / CollFold protocols)
• Think of what this means for tree-like structures such as
vectors
• This is key to leveraging the Fork/Join framework


Now we can use mapping to create reducing functions
(reduce ((mapping inc) +) 0 [1 2 3 4])
;; 14


(reduce ((mapping inc) +) 0 [1 2 3 4])
;; 14

(fn [result input]
(+ result (inc input)))


(reduce ((mapping inc) conj) [] [1 2 3 4])
;; [2 3 4 5]


;; [2 3 4 5]

(fn [result input]
(conj result (inc input)))


;; [2 3 4 5]

(fn [result input]
(conj result (inc input)))

But it feels awkward to use it in this form


What do we have so far?

• Performance has been improved due to less allocations
• No intermediary lists need to be built (see Haskell’s StreamFusion [4])
• However reduce is still sequential

[4] - http://bit.ly/streamFusion

Enters fold


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce


Enters fold

• Potentially parallel (fallsback to standard reduce otherwise)


Enters fold

• Reduce/Combine strategy (think Fork/Join Framework)


Enters fold

• Segments the collection


Enters fold

• Runs multiple reduces in parallel


Enters fold

• Uses a combining function to join/reduce results


Enters fold

• Uses a combining function to join/reduce results

(defn fold [combinef reducef coll]
...)


The combining function is a monoid
• A binary function with an identity element
• All the following functions are equivalent monoids



+
(+ 2 3) ; 5
(+) ; 0



(defn my-+
([] 0)
([a b] (+ a b)))

(my-+ 2 3) ; 5
(my-+) ; 0



(require ‘[clojure.core.reducers :as r])

(def my-+
(r/monoid + (fn [] 0)))

(my-+ 2 3) ; 5
(my-+) ; 0


fold by examples

;; all examples assume the reducers library
is available as r
(ns reducers-playground.core
(:require [clojure.core.reducers :as r]))


fold by examples:
increment all even positive integers up to 10 million
and add them all up


fold by examples:
and add them all up
;; these were taken from Rich’s reducers talk


fold by examples:
and add them all up
(def my-vector (into [] (range 10000000)))


fold by examples:
and add them all up

(time (reduce + (map inc (filter even? my-vector))))


fold by examples:
and add them all up

;; 500msecs


fold by examples:
and add them all up

;; 500msecs

(time (reduce + (r/map inc (r/filter even? my-vector))))


fold by examples:
and add them all up

;; 500msecs

;; 260msecs


fold by examples:
and add them all up

;; 500msecs

;; 260msecs

(time (r/fold + (r/map inc (r/filter even? my-vector))))


fold by examples:
and add them all up

;; 500msecs

;; 260msecs

(time (r/fold + (r/map inc (r/filter even? my-vector))))
;; 130msecs


fold by examples:
standard word count

(def wiki-dump (slurp "subset-wiki-dump50")) ;50 MB

(defn count-words [text]
(reduce
(fn [memo word]
(assoc memo word (inc (get memo word 0))))
{}
(map #(.toLowerCase %) (into [] (re-seq #"w+" text)))))


fold by examples:
standard word count


(defn count-words [text]
(reduce
(fn [memo word]
{}
(map #(.toLowerCase %) (into [] (re-seq #"w+" text)))))

(time (count-words wiki-dump)) ;; 45 secs


fold by examples:
parallel word count


(defn p-count-words [text]
(r/fold
(r/monoid (partial merge-with +) hash-map)
(fn [memo word]
(r/map #(.toLowerCase %) (into [] (re-seq #"w+" text)))))


fold by examples:
parallel word count


(r/fold
(r/monoid (partial merge-with +) hash-map) Combining fn
(fn [memo word]


fold by examples:
parallel word count

Will be called at the leaves to merge the
(defn p-count-words [text] partial computations
(r/fold
(fn [memo word]


fold by examples:
parallel word count

Will be called with no arguments to
(defn p-count-words [text] provide a seed value
(r/fold
(fn [memo word]


fold by examples:
parallel word count


(r/fold
(fn [memo word]

(time (p-count-words wiki-dump)) ;; 30 secs


fold by examples:
Load 100k records into PostgreSQL

(def records
(into [] (line-seq
(BufferedReader. (FileReader. "dump.txt")))))


fold by examples:

(time (doseq [record records]
(let [tokens (clojure.string/split record #"t" )]
(insert users/users
(values {
:account-id (nth tokens 0)
...
})))))


fold by examples:

(time (doseq [record records]
(insert users/users
(values {
...
})))))

;; 90 secs

fold by examples:
Load 100k records into PostgreSQL in parallel
(time (r/fold
+
(r/map (fn [record]
(do (insert users/users
(values {
...
}))
1))) records)))


fold by examples:
Load 100k records into PostgreSQL in parallel
(time (r/fold
+
(r/map (fn [record]
(do (insert users/users
(values {
...
}))
1))) records)))

;; 50 secs

When to use it


When to use it

• Exploring decision trees


When to use it

• Image processing


When to use it

• As a building block for bigger, distributed systems such as Datomic and
Cascalog (maybe around parallel agregators)


When to use it

• Basically any list intensive program


When to use it

• Basically any list intensive program

But the tools are available to anyone so be creative!


Resources

• The Anatomy of a Reducer - http://bit.ly/anatomyReducers
• Rich’s announcement post on Reducers - http://bit.ly/reducersANN
• Rich Hickey - Reducers - EuroClojure 2012 - http://bit.ly/reducersVideo
(this presentation was heavily inspired by this video)
• The Source on github - http://bit.ly/reducersCore

Leonardo Borges
@leonardo_borges

Thanks!

Questions?

Leonardo Borges
@leonardo_borges


Clojure Reducers / clj-syd Aug 2012

Recomendados

Recomendados

Más contenido relacionado

Más de Leonardo Borges

Más de Leonardo Borges (14)

Último

Último (20)

Clojure Reducers / clj-syd Aug 2012