Incanter bigdata jsc2012

BigData avec Incanter et Clojure
JUG Summer Camp 2012

Claude Falguière @cfalguiere

goo.gl/4mmJQ
1
jeudi 13 septembre 2012 1

AU PROGRAMME
Lire et Stocker les
explorer Construire données Générer Faire un job
des un projet dans un rapport MapReduce
données Clojure MongoDB HTML Hadoop

avec Clojure
JUG Summer Camp 2012 2 Claude Falguiere

VOTRE MISSION SI VOUS
L’ACCEPTEZ ...
rapports d’analyse de tests de performance

500 utilisateurs x 5h de test
environ 1 000 000 relevés

ﬁltrage des erreurs
transformer les données
regrouper par catégories
charts, distributions, corrélations


INCANTER

= +
Math,
Stats,
Graps


INCANTER
DSL

manipulation de matrices et tables

statistiques
charts
modèles
test d’hypothèse


INCANTER
Ecosystème Java
Netlib
Parallel Colt
JFreeChart

Motivation pour se mettre à Clojure
Langage Fonctionnel


CLAUDE FALGUIERE

goo.gl/4mmJQ

http://cfalguiere.wordpress.com
@cfalguiere


ET VOUS ?

? ? ?

Petit rappel
Clojure


HELLO WORLD

(println "Hello JUG Summer Camp")
-> "Hello JUG Summer Camp"

(def conference "JUG Summer Camps")
(println "Hello" conference)

(count (str "Hello" conference))


LES NOMBRES

(+ 1 2 3) -> 7
(+ 4 (* 2 3)) -> 10

(= 2 (+ 1 1)) -> true


Lire et
explorer
des données

SE LANCER
La voie du novice

1) downloader Incanter
2) lancer le REPL


LES EXEMPLES
https://github.com/cfalguiere/
ClojureBigDataJugSummerCamp2012


PREMIER SCRIPT


LIRE DES DONNÉES
(use '(incanter core io stats charts))

(def ds
(read-dataset "readings.csv" :header true))

équivalent pour Excel

(save ds "./data.csv")


LES COLLECTIONS
Clojure

,
(def colors ["red" "magenta" "orange"])

(def colors ‘("red" "magenta" "orange"))

(def colors #{"red" "magenta" "orange"})


LES COLLECTIONS
Incanter

Matrix
Dataset


LES MAPS
Clojure

(def conf
{:name "JUG Summer Camp",
:location "La Rochelle" })


JEU DE DONNÉES
t lb ts s rc shop srch userId ...

754 "SU01-HomePage-Page" 1330419301862 "true" 200 "MAF" "gants" "PT1022809551"


LES COLONNES
user>($ :t ds)

user> (mean ($ :t ds))
994,75


QUANTILE 95%

(quantile ($ :t ds) :probs [0.95])
911.1758241758242

Temps maximum pour 95% des relevés


LES COLONNES

(view ($ [:lb :t :shop] ds )) ->
)


LES LIGNES

($where {:s "true"} ds)
-> les lignes OK


LES LIGNES

($where {:s "true"
:shop {:$in #{"MAF", "FAB"}}
:t {:$gt 3000 :$lt 5000}} ds)


WITH-DATA

(with-data ds
ds
(view ($ :t ($where {:shop "MAF"} ))))
ds)))
ds))
ds)


INCANTER CHARTS
Time-series
Scatter Plot

Bar charts

Histograms Box Plot


(view (time-series-plot :ts :t :data ds)
ds
ds))
:title "Readings"
:x-label "time"
:y-label "resp. time (ms)"
:legend true
:series-label "duration ms"))

JUG Summer Camp 2012 Claude Falguiere

(doto (time-series-plot :ts :t :data ds :title "Readings"
:x-label "time" :y-label "resp. time (ms)"
:legend true :series-label "duration ms"
)
(add-lines ts (repeat 3000) :series-label "threshold")
(set-stroke-color java.awt.Color/blue :series 0)
(set-stroke :width 1 :series 0)
(set-stroke-color java.awt.Color/red :series 1)
(set-stroke :width 3 :series 1)
view)


DOTO
(doto (time-series-plot :ts :t :data ds :title "Readings"
(time-series-plot :ts :t :data ds :title "Readings"
:x-label "time" :y-label "resp. time (ms)"
:legend true :series-label "duration ms"
)
(add-lines ts (repeat 3000) :series-label "threshold")
(set-stroke-color java.awt.Color/blue :series 0)
view
view)

(view
(set-stroke-color
(set-stroke-color
(add-lines
(add-lines
(add-lines
(time-series-plot...


REGROUPEMENT

(def groups ($group-by :shop ds))

{ "MAF"
{:shop "MAF"} "FAB"
{:shop "FAB"} }}


LIRE UN GROUPE

(get groups {:shop "MAF"})


$ROLLUP

user> ($rollup count :t :shop ds)
[:shop :t] fonction regroupement
["MSK" 170]
["KIE" 76]
["OIB" 342]
["FAB" 171]
...

Applique une fonction à chaque entrée d’un $group-by


MOYENNE
Moyenne par groupe
user> ($rollup mean :t :lb ds)

[:lb :t]
["SU61-BrowseCat2-Page" 911.1758241758242]
["SU43-Paiment-Page" 1770.6004566210045]
["SU42-DeliveryAndBilling-Page" 732.0340136054422]
["SU41-CustomerDetails-Page" 608.2222222222222]
...


DIAGRAMMES EN BARRES

user>(view (bar-chart
:shop
:t
:vertical false
:data ($rollup count :t :shop ds)
($rollup count :t :shop ds)))


ENREGISTRER LES CHARTS

(save (histogram :t :data ds) "histogram.png")


REJOUER LE SCRIPT
(use '(incanter core io stats charts))
(def ds (read-dataset "../data/readings.csv" :header true) )
(save (histogram :ts :t :data ds) "histogram.png")
(println ($rollup mean :t :lb ds))

analyse.clj

user>(load-file "analyse.clj")


ET ENSUITE ...

Les limitations du REPL seul
- difﬁcile d’écrire des commandes complexes
- pas de tests unitaires
- que les libs fournies dans l’uberjar incanter

lein ...


Construire
un projet Clojure

L’ECOSYSTEME CLOJURE
build et gestion de dépendance
lein (leiningen)

repos Maven
clojars, sonatype

tests unitaires
clojure.test (built-in), midge

intégration avec les IDE
swank, jark


L’ATELIER

REPL + editeur de texte
ou
Emacs + Slime + Swank
Eclipse + CounterClockWise
IDEA + La Clojure + Leiningen
Netbeans + Enclojure
Vim + VimClojure


SE LANCER
La voie du gourou

installer leiningen
lein new myproject
conﬁgurer project.clj
lein <task>


NEW
demo
classes
lib
project.clj
src
demo
core.clj
test
demo
test
core.clj


PROJECT.CLJ

(defproject demo "1.0.0-SNAPSHOT"
:description "demos incanter "
incanter"
:dependencies [[org.clojure/clojure "1.3.0"]
[incanter "1.3.0"] ]
)
:dev-dependencies [[swank-clojure "1.4.2"]])


DEPENDANCES
Clojars .m2
Sonatype repository
incanter
incanter
1.3.0
incanter-1.3.0.jar
>lein deps
demo
myproject
classes
lib
incanter-1.3.0.jar
project.clj

TEST UNITAIRES
clojure.test (built-in)

(defn f1 [s] (deftest test-f1
(* 2 s)) (is 6 (f1 3)))


NAMESPACES
demo
(ns demo.core)
classes
lib (defn f1 [s]
project.clj (* 2 s))
src
demo
core.clj (ns demo.test.core
test (:use demo.core)
demo (:use clojure.test))
test
core.clj (deftest test-f1
(is 6 (f1 3)))


RECHERCHER PAR DATE

(ns demo.utils
(:use [clj-time.core :only date-time]) )

user> (let [ start-time{:timestamp {:$gt02 28 8 55)
(view ($where (date-time 2012 start-time
end-time (date-time 2012 02 end-time}}
end-time ] :$lt 28 8 57) ] ds))
(view ($where {:timestamp {:$gt start-time
:$lt end-time}} ds)))


TEST TIME
(deftest test-readable-timestamp
(let [ ds (dataset
[:ts]
[{:ts 1330418007548}
{:ts 1330418007549}])
result (readable-timestamp ds) ]
(is (= 2012
)
(year (sel result
adate )))
:rows 0
:cols :timestamp ))))
))
)


MAPPER UNE COLONNE

($map fct-to-date :ts ds )


CLJ-TIME (JODA)
(ns demo.utils
(:require [clj-time.coerce :as coerce]) )

(coerce/from-long 1330418007548)

#<Date Tue Feb 28 09:33:27 CET 2012>


MAPPER UNE COLONNE

arg

($map #(coerce/from-long %):ts ds ))
:ts ds

fonction anonyme


AUGMENTER LE DATASET

(defn readable-timestamp [ds]
(conj-cols ds
(dataset [:timestamp]
($map #(coerce/from-long %) :ts ds )
($map #(coerce/from-long %) :ts ds ))
))))
)))


LEIN TEST
>lein test
Testing loaderdemo.test.jmeterloader
Ran 3 tests containing 5 assertions.
0 failures, 0 errors.
Testing loaderdemo.test.xmlloader
Testing loaderdemo.test.core
Testing loaderdemo.test.jmeterloader
Testing loaderdemo.test.xmlloader


SCRIPT
demo/core.clj
(ns demo.core
:use [[incanter.core]
[demo.utils]] ;; readable-timestamp
:require [[incanter.io :as io]
[incanter.stats :as stats]
[incanter.charts :as charts]])

(let [ds (readable-timestamp(io/read-dataset "readings.csv"))]
(save (charts/time-series-plot :ts :t :data ds) "times.png")
(println ($rollup mean :t :lb ds)))


MAIN
demo/core.clj
(ns demo.core
:use [[incanter.core]
[demo.utils]] ;; readable-timestamp
:require [[incanter.io :as io]
[incanter.stats :as stats]
[incanter.charts :as charts]])

(defn -main [& args]
(let [ds (readable-timestamp(io/read-dataset "readings.csv"))]
(save (charts/time-series-plot :ts :t :data ds) "times.png")
(println ($rollup mean :t :lb ds)))

lein run
No :main namespace speciﬁed in project.clj.


LEIN RUN
(defproject demo "1.0.0-SNAPSHOT"
:description "demos incanter"
:dependencies ...
:dev-dependencies ...
:main demo.core)

lein run
read 1738


LEIN REPL

lein repl
REPL started; server listening on localhost port 61426
demo.core=>


LES GRIMOIRES
incanter.org
data-sorcery.org
github.com/liebke/incanter

Incanter sur Stackoverﬂow
Incanter Google Group


Stocker
les données

ARCHITECTURE

Document

Incanter.mongodb
MongoDB
CongoMongo
Server
Clojure


SE CONNECTER

(ns dm
(:require [somnium.congomongo :as cm]
[incanter.core :as incanter]
[incanter.mongodb :as im]))

(def conn (cm/make-connection :demodb))
#'demomongo/conn


MASS INSERT

(cm/with-mongo conn
(im/insert-dataset :readings ds))
ds)
({:ts "1330419301862",
:search__phrase "gants",
:userId "PT1022809551", ...


DANS MONGO
mongo
connecting to: test
> show dbs
demodb

0.203125GB
local

(empty)

db.readings.ﬁnd( {'lb':'SU12b-PickProduct-Page', 'shop':'MAF',
'search__phrase:‘gants’} )
{ "_id" : ObjectId("5034e71703643050fef59474"), "ts" : "1330419324275", "search__phrase" :
"gants", "userId" : "PT1022809551", "hrts" : "2012-02-28T08:55:24.275Z", "rc" : NumberLong(200),
"productId" : "8519304225180894762", "shop" : "MAF", "lb" : "SU12b-PickProduct-Page", "t" :
NumberLong(1177), "s" : "true" }
...


LIRE
(im/fetch-dataset :readings ))

(cm/fetch :readings
:where {:shop "MAF"} )
({:ts "1330419301862",
:search__phrase "gants",
:userId "PT1022809551",
:hrts "2012-02-28T08:55:01.862Z", ...


JDBC

pas d’intégration Incanter JDBC

des librairies Clojure
clojure.java.jdbc
Korma
ClojureQL


Générer
un rapport HTML

LE WEB ET CLOJURE

Enlive ou Hiccup

Moustache ou Compojure

Ring


LE RAPPORT


HICCUP
(ns hiccupdemo.core
(:use hiccup.core)
(:use hiccup.page)
(:use hiccup.element)


FORMATER LA TABLE

(defn build-table [{:keys [rows]}]
(html [:table
(table-headers)
(map table-row rows)
]))
))


FORMATER LA LIGNE

(defn table-row [{:keys
[lb count mean min max q95]}]
(html
[:tr
[:td lb]
[:td {:class "number"} count]
[:td {:class "number"} mean ]
(format "%6.2f" mean)]
...
]]))


DESTRUCTURING
DE MAP
user=> (keys statsds)
(:column-names :rows)

(defn build-table [statsds]
(html [:table
(table-headers)
(map table-row (:rows statsds))
]))


DESTRUCTURING
DE MAP
user=> (keys statsds)
(:column-names :rows)

(defn build-table [{:keys [rows]}]
(html [:table
(table-headers)
(map table-row rows)
]))


45350 relevés
"Elapsed time: 24940.557 msecs" => loader
"Elapsed time: 24542.737 msecs" => 1er chart
"Elapsed time: 3042.975 msecs" => 2eme chart
"Elapsed time: 24317.2 msecs" => stats
"Elapsed time: 122.479 msecs" => rapport

OPTIONS JVM

lein run
Exception in thread "AWT-Shutdown"
java.lang.OutOfMemoryError: Java heap space
! at java.util.IdentityHashMap$Values.iterator
(IdentityHashMap.java:1009)
! at java.util.Collections
$SynchronizedCollection.iterator(Collections.java:1573)

project.clj
:main perfdemo.core
:jvm-opts ["-Xmx1g"] )


PERFORMANCE

Console
surtout ne listez pas les données :o

Mémoire
données accumulées dans le dataset

IO Disque
si beaucoup de graphes à sauver sur disque


Faire un job
MapReduce Hadoop

MAP REDUCE


CLOJURE-HADOOP
:input-format source format
input

:map-reader fn
:map fn :reduce fn

(defjob/defjob job
:map my-map
output
:map-reader wrap/int-string-map-reader
:reduce my-reduce
:input-format :text)


MAP
(defn csv-line-parser [line]
(map #(.trim %) (first (csv/parse-csv line))))
;; require clojure-csv

(defn my-map [key line]
(cond
(= key 0) []
:else (let [ ))
... cells (csv-line-parser line)
t (nth cells 0)
(Long/parseLong (nth cells 0))
[lb, shop] (map #(nth cells %) [1,6])]
] ]
[["all" t]
[(str "lb:" lb) t]
[ lb t]
[(str t]] )))
[ shop"shop:" shop) t]] )))


REDUCE

(defn my-reduce [key values-fn]
[[(str key ":count") (count values-fn)]
[(str key ":mean") (stats/mean values-fn)]])


LEIN JAR

lein jar

et installer déployer dans hadoop


CASCALOG
Cascading + Datalog + Clojure

Dites, vous aimez Prolog ?

user=> (?<- (stdout)
[?person ?age]
(age ?person ?age)
(< ?age 30))


LES ÉCOLES DE MAGIE
4clojure.com
labrepl

Clojure sur Stackoverﬂow

Paris Clojure User Group

http://dev.clojure.org/display/community/Clojure+User+Groups


CLAUDE FALGUIERE

@cfalguiere
goo.gl/4mmJQ Merci


Incanter bigdata jsc2012

Recomendados

Recomendados

Más contenido relacionado

Más de Claude Falguiere

Más de Claude Falguiere (15)

Incanter bigdata jsc2012