Loom and Graphs in Clojure

Loom and Graphs
in Clojure
github.com/aysylu/loom
Aysylu Greenberg
@aysylu22; http://aysy.lu
LispNYC, August 13th 2013

Overview
•  Loom's Graph API
•  Graph Algorithms in Loom
•  Titanium Loom
•  Single Static Assignment (SSA) Loom

Overview
•  Graph Algorithms in Loom
•  SSA Loom

Loom's Graph API
•  Graph, Digraph, Weighted Graph

Loom's Graph API
•  FlyGraph

Loom's Graph API
•  FlyGraph
o  read-only, ad-hoc

Loom's Graph API
•  FlyGraph
o  edges from nodes + successors

Loom's Graph API
•  FlyGraph
o  edges from nodes + successors
o  nodes and edges from successors + start

Loom's Graph API
•  Uses Clojure protocols
(clojure.org/protocols)

Loom's Graph API
o  specification only, no implementation

Loom's Graph API
o  single type can implement multiple
protocols

Loom's Graph API
o  single type can implement multiple
protocols
o  interfaces: design-time choice of the type
author, protocols: can be added to a type
at runtime

Loom's Graph API
(defprotocol Graph
(add-nodes* [g nodes] "Add nodes to graph g. See add-nodes”
(add-edges* [g edges] "Add edges to graph g. See add-edges")

Loom's Graph API
(defprotocol Graph
(add-edges* [g edges] "Add edges to graph g. See add-edges”)
(remove-nodes* [g nodes] "Remove nodes from graph g. See
remove-nodes”)
(remove-edges* [g edges] "Removes edges from graph g. See
remove-edges”)
(remove-all [g] "Removes all nodes and edges from graph g")

Loom's Graph API
(defprotocol Graph
(add-edges* [g edges] "Add edges to graph g. See add-edges”
remove-nodes”)
remove-edges”)
(remove-all [g] "Removes all nodes and edges from graph g”
(nodes [g] "Return a collection of the nodes in graph g”)
(edges [g] "Edges in g. May return each edge twice in an undirected
graph")

Loom's Graph API
(defprotocol Graph
remove-nodes”)
remove-edges”)
graph”)
(has-node? [g node] "Return true when node is in g”)
(has-edge? [g n1 n2] "Return true when edge [n1 n2] is in g")

Loom's Graph API
(defprotocol Graph
remove-nodes”)
remove-edges”)
graph”)
(has-node? [g node] "Return true when node is in g”)
(has-edge? [g n1 n2] "Return true when edge [n1 n2] is in g”)
(successors [g] [g node] "Return direct successors of node, or
(partial successors g)”)
(out-degree [g node] "Return the number of direct successors of
node"))

Loom's Graph API
(defprotocol Digraph
(predecessors [g] [g node] "Return direct
predecessors of node, or (partial predecessors g)”)
(in-degree [g node] "Return the number direct
predecessors to node")

Loom's Graph API
(defprotocol Digraph
(predecessors [g] [g node] "Return direct
predecessors of node, or (partial predecessors g)”)
(in-degree [g node] "Return the number direct
predecessors to node”)
(transpose [g] "Return a graph with all edges
reversed"))

Loom's Graph API
(defprotocol WeightedGraph
(weight [g] [g n1 n2] "Return weight of edge [n1 n2]
or (partial weight g)"))

Graph Algorithms in Loom
•  DFS/BFS (+ bidirectional)

•  Topological Sort

•  Single Source Shortest Path (Dijkstra, Bellman-Ford)

•  Strongly Connected Components (Kosaraju)

•  Density (edges/nodes)

•  Loner nodes

•  Loner nodes
•  2 coloring

•  Loner nodes
•  2 coloring
•  Max-Flow (Edmonds-Karp)

•  Loner nodes
•  2 coloring
•  Max-Flow (Edmonds-Karp)
•  alg-generic requires only successors + start (where
appropriate)

Graph Algorithms: Bellman-Ford
A B C
D E
3 4
5
2
-8

CLRS Introduction to Algorithms

(defn- init-estimates
"Initializes path cost estimates and paths from source to all vertices, for
Bellman-Ford algorithm”
[graph start]
(let [nodes (disj (nodes graph) start)
path-costs {start 0}
paths {start nil}
infinities (repeat Double/POSITIVE_INFINITY)
nils (repeat nil)
init-costs (interleave nodes infinities)
init-paths (interleave nodes nils)]
[(apply assoc path-costs init-costs)
(apply assoc paths init-paths)]))

(defn- can-relax-edge?
"Test for whether we can improve the shortest path to v found so far by
going through u.”
[[u v :as edge] weight costs]
(let [vd (get costs v)
ud (get costs u)
sum (+ ud weight)]
(> vd sum)))

(defn- relax-edge
"If there's a shorter path from s to v via u, update our map of
estimated path costs and map of paths from source to vertex v”
[[u v :as edge] weight [costs paths :as estimates]]
(let [ud (get costs u)
sum (+ ud weight)]
(if (can-relax-edge? edge weight costs)
[(assoc costs v sum) (assoc paths v u)]
estimates)))

(defn- relax-edges
"Performs edge relaxation on all edges in weighted directed graph”
[g start estimates]
(->> (edges g)
(reduce (fn [estimates [u v :as edge]]
(relax-edge edge (wt g u v) estimates))
estimates)))

(defn bellman-ford
"Given a weighted, directed graph G = (V, E) with source start, the Bellman-Ford algorithm produces map of single source shortest paths and their
costs if no negative-weight cycle that is reachable from the source exits, and false otherwise, indicating that no solution exists."
[g start]
(let [initial-estimates (init-estimates g start)
;relax-edges is calculated for all edges V-1 times
[costs paths] (reduce (fn [estimates _]
(relax-edges g start estimates))
initial-estimates
(-> g nodes count dec range))
edges (edges g)]
(if (some
(fn [[u v :as edge]]
(can-relax-edge? edge (wt g u v) costs))
edges)
false
[costs
(->> (keys paths)
;remove vertices that are unreachable from source
(remove #(= Double/POSITIVE_INFINITY (get costs %)))
(reduce
(fn [final-paths v]
(assoc final-paths v
; follows the parent pointers
; to construct path from source to node v
(loop [node v
path ()]
(if node
(recur (get paths node) (cons node path))
path))))
{}))])))

(defn bellman-ford
"Given a weighted, directed graph G = (V, E) with source start,
the Bellman-Ford algorithm produces map of single source
shortest paths and their costs if no negative-weight cycle that is
reachable from the source exits, and false otherwise, indicating
that no solution exists."

[g start]
(let [initial-estimates (init-estimates g start)
;relax-edges is calculated for all edges V-1 times
[costs paths] (reduce (fn [estimates _]
(relax-edges g start estimates))
initial-estimates
(->> g (nodes) (count) (dec) (range)))
edges (edges g)]

(if (some (fn [[u v :as edge]]
(can-relax-edge? edge (wt g u v) costs))
edges)
false

[costs
(->> (keys paths)
(remove
#(= Double/POSITIVE_INFINITY (get costs %)))

[costs
(->> (keys paths)
(remove
#(= Double/POSITIVE_INFINITY (get costs %)))
(reduce
(fn [final-paths v]
(assoc final-paths v
; follows the parent pointers
; to construct path from source to node v
(loop [node v path ()]
(if node
(recur (get paths node) (cons node path))
path))))
{}))])))

Overview
•  Graph Algorithms
•  SSA Loom

Titanium Loom
•  Titanium by Clojurewerkz
(titanium.clojurewerkz.org)

Titanium Loom
•  Clojure graph library built on top of Aurelius
Titan (thinkaurelius.github.com/titan)

Titanium Loom
•  Various storage backends: Cassandra,
HBase, BerkeleyDB Java Edition

Titanium Loom
•  Various storage backends: Cassandra,
HBase, BerkeleyDB Java Edition
•  No graph visualization

Titanium Loom
(let [in-mem-graph (tg/open {"storage.backend" "inmemory"})]
(tg/transact!
(let [
a (nodes/create! {:name "Node A"})
b (nodes/create! {:name "Node B"})
c (nodes/create! {:name "Node C"})

Titanium Loom
(let [in-mem-graph (tg/open {"storage.backend" "inmemory"})]
(tg/transact!
(let [
a (nodes/create! {:name "Node A"})
b (nodes/create! {:name "Node B"})
c (nodes/create! {:name "Node C"})
e1 (edges/connect! a "edge A->B" b)
e2 (edges/connect! b "edge B->C" c)
e3 (edges/connect! c "edge C->A” a)
graph (titanium->loom in-mem-graph)])

Titanium Loom
(defn titanium->loom
"Converts titanium graph into Loom representation”
([titanium-graph &
{:keys [node-fn edge-fn weight-fn]
:or {node-fn (nodes/get-all-vertices)
edge-fn (map (juxt edges/tail-vertex
edges/head-vertex)
(edges/get-all-edges))
weight-fn (constantly 1)}}]
(let [nodes-set (set node-fn)
edges-set (set edge-fn)]

Titanium Loom
(reify
Graph
(nodes [_] nodes-set)
(edges [_] edges-set)
(has-node? [g node] (contains? (nodes g) node))
(has-edge? [g n1 n2] (contains? (edges g) [n1 n2]))
(successors [g] (partial successors g))
(successors [g node] (filter (nodes g)
(seq (nodes/connected-out-vertices node))))
(out-degree [g node] (count (successors g node)))

Titanium Loom
(reify
Graph
(nodes [_] nodes-set)
(edges [_] edges-set)
(has-node? [g node] (contains? (nodes g) node))
(has-edge? [g n1 n2] (contains? (edges g) [n1 n2]))
(successors [g node] (filter (nodes g)
(seq (nodes/connected-out-vertices node))))
Digraph
(predecessors [g] (partial predecessors g))
(predecessors [g node] (filter (nodes g)
(seq (nodes/connected-in-vertices node))))
(in-degree [g node] (count (predecessors g node)))
WeightedGraph
(weight [g] (partial weight g))
(weight [g n1 n2] (weight-fn n1 n2))))))

SSA Loom
•  Single Static Assignment (SSA) form
produced by core.async

SSA Loom
•  Single Static Assignment (SSA) form
produced by core.async
•  Generated by parse-to-state-machine
function

SSA Loom
(parse-to-state-machine
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])

SSA Loom
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
[inst_4938
{:current-block 76,
:start-block 73,
:block-catches {76 nil, 75 nil, 74 nil, 73 nil},
:blocks
{76
[{:value :clojure.core.async.impl.ioc-macros/value, :id
inst_4937}
{:value inst_4937, :id inst_4938}],
75
[{:refs [clojure.core/+ x 2], :id inst_4935}
{:value inst_4935, :block 76, :id inst_4936}],
74
73
[{:refs [clojure.core/+ x 1 2 y], :id inst_4930}
{:refs [clojure.core/> inst_4930 0], :id inst_4931}
{:test inst_4931, :then-block 74, :else-block 75, :id
inst_4932}]}}]

SSA Loom
(def ssa
(->>
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
second
:blocks))

SSA Loom
{76
[{:value :clojure.core.async.impl.ioc-macros/
value, :id inst_4937}
{:value inst_4937, :id inst_4938}],
75
74
73
[{:refs [clojure.core/+ x 1 2 y], :id inst_4930}
{:refs [clojure.core/> inst_4930 0], :id
inst_4931}
{:test inst_4931, :then-block 74, :else-block
75, :id inst_4932}]}}]
(def ssa
(->>
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
second
:blocks))

SSA Loom
(view (ssa->loom ssa ssa-nodes-fn ssa-edges-fn))

SSA Loom
(defn ssa->loom
"Converts the SSA form generated by core.async into Loom
representation.”
([ssa node-fn edge-fn]
(let [nodes (delay (node-fn ssa))
edges (delay (edge-fn ssa))]

SSA Loom
{:graph (reify
Graph
(nodes [g] @nodes)
(edges [g] @edges)
(has-node? [g node] (contains? @nodes node))
(has-edge? [g n1 n2] (contains? @edges [n1 n2]))
(successors [g node]
(->> @edges
(filter (fn [[n1 n2]] (= n1 node)))
(map second)))

SSA Loom
Digraph
(predecessors [g] (partial predecessors g))
(predecessors [g node]
(->> @edges
(filter (fn [[n1 n2]] (= n2 node)))
(map first)))
(in-degree [g node] (count (predecessors g node))))
:data ssa})))

SSA Loom: Dataflow Analysis
•  For each basic block, solve system of
equations until reaching fixed point:
•  Use worklist approach

(defn dataflow-analysis
"Performs dataflow analysis. Nodes have value nil initially.”
[& {:keys [start graph join transfer]}]
(let [start (cond
(set? start) start
(coll? start) (set start)
:else #{start})]

(loop [out-values {}
[node & worklist]
(into clojure.lang.PersistentQueue/EMPTY start)
(let [in-value (join
(mapv out-values (predecessors graph node)))
out (transfer node in-value)
update? (not= out (get out-values node))
out-values (if update?
(assoc out-values node out)
out-values)
worklist (if update?
(into worklist (successors graph node))
worklist)]

(loop [out-values {}
[node & worklist]
(into clojure.lang.PersistentQueue/EMPTY start
(let [in-value (join
(mapv out-values (predecessors graph node)))
out (transfer node in-value)
update? (not= out (get out-values node))
out-values (if update?
(assoc out-values node out)
out-values)
worklist (if update?
(into worklist (successors graph node))
worklist)]
(if (seq worklist)
(recur out-values worklist)
out-values)))))

SSA Loom: Global Availability
(defn global-cse
[ssa]
(let [{graph :graph node-data :data}
(ssa->loom (:blocks ssa) ssa-nodes-fn ssa-edges-fn)
start (:start-block ssa)]
(letfn [(pure? [instr] (contains? instr :refs))
(global-cse-join
[values]
(if (seq values)
(apply set/intersection values)
#{}))
(global-cse-transfer
[node in-value]
(into in-value
(map :refs (filter pure? (node-data node)))))]

SSA Loom: Global Availability
(defn global-cse
[ssa]
(let [{graph :graph node-data :data}
(ssa->loom (:blocks ssa) ssa-nodes-fn ssa-edges-fn)
start (:start-block ssa)]
(letfn [(pure? [instr] (contains? instr :refs))
(global-cse-join
[values]
(if (seq values)
(apply set/intersection values)
#{}))
(global-cse-transfer
[node in-value]
(into in-value
(map :refs (filter pure? (node-data node)))))]
(dataflow-analysis :start start :graph graph
:join global-cse-join
:transfer global-cse-transfer))))

• Reaching definitions

• Liveness analysis (dead code elimination)

• Available expressions

• Constant propagation

• Other Applications:

o  Erdős number

o  Erdős number
o  Spread of information in systems (e.g. taint)

My Experience
• Intuitive way to implement algorithms
functionally
• Some mental overhead of transforming data
structures

Open Questions
• How general should a graph API be?

Open Questions
• How general should a graph API be?
• How feature-rich should a graph API be?

Loom and Graphs in Clojure

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (17)

Similar a Loom and Graphs in Clojure

Similar a Loom and Graphs in Clojure (20)

Más de Aysylu Greenberg

Más de Aysylu Greenberg (20)

Último

Último (20)

Loom and Graphs in Clojure