Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Experimental.Flow
An overview of experimental Elixir Flow module that allows developers
to express processing steps for co...
Resources
ElixirConf 2016 - Keynote by José Valim
www.youtube.com/watch?v=srtMWzyqdp8
Kyiv Elixir Meetup 3 - Flow-Based Pr...
Resources
Announcing GenStage
elixir-lang.org/blog/2016/07/14/announcing-genstage/
gen_stage
hex.pm/packages/gen_stage
htt...
Task
Implement word counting algorithm using Eager, Lazy and Concurrent
approaches
• Eager – Enum
• Lazy – Stream
• Concur...
Eager
+ High processing speed (for small collections)
- May require large amounts of memory
Good for fast processing of sm...
Eager (Enum)
def process_eager(path_to_file) do
path_to_file
|> File.read!()
|> String.split()
|> Enum.reduce(%{}, &words_...
Helper functions
defp words_to_map(word, map) do
word
|> String.replace(~r/W/u, "")
|> filter_map(map)
end
defp filter_map...
Eager (Enum)
def process_eager(path_to_file) do
path_to_file
|> File.read!()
"The Project Gutenberg EBook of The Complete ...
Eager (Enum)
def process_eager(path_to_file) do
path_to_file
|> File.read!()
|> String.split()
["The", "Project", "Gutenbe...
Eager (Enum)
def process_eager(path_to_file) do
path_to_file
|> File.read!()
|> String.split()
|> Enum.reduce(%{}, &words_...
Lazy (Stream)
+ Allows us to “control” memory consumption
- Processing overhead
Allows us to work with large datasets with...
Lazy (Stream)
def process_lazy(path_to_file) do
path_to_file
|> File.stream!()
|> Stream.flat_map(&String.split/1)
|> Enum...
Lazy (Stream)
def process_lazy(path_to_file) do
path_to_file
|> File.stream!()
%File.Stream{line_or_bytes: :line, modes: [...
Lazy (Stream)
def process_lazy(path_to_file) do
path_to_file
|> File.stream!()
|> Stream.flat_map(&String.split/1)
#Functi...
Lazy (Stream)
"The Project Gutenberg EBook of The Complete Works
of William Shakespeare, byn"
"The", "Project", "Gutenberg...
Lazy (Stream)
"William Shakespearen"
"William", "Shakespeare“
|> Enum.reduce(%{}, &words_to_map/2)
Concurrent (Flow)
+ Concurrency
+ Allows us to “control” memory consumption
- Processing overhead
- Processing order
Allow...
GenStage is a new Elixir behaviour for exchanging events
with back-pressure between Elixir processes
producer
producer
con...
GenStage: demand-driven message exchange
producer
producer
consumer
consumer
Asks 10
Sends 10 max Sends 10 max
Asks 10
Dispatcher defines how the events are dispatched to
multiple consumers
P
C
C
C
Dispatcher
DemandDispatcher - dispatches events according to a
demand
P
C
C
C
1, 2, 3, 4, 5, 6, 7
1, 4, 7
2, 5
3, 6
PartitionDispatcher - dispatches events according to a
hash
P
C
C
C
“a”, “b”, “c”, “a”,
“d”, “c”, “a”
“c”, “c”
“a”, “a”,
“...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
%Experimental...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow)
P
PC PC
PC PC
C C
%{} %{}
"The Project Gutenberg EBook of
The Complete Works of William
Shakespeare, byn...
Concurrent (Flow)
def process_flow(path_to_file) do
path_to_file
|> File.stream!()
|> Flow.from_enumerable()
|> Flow.flat_...
Concurrent (Flow): multiple sources
streams =
for file <- File.ls!(path_to_dir) do
File.stream!(path_to_dir <> "/" <> file...
Concurrent (Flow): multiple sources
streams
|> Flow.from_enumerables()
|> Flow.flat_map(&String.split/1)
|> Flow.map(&Stri...
Concurrent (Flow): multiple sources
streams
|> Flow.from_enumerables()
|> Flow.flat_map(&String.split/1)
|> Flow.map(&Stri...
Configuration (demand, the number of stages)
Flow.partition(stages: 8)
• :stages - the number of partitions (reducer stage...
Experimental.Flow.Window
Splits a flow into windows that are materialized at certain triggers.
window = Flow.Window.global...
Experimental.Flow.Window
Flow.from_enumerable(1..100)
|> Flow.partition(window: window, stages: 1)
|> Flow.reduce(fn -> 0 ...
Questions?
twitter.com/bodarev_yurii
yuriibodarev@gmail.com
THANK YOU!!!
https://github.com/yuriibodarev/elixir_flow
Próxima SlideShare
Cargando en…5
×

Experimental.flow

56 visualizaciones

Publicado el

Presentation of Yurii Bodarev on Kyiv Elixir Meetup 3.1 (3.11.2016)

Publicado en: Software
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Experimental.flow

  1. 1. Experimental.Flow An overview of experimental Elixir Flow module that allows developers to express processing steps for collections, like Stream, but utilizing the power of parallel execution.
  2. 2. Resources ElixirConf 2016 - Keynote by José Valim www.youtube.com/watch?v=srtMWzyqdp8 Kyiv Elixir Meetup 3 - Flow-Based Programming with Elixir, Anton Mishchuk www.youtube.com/watch?v=yDaPaCxAVq8 www.slideshare.net/AntonMishchuk/flowbased-programming-with- elixir
  3. 3. Resources Announcing GenStage elixir-lang.org/blog/2016/07/14/announcing-genstage/ gen_stage hex.pm/packages/gen_stage https://hexdocs.pm/gen_stage/ https://hexdocs.pm/gen_stage/Experimental.Flow.html
  4. 4. Task Implement word counting algorithm using Eager, Lazy and Concurrent approaches • Eager – Enum • Lazy – Stream • Concurrent - Flow
  5. 5. Eager + High processing speed (for small collections) - May require large amounts of memory Good for fast processing of small collections
  6. 6. Eager (Enum) def process_eager(path_to_file) do path_to_file |> File.read!() |> String.split() |> Enum.reduce(%{}, &words_to_map/2) end
  7. 7. Helper functions defp words_to_map(word, map) do word |> String.replace(~r/W/u, "") |> filter_map(map) end defp filter_map("", map), do: map defp filter_map(word, map) do word = String.downcase(word) Map.update(map, word, 1, &(&1 + 1)) end
  8. 8. Eager (Enum) def process_eager(path_to_file) do path_to_file |> File.read!() "The Project Gutenberg EBook of The Complete Works of William Shakespeare, byrnWilliam Shakespearernrn …”
  9. 9. Eager (Enum) def process_eager(path_to_file) do path_to_file |> File.read!() |> String.split() ["The", "Project", "Gutenberg", "EBook", "of", "The", "Complete", "Works", "of", "William", "Shakespeare,", "by", "William", "Shakespeare", …]
  10. 10. Eager (Enum) def process_eager(path_to_file) do path_to_file |> File.read!() |> String.split() |> Enum.reduce(%{}, &words_to_map/2) %{"citizens" => 82, "bestrides" => 1, "oercoverd" => 1, "roots" => 10, "quintus" => 20, "ordain" => 1, "sop" => 3, "inset" => 1, …}
  11. 11. Lazy (Stream) + Allows us to “control” memory consumption - Processing overhead Allows us to work with large datasets without loading them all into memory
  12. 12. Lazy (Stream) def process_lazy(path_to_file) do path_to_file |> File.stream!() |> Stream.flat_map(&String.split/1) |> Enum.reduce(%{}, &words_to_map/2) end
  13. 13. Lazy (Stream) def process_lazy(path_to_file) do path_to_file |> File.stream!() %File.Stream{line_or_bytes: :line, modes: [:raw, :read_ahead, {:read_ahead, 100000}, :binary], path: “…/small.txt", raw: true}
  14. 14. Lazy (Stream) def process_lazy(path_to_file) do path_to_file |> File.stream!() |> Stream.flat_map(&String.split/1) #Function<57.64528250/2 in Stream.transform/3>
  15. 15. Lazy (Stream) "The Project Gutenberg EBook of The Complete Works of William Shakespeare, byn" "The", "Project", "Gutenberg", "EBook", "of", "The", "Complete", "Works", "of", "William", "Shakespeare", "by" |> Enum.reduce(%{}, &words_to_map/2)
  16. 16. Lazy (Stream) "William Shakespearen" "William", "Shakespeare“ |> Enum.reduce(%{}, &words_to_map/2)
  17. 17. Concurrent (Flow) + Concurrency + Allows us to “control” memory consumption - Processing overhead - Processing order Allows us to process large or infinite collections concurrently (on multicore machines)
  18. 18. GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes producer producer consumer producer consumer consumer
  19. 19. GenStage: demand-driven message exchange producer producer consumer consumer Asks 10 Sends 10 max Sends 10 max Asks 10
  20. 20. Dispatcher defines how the events are dispatched to multiple consumers P C C C Dispatcher
  21. 21. DemandDispatcher - dispatches events according to a demand P C C C 1, 2, 3, 4, 5, 6, 7 1, 4, 7 2, 5 3, 6
  22. 22. PartitionDispatcher - dispatches events according to a hash P C C C “a”, “b”, “c”, “a”, “d”, “c”, “a” “c”, “c” “a”, “a”, “a”, “d” “b”
  23. 23. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end
  24. 24. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() %Experimental.Flow{operations: [], options: [stages: 4], producers: {:enumerables, [%File.Stream{line_or_bytes: :line, modes: [:raw, :read_ahead, {:read_ahead, 100000}, :binary], path: “…/small.txt", raw: true}]}, window: %Experimental.Flow.Window.Global{periodically: [], trigger: nil}}
  25. 25. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) %Experimental.Flow{operations: [{:mapper, :flat_map, [&String.split/1]}], options: [stages: 4], producers: {:enumerables, [%File.Stream{line_or_bytes: :line, modes: [:raw, :read_ahead, {:read_ahead, 100000}, :binary], path: "d:/Elixir/flow/files/small.txt", raw: true}]}, window: %Experimental.Flow.Window.Global{periodically: [], trigger: nil}}
  26. 26. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end P
  27. 27. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end P PC PC DemandDispatcher
  28. 28. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end P PC PC DemandDispatcher PC PC PartitionDispatcher
  29. 29. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end P PC PC DemandDispatcher PartitionDispatcher PC PC C CReducers %{} %{}
  30. 30. Concurrent (Flow) P PC PC PC PC C C %{} %{} "The Project Gutenberg EBook of The Complete Works of William Shakespeare, byn" "William Shakespearen" "The", "Project", "Gutenberg", "EBook", "of", "The", "Complete", "Works", "of", "William", "Shakespeare,", "by" "William", "Shakespeare" "the", "project", "of", “the", "william", "of", "by ", "william" "gutenberg", "ebook", "complete", "shakespeare", "works", "shakespeare"
  31. 31. Concurrent (Flow) def process_flow(path_to_file) do path_to_file |> File.stream!() |> Flow.from_enumerable() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) end
  32. 32. Concurrent (Flow): multiple sources streams = for file <- File.ls!(path_to_dir) do File.stream!(path_to_dir <> "/" <> file, read_ahead: 100_000) end
  33. 33. Concurrent (Flow): multiple sources streams |> Flow.from_enumerables() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{})
  34. 34. Concurrent (Flow): multiple sources streams |> Flow.from_enumerables() |> Flow.flat_map(&String.split/1) |> Flow.map(&String.replace(&1, ~r/W/u, "")) |> Flow.filter_map(fn w -> w != "" end, &String.downcase/1) |> Flow.partition() |> Flow.reduce(fn -> %{} end, fn word, map -> Map.update(map, word, 1, &(&1 + 1)) end) |> Enum.into(%{}) FS P P FSFS P C %{} C %{} 2 cores 3 files
  35. 35. Configuration (demand, the number of stages) Flow.partition(stages: 8) • :stages - the number of partitions (reducer stages) • :hash - the hashing function • :max_demand - the maximum demand for this subscription • :min_demand - the minimum demand for this subscription • …
  36. 36. Experimental.Flow.Window Splits a flow into windows that are materialized at certain triggers. window = Flow.Window.global |> Flow.Window.trigger_every(10, :keep) window = Flow.Window.global |> Flow.Window.trigger_every(10, :reset)
  37. 37. Experimental.Flow.Window Flow.from_enumerable(1..100) |> Flow.partition(window: window, stages: 1) |> Flow.reduce(fn -> 0 end, & &1 + &2) |> Flow.emit(:state) |> Enum.to_list() keep> [55, 210, 465, 820, 1275, 1830, 2485, 3240, 4095, 5050, 5050] reset> [55, 155, 255, 355, 455, 555, 655, 755, 855, 955, 0]
  38. 38. Questions? twitter.com/bodarev_yurii yuriibodarev@gmail.com
  39. 39. THANK YOU!!! https://github.com/yuriibodarev/elixir_flow

×