Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

ZIO Queue

1.988 visualizaciones

Publicado el

At the heart of data processing, event-sourcing, actors, and much more is the queue—a data structure that allows producers to pass work to consumers in a flexible way. On the JVM, most libraries, including Akka, are powered by Java concurrent queues, which are reliable but were designed in a different era, for synchronous (blocking) procedural code. In this presentation, John A. De Goes—architect of the Scalaz 8 effect system—introduces IOQueue, a new type of queue powered by the Scalaz 8 IO monad. IOQueue never blocks and provides seamless, composable back-pressure across an entire application, without users having to think about the problem or write any special code. John will discuss how IOQueue achieves these remarkable properties, and show how the structure can be used to solve hard problems with just a few lines of type-safe, leak-free, composable Scala code. Come learn about the power of Scalaz to solve the hard problems of software development, in a principled way, without compromises.

Publicado en: Tecnología
  • Sé el primero en comentar

ZIO Queue

  1. 1. ZIO Queue John A. De Goes — @jdegoes - http://degoes.net ScalaWave 2018 ZIO Queue ScalaWave 2018 - Gdańsk, Poland John A. De Goes @jdegoes - http://degoes.net
  2. 2. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  3. 3. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up ZIO ZIO lets you build high-performance, type-safe, concurrent, asynchronous applications that don’t leak resources or deadlock, and are easy to reason about compositionally, test, and refactor.
  4. 4. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up github.com/scalaz/scalaz-zio ZIO
  5. 5. def main: Unit = { println("Hello, what is your name?") val name = readLine() println(s"Good morning, $name!") } Second-Class Effects ✗ Pass to functions ✗ Return from functions ✗ Store in data structures ✗ Async/sync/concurrency ✓ Resource-safety What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  6. 6. def main: IO[IOException, Unit] = for { _ <- putStrLn("Hello, what is your name?") name <- getStrLn _ <- putStrLn(s"Good morning, $name!") } yield () First-Class Effects ✓ Pass to functions ✓ Return from functions ✓ Store in data structures ✓ Async/sync/concurrency ✓ Resource-safety What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  7. 7. IO[E, A] IO[E, A] is an immutable value that describes an effectful computation, which may fail with a value of type E, or compute a value of type A. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  8. 8. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  9. 9. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  10. 10. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  11. 11. def runInParallel[E, A, B]( leftIO : IO[E, A], rightIO: IO[E, B]): IO[E, (A, B)] = for { leftFiber <- leftIO.fork rightFiber <- rightIO.fork a <- leftFiber.join b <- rightFiber.join } yield (a, b) Fork/Join Concurrency with Fibers ✓ Massive scalability over threads ✓ Non-blocking ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up
  12. 12. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Powered by Queues
  13. 13. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Producer Consumer offer take Queue 1-to-1
  14. 14. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Producer Consumer offer take Queue Producer n-to-1
  15. 15. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Producer Consumer offer take Queue Consumer 1-to-m
  16. 16. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Consumer offer take Queue Consumer Producer Producer n-to-m
  17. 17. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up FIFO LIFO PRIO Bounded Unbounded Dropping Sliding dimensions of variation
  18. 18. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up queue actors send
  19. 19. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up concurrent graph search
  20. 20. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up resource sharing scarce resource e.g. threads Thread 1 take Thread 2offer
  21. 21. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up loose coupling offer take Fast Producers Slow Consumers buffer
  22. 22. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Troubled Queues
  23. 23. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue
  24. 24. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue trait BlockingQueue[A] { def take: A def offer(a: A): Boolean }
  25. 25. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue Blocks one thread until element added trait BlockingQueue[A] { def take: A def offer(a: A): Boolean }
  26. 26. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue Fast Producer Slow Consumer offer take Queue
  27. 27. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue Fast Producer Slow Consumer offer take Queue manual backpressure
  28. 28. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue def fastProducer = { if (!queue.offer(a)) { // Slow down!!! } else { // Keep producing... } }
  29. 29. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.BlockingQueue Fast Producer Slow Consumer offer take Queue manual backpressure manual backpressure take
  30. 30. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.ConcurrentLinkedQueue trait ConcurrentLinkedQueue[A] { def poll: A def offer(a: A): Boolean } Returns null if empty
  31. 31. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up java.util.concurrent.ConcurrentLinkedQueue Fast Producer Slow Consumer offer poll Queue
  32. 32. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up github.com/scalaz/scalaz-zio ZIO QUEUE
  33. 33. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up ZIO Queue trait Queue[A] { def take: IO[Nothing, A] def offer(a: A): IO[Nothing, Unit] def shutdown(ts: Throwable*): IO[Nothing, Unit] } object Queue { def bounded[A](capacity: Int): IO[Nothing, Queue[A]] = ??? def unbounded[A]: IO[Nothing, Queue[A]] = ??? }
  34. 34. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Example for { queue <- Queue.unbounded[String] _ <- queue.offer("Give me Coffee!").forever.fork _ <- queue.take.flatMap(putStrLn).forever } yield ()
  35. 35. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 5. Concurrent Consumer offer take Queue Consumer Producer Producer ZIO
  36. 36. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 4. Interruptible for { queue <- Queue.unbounded[String] fiber <- queue.offer("Give me Coffee!").forever.fork _ <- fiber.interrupt } yield ()
  37. 37. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 3. Clean Shutdown for { queue <- Queue.unbounded[String] fiber1 <- queue.offer("Give me Coffee!").forever.fork fiber2 <- queue.take.forever.fork _ <- queue.shutdown } yield ()
  38. 38. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 2. Asynchronous for { queue <- Queue.unbounded[String] worker = queue.take.flatMap(putStrLn).forever workers10k = List.fill(10000)(worker) _ <- IO.forkAll(workers10k) _ <- queue.offer("More Coffee!").forever.fork } yield ()
  39. 39. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 2. Asynchronous trait BlockingQueue[A] { def take: A ; def offer(a: A): Boolean }
  40. 40. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up 1. Composable Backpressure for { queue <- Queue.bounded[String](10) worker = queue.offer("Coffee").forever workers10k = List.fill(10000)(worker) _ <- IO.forkAll(workers10k) _ <- queue.take.flatMap(putStrLn).forever.fork } yield ()
  41. 41. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors case class Actor[+E, -A, +B](run: A => IO[E, B]) { self => def ! (a: A): IO[E, B] = run(a) }
  42. 42. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  43. 43. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  44. 44. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  45. 45. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  46. 46. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  47. 47. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  48. 48. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  49. 49. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  50. 50. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  51. 51. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  52. 52. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  53. 53. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeActor[S, E, A, B](supervisor: Supervisor)(s: S)(receive: (S, A) => IO[E, (S, B)]): IO[Nothing, Actor[E, A, B]] = for { state <- Ref(s) queue <- Queue.bounded[(A, Promise[E, B])](QueueSize) fiber <- (for { t <- queue.take s <- state.get (a, promise) = t receiver = receive(s, a) completer = (s: S, b: B) => state.set(s) *> promise.complete(b) _ <- receiver.redeem( e => supervise(receiver, e) .redeem(promise.error, completer), completer) } yield ()).forever.fork } yield Actor[E, A, B]((a: A) => Promise.make[E, B].flatMap(p => queue.offer((a, p)).flatMap(_ => p.get))
  54. 54. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Application: Scalaz Actors def makeCounter: IO[Nothing, Actor[Nothing, Int, Int]] = makeActor(Supervisor.logDefects(Logging))(0) { (state: Int, n: Int) => IO.now((state + n, state + n)) } for { counter <- makeCounter _ <- counter ! 40 _ <- counter ! -2 v <- counter ! 0 } yield v
  55. 55. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up WRAP UP
  56. 56. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up github.com/scalaz/scalaz-zio ZIO CFC
  57. 57. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up ZIO ROADMAP Performance Sliding Dropping
  58. 58. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up https://gitter.im/scalaz/scalaz/ https://gitter.im/scalaz/scalaz-ioqueue/
  59. 59. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up Functional Scala by John A. De Goes Local Remote Date Scotland European Time Sep 10 - 14 Toronto East Coast Time Oct 1 - 5 ? European Time Oct 22 - 26 SF / SV Pacific Coast Time Nov 18 - 22
  60. 60. What is ZIO? Powered by Queues Troubled Queues Introducing ZIO Queue Wrap Up THANK YOU! @jdegoes - http://degoes.net

×