Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Reactive Streams 1.0 and Akka Streams

Slides for the Streaming meetup http://www.meetup.com/streams/events/224961170/

  • Inicia sesión para ver los comentarios

Reactive Streams 1.0 and Akka Streams

  1. 1. Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Konrad `@ktosopl` Malawski (Channeled by @deanwampler) streams Rise of the 1.0 : reactive streams
  2. 2. Konrad `@ktosopl` Malawski akka.io typesafe.com geecon.org Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London GDGKrakow.pl lambdakrk.pl (we’re renaming soon!)
  3. 3. @deanwampler dean.wampler@typesafe.com (we’re renaming soon!)
  4. 4. Aside…
  5. 5. Streams
  6. 6. A Real Stream…
  7. 7. An Abstract Stream
  8. 8. Streams “You cannot enter the same river twice” ~ Heraclitus http://en.wikiquote.org/wiki/Heraclitus
  9. 9. Streams Real Time Stream Processing When you attach “late” to a Publisher, you may miss initial elements – it’s a river of data. http://en.wikiquote.org/wiki/Heraclitus
  10. 10. Reactive Streams 1.0
  11. 11. Reactive Streams Stream processing
  12. 12. Reactive Streams Back-pressured Stream processing
  13. 13. Reactive Streams Back-pressured Asynchronous Stream processing
  14. 14. Reactive Streams Back-pressured Asynchronous Stream processing Standardised (!)
  15. 15. Reactive Streams: Goals 1. Back-pressured Asynchronous Stream processing 2. Standard implemented by many libraries
  16. 16. Reactive Streams http://www.reactive-streams.org/
  17. 17. Reactive Streams
  18. 18. Reactive Streams - Who? http://reactive-streams.org Kaazing Corp. RxJava @ Netflix, Reactor @ Pivotal (SpringSource), Vert.x @ Red Hat, Twitter, Akka Streams, Slick @ Typesafe, Spray @ Spray.io, Oracle, OpenJDK (Java 9) – Doug Lea - SUNY Oswego …
  19. 19. Reactive Streams - Inter-op http://reactive-streams.org We want to make different implementations co-operate with each other.
  20. 20. Reactive Streams - Inter-op http://reactive-streams.org The different implementations “talk to each other” using the Reactive Streams protocol.
  21. 21. Reactive Streams - Inter-op http://reactive-streams.org The Reactive Streams SPI is NOT meant to be user-api. You should use one of the implementing libraries.
  22. 22. Reactive Streams - Inter-op example import ratpack.rx.RxRatpack import ratpack.test.embed.EmbeddedApp import ratpack.handling.Handler import ratpack.handling.Context import rx.Observable import scala.collection.JavaConverters._ import akka.stream.scaladsl.Flow import akka.stream.scaladsl.Source import rx.RxReactiveStreams import akka.stream.scaladsl.Sink import akka.actor.ActorSystem import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient
  23. 23. import ratpack.rx.RxRatpack import ratpack.test.embed.EmbeddedApp import ratpack.handling.Handler import ratpack.handling.Context import rx.Observable import scala.collection.JavaConverters._ import akka.stream.scaladsl.Flow import akka.stream.scaladsl.Source import rx.RxReactiveStreams import akka.stream.scaladsl.Sink import akka.actor.ActorSystem import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient import reactor.rx.Streams
  24. 24. import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient import reactor.rx.Streams object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava)
  25. 25. import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient import reactor.rx.Streams object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava)
  26. 26. import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient import reactor.rx.Streams object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava)
  27. 27. RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs) // Akka Streams Source val stringSource = Source(intPub).map(_.toString) // Reactive Streams Publisher val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1))
  28. 28. // Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs) // Akka Streams Source val stringSource = Source(intPub).map(_.toString) // Reactive Streams Publisher val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1)) // Reactor Stream val reactor.function.Function val linesStream = Streams.create(stringPub).map[String]( new Function[String, String] { override def apply(in: String) = in+"n" })
  29. 29. val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1)) // Reactor Stream val reactor.function.Function val linesStream = Streams.create(stringPub).map[String]( new Function[String, String] { override def apply(in: String) = in+"n" }) // and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } }) EmbeddedApp.fromHandler(handler).test(
  30. 30. // and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } }) EmbeddedApp.fromHandler(handler).test( new Consumer[TestHttpClient] { override def accept( client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } }) }
  31. 31. // and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } }) EmbeddedApp.fromHandler(handler).test( new Consumer[TestHttpClient] { override def accept( client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } }) }
  32. 32. Reactive Streams - Inter-op public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); } public interface Subscription { public void request(long n); public void cancel(); }
  33. 33. What is back-pressure?
  34. 34. Back-pressure? Example Without Publisher[T] Subscriber[T]
  35. 35. Back-pressure? Example Without Fast Publisher Slow Subscriber
  36. 36. Back-pressure? “Why would I need that!?”
  37. 37. Back-pressure? Push + NACK model
  38. 38. Back-pressure? Push + NACK model Subscriber usually has some kind of buffer.
  39. 39. Back-pressure? Push + NACK model
  40. 40. Back-pressure? Push + NACK model
  41. 41. Back-pressure? Push + NACK model What if the buffer overflows?
  42. 42. Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending
  43. 43. Back-pressure? Push + NACK model (a) Kernel does this! Routers do this! (TCP) Use bounded buffer, drop messages + require re-sending
  44. 44. Back-pressure? Push + NACK model (b) Increase buffer size… Well, while you have memory available!
  45. 45. Back-pressure? Push + NACK model (b)
  46. 46. NACKing is NOT enough! Negative ACKnowledgement
  47. 47. Back-pressure? Example NACKing Buffer overflow is imminent!
  48. 48. Back-pressure? Example NACKing Telling the Publisher to slow down / stop sending…
  49. 49. Back-pressure? Example NACKing NACK did not make it in time, because M was in-flight!
  50. 50. Back-pressure? speed(publisher) < speed(subscriber)
  51. 51. Back-pressure? Fast Subscriber, No Problem No problem!
  52. 52. Back-pressure? Reactive-Streams = “Dynamic Push/Pull”
  53. 53. Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber Back-pressure? RS: Dynamic Push/Pull
  54. 54. Solution: Dynamic adjustment Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber
  55. 55. Back-pressure? RS: Dynamic Push/Pull Slow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.
  56. 56. Back-pressure? RS: Dynamic Push/Pull Fast Publisher will send at-most 3 elements. This is pull-based-backpressure.
  57. 57. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives!
  58. 58. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives. Publisher can accumulate demand.
  59. 59. Back-pressure? RS: Accumulate demand Publisher accumulates total demand per subscriber.
  60. 60. Back-pressure? RS: Accumulate demand Total demand of elements is safe to publish. Subscriber’s buffer will not overflow.
  61. 61. Back-pressure? RS: Requesting “a lot” Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)
  62. 62. streams
  63. 63. Akka • Fault tolerant • Supervision hierarchies • Failure detection • Asynchronous data processing • Optimised for high performance • both in-jvm and across network • Adaptive Cluster • Load balancing among Cluster Nodes Actor 131 Actor 132 Supervisor 1 Actor 12 Actor 13 Actor 111 Actor 112 Supervisor 11
  64. 64. Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
  65. 65. Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
  66. 66. An Actor can only: • Send and receive messages • Create Actors • Change it’s behaviour Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
  67. 67. class Player extends Actor { def receive = { case NextTurn => sender() ! decideOnMove() } def decideOnMove(): Move = ??? } Akka Actors
  68. 68. Akka Actors are: a distribution and concurrency abstraction. Streams are: a way to describe how data flows through a system.
  69. 69. Akka Streams – Linear Flow
  70. 70. Akka Streams – Linear Flow
  71. 71. Akka Streams – Linear Flow
  72. 72. Akka Streams – Linear Flow
  73. 73. Akka Streams – Linear Flow Flow[Double].map(_.toInt). [...] No Source attached yet. “Pipe ready to work with Doubles”.
  74. 74. Akka Streams – Linear Flow implicit val sys = ActorSystem() implicit val mat = ActorMaterializer() Source(1 to 3).runWith(Sink.foreach(println))
  75. 75. Akka Streams – Linear Flow Source(1 to 3).runWith(Sink.foreach(println)) // sugar for runWith Source(1 to 3).foreach(println) implicit val sys = ActorSystem() implicit val mat = ActorMaterializer()
  76. 76. Akka Streams – Linear Flow Source(1 to 3).runWith(Sink.foreach(println)) // sugar for runWith Source(1 to 3).foreach(println) Sink.fold Sink.head Sink.ignore Sink.publisher Sink.cancelled // your own Sink … implicit val sys = ActorSystem() implicit val mat = ActorMaterializer()
  77. 77. Akka Streams – Flows are reusable sink.runWith(Source(1 to 10)) sink.runWith(Source(1 to 100)) sink.runWith(Source(1 to 1000)) source.runWith(Sink.ignore) source.runWith(Sink.foreach(println)) Multiple materializations val ready = source.to(Sink.ignore) ready.run() ready.run()
  78. 78. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”) ) Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber) All the usual ops available for Linear Flows.
  79. 79. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”) ) Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber) Aggregating values until downstream demand comes.
  80. 80. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”) ) Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber) Creates a stream of streams: Source[(Int, Source[String])]
  81. 81. Akka Streams: Graphs val p: Publisher[String] = FlowGraph.closed(out) { implicit b o val merge = b.add(new StrictRoundRobin[String])
 in1 ~> merge.in(0) in2 ~> merge.in(1) 
 merge.out ~> o.inlet }.run() val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f")) val out = Sink.publisher[String]
  82. 82. Akka Streams: Graphs val p: Publisher[String] = FlowGraph.closed(out) { implicit b o val merge = b.add(new StrictRoundRobin[String])
 in1 ~> merge.in(0) in2 ~> merge.in(1) 
 merge.out ~> o.inlet }.run() val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f")) val out = Sink.publisher[String] Sink[String, Publisher[String]] imports Graphs
  83. 83. Akka Streams: Graphs val p: Publisher[String] = FlowGraph.closed(out) { implicit b o val merge = b.add(new StrictRoundRobin[String])
 in1 ~> merge.in(0) in2 ~> merge.in(1) 
 merge.out ~> o.inlet }.run() val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f")) val out = Sink.publisher[String] Sink[String, Publisher[String]] materializes a Publisher[String]
  84. 84. Reactive Streams Bigger than Scala-ecosystem - JDK-wide (and wider). Inter-operable back-pressure protocol. Future work: reactive-streams-io, reactive-streams-js Akka Streams - one of the leading Reactive Streams impls. Complex in-memory stream processing. SlickR - provides Reactive Stream from DB queries Akka Http - Akka Streams based; “Spray 2.0”
  85. 85. Wrapping up
  86. 86. Available Sources •FutureSource •IterableSource •IteratorSource •PublisherSource •SubscriberSource •ThunkSource •SynchronousFileSource •TickSource (timer based) •… easy to add your own!
  87. 87. Available operations • drop / dropWithin • take / takeWithin • filter • groupBy • grouped • transform • buffer • collect • expand • splitWhen / splitAfter • map • scan • prefixAndTail • … easy to add your own! “Rate – detaching” operations: • buffer • collect • concat • conflate
  88. 88. Available Sinks • BlackHoleSink • FoldSink • ForeachSink • FutureSink • OnCompleteSink • UdpSink [next release] • SynchronousFileSink • PublisherSink / FanoutPublisherSink • SubscriberSink • FileSink • … easy to add your own!
  89. 89. Available Junctions • Broadcast • Merge • FlexiMerge • Route • FlexiRoute • Zip • ZipWith • Unzip • UnzipWith • Concat • … easy to add your own!
  90. 90. Community ecosystem • Kafka • Spark (soon?) • Slick • S3 • Files • MongoDB • …
  91. 91. Links • The projects: • http://akka.io • http://reactive-streams.org
 • Akka User - mailing list: • https://groups.google.com/group/akka-user

  92. 92. Tänan! Dzięki! Thanks! ありがとう! ktoso @ typesafe.com t: ktosopl / g: ktoso blog: project13.pl
  93. 93. ©Typesafe 2015 – All Rights Reserved The company soon to be previously known as

×