Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Drinking from the firehose, with virtual streams and virtual actors

17 visualizaciones

Publicado el

Event Stream Processing is a popular paradigm for building robust and performant systems in many different domains, from IoT to fraud detection to high-frequency trading. Because of the wide range of scenarios and requirements, it is difficult to conceptualize a unified programming model that would be equally applicable to all of them. Another tough challenge is how to build streaming systems with cardinalities of topics ranging from hundreds to billions while delivering good performance and scalability.

In this session, Sergey Bykov will talk about the journey of building Orleans Streams that originated in gaming and monitoring scenarios, and quickly expanded beyond them. He will cover the programming model of virtual streams that emerged as a natural extension of the virtual actor model of Orleans, the architecture of the underlying runtime system, the compromises and hard choices made in the process. Sergey will share the lessons learned from the experience of running the system in production, and future ideas and opportunities that remain to be explored.

Publicado en: Software
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Drinking from the firehose, with virtual streams and virtual actors

  1. 1. Drinking from the Firehose With Virtual Streams & Virtual Actors Sergey Bykov Microsoft, @sergeybykov
  2. 2. Event Streaming
  3. 3. Compute Storage Typical Workloads
  4. 4. User X Event User X State Updated User X State User X Event User X Event Processing an Event
  5. 5. Challenges Y X X Z X Z Y Z C A C B B B A C D E F F DD D E B A C E F D Z A Z X X Z X Z Y X A A B C C C B A E D E F F F E D A Y Z C F D B A Storage High volume of events ▪Need multiple servers Each event requires context ▪Load state from storage ▪Write results to storage Uncoordinated IO is expensive ▪Caching ▪Write conflicts Servers may go down ▪Clustering / coordination needed
  6. 6. User X Event User X Event Processing an Event User X Event User X State Updated User X State
  7. 7. User X Event User X State Updated User X State Benefits of Single Processor User X Event User X Event User Y Event User Y Event User Y Event User Z Event User Z Event User Z Event
  8. 8. Orleans Architecture Frontends Orleans Cluster Storage Grain Persisted Activating Active in memory Deactivating
  9. 9. Challenges Y X X Z X Z Y Z C A C B B B A C D E F F DD D E B A C E F D Z A Z X X Z X Z Y X A A B C C C B A E D E F F F E D A Y Z C F D B A Storage High volume of events ▪Need multiple servers Each event requires context ▪Load state from storage ▪Write results to storage Uncoordinated IO is expensive ▪Caching ▪Write conflicts Servers may go down ▪Clustering / coordination needed
  10. 10. PA PA PA PA Messaging Cluster Membership Grain Directory PA PA PA PA Messaging Cluster Membership Grain Directory Pulling Agents Y X X Z X Z Y Z C A C B B B A C D E F F DD D E B A C E F D Z A Z X X Z X Z Y X A A B C C C B A E D E F F F E D A Y Z C F D B A
  11. 11. Virtual Streams A A A A A A A A B B B B B B B B C C C C C C C C A A A A A A A A B B B B B B B B C C C C C C C C A B B C B C A A A B B C A B B B C A A C A B B A DD E F E E D F D E F F DD E E F F E F D E DD X Y Y Z Y Z X X X Y Y Z X Y Y Y Z X X Z X Y Y X A B C D D Logical (Virtual) Streams 1000s to Ms to Bs Physical Persistent Queues 10s to 1000s
  12. 12. Programming Model public class UserGrain : Grain, IAsyncObserver<GameEvent> { public async Task Initialize() { var provider = GetStreamProvider(“EventHub Provider”); var stream = provider.GetStream<GameEvent>(userId, “PlayerEvents”); await stream.Subscribe(this); } public async Task OnNext(GameEvent event) { // Processing logic ... await AwardAchievement(Achievements.MasterChief); } ... public async Task AwardAchievement(Achievements achievement) { var stream = provider.GetStream<Achievements>(streamId, “Achievements”); await stream.OnNext(achievement); } } // Rx-like interface to implement public interface IAsyncObserver<T> { Task OnNext(T event); Task OnComplete(); Task OnError(); }
  13. 13. Event Streaming is a powerful paradigm Routing events to affected entity’s state is much better ▪ Reduces read and write load on storage ▪ Can serve RPC calls from same in-memory state Virtual Actors are natural fit for processing events ▪ Cluster objects with stable identities ▪ Automatic resource management Virtual Streams provide a simple but powerful model ▪ Orleans code is open source (MIT license) ▪ Runs on Windows, Linux, MacOS (via .NET Core) Takeaways
  14. 14. Muchas Gracias! Sergey Bykov Microsoft, @sergeybykov

×