Microservice Plumbing - Glynn Bird - Codemotion Rome 2017

Más contenido relacionado

La actualidad más candente

Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...

Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture. Things we’ll discuss decoupling your system at scale via event-driven architectures include: – Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows. – Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern – How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea) – Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way. – Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems – Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem. – The simplicity and power of streaming SQL with microservices

Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...

confluent

James Watters, Pivotal | Kafka Summit NYC 2019 Keynote (Spring Boot+Kafka: Th...

confluent

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

Creating a Kafka Topic. Super easy? | Andrew Stevenson and Marios Andreopoulo...

HostedbyConfluent

Pivoting Spring XD to Spring Cloud Data Flow: A microservice based architecture for stream processing Microservice based architectures are not just for distributed web applications! They are also a powerful approach for creating distributed stream processing applications. Spring Cloud Data Flow enables you to create and orchestrate standalone executable applications that communicate over messaging middleware such as Kafka and RabbitMQ that when run together, form a distributed stream processing application. This allows you to scale, version and operationalize stream processing applications following microservice based patterns and practices on a variety of runtime platforms such as Cloud Foundry, Apache YARN and others. About Sabby Anandan Sabby Anandan is a Product Manager at Pivotal. Sabby is focused on building products that eliminate the barriers between application development, cloud, and big data.

Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan

PivotalOpenSourceHub

MongoDB World 2016: NOW TV and Linear Streaming: Scaling MongoDB for High Loa...

MongoDB

Le applicazioni moderne sono sempre più orientate ad essere una composizione di API e ad avere un architettura serverless, per questo motivo chi sviluppa API non può limitarsi ad esporre i più comuni endpoint REST. I Webhook non possono mancare in un' API moderna ma non c'è nulla nella letteratura delle API HTTP che si avvicini ad un formato standard per la loro progettazione dando vita alle implementazioni più disparate. Dopo aver integrato oltre 100 API con Stamplay vi raccontiamo quali sono i pro e i contro delle scelte progettuali che si fanno nello sviluppo di Webhook.

Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...

Codemotion

MongoDB World 2016: Scaling Targeted Notifications in the Music Streaming Wor...

MongoDB

From Kafka to BigQuery - Strata Singapore

Ofir Sharony

Introduction to Google Cloud Platform

Opsta

Crash Course in Cloud Computing

All Things Open

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

Cncf event driven autoscaling with keda

JurajHantk

While SQL is a simple declarative language, it can be used in very advanced ways when querying streams of data on Kafka - in this talk Kenny will discuss techniques like advanced time specification, complex event processing (CEP), unifying sparse events, restart from failure, and even using Kafka metadata like message size.He will deep dive into how schema management, data serialization formats, Apache Flink and SQL all work together to successfully process data. He will cover advanced SQL techniques, architecture, recovery and scalability strategies from a full stack point of view. Attendees will see a demo of end-to-end processing pipeline showing features and capabilities of SQL Stream Builder that show new powerful capabilities within the SQL engine itself. They will leave well versed in rich techniques for processing data with SQL at scale and gain new tips and tricks to use in their day to day work.

Why SQL? | Kenny Gorman, Cloudera

HostedbyConfluent

Convincing developers to write tests for new code is hard; convincing developers to write tests for new event data is even harder. At Reddit, engineers have often deployed new app versions, only to find out later that the event wasn’t firing at all, or it was missing critical fields. So this begs the question, “How can engineers at Reddit be confident that the events they instrument are accurate and complete?” In this session, we will learn about an internal tool developed at Reddit to QA events in real-time. This KSQL-powered web app streams events from our pipeline, allowing developers to filter events they care about using criteria like User ID, Device ID or the type of user interaction. With a backbone of KSQL and Kafka Streams, engineers can get real-time feedback on how accurate (or how erroneous) their event data is.

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

HostedbyConfluent

The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.

Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...

confluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

La actualidad más candente (19)