Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

•

0 recomendaciones•1,548 vistas

In this talk, Peter will cover his experience using Spark, Cassandra & Kafka to build a real time analytics platform that processed billions events a day. He will cover the challenges in how to turn all those raw events into actionable insights. He will also cover scaling the platform across multiple regions, as well as across multiple cloud environments.

Datos y análisis

Zero to Insights
Real time analytics with Kafka, C*, and Spark
Peter Bakas

Peter Bakas | @peter_bakas
@ Netflix : Cloud Platform Engineering - Event and Data Pipelines
@ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure
@ Yahoo : Display Advertising, Behavioral Targeting, Payments
@ PayPal : Site Engineering and Architecture
@ Play : Advisor to various Startups (Data, Security, Containers)
Who is this guy?

● 450 billion events per day
● 8 million events & 17 GB per second during
peak
● Hundreds of event types
By the Numbers

Publish, Collect, Process, Aggregate & Move Data

Event
Producer
Druid
EMR
Stream
Consumers
Kafka
Router
Suro
Event
Producer

Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone

Consumer
Kafka
Custom Apps
Real time processing

Consumer
Kafka
Custom Apps
Real time processing
Fronting
Kafka

About Ooyala
Powering personalized
video experiences
across all screens.

● 5 billion events per day
● 1 billion videos per month
● 200 million unique users per month
● 130 countries
● 25% of U.S. online viewers watch video powered by Ooyala
By the Numbers

What were the options?
100% Precomputation 100% Dynamic

Where we wanted to be
100% Precomputation 100% Dynamic
Partly dynamic

API
loggersloggersloggersloggersloggers
loggersloggersloggersloggersingest
loggersloggersloggersloggersjob server
Delphi - Real time Analytics
Kafka

● Hiring
● Rapidly evolving ecosystem
● Enterprise Service for Enterprise Software
Challenges

Obligatory...
Everyone is hiring
pbakas@netflix.com

Más contenido relacionado

La actualidad más candente

APIdays Paris 2018 - What a Mesh! Laurent Doguin, DevRel VP, Clever Cloud

apidays

Realtime Conf - Lightning node.js dev

Glenn Block

Ingest, store, analyze, and monitor a large volume of real-time streaming data using cloud-based managed services. The reference architecture will showcase the following solution areas using managed services/serverless technologies.- Setup and Manage Mult-Region infrastructure via CI/CD - Ingest and Store a large volume of streaming data (Ex: 100K records/second) - Analyze and derive insights in near real-time - Monitor the infrastructure and pipelines

Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...

Data Con LA

Meetup roundupazure

Simon Hayward

Event Grid is an Azure service that enables developers to route events between endpoints. It provides support for virtually any existing Azure service from storage activity to serverless Azure Function calls. It also has support for custom end points. Instead of taking on the costly overhead of continuous polling in your applications, Event Grid will manage the events for you and publish them to your app when ready. In this hands-on presentation, learn about the true value of Event Grid by seeing an example in action and understand what sets Event Grid apart from other services like Service Bus and messaging queues.

Azure Event Grid: Glue for the Internet

Jeremy Likness

APIdays Paris 2018 - Cloud computing - we went through every steps of the Gar...

apidays

Rockaway AWS Hackathon

fersman

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Using Telegraf, InfluxDB and Mist Mist.io are the creators of Mist, the open source multi-cloud management platform. Their solution helps clients manage the complexities of using a mix of public and private clouds, hypervisors, containers and bare metals. Mist streamlines its customers’ operations and optimizes costs by improving DevOps practices. Discover how Mist achieves that by building on top of Telegraf and InfluxDB. Join this webinar as Chris Psaltis and Dimitris Moraitis dive into: Mist’s approach to improving visibility across environments How a time series database is used to monitor infrastructures Mist’s ability to automate processes and reduce infrastructure costs

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...

InfluxData

Streaming applications are at the core of realtime information system. With microservices streaming components are becoming increasingly important. With the rise of cloud architecture, it is now easier and more elegant to use Go to write realtime high-performance components for streaming architecture than to use traditional tools. Mickaël explains what are streaming architectures and demonstrates the benefits of using Go to build such platform.

Go for Real Time Streaming Architectures - DotGo 2017

Mickaël Rémond

Modern application delivery with Consul

Mitchell Pronschinske

Building a multi-tenant PaaS is not a walk in the part, certainly if the platform you are building on is constantly changing. In this session I'll walk you through the adventure we've been on where you'll learn about the challenges we've had and how we approached them and whether or not our decisions worked out or not. – How to design for scale – How to operate the platform – How to grow a platform mindset and force ownership – How to run tests for your whole platform – How to design for multi-tenancy – How to approach constant change – etc Cloud projects are never finished so you'd better come prepared.

Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...

Tom Kerkhove

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

InfluxData

Cloud@eBay

Open Networking Summits

SnapLogic Live: IoT Integration

SnapLogic

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021

InfluxData

Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...

Codemotion

Using FLaNK with InfluxDB for EdgeAI IoT at Scale Timothy from StreamNative take you on a hands-on deep-dive on using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team run deep learning models on the edge devices and send images, and capture real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

InfluxData

Spring Cloud Kubernetes

Mauricio (Salaboy) Salatino

Infrastructure management and observability can be challenging in industries like financial services where uptime is of paramount importance. Learn how AWS and InfluxDB solutions overcome these challenges by delivering real-time visibility into systems to help quickly identify and mitigate potential outages. We’ll share real examples of how these solutions were implemented, providing better observability into infrastructure and transactions, and how these organizations were able to pass on these performance improvements to their customers as a result. Finally, this session will share how you can get started with a free trial of InfluxDB on AWS. 

Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I...

InfluxData

Open data is on the rise, and freely available data sets - like municipal data - can bring huge value and new features to applications, at no upfront cost. For example, open crime data can be leveraged to support features that make location based apps safer for users. However, the challenge for developers is to efficiently wrangle, store and deliver open data sets without having to build complex data architectures. Using the example of crime data applied to a new safety feature for Pokemon Go, this webinar will demonstrate how to harvest open geo data, store it natively in a cloud database (Cloudant), and ensure it is highly available for applications and analytics platforms.

Optimizing location-based apps with open data

Raj Singh

La actualidad más candente (20)

APIdays Paris 2018 - What a Mesh! Laurent Doguin, DevRel VP, Clever Cloud

Realtime Conf - Lightning node.js dev

Data Con LA 2019 - Large scale streaming analytics using cloud based managed ...

Meetup roundupazure

Azure Event Grid: Glue for the Internet

APIdays Paris 2018 - Cloud computing - we went through every steps of the Gar...

Rockaway AWS Hackathon

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...

Go for Real Time Streaming Architectures - DotGo 2017

Modern application delivery with Consul

Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Cloud@eBay

SnapLogic Live: IoT Integration

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021

Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Spring Cloud Kubernetes

Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I...

Optimizing location-based apps with open data

Similar a Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

The Internet of Things (IoT) is getting more and more traction as valuable use cases come to light. Whether you are in Healthcare, Telecommunications, Manufacturing, Banking or Retail to name a few industries, there is one key challenge and that's the integration of backend IoT data logs and applications, business services and cloud services to process the data in real time and at scale. In this talk, we will be sharing how Kafka has become the leading technology used throughout the business to provide Real Time Event Streaming. Explore real life use cases of Kafka Connect, Kafka Streams and KSQL independent of the data deployment be it on a private or public Cloud, On Premise or at the Edge. Audi - Connected car infrastructure Robert Bosch Power Tools - Track and Trace of devices and people at construction areas Deutsche Bahn - Customer 360 for train timetable updates E.ON - IoT Streaming Platform to integrate and build smart home, smart building and smart grid infrastructures

IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...

Kai Wähner

IoT and Event Streaming at Scale with Apache Kafka

confluent

Swissport ist weltweit der führende Dienstleister im Groundhandling und bei Cargo. Über 300 Flughäfen in 50 Ländern werden bedient. Dabei spielen Daten eine führende Rolle: wann und wo hebt ein Flugzeug ab, wie lange dauerte die Betankung, welche Gepäcktücke sind auszuladen? Aufgrund von Wachstum durch Übernahmen ist die IT Landschaft durch eine Vielzahl von Silos charakterisiert - was übergreifende Analytik und die Nutzung von Daten in neuen Kontexten erschwert. Gleichzeitig werden weltweit verschiedene IT Governance Modelle verfolgt, was zu Inkonsistenzen in Prozessen, Datenzugriffen und Datenqualitätsverlusten führt. Diese Situation wird adressiert durch eine Vision für eine Eventgetriebene Architektur und deren Verankerung im Management Prozessen und Prinzipien zur Realisierung der Vision - sowie deren Umsetzung den Aufbau eines Enterprise Datenmodells, Governance für den Zugriff und die Dokumentation von Daten. Erfahrungen aus diesen Schritten werden in der Präsentation reflektiert. Dabei ist zu beachten, dass der Aufbau der zugrundeliegenden Plattform unter strikter Kostenkontrolle stand und lediglich 10 zweiwöchentliche Sprints für die erste Produktivsetzung genutzt werden konnten.

batbern43 Events - Lessons learnt building an Enterprise Data Bus

BATbern

Apache Kafka® and Analytics in a Connected IoT World

confluent

Slides from JupyterCon 2018 in NYC on 8/23/2018. Notebooks have moved beyond a niche solution at Netflix; they are now the critical path for how everyone runs jobs against the company’s data platform. From creating original content to delivering bufferless streaming, Netflix relies on notebooks to inform decisions and fuel experiments across the company. Netflix also uses notebooks to power its machine learning infrastructure and run over 150,000 jobs against its 100 PB cloud-based data warehouse every day. The goal is to deliver a compelling notebooks experience that simplifies end-to-end workflows for every type of user. To enable this, Netflix is investing deeply in notebook infrastructure and open source projects such as nteract. In this talk, Michelle Ufford and Kyle Kelley share interesting ways Netflix uses data and some of the big bets the company is making on notebooks. Topics will include architecture, kernels, UIs, and Netflix’s open source collaborations with projects such as Jupyter, nteract, pandas, and Spark.

Notebooks @ Netflix: From analytics to engineering with Jupyter notebooks

Michelle Ufford

Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025

Nicola Sandoli

Set Your Data In Motion - CTO Roundtable

confluent

Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage. Agenda: - Data in Motion with Event Streaming and Apache Kafka - Streaming ETL Pipelines - IT Modernisation and Hybrid Multi-Cloud - Customer Experience and Customer 360 - IoT and Big Data Processing - Machine Learning and Analytics

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Kai Wähner

The Briefing Room with Dez Blanchfield and Striim Living in the moment is often touted as solid advice for a happy life. The same can now be said for business. Thanks to a confluence of innovations, the practice of streaming analytics is fast becoming the gold standard for today's most innovative enterprises. Whether for real-time responsiveness, optimal operations, customer relations, or any number of use cases, the immediacy of analytics has taken a turn for the better. Register for this episode of The Briefing Room to hear Data Scientist Dez Blanchfield explain why streaming analytics is taking the enterprise by storm. He'll be briefed by Steve Wilkes of Striim, who will demonstrate how his company's platform was designed to leverage a new generation of information architectures. He'll show several use cases, including hybrid cloud, replication validation, and multi-log correlation, which can tackle a variety of business needs.

A Real-Time Version of the Truth

Eric Kavanagh

Canzoni sotto l'albero con Azure IoT

Riccardo Cappello

Real Time analytics with Druid, Apache Spark and Kafka

Daria Litvinov

Introducing Pico - Object Detection & Analytics using Docker, IoT & Amazon Re...

Ajeet Singh Raina

You want to implement a Big Data or Internet of Things (IoT) solution and like to know if it should be implemented in the cloud or on-premises. You are interested in the cloud offerings of vendors and what benefits they provide and if a similar solution would not be possible on-premises. This presentation deals with this and other questions. Starting from a vendor-independent reference architecture and corresponding design patterns, different cloud solutions from various vendors are compared and rated. Additionally, it will be shown how such solution could be implemented on-premises and how a hybrid IoT solution could look like.

Internet of Things (IoT) - in the cloud or rather on-premises?

Guido Schmutz

Druid meetup @ Netflix (11/14/2018 )

Jaebin Yoon

Watch this talk here: https://www.confluent.io/online-talks/gcp-for-apache-kafka-users-stream-ingestion-processing In private and public clouds, stream analytics commonly means stateless processing systems organized around Apache Kafka® or a similar distributed log service. GCP took a somewhat different tack, with Cloud Pub/Sub, Dataflow, and BigQuery, distributing the responsibility for processing among ingestion, processing and database technologies. We compare the two approaches to data integration and show how Dataflow allows you to join and transform and deliver data streams among on-prem and cloud Apache Kafka clusters, Cloud Pub/Sub topics and a variety of databases. The session will have a mix of architectural discussions and practical code reviews of Dataflow-based pipelines.

GCP for Apache Kafka® Users: Stream Ingestion and Processing

confluent

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

Slides from Michelle Ufford's talk, Data-Driven @ Netflix. Talk given at PASS Summit 2016 in October 2016. Netflix is the quintessential data-driven company. It’s 83 million members stream more than 125 million hours in over 190 countries every day and generate more than 700 billion events in the process. In this session, we’ll share how data is used to make informed decisions across the entire business — from content acquisition to content delivery, and everything in between. We’ll look at how Netflix successfully employs a scalable cloud-based data platform to support a constant deluge of data and a small army of data analysts, engineers, and scientists. We’ll discuss the advanced analytical capabilities that are enabled through modern data technologies. Lastly, we’ll explore some of the architectural & operational principals that enable Netflix to so effectively make use of its data.

Data-Driven @ Netflix

Michelle Ufford

OIT Technology, Communications, Japan

Christos Makiyama

Both CQRS and Event Sourcing are by no means “new stuff” anymore, yet a lot can be told about how to use Azure’s PaaS to implement such patterns and unleash their power. The ingredients are: DocumentDB as the event storage, Service Bus as the events’ dispatcher, Could Services/Service Fabric as the scalable, fault tolerant business logic container, SQL Azure as the read model and ASP .NET Core as the application framework used to implement views and back-end services. Eager to know the recipe? Don’t miss this talk then.

Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello

ITCamp

Net conf uy v2018 real time analytics

Gaston Cruz

Similar a Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015 (20)

IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...

IoT and Event Streaming at Scale with Apache Kafka

batbern43 Events - Lessons learnt building an Enterprise Data Bus

Apache Kafka® and Analytics in a Connected IoT World

Notebooks @ Netflix: From analytics to engineering with Jupyter notebooks

Tibco Augmented Intelligence - Analytics, IoT, Big Data, Streaming 20161025

Set Your Data In Motion - CTO Roundtable

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

A Real-Time Version of the Truth

Canzoni sotto l'albero con Azure IoT

Real Time analytics with Druid, Apache Spark and Kafka

Introducing Pico - Object Detection & Analytics using Docker, IoT & Amazon Re...

Internet of Things (IoT) - in the cloud or rather on-premises?

Druid meetup @ Netflix (11/14/2018 )

GCP for Apache Kafka® Users: Stream Ingestion and Processing

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

Data-Driven @ Netflix

OIT Technology, Communications, Japan

Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello

Net conf uy v2018 real time analytics

Más de NoSQLmatters

While metrics generated by static code analysis are well established as predictors of possible future defects, there is another untapped source of useful information, namely your source code revision history. This presentation will discuss converting this revision information into a graph representation, various defect prediction models and how to generate their related change metrics through graph traversal, as well as the potential applications and benefits of these graph enabled prediction models.

Nathan Ford- Divination of the Defects (Graph-Based Defect Prediction through...

NoSQLmatters

Stefan Hochdörfer - The NoSQL Store everyone ignores: PostgreSQL - NoSQL matt...

NoSQLmatters

Adrian Colyer - Keynote: NoSQL matters - NoSQL matters Dublin 2015

NoSQLmatters

Data analysis is an exploratory process that requires a variety of tools and a flexible data store. Data analysis projects are easy to start but quickly become difficult to manage and error prone when depending on file-based data storage. Relational databases are poorly equipped to accommodate the dynamic demands complex analysis. This talk describes best practices for using MongoDB for analytics projects. Examples will be drawn from a large scale text mining project (approximately 25 million documents) that applies machine learning (neural networks and support vector machines) and statistical analysis. Tools discussed include R, Spark, Python scientific stack, and custom pre-processing scripts but the focus is on using these with the document database.

Dan Sullivan - Data Analytics and Text Mining with MongoDB - NoSQL matters Du...

NoSQLmatters

Sometimes we need to step back and take a look at the bigger picture - not just counting huge piles of individual log records, but reasoning about the behaviors of the people who are ultimately generating this firehose of data. While your DevOps folks care deeply about log records from a machine utlization perspective, marketing wants to know what these records tell us about the customers' needs. Elasticsearch Aggregations are a great feature but are not a panacea. We can happily use them to summarise complex things like the number of web requests per day broken down by geography and browser type on a busy website, but we would quickly run out of memory if we tried to calculate something as simple as a single number for the average duration of visitor web sessions when using the very same dataset. Why does this occur? A web session duration is an example of a behavioural attribute not held on any one log record; it has to be derived by finding the first and last records for each session in our weblogs, requiring some complex query expressions and a lot of memory to connect all the data points. We can maintain a more useful joined-up-picture if we run an ongoing background process to fuse related events from one index into ?entity-centric? summaries in another index e.g: • Web log events summarised into ?web session? entities • Road-worthiness test results summarised into ?car? entities • Reviews in a marketplace summarised into a ?reviewer? entity Using real data, this session will demonstrate how to incrementally build entity-centric indexes alongside event-centric indexes by using simple scripts to uncover interesting behaviours that accumulate over time. We'll explore: • Which cars are driven long distances after failing roadworthiness tests? • Which website visitors look to be behaving like ?bots?? • Which seller in my marketplace has employed an army of ?shills? to boost his feedback rating? Attendees will leave this session with all the tools required to begin building entity-centric indexes and using that data to derive richer business insights across every department in their organization.

Mark Harwood - Building Entity Centric Indexes - NoSQL matters Dublin 2015

NoSQLmatters

Relevance and Personalization is crucial to building personalized local commerce experience at Groupon. Talk covers overview of the real time analytics infrastructure that handles over 3 million events/ second and stores and scales to billions of data points. Solution covers how our Kafka -> Storm -> Redis/ HBase pipeline is used to generate real time analytics for hundreds of millions of users of Groupon. Solution includes various architecture design choices and tradeoffs including some interesting algorithmic choices such as Bloom Filters & Hyper Log Log. Attendees can take away learnings from our real-life experience that can help them understand various tuning methods, their tradeoffs and apply them in their solutions

Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase -...

NoSQLmatters

Building applications on streaming data has its challenges. If you are trying to use programs such as Apache Spark or Storm to build applications, this presentation will explain the advantages and disadvantages of each solution and how to choose the right tool for your next streaming data project. Building streaming data applications that can manage the massive quantities of data generated from mobile devices, M2M, sensors and other IoT devices, is a big challenge that many organizations face today. Traditional tools, such as conventional database systems, do not have the capacity to ingest data, analyze it in real-time, and make decisions. New technologies such as Apache Spark and Storm are now coming to the forefront as possible solutions to handing fast data streams. Typical technology choices fall into one of three categories: OLAP, OLTP, and stream-processing systems. Each of these solutions has its benefits, but some choices support streaming data and application development much better than others. Employing a solution that handles streaming data, provides state, ensures durability, and supports transactions and real-time decisions is key to benefitting from fast data. During this presentation you will learn: - The difference between fast OLAP, stream-processing, and OLTP database solutions. - The importance of state, real-time analytics and real-time decisions when building applications on streaming data. - How streaming applications deliver more value when built on a super-fast in-memory, SQL database.

Akmal Chaudhri - How to Build Streaming Data Applications: Evaluating the Top...

NoSQLmatters

Just a few years ago all software systems were designed to be monoliths running on a single big and powerful machine. But nowadays most companies desire to scale out instead of scaling up, because it is much easier to buy or rent a large cluster of commodity hardware then to get a single machine that is powerful enough. In the database area scaling out is realized by utilizing a combination of polyglot persistence and sharding of data. On the application level scaling out is realized by microservices. In this talk I will briefly introduce the concepts and ideas of microservices and discuss their benefits and drawbacks. Afterwards I will focus on the point of intersection of a microservice based application talking to one or many NoSQL databases. We will try and find answers to these questions: Are the differences to a monolithic application? How to scale the whole system properly? What about polyglot persistence? Is there a data-centric way to split microservices?

Michael Hackstein - NoSQL meets Microservices - NoSQL matters Dublin 2015

NoSQLmatters

In this talk we'll focus on the use of Crate alongside Weave in Docker containers, the technical challenges, best practices learned, and getting a big data application running alongside it. You'll learn about the reasons why Crate.IO is building "yet another NoSQL database" and why it's unique and important when running web scale containerized applications. We'll show why the shared-nothing architecture is so important when deploying large clusters in containers and how it addresses the issues and fears of a Docker-based persistence layer. You will learn how to deploy a Crate cluster in the cloud within minutes using Docker, some of the challenges you'll encounter, and how to overcome them in order to scale your backends efficiently. We focused on super simple integration with any cloud provider, striving it to be as turnkey as possible with minimal up-front configuration required to establish a cluster. Once established, we'll show how to scale the cluster horizontally by simply adding more nodes. The session will also give you examples when you should use Crate compared to other similar technologies such as MongoDB, Hadoop, Cassandra or FoundationDB. We'll talk about this approach's strengths and what types of applications are well-suited for this type of data store, as well what is not. Finally we'll outline how to architect an application that is easy to scale using Crate and Docker.

Chris Ward - Understanding databases for distributed docker applications - No...

NoSQLmatters

More than two years ago we faced the decision whether to run our MongoDB database on Amazon's EC2 ourselves or to rely on a Database as a Service provider. Common wisdom told us that a well known provider, focusing all its knowledge and energy on running MongoDB, would be a better choice than us trying it on the side. Well, this talk describes what can go wrong, since we have seen a lot of interesting minor and major hiccups — including stopped instances, broken backups, a major security incident, and more broken backups. Additionally, we discuss some reasons why a hosted solution is not always the better choice and which new challenges arise from it.

Philipp Krenn - Host your database in the cloud, they said... - NoSQL matters...

NoSQLmatters

What if we would try to make Elasticsearch SQL 92 compliant (http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt)? This wouldn't serve that much nowadays, you would say. Well, we actually tried to do the exercise and we have some interesting conclusions. While we take Elasticsearch as an example for this "side by side", the issues we are addressing also apply to nosql in general. With this unusual exercise, we take the occasion to compare relational databases / sql with Elasticsearch / nosql on all the levels : functionality, semantics, performance and user experience.

Lucian Precup - Back to the Future: SQL 92 for Elasticsearch? - NoSQL matters...

NoSQLmatters

Bruno Guedes - Hadoop real time for dummies - NoSQL matters Paris 2015

NoSQLmatters

Apache Spark is a general data processing framework which allows you perform map-reduce tasks (but not only) in memory. Apache Cassandra is a highly available and massively scalable NoSQL data-store. By combining Spark flexible API and Cassandra performance, we get an interesting alternative to the Hadoop eco-system for both real-time and batch processing. During this talk we will highlight the tight integration between Spark & Cassandra and demonstrate some usages with live code demo.

DuyHai DOAN - Real time analytics with Cassandra and Spark - NoSQL matters Pa...

NoSQLmatters

When deploying your service to Microsoft Azure, you have a number of options in terms of noSQL: you can install databases on Linux or Windows virtual machines by yourself, or via the marketplace, or you can use open source databases available as a service like HBase or proprietary and managed databases like Document DB. After showing these options, we'll show Document DB in more details. This is a noSQL database as a service that stores JSON.

Benjamin Guinebertière - Microsoft Azure: Document DB and other noSQL databas...

NoSQLmatters

How do you mix SQL and NoSQL worlds without starting a messy revolution?This live coding talk will show you how to add Elasticsearch to your legacy application without changing all your current development habits. Your application will have suddenly have advanced search features, all without the need to write complex SQL code!David will start from a Spring, Hibernate and Postgresql based application and will add a complete integration of Elasticsearch, all live from the stage during his presentation.

David Pilato - Advance search for your legacy application - NoSQL matters Par...

NoSQLmatters

During this live-coding session, Tugdual will move an old fashion full SQL application (JavaEE) to the new NoSQL world.Using MongoDB, and REST, he will show the benefits of this new architecture: * Easyness * Flexibility * High availability * Scalability; During this presentation, you will learn more about: * Document Oriented Model * JSON * REST * Iterative development; This demonstration is also a good opportunity to see how you can migrate data from a relational database, and the various schema options.

Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015

NoSQLmatters

How do you monitor performance for one of your clients on a specific user segmentation when dealing with billions of events a day ? With over 2 billion ads served and 230Tb of data processed a day, we at Criteo have a comprehensive need for an interactive analytics stack. And by interactive, we mean a querying system with dynamic filtering to drill down over multiple dimensions, answering within sub-second latency. This session will take you on our journey with Druid, ""an open-source data store designed for real-time exploratory analytics on large data sets"". We will explore Druid's architecture and noticeable concepts, how relevant they are for some use cases and how it really performs.

Gregorry Letribot - Druid at Criteo - NoSQL matters 2015

NoSQLmatters

In many modern applications the database side is realized using polyglot persistence – store each data format (graphs, documents, etc.) in an appropriate separate database. This approach yields several benefits, databases are optimized for their specific duty, however there are also drawbacks: * keep all databases in sync * queries might require data from several databases * experts needed for all used systems A multi-model database is not restricted to one data format, but can cope with several of them. In this talk i will present how a multi-model database can be used in a polyglot persistence setup and how it will reduce the effort drastically.

Michael Hackstein - Polyglot Persistence & Multi-Model NoSQL Databases - NoSQ...

NoSQLmatters

The impact that NoSQL has had on the technology community cannot be overstated. The proliferation of new and exciting data systems has led to a slew of interesting solutions to problems that were once solved the relational way. In this session we explore all that is great and good about NoSQL: the innovative software, the clever storage paradigms and the reigniting of developer interest in data access. It is unfortunate that NoSQL is not only a force for good in our community. We'll explore some of the darker corners of NoSQL: the disregard for years of proven technology, the overbearing hype, the overblown marketing and the ever present arguments over which technology is best. We close the session by exploring what can be done to extract even more value from the NoSQL movement, where we can improve how the community interacts with the larger technology community and what the future holds for data access technologies.

Rob Harrop- Key Note The God, the Bad and the Ugly - NoSQL matters Paris 2015

NoSQLmatters

Come to this deep dive on how Pivotal's Data Lake Vision is evolving by embracing next generation in-memory data exchange and compute technologies around Spark and Tachyon. Did we say Hadoop, SQL, and what's the shortest path to get from past to future state? The next generation of data lake technology will leverage the availability of in-memory processing, with an architecture that supports multiple data analytics workloads within a single environment: SQL, R, Spark, batch and transactional.

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

NoSQLmatters

Más de NoSQLmatters (20)