Converging Big Data and Application Infrastructure by Steven Poutsy

•

1 recomendación•370 vistas

https://www.bigdataspain.org/2016/program/fri-converging-big-data-application-infrastructure.html https://www.youtube.com/watch?v=zwmbI47gLEg&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=44&t=8s

Tecnología

Converging Big Data and
Application Infrastructure
Steven Pousty
OpenShift/Red Hat
@TheSteve0 on Twitter, IRC,SmugMug,Github,
Ingress, Instagram, Skype,and

Goals
1. Brief introduction to how things have evolved
2. Talk about what we have now
3. Demo of this in practice
4. Have fun!

Assumptions
1. You have written a web app at some point in your life
2. You know some basics of Spark and such

We have had some great developments
1. People figured out how to make commodity hardware do great things
2. Distributed software has gotten better
3. Programming languages have gotten easier
4. CS has put effort into statistical libraries
5. The browser has grown up

Example framework
Docker Containers
Kubernetes (via OpenShift)
Node.JS and Python
Spark (via Oshinko)

We are lucky
We can still do analysis on our desktop if we want
But now we can easily integrate
1. Data or APIs
2. Statistics and Machine Learning
3. Interactive Web Applications

To Do this on your own:
1. To run your own OpenShift either:
https://www.openshift.org/vm/
Or
oc cluster up (get oc here - https://github.com/openshift/origin/releases )
2. To get Oshinko:
https://github.com/radanalyticsio then you need oshinko-rest and oshinko-
console
3. For the word analysis app get it here:
https://github.com/ophicleide
We LOVE pull requests

Más contenido relacionado

La actualidad más candente

Apache Flink community Update for March 2016 - Slim Baltagi

Slim Baltagi

The Power of Unified Analytics with Ali Ghodsi

Databricks

At Netflix, we've spent a lot of time thinking about how we can make our analytics group move quickly. Netflix's Data Engineering & Analytics organization embraces the company's culture of "Freedom & Responsibility". How does a company with a $40 billion market cap and $6 billion in annual revenue keep their data teams moving with the agility of a tiny company? How do hundreds of data engineers and scientists make the best decisions for their projects independently, without the analytics environment devolving into chaos? We'll talk about how Netflix equips its business intelligence and data engineers with: the freedom to leverage cloud-based data tools - Spark, Presto, Redshift, Tableau and others - in ways that solve our most difficult data problems the freedom to find and introduce right software for the job - even if it isn't used anywhere else in-house the freedom to create and drop new tables in production without approval the freedom to choose when a question is a one-off, and when a question is asked often enough to require a self-service tool the freedom to retire analytics and data processes whose value doesn't justify their support costs Speaker Bios Monisha Kanoth is a Senior Data Architect at Netflix, and was one of the founding members of the current streaming Content Analytics team. She previously worked as a big data lead at Convertro (acquired by AOL) and as a data warehouse lead at MySpace. Jason Flittner is a Senior Business Intelligence Engineer at Netflix, focusing on data transformation, analysis, and visualization as part of the Content Data Engineering & Analytics team. He previously led the EC2 Business Intelligence team at Amazon Web Services and was a business intelligence engineer with Cisco. Chris Stephens is a Senior Data Engineer at Netflix. He previously served as the CTO at Deep 6 Analytics, a machine learning & content analytics company in Los Angeles, and on the data warehouse teams at the FOX Audience Network and Anheuser-Busch.

Rapid Data Analytics @ Netflix

Data Con LA

Extracting Insights from Data at Twitter

Prasad Wagle

Technology behind-real-time-log-analytics

Data Science Thailand

Organising for Data Success

Lars Albertsson

The Business Economics and Opportunity of Open Source Data Science

Revolution Analytics

No sql and sql - open analytics summit

Open Analytics

The presentation aims to demystify the practice of building reliable data processing pipelines. It includes a brief overview of the pieces needed to build a stable processing platform: data ingestion,processing engines, workflow management, and schemas. For each component, suitable components are suggested, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes. Original document: https://goo.gl/rmKxZM

Data pipelines from zero

Lars Albertsson

“The ability to detect malware has needed to drastically change in the past few years away from traditional signature or list based techniques. Couple this with the rise of mobile device based attacks, where the scale of the data is predicted to be 60% of the internet in 2018*, our online lives will need Machine Learning (ML) and Data Science to ensure its security. At Wandera we have successfully implemented a malware detection (and classification) ML model at scale with the use of Apache Spark (MLib) and the PMML via OpenScoring paradigm. In this talk we will touch on the training data and why we use Spark at all, the features we extract from mobile phone applications and how we then obtain our high accuracy scores in the cloud. At Wandera we have successfully implemented a Malware detection (and classification) ML model at scale with the use of Apache Spark (MLib) and the PMML via OpenScoring paradigm. *https://blog.cloudflare.com/our-predictions-for-2018/”

Detecting Mobile Malware with Apache Spark with David Pryce

Databricks

Realtime streaming architecture in INFINARIO

Jozo Kovac

"Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT YOU’LL LEARN: – Understand the key data reliability and performance data pipelines challenges – How Databricks Delta helps build robust pipelines at scale – Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements – How to deliver performance gains using Delta PREREQUISITES: – A fully-charged laptop (8-16GB memory) with Chrome or Firefox – Pre-register for Databricks Community Edition" Speakers: Steven Yu, Burak Yavuz

Building Robust Production Data Pipelines with Databricks Delta

Databricks

The Spark Listener interface provides a fast, simple and efficient route to monitoring and observing your Spark application - and you can start using it in minutes. In this talk, we'll introduce the Spark Listener interfaces available in core and streaming applications, and show a few ways in which they've changed our world for the better at SpotX. If you're looking for a "Eureka!" moment in monitoring or tracking of your Spark apps, look no further than Spark Listeners and this talk!

Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring

Databricks

Bay Area Apache Flink Meetup Community Update August 2015

Henry Saputra

Upwork has the biggest closed-loop online dataset of jobs and job seekers in labor history (>10M Profiles, >100M Job Posts, Job Proposals and Hiring Decisions, >10B of Messages, Transaction and Feedback Data). Besides sheer quantity, our data is also contextually very rich. We have client and contractor data for the entire job-funnel – from finding jobs to getting the job done. For various machine learning applications including search and recommendations and labor marketplace optimization (rate, supply and demand), we heavily relied on a Greenplum-based data warehouse solution for data processing and ad-hoc ML pipelines (weka, scikit-learn, R) for offline model development and online model scoring. In this talk, we present our modernization efforts in moving towards a 1) holistic data processing infrastructure for batch and stream data processing using S3, Kinesis, Spark and Spark Structured Streaming 2) model development using Spark MLlib and other ML libraries for Spark 3) model serving using Databricks Model Scoring, Scoring over Structured Streams and microservices and 3) how we orchestrate and streamline all these processes using Apache Airflow and a CI/CD workflow customized to our Data Science product engineering needs. The focus of this talk is on how we were able to leverage the Databricks service offering to reduce DevOps overhead and costs, complete the entire modernization with moderate efforts and adopt a collaborative notebook-based solution for all our data scientists to develop model, reuse features and share results. We will shared the core lessons learned and pitfalls we encountered during this journey.

How to Rebuild an End-to-End ML Pipeline with Databricks and Upwork with Than...

Databricks

These are the slides of my talk on June 30, 2015 at the first event of the Chicago Apache Flink meetup. Although most of the current buzz is about Apache Spark, the talk shows how Apache Flink offers the only hybrid open source (Real-Time Streaming + Batch) distributed data processing engine supporting many use cases: Real-Time stream processing, machine learning at scale, graph analytics and batch processing. In these slides, you will find answers to the following questions: What is Apache Flink stack and how it fits into the Big Data ecosystem? How Apache Flink integrates with Apache Hadoop and other open source tools for data input and output as well as deployment? What is the architecture of Apache Flink? What are the different execution modes of Apache Flink? Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark? Who is using Apache Flink? Where to learn more about Apache Flink?

Overview of Apache Flink: Next-Gen Big Data Analytics Framework

Slim Baltagi

Around the world, businesses are turning to AI to transform the way they operate and serve their customers. But before they can implement these technologies, companies must address the roadblock of moving from batch analytics to making real-time decisions by rapidly accessing and analyzing the relevant information amidst a sea of data. Yaron will explain how to make Spark handle multivariate real-time, historical and event data simultaneously to provide immediate and intelligent responses. He will present several time sensitive use-cases including fraud detection, prevention of outages and customer recommendations to demonstrate how to perform predictive analytics and real-time actions with Spark. Speaker: Yaron Ekshtein

Real-Time Analytics and Actions Across Large Data Sets with Apache Spark

Databricks

Zalandos AI-driven products and distributed landscape of analytical data marts cannot wait for long-running, hard-to-recover, monolithic batch jobs taking all night to calculate already outdated data. Modern data integration pipelines need to deliver fast and easy to consume data sets in high quality. Based on Spark Streaming and Delta, the central data warehousing team was able to deliver widely-used master data as S3 or Kafka streams and snapshots at the same time. The talk will cover challenges in our fashion data platform and a detailed architectural deep dive about separation of integration from enrichment, providing streams as well as snapshots and feeding the data to distributed data marts. Finally, lessons learned and best practices about Delta’s MERGE command, Scala API vs Spark SQL and schema evolution give more insights and guidance for similar use cases.

Data Warehousing with Spark Streaming at Zalando

Databricks

PyCon Singapore 2013 Keynote

Wes McKinney

La actualidad más candente (19)

Apache Flink community Update for March 2016 - Slim Baltagi

The Power of Unified Analytics with Ali Ghodsi

Rapid Data Analytics @ Netflix

Extracting Insights from Data at Twitter

Technology behind-real-time-log-analytics

Organising for Data Success

The Business Economics and Opportunity of Open Source Data Science

No sql and sql - open analytics summit

Data pipelines from zero

Detecting Mobile Malware with Apache Spark with David Pryce

Realtime streaming architecture in INFINARIO

Building Robust Production Data Pipelines with Databricks Delta

Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring

Bay Area Apache Flink Meetup Community Update August 2015

How to Rebuild an End-to-End ML Pipeline with Databricks and Upwork with Than...

Overview of Apache Flink: Next-Gen Big Data Analytics Framework

Real-Time Analytics and Actions Across Large Data Sets with Apache Spark

Data Warehousing with Spark Streaming at Zalando

PyCon Singapore 2013 Keynote

Destacado

Advanced data science algorithms applied to scalable stream processing by Dav...

Big Data Spain

Enabling the Bank of the Future by Ignacio Bernal

Big Data Spain

Why Apache Flink is better than Spark by Rubén Casado

Big Data Spain

Big Migrations: Moving elephant herds by Carlos Izquierdo

Big Data Spain

Case of success: Visualization as an example for exercising democratic transp...

Big Data Spain

Finding the needle in the haystack: how Nestle is leveraging big data to defe...

Big Data Spain

Big data in 140 characters by Joe Rice

Big Data Spain

From data to AI with the Machine Learning Canvas by Louis Dorard Slides

Big Data Spain

Assessing spatial accessibility to primary health care services in the Metrop...

Big Data Spain

Stream Processing use cases and applications with Apache Apex by Thomas Weise

Big Data Spain

VP of WW Partners by Alan Chhabra

Big Data Spain

Migration and Coexistence between Relational and NoSQL Databases by Manuel H...

Big Data Spain

The Rise of Engineering-Driven Analytics by Loren Shure

Big Data Spain

From data to numbers to knowledge: semantic embeddings By Alvaro Barbero

Big Data Spain

The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...

Big Data Spain

Open data : from Insight to Visualisation with Google BigQuery and Carto.com ...

Big Data Spain

HOW TO APPLY BIG DATA ANALYTICS AND MACHINE LEARNING TO REAL TIME PROCESSING ...

Big Data Spain

Next generation Polyglot Architectures using Neo4j by Stefan Kolmar

Big Data Spain

Apache Hive 2.0 SQL, Speed, Scale by Alan Gates

Big Data Spain

Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...

Big Data Spain

Destacado (20)

Advanced data science algorithms applied to scalable stream processing by Dav...

Enabling the Bank of the Future by Ignacio Bernal

Why Apache Flink is better than Spark by Rubén Casado

Big Migrations: Moving elephant herds by Carlos Izquierdo

Case of success: Visualization as an example for exercising democratic transp...

Finding the needle in the haystack: how Nestle is leveraging big data to defe...

Big data in 140 characters by Joe Rice

From data to AI with the Machine Learning Canvas by Louis Dorard Slides

Assessing spatial accessibility to primary health care services in the Metrop...

Stream Processing use cases and applications with Apache Apex by Thomas Weise

VP of WW Partners by Alan Chhabra

Migration and Coexistence between Relational and NoSQL Databases by Manuel H...

The Rise of Engineering-Driven Analytics by Loren Shure

From data to numbers to knowledge: semantic embeddings By Alvaro Barbero

The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...

Open data : from Insight to Visualisation with Google BigQuery and Carto.com ...

HOW TO APPLY BIG DATA ANALYTICS AND MACHINE LEARNING TO REAL TIME PROCESSING ...

Next generation Polyglot Architectures using Neo4j by Stefan Kolmar

Apache Hive 2.0 SQL, Speed, Scale by Alan Gates

Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...

Similar a Converging Big Data and Application Infrastructure by Steven Poutsy

Dean4j@Njug5

ทวิร พานิชสมบัติ

Full-Stack Development

Dhilipsiva DS

Mind Grill

pritamkumar

A Technical Driven Seminar

Deepak Chawla

Building an Open Source iOS app: lessons learned Dec 12, 2016, Hacker Dojo (Santa Clara), 6pm In this talk I'm going to talk about lessons learned from building Sensorama (http://www.sensorama.org), an Open Source sensor platform for data science. The main theme of the talk will be Open Source: what is great about it, what is bad and how you must become a part of the Open Source community to really move quickly and benefit from it. For this project, I did both the code and the design, so you'll have a chance to see how solo-developer deals with time/feature constraints, which tools I've used and what my approach towards development in this mode is. In other words: I'll tell you what I did to stay sane. If the iOS development were a walk in a dark city park, this talk may turn out to be your flashlight. If you like it, star it at GitHub: https://github.com/wkoszek/sensorama-ios Agenda https://www.meetup.com/svmobiledev/events/235836893/ Materials https://github.com/wkoszek/talks/tree/master/svmobiledev2016 Some links from the slides Fake it till you make it presentation https://developer.apple.com/videos/play/wwdc2014/223 Designing for Future Hardware https://developer.apple.com/videos/play/wwdc2015/801/ References WWW: http://www.sensorama.org GitHub (code): https://github.com/wkoszek/sensorama-ios GitHub (artwork): https://github.com/wkoszek/sensorama-artwork Author WWW: http://www.koszek.com Twitter: https://twitter.com/wkoszek LinkedIn: https://www.linkedin.com/in/wkoszek/ Email: wojciech (at) koszek.com

Building an Open Source iOS app: lessons learned

Wojciech Koszek

Intro to web development

MusTufa Nullwala

Introduction to coding and Python

Haris NP

Selje_Fox on the Run.pdf

Eric Selje

Walter api

Nicholas Schiller

This presentation on Pig will help you understand why Pig is required, what is Pig, MapReduce vs Hive vs Pig, Pig architecture, working of Pig, Pig Latin data model, Pig Execution modes, and finally a demo which shows Pig Latin scripts. Pig is a scripting platform that runs on Hadoop clusters, designed to process and analyze large datasets. It operates on various types of data like structured, semi-structured and unstructured data. Pig Latin is the procedural data flow language used in Pig to analyze data. It is easy to program using Pig Latin as it is similar to SQL. Now, let us get started with Pig. Below topics are explained in this Pig presentation: 1. Why Pig? 2. What is Pig? 3. MapReduce vs Hive vs Pig 4. Pig architecture 5. Working of Pig 6. Pig Latin data model 7. Pig Execution modes 8. Use case – Twitter 9. Features of Pig What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? This course will enable you to: 1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark 2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management 3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts 4. Get an overview of Sqoop and Flume and describe how to ingest data using them 5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning 6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution 7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations 8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS 9. Gain a working knowledge of Pig and its components 10. Do functional programming in Spark 11. Understand resilient distribution datasets (RDD) in detail 12. Implement and build Spark applications 13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques 14. Understand the common use-cases of Spark and the various interactive algorithms 15. Learn Spark SQL, creating, transforming, and querying Data frames Learn more at https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training

Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...

Simplilearn

How Open Source / Open Technology Could Help On Your Project

Wan Leung Wong

Grid reader

ctao01

Meetups - The Oracle Ace Way

Phil Wilkins

Breaking out of the endless callback look - #jsday Italy keynote

Christian Heilmann

Jsday

Christian Heilmann

Semantic Web 2.0

hchen1

WebRTC Reborn Hackference

Dan Jenkins

Of innovation and impatience - Future Decoded 2015

Christian Heilmann

Talk for FOSSASIA 2016 (http://2016.fossasia.org) ---- This talk will give a brief and enlightening look into how GraphQL can help you address common weaknesses that you, as a web / mobile developer, would normally face with using / building typical REST API systems. Let's stop fighting about whether we should implement the strictest interpretation of REST or how pragmatic REST-ful design is the only way to go, or debate about what REST is or what it should be. A couple of demos (In Golang! Yay!) will be shown that are guaranteed to open up your eyes and see that the dawn of liberation for product developers is finally here. Background: GraphQL is a data query language and runtime designed and used at Facebook to request and deliver data to mobile and web apps since 2012. Hafiz Ismail (@sogko) is a contributor to Go / Golang implementation of GraphQL server library (https://github.com/graphql-go/graphql) and is looking to encourage fellow developers to join in the collaborative effort.

Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)

Hafiz Ismail

Top Deep Learning Frameworks.pdf

Appdeveloper10

Similar a Converging Big Data and Application Infrastructure by Steven Poutsy (20)

Dean4j@Njug5

Full-Stack Development

Mind Grill

A Technical Driven Seminar

Building an Open Source iOS app: lessons learned

Intro to web development

Introduction to coding and Python

Selje_Fox on the Run.pdf

Walter api

Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...

How Open Source / Open Technology Could Help On Your Project

Grid reader

Meetups - The Oracle Ace Way

Breaking out of the endless callback look - #jsday Italy keynote

Jsday

Semantic Web 2.0

WebRTC Reborn Hackference

Of innovation and impatience - Future Decoded 2015

Introduction to GraphQL (or How I Learned to Stop Worrying about REST APIs)

Top Deep Learning Frameworks.pdf

Más de Big Data Spain

Big Data, Big Quality? by Irene Gonzálvez at Big Data Spain 2017

Big Data Spain

Scaling a backend for a big data and blockchain environment by Rafael Ríos at...

Big Data Spain

AI: The next frontier by Amparo Alonso at Big Data Spain 2017

Big Data Spain

Disaster Recovery for Big Data by Carlos Izquierdo at Big Data Spain 2017

Big Data Spain

Presentation: Boost Hadoop and Spark with in-memory technologies by Akmal Cha...

Big Data Spain

Data science for lazy people, Automated Machine Learning by Diego Hueltes at ...

Big Data Spain

Training Deep Learning Models on Multiple GPUs in the Cloud by Enrique Otero ...

Big Data Spain

Unbalanced data: Same algorithms different techniques by Eric Martín at Big D...

Big Data Spain

State of the art time-series analysis with deep learning by Javier Ordóñez at...

Big Data Spain

Trading at market speed with the latest Kafka features by Iñigo González at B...

Big Data Spain

Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...

Big Data Spain

The Analytic Platform behind IBM’s Watson Data Platform by Luciano Resende a...

Big Data Spain

Artificial Intelligence and Data-centric businesses by Óscar Méndez at Big Da...

Big Data Spain

Why big data didn’t end causal inference by Totte Harinen at Big Data Spain 2017

Big Data Spain

Meme Index. Analyzing fads and sensations on the Internet by Miguel Romero at...

Big Data Spain

Vehicle Big Data that Drives Smart City Advancement by Mike Branch at Big Dat...

Big Data Spain

End of the Myth: Ultra-Scalable Transactional Management by Ricardo Jiménez-P...

Big Data Spain

Attacking Machine Learning used in AntiVirus with Reinforcement by Rubén Mart...

Big Data Spain

More people, less banking: Blockchain by Salvador Casquero at Big Data Spain ...

Big Data Spain

Make the elephant fly, once again by Sourygna Luangsay at Big Data Spain 2017

Big Data Spain

Más de Big Data Spain (20)