DataOps or how I learned to love production - Michael Hausenblas

•

1 recomendación•133 vistas

A plethora of data processing tools, most of them open source, is available to us. But who actually runs data pipelines? What about dynamically allocating resources to data pipeline components? In this talk we will discuss options to operate elastic data pipelines with modern, cloud native platforms such as DC/OS with Apache Mesos, Kubernetes and Docker Swarm. We will review good practices, from containerizing workloads to making things resilient and show elastic data pipelines in action.

Datos y análisis

© 2017 Mesosphere, Inc. All Rights Reserved.
DATAOPS 
OR: HOW I LEARNED
TO LOVE PRODUCTION
1
Michael Hausenblas, Distributed Systems Jester | 2017-02-09 | Big Data Technology Summit Warsaw

© 2016 Mesosphere, Inc. All Rights Reserved.
sys admin/SRE
developer
data engineer
architect
data scientist
manager/PM

© 2017 Mesosphere, Inc. All Rights Reserved.
FROM THE TRENCHES …
3

© 2017 Mesosphere, Inc. All Rights Reserved.
CONCEPTUAL SETUP
4
data engineer
operationsdata scientist

© 2017 Mesosphere, Inc. All Rights Reserved.
CHALLENGES &
OPPORTUNITIES
5

© 2017 Mesosphere, Inc. All Rights Reserved.
CHALLENGES
6
• No or long feedback cycles
• Creating new stuff more valued than keeping it alive
• Need to move fast
• software is eating the world
• 24/7 online mentality
• expectations from consumer apps determine enterprise apps

© 2016 Mesosphere, Inc. All Rights Reserved. 7
appops
The person who writes an
app is also the person
responsible for operating
the app in prod.

© 2016 Mesosphere, Inc. All Rights Reserved. 8
It's not about provisioning 
a VM or installing a DC/OS
cluster or replacing a faulty
HDD … 
 
… this would be on the
infrastructure team.
appops

© 2016 Mesosphere, Inc. All Rights Reserved. 9
appops
speakerdeck.com/charity/devops-for-developers-building-an-effective-ops-org-1

© 2017 Mesosphere, Inc. All Rights Reserved.
IN AN IDEAL WORLD
10
• Small batches
• Ship early and often
• Short feedback loops allowing to move fast
• Instill mutual respect (DS-DE-ops)

© 2017 Mesosphere, Inc. All Rights Reserved.
HANDS-ON
11
https://github.com/dcos/demos/tree/master/1.8/ﬁntrans

© 2017 Mesosphere, Inc. All Rights Reserved.
LEARNING
RESOURCES
12

© 2017 Mesosphere, Inc. All Rights Reserved. 13
WHERE CAN 
I LEARN
MORE?
13
usenix.org/conference/lisa16/conference-program/presentation/eckhardt

© 2017 Mesosphere, Inc. All Rights Reserved. 14
WHERE CAN 
I LEARN
MORE?
14
some.ops4devs.info

© 2017 Mesosphere, Inc. All Rights Reserved.
Q & A
15
• @mhausenblas
• mhausenblas.info
• michael@dcos.io
https://dcos.io

Más contenido relacionado

La actualidad más candente

PgConf 2018 - Postgres in a World of DevOpsEDB

Creating Agility Through Data Governance and Self-service Integration with S...SnapLogic

The Benefits of Data FabricNetApp

Scaling Your Skillset with Your Data with Jarrett Garcia (Nielsen)Spark Summit

Modernizing to a Cloud Data ArchitectureDatabricks

Smart App@Pivotal by Dat TranVMware Tanzu Korea

The API LieSnapLogic

The Transformation of your Data in modern IT (Presented by DellEMC)Cloudera, Inc.

The Changing Role of the DBA in the CloudEDB

Azure databricks by usama whaba khanUsama Wahab Khan Cloud, Data and AI

Postgres Vision 2018: The Pragmatic CloudEDB

NYC Data Amp - Microsoft Azure and Data Services OverviewTravis Wright

Instantaneous Replication of Build Artifacts with NetAppNetApp

Postgres Vision 2018: Making Modern an Old Legacy SystemEDB

Why do the majority of Data Science projects never make it to production?Itai Yaffe

Part 1: Introducing the Cloudera Data Science WorkbenchCloudera, Inc.

Data Warehousing in the Cloud: Practical Migration Strategies SnapLogic

The Practice of Big Data - The Hadoop ecosystem explained with usage scenarioskcmallu

Analyzing Hadoop Data Using Sparklyr Cloudera, Inc.

Hadoop Essentials -- The What, Why and How to Meet Agency ObjectivesCloudera, Inc.

La actualidad más candente (20)

PgConf 2018 - Postgres in a World of DevOps

Creating Agility Through Data Governance and Self-service Integration with S...

The Benefits of Data Fabric

Scaling Your Skillset with Your Data with Jarrett Garcia (Nielsen)

Modernizing to a Cloud Data Architecture

Smart App@Pivotal by Dat Tran

The API Lie

The Transformation of your Data in modern IT (Presented by DellEMC)

The Changing Role of the DBA in the Cloud

Azure databricks by usama whaba khan

Postgres Vision 2018: The Pragmatic Cloud

NYC Data Amp - Microsoft Azure and Data Services Overview

Instantaneous Replication of Build Artifacts with NetApp

Postgres Vision 2018: Making Modern an Old Legacy System

Why do the majority of Data Science projects never make it to production?

Part 1: Introducing the Cloudera Data Science Workbench

Data Warehousing in the Cloud: Practical Migration Strategies

The Practice of Big Data - The Hadoop ecosystem explained with usage scenarios

Analyzing Hadoop Data Using Sparklyr 

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

Similar a DataOps or how I learned to love production - Michael Hausenblas

SMACK stack and beyondMatt Jarvis

The Open Sourcing of InfrastructureAll Things Open

Journey to the Modern App with Containers, Microservices and Big DataLightbend

Partner Webinar: Mesosphere and DSE: Production-Proven Infrastructure for Fas...DataStax

Hyperscale Computing, Enterprise Agility with MesosphereMarkus Eisele

A Journey to Modern Apps with Containers, Microservices and Big DataEdward Hsu

OSDC 2018 | From batch to pipelines – why Apache Mesos and DC/OS are a soluti...NETWAYS

Modern Container Orchestration (Without Breaking the Bank)All Things Open

Downtime is not an option - day 2 operations - Jörg SchadCodemotion

Introduction to DC/OSMatt Jarvis

Snowflake OverviewSnowflake Computing

EMC World 2016 - cnaITL.04 Open Source has changed how you run Infrastructure{code}

Five Cloud Native Ops Superpowers: Yes, you can do that!VMware Tanzu

Develop, Deploy, Iterate Often.J On The Beach

Flink forward SF 2017: Elizabeth K. Joseph and Ravi Yadav - Flink meet DC/OS ...Flink Forward

Flink forward sf 17Ravi Yadav

Flink Forward San Francisco 2017 - Flink meet DC/OSpleia2

Alluxio Mesos Meetup - SMACK to SMAACKAlluxio, Inc.

19. Graham Smith & Sarah Hurley - Implementing the Oracle Vision For PeopleSo...Cedar Consulting

Similar a DataOps or how I learned to love production - Michael Hausenblas (20)

SMACK stack and beyond

The Open Sourcing of Infrastructure

Journey to the Modern App with Containers, Microservices and Big Data

Partner Webinar: Mesosphere and DSE: Production-Proven Infrastructure for Fas...

Hyperscale Computing, Enterprise Agility with Mesosphere

A Journey to Modern Apps with Containers, Microservices and Big Data

OSDC 2018 | From batch to pipelines – why Apache Mesos and DC/OS are a soluti...

Modern Container Orchestration (Without Breaking the Bank)

Downtime is not an option - day 2 operations - Jörg Schad

Introduction to DC/OS

Snowflake Overview

EMC World 2016 - cnaITL.04 Open Source has changed how you run Infrastructure

Five Cloud Native Ops Superpowers: Yes, you can do that!

Develop, Deploy, Iterate Often.

Flink forward SF 2017: Elizabeth K. Joseph and Ravi Yadav - Flink meet DC/OS ...

Flink forward sf 17

Flink Forward San Francisco 2017 - Flink meet DC/OS

Alluxio Mesos Meetup - SMACK to SMAACK

19. Graham Smith & Sarah Hurley - Implementing the Oracle Vision For PeopleSo...

Más de Evention

The Factorization Machines algorithm for building recommendation system - Paw...Evention

A/B testing powered by Big data - Saurabh Goyal, Booking.comEvention

Near Real-Time Fraud Detection in Telecommunication Industry - Burak Işıklı, ...Evention

Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...Evention

Machine learning security - Pawel Zawistowski, Warsaw University of Technolog...Evention

Building a Modern Data Pipeline: Lessons Learned - Saulius Valatka, AdformEvention

Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data ArtisansEvention

Privacy by Design - Lars Albertsson, MapflatEvention

Elephants in the cloud or how to become cloud ready - Krzysztof Adamski, GetI...Evention

Deriving Actionable Insights from High Volume Media Streams - Jörn Kottmann, ...Evention

Enhancing Spark - increase streaming capabilities of your applications - Kami...Evention

7 Days of Playing Minesweeper, or How to Shut Down Whistleblower Defense with...Evention

Big Data Journey at a Big Corp - Tomasz Burzyński, Maciej Czyżowicz, Orange P...Evention

Stream processing with Apache Flink - Maximilian Michels Data ArtisansEvention

Scaling Cassandra in all directions - Jimmy Mardell SpotifyEvention

Big Data for unstructured data Dariusz ŚliwaEvention

Elastic development. Implementing Big Data search Grzegorz KołpućEvention

H2 o deep water making deep learning accessible to everyone -jo-fai chowEvention

That won’t fit into RAM - Michał BrzezickiEvention

Stream Analytics with SQL on Apache Flink - Fabian HueskeEvention

Más de Evention (20)

The Factorization Machines algorithm for building recommendation system - Paw...

A/B testing powered by Big data - Saurabh Goyal, Booking.com

Near Real-Time Fraud Detection in Telecommunication Industry - Burak Işıklı, ...

Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...

Machine learning security - Pawel Zawistowski, Warsaw University of Technolog...

Building a Modern Data Pipeline: Lessons Learned - Saulius Valatka, Adform

Apache Flink: Better, Faster & Uncut - Piotr Nowojski, data Artisans

Privacy by Design - Lars Albertsson, Mapflat

Elephants in the cloud or how to become cloud ready - Krzysztof Adamski, GetI...

Deriving Actionable Insights from High Volume Media Streams - Jörn Kottmann, ...

Enhancing Spark - increase streaming capabilities of your applications - Kami...

7 Days of Playing Minesweeper, or How to Shut Down Whistleblower Defense with...

Big Data Journey at a Big Corp - Tomasz Burzyński, Maciej Czyżowicz, Orange P...

Stream processing with Apache Flink - Maximilian Michels Data Artisans

Scaling Cassandra in all directions - Jimmy Mardell Spotify

Big Data for unstructured data Dariusz Śliwa

Elastic development. Implementing Big Data search Grzegorz Kołpuć

H2 o deep water making deep learning accessible to everyone -jo-fai chow

That won’t fit into RAM - Michał Brzezicki

Stream Analytics with SQL on Apache Flink - Fabian Hueske

Último

Dubai Call Girls Wifey O52&786472 Call Girls Dubaihf8803863

Brighton SEO | April 2024 | Data StorytellingNeil Barnes

E-Commerce Order PredictionShraddha Kamble.pptxBoston Institute of Analytics

Market Analysis in the 5 Largest Economic Countries in Southeast Asia.pdfRachmat Ramadhan H

April 2024 - Crypto Market Report's Analysismanisha194592

Invezz.com - Grow your wealth with trading signalsInvezz1

Industrialised data - the key to AI success.pdfLars Albertsson

VIP Call Girls Service Charbagh { Lucknow Call Girls Service 9548273370 } Boo...Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure

Ravak dropshipping via API with DroFx.pptxolyaivanovalion

Smarteg dropshipping via API with DroFx.pptxolyaivanovalion

Unveiling Insights: The Role of a Data AnalystSamantha Rae Coolbeth

Ukraine War presentation: KNOW THE BASICSAishani27

Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al BarshaAroojKhan71

Generative AI on Enterprise Cloud with NiFi and MilvusTimothy Spann

CebaBaby dropshipping via API with DroFX.pptxolyaivanovalion

Delhi 99530 vip 56974 Genuine Escort Service Call Girls in Kishangarh9953056974 Low Rate Call Girls In Saket, Delhi NCR

꧁❤ Greater Noida Call Girls Delhi ❤꧂ 9711199171 ☎️ Hard And Sexy Vip Callshivangimorya083

Low Rate Call Girls Bhilai Anika 8250192130 Independent Escort Service BhilaiSuhani Kapoor

(PARI) Call Girls Wanowrie ( 7001035870 ) HI-Fi Pune Escorts Serviceranjana rawat

dokumen.tips_chapter-4-transient-heat-conduction-mehmet-kanoglu.pptSonatrach