2016 Spark Summit East Keynote: Ali Ghodsi and Databricks Community Edition demo

HP ships millions of PCs, Printers, and other devices every year to customers in all market segments. More customers are seeking services provided with our products enabling new opportunities for HP to create services from the data we can collect from our devices. Every device we ship is an IoT endpoint with powerful CPU to capture rich data. Insights from this data are used internally to improve our products and focus on customer needs. In this presentation, John will focus on HP’s journey to enabling Big Data analytics from within a large enterprise environment. He will review the challenges and how HP decided on AWS, Apache Spark and Databricks as the foundation for their entry into Big Data Analytics. John will also review how HP uses Spark to build analytic services from the data they generate from their devices.

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

Accelerating Innovation with Unified Analytics with Ali Ghodsi

Operationalizing Machine Learning at Scale at Starbucks

In the on-demand economy real-time analytics is both a necessity and a competitive advantage. The next evolution in the on-demand economy is in predictive analytics fueled by live streams of data—in effect knowing what customers want before they do. This session will feature technical examples of real-time pipelines, machine learning, and custom dashboards as well as off-the-shelf dashboards with Tableau.

CTO View: Driving the On-Demand Economy with Predictive Analytics

SingleStore

Machine learning algorithm itself is rarely the main barrier in building AI applications. Instead, the real culprit is the set of complex systems that prepares large-scale training and test data for the ML algorithms. Apache Spark is a huge leap forward in democratizing AI. However, it does not solve all the problems. Databricks CEO Ali Ghodsi explains how Databricks democratizes AI by making it easier to build end-to-end machine learning pipelines with Apache Spark.

Spark Summit Europe 2016 Keynote - Databricks CEO

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

Learn to Use Databricks for the Full ML Lifecycle

This presentation starts off by discussing powerful examples of The Power of Data and the benefits of Data Driven architectures. A Data Governance program is important for the success of Data Driven architectures. We then discuss the challenges of implementing a Data Governance framework on a Big Data Data Lake with open source software including DataPlane, Apache Atlas and Apache Ranger. And finally, we discuss the importance of the democratization of data and the switching to a speed of thought framework with Hive LLAP.

Eugene Polonichko "Architecture of modern data warehouse"

Lviv Startup Club

SnapLogic Live: Salesforce Integration

SnapLogic

The Power of Data

DataWorks Summit

At Sams Club we have a long history of using Apache Spark and Hadoop. Projects from all parts of the company use Apache Spark, from fraud detection to product recommendations. Because of the scale of our business with billions of transactions and trillions of events it is often essential to use big data technologies. Until recently all of this work has run on several large on-premise Hadoop clusters. As part of our transition to public cloud we needed to build out an enterprise scale data platform. Azure Databricks is a key component of this platform giving our data scientist, engineers, and business users the ability to easily work with the companies data. We will discuss our architecture considerations that lead to using multiple Databricks workspaces and external Azure blob storage. We will also discuss how we move massive amounts of data to Azure on a daily basis with Airflow. Further we will discuss the self-service tools that we created to help users get their data to Azure and for us to manage the platform. Finally we will discuss our security considerations and how that played out in our architecture. Authors: Andrew Ray, Craig Covey

Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...

Insnap, a hyper-personalized ML-based platform acquired by The Honest Company, has been used to build a real-time data platform based on Apache Spark, Cassandra and Redshift. Users’ behavioral and transactional data have been used to build data models and ML models, and to drive use cases for marketing, growth, finance and operations. Learn how Honest Company has used Spark as a workhorse for 1) collecting, ETL and storing data from various sources including mysql, mongo, jde, Google analytics, Facebook, Localytics and REST API; 2) building data models and aggregating and generating reports of revenue, order fulfillment tracking, data pipeline monitoring and subscriptions; 3) Using ML to build model for user acquisitions, LTV and recommendations use cases. Spark replaced the monolithic codebase with flexible, scalable and robust pipelines. Databricks helped The Honest Company to focus on data instead of maintaining infrastructure. While Honest users got delightful recommendations to improve experience, data users at Honest understood users much better in terms of segmenting with behavioral information and advanced ML models, leading to increased revenue and retention.

Real-time Microservices and In-Memory Data Grids

Ali Hodroj

Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq Abdullah

Intuit Analytics Cloud 101

In the race to invent multi-million dollar business opportunities with exclusive insights, data scientists and engineers are hampered by a multitude of challenges just to make one use case a reality – the need to ingest data from multiple sources, apply real-time analytics, build machine learning algorithms, and intermix different data processing models, all while navigating around their legacy data infrastructure that is just not up to the task. This need has created the demand for Virtual Analytics, where the complexities of disparate data and technology silos have been abstracted away, coupled with a powerful range of analytics and processing horsepower, all in one unified data platform. This talk describes how Databricks is powering this revolutionary new trend with Apache Spark.

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

In this webinar, we talk about the shift in data gravity as more and more business applications are moving to the cloud, and how the ability to deliver analytics in the cloud has evolved from idea to enterprise reality with new solutions being announced constantly that appeal to the need for speed, simplicity and access to insight on demand. Joining us in this webinar is David Glueck, Sr. Director of Data Science and Engineering at Bonobos. To learn more, visit: www.SnapLogic.com/salesforce-analytics

Webinar: BI in the Sky - The New Rules of Cloud Analytics

SnapLogic

Optimize Data for the Logical Data Warehouse

Attunity

The Fast Path to Building Operational Applications with Spark

SingleStore

In-Memory Computing Webcast. Market Predictions 2017

SingleStore

La actualidad más candente (20)

Azure databricks by usama whaba khan

Intro to Delta Lake

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

Accelerating Innovation with Unified Analytics with Ali Ghodsi

Operationalizing Machine Learning at Scale at Starbucks

CTO View: Driving the On-Demand Economy with Predictive Analytics

Spark Summit Europe 2016 Keynote - Databricks CEO

Learn to Use Databricks for the Full ML Lifecycle

Eugene Polonichko "Architecture of modern data warehouse"

SnapLogic Live: Salesforce Integration

The Power of Data

Building an Enterprise Data Platform with Azure Databricks to Enable Machine ...

Real-time Microservices and In-Memory Data Grids

Leveraging Spark to Democratize Data for Omni-Commerce with Shafaq Abdullah

Intuit Analytics Cloud 101

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

Webinar: BI in the Sky - The New Rules of Cloud Analytics

Optimize Data for the Logical Data Warehouse

The Fast Path to Building Operational Applications with Spark

In-Memory Computing Webcast. Market Predictions 2017

Destacado

Spark meetup TCHUG

Ryan Bosshart

Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop

Big Data Platform Industrialization

Migrating Clinical Data in Various Formats to a Clinical Data Management System

Perficient, Inc.

Cloudera Impala 1.0

Minwoo Kim

Building a modern Application with DataFrames

Building a modern Application with DataFrames

These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016. --- Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download.

Jump Start into Apache® Spark™ and Databricks

Scalable And Incremental Data Profiling With Spark

Jen Aman

Hadoop application architectures - using Customer 360 as an example

hadooparchbook

How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...

빅데이터 개념 부터 시작해서 빅데이터 분석 플랫폼의 출현(hadoop)과 스파크의 등장배경까지 풀어서 작성된 spark 소개 자료 입니다. 스파크는 RDD에 대한 개념과 spark SQL 라이브러리에 대한 자료가 조금 자세히 설명 되어있습니다. (텅스텐엔진, 카탈리스트 옵티마이져에 대한 간략한 설명이 있습니다.) 마지막에는 간단한 설치 및 interactive 분석 실습자료가 포함되어 있습니다. 원본 ppt 를 공개해 두었으니 언제 어디서든 필요에 따라 변형하여 사용하시되 출처만 잘 남겨주시면 감사드리겠습니다. 다른 슬라이드나, 블로그에서 사용된 그림과 참고한 자료들은 작게 출처를 표시해두었는데, 본 ppt의 초기버전을 작성하면서 찾았던 일부 자료들은 출처가 불분명한 상태입니다. 자료 출처를 알려주시면 반영하여 수정해 두도록하겠습니다. (제보 부탁드립니다!)

Apache spark 소개 및 실습

동현 강

The next release of Apache Spark will be 2.0, marking a big milestone for the project. In this talk, I’ll cover how the community has grown to reach this point, and some of the major features in 2.0. The largest additions are performance improvements for Datasets, DataFrames and SQL through Project Tungsten, as well as a new Structured Streaming API that provides simpler and more powerful stream processing. I’ll also discuss a bit of what’s in the works for future versions.

Spark Summit San Francisco 2016 - Matei Zaharia Keynote: Apache Spark 2.0

“As Apache Spark becomes more widely adopted, we have focused on creating higher-level APIs that provide increased opportunities for automatic optimization. In this talk, I give an overview of some of the exciting new API’s available in Spark 2.0, namely Datasets and Structured Streaming. Together, these APIs are bringing the power of Catalyst, Spark SQL's query optimizer, to all users of Spark. I'll focus on specific examples of how developers can build their analyses more quickly and efficiently simply by providing Spark with more information about what they are trying to accomplish.” - Michael Databricks Blog: "Deep Dive into Spark SQL’s Catalyst Optimizer" https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html // About the Presenter // Michael Armbrust is the lead developer of the Spark SQL project at Databricks. He received his PhD from UC Berkeley in 2013, and was advised by Michael Franklin, David Patterson, and Armando Fox. His thesis focused on building systems that allow developers to rapidly build scalable interactive applications, and specifically defined the notion of scale independence. His interests broadly include distributed systems, large-scale structured storage and query optimization. Follow Michael on - Twitter: https://twitter.com/michaelarmbrust LinkedIn: https://www.linkedin.com/in/michaelarmbrust

[D2 COMMUNITY] Spark User Group - 스파크를 통한 딥러닝 이론과 실제

NAVER D2

Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...

File Format Benchmark - Avro, JSON, ORC & Parquet

Spark Summit Presentation by Anjul Bhambhri

R is the latest language added to Apache Spark, and the SparkR API is slightly different from PySpark. With the release of Spark 2.0, the R API officially supports executing user code on distributed data. This is done through a family of apply() functions. In this talk, Hossein Falaki gives an overview of this new functionality in SparkR. Using this API requires some changes to regular code with dapply(). This talk will focus on how to correctly use this API to parallelize existing R packages. Most important topics of consideration will be performance and correctness when using the apply family of functions in SparkR. Speaker: Hossein Falaki This talk was originally presented at Spark Summit East 2017.

Parallelizing Existing R Packages with SparkR

At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

StampedeCon

Destacado (19)

Spark meetup TCHUG

Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop

Big Data Platform Industrialization

Migrating Clinical Data in Various Formats to a Clinical Data Management System

Cloudera Impala 1.0

Building a modern Application with DataFrames

Jump Start into Apache® Spark™ and Databricks

Scalable And Incremental Data Profiling With Spark

Hadoop application architectures - using Customer 360 as an example

How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...

Apache spark 소개 및 실습

Spark Summit San Francisco 2016 - Matei Zaharia Keynote: Apache Spark 2.0

[D2 COMMUNITY] Spark User Group - 스파크를 통한 딥러닝 이론과 실제

Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...

File Format Benchmark - Avro, JSON, ORC & Parquet

Spark Summit Presentation by Anjul Bhambhri

Parallelizing Existing R Packages with SparkR

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

Similar a 2016 Spark Summit East Keynote: Ali Ghodsi and Databricks Community Edition demo

ICP for Data- Enterprise platform for AI, ML and Data Science

Karan Sachdeva

Originally Published on Sep 23, 2014 IBM InfoSphere BigInsights, an enterprise-ready distribution of Hadoop, is designed to address the challenges of big data and modern IT by analyzing larger volumes of data more cost-effectively. Deployed on the cloud, it enables rapid deployment of clusters and real-time analytics. FYI: The value of Hadoop and many more questions will be pondered at this year’s Strata/Hadoop World event in NYC (October 15-17, 2014) and certainly at IBM Insight (October 26-30, 2014).

Hadoop in the Cloud

IBM Analytics

IT Modernization in Practice

Tom Diederich

Accelerating Data Warehouse Modernization

In this webinar, we talk with experts from Integration Developer News about the SnapLogic Elastic Integration Platform and adoption trends for iPaaS in the enterprise. During the discussion, we address cloud application adoption challenges and 5 signs you need better cloud integration, including struggles with the "Integrator's Dilemma" and segregated integration. To learn more, visit: www.snaplogic.com/connect-faster

Cloud Con 2015 - Integration & Web APIs

SnapLogic

Crimson 3 - Final case presentation

Pragnya Balamurukesan

3 Things to Learn: -How data is driving digital transformation to help businesses innovate rapidly -How Choice Hotels (one of largest hoteliers) is using Cloudera Enterprise to gain meaningful insights that drive their business -How Choice Hotels has transformed business through innovative use of Apache Hadoop, Cloudera Enterprise, and deployment in the cloud — from developing customer experiences to meeting IT compliance requirements

How Data Drives Business at Choice Hotels

Cloudera, Inc.

Danny Bickson - Python based predictive analytics with GraphLab Create

PyData

Mainframe and IBM i data continues to be prevalent in several industries including financial services, insurance, and retail where critical customer information lives on legacy systems. In fact, in 2019 alone, studies show that there was a 55% increase in transaction volumes on the mainframe across all industries. To thrive in highly competitive markets, you must quickly break down legacy data silos to swiftly gain a full picture of data for insights for strategic action. Traditional storage solutions that are mainframe proprietary struggle to scale for high data volumes and real-time analytics use cases. This results in increased costs, diminished performance, and missed SLAs. To solve this, Precisely and Databricks provide a modern approach for organizations to optimize volumes of data by leveraging the massive scalability of the cloud to power high-performance analytics, AI, and machine learning, regardless of where data lives. In this webinar, we discuss: - Quickly ingesting data from on-premises sources – such as mainframe and IBM i – to the cloud with the Databricks Unified Data Analytics Platform and Delta Lake - Modernizing ETL processes and reduce development costs with visual data pipelines that uses the elastic scalability of Databricks - Empowering business users with the most up to date data by populating Delta Lake with realtime data changes from legacy systems View this webinar on-demand to see a live demo of the joint solution and how it can modernize your legacy infrastructure

Liberate Legacy Data Sources with Precisely and Databricks

Precisely

Redis Labs' CMO is hosting a fireside chat with leaders from multiple industries including Groupon (e-commerce ), Intuit (Finance ), and LifeLock (Identity Protection ). This conversation-style session will cover the Big Data related challenges faced by these leading companies as they scale their applications, ensure high availability, serve the best user experience at lowest latencies, and optimize between cloud and on-premises operations. The introductory level session will appeal to both developer and DevOps functions. They will hear about diverse use cases such as recommendations engine, hybrid transactions and analytics operations, and time-series data analysis. The audience will learn how the Redis in-memory database platform addresses the above use cases with its multi-model capability and in a cost effective manner to meet the needs of the next generation applications. Session sponsored by Redis Labs.

AWS re:Invent 2016: Fireside chat with Groupon, Intuit, and LifeLock on solvi...

Amazon Web Services

Effective AI and ML projects require a perfect blend of scalable, clean data funneled from a variety of sources across the business. The only problem? Uncleaned data often lives in hard-to-access legacy systems, and it costs time and money to build the right foundation to deliver that data to answer ever-changing questions from business users. Together, Cloudera and Syncsort enable you to build a scalable foundation of data connections to reinvent the data lifecycle of all your projects in the most efficient way possible. View this webinar on-demand to learn how innovative solutions from Cloudera and Syncsort enable AI and ML success. You will learn: • Best practices for transforming complex data into clear, actionable insights for AI and ML projects • How to visually assess the quality of the sources in your data lake and their completeness, consistency, and accuracy • The value of an Enterprise Data Cloud and the newly unveiled Cloudera Data Platform • How Syncsort Connect integrates natively with the Cloudera Data Platform

Cloudera + Syncsort: Fuel Business Insights, Analytics, and Next Generation T...

Precisely

IBM Relay 2015: Open for Data

IBM

Analytic workloads and the ability to determine “what happened” are some of the most common use cases across enterprises today - helping you understand and adapt based on changing trends. However, for most businesses today, they are only able to see a piece of the story. Analytics are limited by the amount of data able to be stored and ultimately accessed, it’s time-intensive to bring in new datasets or fit unstructured data into rigid schemas, and user access is constrained to a select few who must already know the questions they’re trying to answer. It’s no surprise that big data is disrupting this modus operandi for analytics. A modern, Hadoop-based platform is designed to help businesses break free of these analytic limitations, providing a new kind of adaptive, high-performance analytic database. The recent release of Cloudera 5.8 continues to advance Cloudera Enterprise as the foundation for these analytic workloads. Join Justin Erickson, Senior Director of Product Management at Cloudera, and Andy Frey, Chief Technology Officer at Marketing Associates, as they discuss: -What technology is needed to build a modern analytic database with Hadoop -What’s new with Cloudera 5.8 -How to align your teams around agile analytics -Real world success from Marketing Associates -What’s next for Cloudera Enterprise’s Analytic Database

Building a Modern Analytic Database with Cloudera 5.8

Cloudera, Inc.

Watch full webinar here: https://bit.ly/3fBpO2M Data Fabric has been a hot topic in town and Gartner has termed it as one of the top strategic technology trends for 2022. Noticeably, many mid-to-large organizations are also starting to adopt this logical data fabric architecture while others are still curious about how it works. With a better understanding of data fabric, you will be able to architect a logical data fabric to enable agile data solutions that honor enterprise governance and security, support operations with automated recommendations, and ultimately, reduce the cost of maintaining hybrid environments. In this on-demand session, you will learn: - What is a data fabric? - How is a physical data fabric different from a logical data fabric? - Which one should you use and when? - What’s the underlying technology that makes up the data fabric? - Which companies are successfully using it and for what use case? - How can I get started and what are the best practices to avoid pitfalls?

Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...

Denodo

Companies are finding accessing data from a variety of sources can be labor-intensive and costly. Oftentimes these companies are looking to cloud solutions, but are then finding the traditional architecture brittle when trying to move data to the cloud, which can drain organizations of time and resources. Join this webinar to hear several company success stories, the data-to-cloud issues they were encountering, and the steps these companies took to bring their cloud architecture to a successful, real-time analytic solution unlocking massive amounts of fresh enterprise-wide on a continuous basis. In addition, you will learn how to: • Modernize the ETL process to one that’s fast, flexible, and scalable • Supply users with up-to-date, accurate, trusted data • Increase your time to value with data in the cloud • Best practices on how to minimize resource overhead

Slides: Success Stories for Data-to-Cloud

DATAVERSITY

Node.js is a very popular framework for developing asynchronous, event-driven, reactive applications. Red Hat JBoss Data Grid, an in-memory distributed database designed for fast access to large volumes of data and scalability, has recently gained compatibility with Node.js letting reactive applications use it as a persistence layer. Thanks to near caching, JBoss Data Grid offers excellent response times for data queried regularly, and its continuous remote event support means data can get pushed from the data grid to the Node.js application instead of having to wait for the data grid to serve it. In this session, we'll show how to build Node.js applications that use JBoss Data Grid as a persistence layer.

Building Reactive Applications With Node.Js And Red Hat JBoss Data Grid (Gald...

Red Hat Developers

Neo4j Graph Use Cases, Bruno Ungermann, Neo4j

Neo4j

Indonesia new default short msp client presentation partnership with isv

Pandu W Sastrowardoyo

Postgres is the leading open source database management system that is being developed by a very active community for more than 15 years. Gaby Schilders is Sales Engineer at EnterpriseDB, supplier of the EDB Postgres data platform. Gaby Schilders, Sales Engineer at EnterpriseDB, will be explaining why companies take open source as the centerpiece for modernising their IT infrastructure, thus increasing their scalability and taking full advantage today's technologies offer them.

OPEN'17_4_Postgres: The Centerpiece for Modernising IT Infrastructures

Kangaroot

How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)

Ontico

Similar a 2016 Spark Summit East Keynote: Ali Ghodsi and Databricks Community Edition demo (20)

ICP for Data- Enterprise platform for AI, ML and Data Science

Hadoop in the Cloud

IT Modernization in Practice

Accelerating Data Warehouse Modernization

Cloud Con 2015 - Integration & Web APIs

Crimson 3 - Final case presentation

How Data Drives Business at Choice Hotels

Danny Bickson - Python based predictive analytics with GraphLab Create

Liberate Legacy Data Sources with Precisely and Databricks

AWS re:Invent 2016: Fireside chat with Groupon, Intuit, and LifeLock on solvi...

Cloudera + Syncsort: Fuel Business Insights, Analytics, and Next Generation T...

IBM Relay 2015: Open for Data

Building a Modern Analytic Database with Cloudera 5.8

Data Fabric - Why Should Organizations Implement a Logical and Not a Physical...

Slides: Success Stories for Data-to-Cloud

Building Reactive Applications With Node.Js And Red Hat JBoss Data Grid (Gald...

Neo4j Graph Use Cases, Bruno Ungermann, Neo4j

Indonesia new default short msp client presentation partnership with isv

OPEN'17_4_Postgres: The Centerpiece for Modernising IT Infrastructures

How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)

Más de Databricks

DW Migration Webinar-March 2022.pptx

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 4

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Learn to Use Databricks for Data Science

Application performance monitoring (APM) has become the cornerstone of software engineering allowing engineering teams to quickly identify and remedy production issues. However, as the world moves to intelligent software applications that are built using machine learning, traditional APM quickly becomes insufficient to identify and remedy production issues encountered in these modern software applications. As a lead software engineer at NewRelic, my team built high-performance monitoring systems including Insights, Mobile, and SixthSense. As I transitioned to building ML Monitoring software, I found the architectural principles and design choices underlying APM to not be a good fit for this brand new world. In fact, blindly following APM designs led us down paths that would have been better left unexplored. In this talk, I draw upon my (and my team’s) experience building an ML Monitoring system from the ground up and deploying it on customer workloads running large-scale ML training with Spark as well as real-time inference systems. I will highlight how the key principles and architectural choices of APM don’t apply to ML monitoring. You’ll learn why, understand what ML Monitoring can successfully borrow from APM, and hear what is required to build a scalable, robust ML Monitoring architecture.

Why APM Is Not the Same As ML Monitoring

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Stage Level Scheduling Improving Big Data and AI Integration

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

In this talk about zipline, we will introduce a new type of windowing construct called a sawtooth window. We will describe various properties about sawtooth windows that we utilize to achieve online-offline consistency, while still maintaining high-throughput, low-read latency and tunable write latency for serving machine learning features.We will also talk about a simple deployment strategy for correcting feature drift – due operations that are not “abelian groups”, that operate over change data.

Sawtooth Windows for Feature Aggregations

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. To address these challenges, we built whylogs, an open source standard for data logging. It is a lightweight data profiling library that enables end-to-end data profiling across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It can work with different modes of data operations, including streaming, batch and IoT data. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and various integrations. We will demonstrate how the whylogs integration with Apache Spark achieves large scale data profiling, and we will show how users can apply this integration into existing data and ML pipelines.

Re-imagine Data Monitoring with whylogs and Spark

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Raven: End-to-end Optimization of ML Prediction Queries

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Processing Large Datasets for ADAS Applications using Apache Spark

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Massive Data Processing in Adobe Using Delta Lake