AI at Scale

•

0 recomendaciones•59 vistas

This document discusses AI at scale using Apache Spark on Azure. It provides an overview of Apache Spark, how it can be used for machine learning with tools like MLlib and Databricks, and how cognitive services can be combined with Spark. It also discusses using Azure services like Databricks, HDInsight and AKS for running Spark workloads at scale, and the roles of data engineers and data scientists.

Ingeniería

AI at Scale
Adi Polak | Microsoft
@adipolak

Start from the beginning
• Massive datasets
• Intelligent product
@adipolak

• Apache Spark
• Databricks for Production
• Build machine learning models with
Pipelines
• Cognitive Services and Apache Spark
• Data Engineer vs Data Scientist
• MMLSpark
Overview
@adipolak

Apache Spark submit job
storage storage storage
@adipolak

Framework libraries- Software Stracture
@adipolak

Python, R & .NET for Apache Spark
@adipolak

Embedded Service
PySpark
Spark Scala Process
Pyspark
Protocol
Spark Worker
@adipolak

Structured
Streaming
MLlib GraphFrame TensorFrames
SQL SparkSession / DataFrame / Dataset APIs
Data Source
Connectors
Catalyst Optimization & Tungsten Execution
Spark Core (RDD APIs)
SQL
API
More API
@adipolak

Why Apache Spark is fast and efficient?
CATALYST OPTIMIZER IN MEMORY
COMPUTING
TUNGSTEN
@adipolak

Fundamentals of Catalyst Optimizer
SUB
Attribute(x) SUB
some_func(1) some_func(2)
Tree Rules
SUB
Attribute(x) some_func(-1)
@adipolak

Spark SQL Execution Plan
Logical optimization –> Optimization rules
• Constant folding
• Predicate pushdown
• Projection pruning
• …
Catalyst
Frontend Backend
@adipolak

Spark Connector for SQL Server
@adipolak

How to run
Apache
Spark on
Azure?
HDInsight
Azure Databricks
AKS

Build for Production
• - Databricks limits and Azure limits
• - Workspace isolation
• - vnet
• - Security and key vault
• - Enable log analytics for monitoring

Databricks limits and Azure limits
• Key Databricks limits ( per workspace):
• - 150 concurrent jobs
• - 150 max notebooks
• - 1000 max hourly job submission
Key Azure limits :
- Storage accounts per region per
subscription: 250
- VMs per subscription per
region: 25,000
- Resource groups per subscription: 980
@adipolak

Cluster modes and their characteristics
@adipolak

Batch vs. Interactive workloads
@adipolak

Data Scientists and Analysts –
Hight concurrency mode
@adipolak

Troubleshooting with cluster logs
• Azure Databricks provides three kinds of logging of cluster-related
activity:
• Cluster event logs, which capture cluster lifecycle events, like creation,
termination, configuration edits, and so on.
• Apache Spark driver and worker logs, which you can use for
debugging.
• Cluster init-script logs, valuable for debugging init scripts.
@adipolak

Log Analytics Workspace
• Leverage Azure log analytics and connect it to the workspace
• with init scripts and visualize it using Kibana or other tools.
@adipolak

Gather Data
ML Process / Basic Life Cycle
Feature Extract, Clean and Normalize
Select algorithm
Evaluate model
Data visualization
4
5
1
2
3
Repeat!
@adipolak

H O W ?
M L P i p e l i n e s !
@adipolak

Tools
Spark
Streaming
Spark ML/Other
Spark SQL MLflow
@adipolak

But in real life:
Accuracy < 0.5
AUC L
Results are far
from being good

Building machine learning models
from scratch is hard!
@adipolak

Our Data scientist needs tools
@adipolak

Stand on the
shoulder of giants
@adipolak

Cognitive Services capabilities
Infuse your apps, websites, and bots with human-like intelligence

A variety of real-world applications
Vision Speech
Intent: PlayCall
Language Knowledge Search

Apache Spark + Cognitive Services
@adipolak

Data Engineer vs Data Scientist
Advanced programming
Distributed systems
Data pipeline
Basic analytics
Data Engineer
Advance math/statistics
ML/AI
Advanced analytics
Basic programming
Data Science
Big Data constrains
Both should understand:
@adipolak

MMLSpark:
Microsoft Machine
Learning for
Apache Spark
@adipolak

aka.ms/mlflow_and_azure_ml
aka.ms/azure_databricks
aka.ms/twitter_sentiment_analysis
Want to learn more?
@adipolak
Thank you! Dank u! Merci!

Más contenido relacionado

La actualidad más candente

5 reasons why spark is in demand!

Edureka!

My talk at Data Science Labs conference in Odessa. Training a model in Apache Spark while having it automatically available for real-time serving is an essential feature for end-to-end solutions. There is an option to export the model into PMML and then import it into a separated scoring engine. The idea of interoperability is great but it has multiple challenges, such as code duplication, limited extensibility, inconsistency, extra moving parts. In this talk we discussed an alternative solution that does not introduce custom model formats and new standards, not based on export/import workflow and shares Apache Spark API.

Spark ML Pipeline serving

Stepan Pushkarev

The current implementation of ML algorithms in Spark has several disadvantages associated with the transition from standard Spark SQL types to ML-specific types, a low level of algorithms' adaptation to distributed computing, a relatively slow speed of adding new algorithms to the current library. Also, Spark ML doesn't support online-learning by nature for all algorithms, stacking, boosting and a bunch of approximate ML algorithms that gives a significant speedup in many cases. The Apache Ignite could work closely with Apache Spark due to exellent Ignite RDD/Ignite DataFrame implementation (see https://ignite.apache.org/use-cases/spark/shared-memory-layer.html). Also Apache Ignite has Ignite ML module that includes a lot of distributed ML algorithms, NLP package (will be available in next release, 2.8), the bunch of approximate ML algorithms, simple integration with TensorFlow via TensorFlow Ignite Dataset (currently, this is a part of TF.contrib package) and also each algorithm supports the model updating that gives us ability to make online-learning not only for KMeans and LinReg. We suggest to use Apache Ignite ML module to speedup your ML training and use Spark + Ignite as backend for distributed TensorFlow calculations. You will see live demos of ML pipeline building with Apache Ignite ML module, Apache Spark, Apache Kafka, TensorFlow and more.

Distributed ML/DL with Ignite ML Module Using Apache Spark as Database

Databricks

This session will give a new dimension to Apache Spark’s usage. See how Apache Spark and other open source projects can be used together in providing a scalable, real-time monitoring system. Apache Spark plays the central role in providing this scalable solution, since without Spark Streaming we would not be able to process millions of events in real time. This approach can provide a lot of learning to the DevOps/Infrastructure domain on how to build a scalable and automated logging and monitoring solution using Apache Spark, Apache Kafka, Grafana and some other open-source technologies. Sony PlayStation’s monitoring pipeline processes about 40 billion events every day, and generates metrics in near real-time (within 30 seconds). All the components, used along with Apache Spark, are horizontally scalable using any auto-scaling techniques, which enhances the reliability of this efficient and highly available monitoring solution. Sony Interactive Entertainment has been using Apache Spark, and specifically Spark Streaming, for the last three years. Hear about some important lessons they have learned. For example, they still use Spark Streaming’s receiver-based method in certain use cases instead of Direct Streaming, and will share the application of both the methods, giving the knowledge back to the community.

Scalable Monitoring Using Apache Spark and Friends with Utkarsh Bhatnagar

Databricks

Back in the old days of Apache Spark, using Python with Spark was an exercise in patience. Data was moving up and down from Python to Scala, being serialised constantly. Leveraging SparkSQL and avoiding UDFs made things better, likewise did the constant improvement of the optimisers (Catalyst and Tungsten). But, after Spark 2.3, PySpark has sped up tremendously thanks to the addition of the Arrow serialisers. In this talk you will learn how the Spark Scala core communicates with the Python processes, how data is exchanged across both sub-systems and the development efforts present and underway to make it as fast as possible.

Internals of Speeding up PySpark with Arrow

Databricks

>Sarah: My Spark SQL query failed. How can I fix it? >Jeeves: Your Spark query driver went out of memory. >Jeeves: You can set spark.driver.memory to 2.2GB and rerun the query to complete it successfully. Who is Jeeves? An experienced Spark developer? A seasoned administrator? No, Jeeves is a chatbot created to simplify data operations management for enterprise Spark clusters. This chatbot is powered by advanced AI algorithms and an intuitive conversational interface that together provide answers to get users in and out of performance problems quickly. Instead of just being stuck to screens displaying performance logs and metrics, users can now have more refreshing experience; and consume performance insights via a two-way conversation with their own personal Spark expert. This talk will give an overview of the chatbot, its architecture, and how it fits in a complex Spark environment. The chatbot connects to a large number of sources to get the data to power its AI algorithms. It can detect anomalies in performance and push key insights via alerts to users when they need them the most. The chatbot can also be told to take actions like creating tickets and making configuration changes. You will learn how to build chatbots that tackle your complex data operations challenges with AI algorithms and automation, keeping a cool head at all times.

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

Databricks

Spark at Airbnb

Hao Wang

The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...

Spark Summit

At SpotX, we have built and maintained a portfolio of Spark Streaming applications -- all of which process records in the millions per minute. From pure data ingestion, to ETL, to real-time reporting, to live customer-facing products and features, continuous applications are in our DNA. Come along with us as we outline our journey from square one to present in the world of Spark Streaming. We'll detail what we've learned about efficient processing and monitoring, reliability and stability, and long term support of a streaming app. Come learn from our mistakes, and leave with some handy settings and designs you can implement in your own streaming apps.

Headaches and Breakthroughs in Building Continuous Applications

Databricks

Apache Spark has rapidly become a key tool for data scientists to explore, understand and transform massive datasets and to build and train advanced machine learning models. The question then becomes, how do I deploy these model to a production environment? How do I embed what I have learned into customer facing data applications? In this webinar, we will discuss best practices from Databricks on how our customers productionize machine learning models do a deep dive with actual customer case studies, show live tutorials of a few example architectures and code in Python, Scala, Java and SQL.

Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models

Anyscale

Vulnerabilities such as Spectre and Meltdown continue to plague many production servers, based on Intel CPUs. Our solution involves software-based monitoring of hardware counters and sending that data to Apache Spark clusters for threat detection. We leverage Spark's support for support vector machine (SVM) inference. Our machine learning models are trained off-line by a data scientist within a Jupyter notebook environment. As new models are validated, they can be easily deployed to the Spark cluster from the notebook. We have standardized model export and import using the ONNX machine learning open file format. In our presentation, we will demo the full pipeline, from model training to deployment. We will discuss the various challenges when deploying ML-based cyber-threat detection at scale using Apache Spark. For example, we found that gaps in detection can occur when Spark models are updated. We will describe a novel data ingestion architecture, based on Apache Kafka, that we developed to deal with this issue.

Scaling ML-Based Threat Detection For Production Cyber Attacks

Databricks

Spline 2 - Vision and Architecture Overview

Vaclav Kosar

Delight (https://www.datamechanics.co/delight) is a free & cross-platform monitoring dashboard for Apache Spark, which display system metrics (CPU Usage, Memory Usage) along with Spark information (jobs, stages, tasks) on the same timeline. Delight is a great complement to the Spark UI when it comes to troubleshooting your Spark application and understanding its performance bottleneck. It works freely on top of any Spark platform (whether it’s open-source or commercial, in the cloud or on-premise). You can install it using an open-sourced Spark agent (https://github.com/datamechanics/delight). In this session, the co-founders of Data Mechanics will take you through performance troubleshooting sessions with Delight on real-world data engineering pipelines. You will see how Delight and the Spark UI can jointly help you spot the performance bottleneck of your applications, and how you can use these insights to make your applications more cost-effective and stable.

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

Databricks

Interested in learning how Showtime is leveraging the power of Spark to transform a traditional premium cable network into a data-savvy analytical competitor? The growth in our over-the-top (OTT) streaming subscription business has led to an abundance of user-level data not previously available. To capitalize on this opportunity, we have been building and evolving our unified platform which allows data scientists and business analysts to tap into this rich behavioral data to support our business goals. We will share how our small team of data scientists is creating meaningful features which capture the nuanced relationships between users and content; productionizing machine learning models; and leveraging MLflow to optimize the runtime of our pipelines, track the accuracy of our models, and log the quality of our data over time. From data wrangling and exploration to machine learning and automation, we are augmenting our data supply chain by constantly rolling out new capabilities and analytical products to help the organization better understand our subscribers, our content, and our path forward to a data-driven future. Authors: Josh McNutt, Keria Bermudez-Hernandez

Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark

Databricks

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

Spark Summit

Tuning ML Models: Scaling, Workflows, and Architecture

Databricks

Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...

Spark Summit

Koalas: Unifying Spark and pandas APIs

Xiao Li

Tuning a Spark ML model with cross-validation can be an extremely computationally expensive process. As the number of hyperparameter combinations increases, so does the number of models being evaluated. The default configuration in Spark is to evaluate each of these models one-by-one to select the best performing. When running this process with a large number of models, if the training and evaluation of a model does not fully utilize the available cluster resources then that waste will be compounded for each model and lead to long run times. Enabling model parallelism in Spark cross-validation, from Spark 2.3, will allow for more than one model to be trained and evaluated at the same time and make better use of cluster resources. We will go over how to enable this setting in Spark, what effect this will have on an example ML pipeline and best practices to keep in mind when using this feature. Additionally, we will discuss ongoing work in progress to reduce the amount of computation required when tuning ML pipelines by eliminating redundant transformations and intelligently caching intermediate datasets. This can be combined with model parallelism to further reduce the run time of cross-validation for complex machine learning pipelines.

Model Parallelism in Spark ML Cross-Validation with Nick Pentreath and Bryan ...

Databricks

Processing data at scale usually results in struggling with performance, strict SLA, limited hardware etc. I've struggled with cutting Spark SQL query run-time and found the culprit! This culprit, and SOLUTION! I would like to share with you. Today in the world of Big Data and Spark we are processing high volume transactions. Catalyst is the Spark SQL query optimizer and in this talk, you will learn how to fully utilize Catalyst optimization power in order to make our queries as fast as possible, by pushing down actions and trying to avoid UDFs as much as possible and maximizing performance.

Adi Polak - Light up the Spark in Catalyst by avoiding UDFs - Codemotion Mila...

Codemotion

La actualidad más candente (20)

5 reasons why spark is in demand!

Spark ML Pipeline serving

Distributed ML/DL with Ignite ML Module Using Apache Spark as Database

Scalable Monitoring Using Apache Spark and Friends with Utkarsh Bhatnagar

Internals of Speeding up PySpark with Arrow

An AI-Powered Chatbot to Simplify Apache Spark Performance Management

Spark at Airbnb

The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...

Headaches and Breakthroughs in Building Continuous Applications

Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models

Scaling ML-Based Threat Detection For Production Cyber Attacks

Spline 2 - Vision and Architecture Overview

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark

How Apache Spark Is Helping Tame the Wild West of Wi-Fi

Tuning ML Models: Scaling, Workflows, and Architecture

Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...

Koalas: Unifying Spark and pandas APIs

Model Parallelism in Spark ML Cross-Validation with Nick Pentreath and Bryan ...

Adi Polak - Light up the Spark in Catalyst by avoiding UDFs - Codemotion Mila...

Similar a AI at Scale

An Insider’s Guide to Maximizing Spark SQL Performance

Takuya UESHIN

Fighting Fraud with Apache Spark

Miklos Christine

Spark + AI Summit 2020 イベント概要

Paulo Gutierrez

Media_Entertainment_Veriticals

Peyman Mohajerian

Apache Spark in Scientific Applications

Dr. Mirko Kämpf

Apache Spark in Scientific Applciations

Dr. Mirko Kämpf

IBM Strategy for Spark

Mark Kerzner

Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2. Continuing forward in that spirit, the upcoming release of Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of soon to be released Spark 2.3 features: • Kubernetes Scheduler Backend • PySpark Performance and Enhancements • Continuous Structured Streaming Processing • DataSource v2 APIs • Structured Streaming v2 APIs

What's New in Upcoming Apache Spark 2.3

Databricks

2018 02-08-what's-new-in-apache-spark-2.3

Chester Chen

End-to-End Data Pipelines with Apache Spark

Burak Yavuz

Apache Spark 2.x has laid the foundation for many new features and functionality. Its main three themes—easier, faster, and smarter—are pervasive in its unified and simplified high-level APIs for Structured data. In this introductory part lecture and part hands-on workshop you’ll learn how to apply some of these new APIs using Databricks Community Edition. In particular, we will cover the following areas: Apache Spark Fundamentals & Concepts What’s new in Spark 2.x SparkSessions vs SparkContexts Datasets/Dataframes and Spark SQL Introduction to Structured Streaming concepts and APIs

Jump Start with Apache Spark 2.0 on Databricks

Anyscale

In this webcast, Patrick Wendell from Databricks will be speaking about Apache Spark's new 1.6 release. Spark 1.6 will include (but not limited to) a type-safe API called Dataset on top of DataFrames that leverages all the work in Project Tungsten to have more robust and efficient execution (including memory management, code generation, and query optimization) [SPARK-9999], adaptive query execution [SPARK-9850], and unified memory management by consolidating cache and execution memory [SPARK-10000].

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

Databricks

Databricks Meetup @ Los Angeles Apache Spark User Group

Paco Nathan

This presentation on Spark Architecture will give an idea of what is Apache Spark, the essential features in Spark, the different Spark components. Here, you will learn about Spark Core, Spark SQL, Spark Streaming, Spark MLlib, and Graphx. You will understand how Spark processes an application and runs it on a cluster with the help of its architecture. Finally, you will perform a demo on Apache Spark. So, let's get started with Apache Spark Architecture. YouTube Video: https://www.youtube.com/watch?v=CF5Ewk0GxiQ What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? Simplilearn’s Apache Spark and Scala certification training are designed to: 1. Advance your expertise in the Big Data Hadoop Ecosystem 2. Help you master essential Apache and Spark skills, such as Spark Streaming, Spark SQL, machine learning programming, GraphX programming and Shell Scripting Spark 3. Help you land a Hadoop developer job requiring Apache Spark expertise by giving you a real-life industry project coupled with 30 demos What skills will you learn? By completing this Apache Spark and Scala course you will be able to: 1. Understand the limitations of MapReduce and the role of Spark in overcoming these limitations 2. Understand the fundamentals of the Scala programming language and its features 3. Explain and master the process of installing Spark as a standalone cluster 4. Develop expertise in using Resilient Distributed Datasets (RDD) for creating applications in Spark 5. Master Structured Query Language (SQL) using SparkSQL 6. Gain a thorough understanding of Spark streaming features 7. Master and describe the features of Spark ML programming and GraphX programming Who should take this Scala course? 1. Professionals aspiring for a career in the field of real-time big data analytics 2. Analytics professionals 3. Research professionals 4. IT developers and testers 5. Data scientists 6. BI and reporting professionals 7. Students who wish to gain a thorough understanding of Apache Spark Learn more at https://www.simplilearn.com/big-data-and-analytics/apache-spark-scala-certification-training

Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...

Simplilearn

ASPgems - kappa architecture

Juantomás García Molina

Apache Spark - A High Level overview

Karan Alang

Started with-apache-spark

Happiest Minds Technologies

This presentation about Apache Spark covers all the basics that a beginner needs to know to get started with Spark. It covers the history of Apache Spark, what is Spark, the difference between Hadoop and Spark. You will learn the different components in Spark, and how Spark works with the help of architecture. You will understand the different cluster managers on which Spark can run. Finally, you will see the various applications of Spark and a use case on Conviva. Now, let's get started with what is Apache Spark. Below topics are explained in this Spark presentation: 1. History of Spark 2. What is Spark 3. Hadoop vs Spark 4. Components of Apache Spark 5. Spark architecture 6. Applications of Spark 7. Spark usecase What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? Simplilearn’s Apache Spark and Scala certification training are designed to: 1. Advance your expertise in the Big Data Hadoop Ecosystem 2. Help you master essential Apache and Spark skills, such as Spark Streaming, Spark SQL, machine learning programming, GraphX programming and Shell Scripting Spark 3. Help you land a Hadoop developer job requiring Apache Spark expertise by giving you a real-life industry project coupled with 30 demos What skills will you learn? By completing this Apache Spark and Scala course you will be able to: 1. Understand the limitations of MapReduce and the role of Spark in overcoming these limitations 2. Understand the fundamentals of the Scala programming language and its features 3. Explain and master the process of installing Spark as a standalone cluster 4. Develop expertise in using Resilient Distributed Datasets (RDD) for creating applications in Spark 5. Master Structured Query Language (SQL) using SparkSQL 6. Gain a thorough understanding of Spark streaming features 7. Master and describe the features of Spark ML programming and GraphX programming Who should take this Scala course? 1. Professionals aspiring for a career in the field of real-time big data analytics 2. Analytics professionals 3. Research professionals 4. IT developers and testers 5. Data scientists 6. BI and reporting professionals 7. Students who wish to gain a thorough understanding of Apache Spark Learn more at https://www.simplilearn.com/big-data-and-analytics/apache-spark-scala-certification-training

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Simplilearn

Apache spark 2.4 and beyond

Xiao Li

Apache Spark 2.0 and subsequent releases of Spark 2.1 and 2.2 have laid the foundation for many new features and functionality. Its main three themes—easier, faster, and smarter—are pervasive in its unified and simplified high-level APIs for Structured data. In this introductory part lecture and part hands-on workshop, you’ll learn how to apply some of these new APIs using Databricks Community Edition. In particular, we will cover the following areas: Agenda: • Overview of Spark Fundamentals & Architecture • What’s new in Spark 2.x • Unified APIs: SparkSessions, SQL, DataFrames, Datasets • Introduction to DataFrames, Datasets and Spark SQL • Introduction to Structured Streaming Concepts • Four Hands On Labs You will use Databricks Community Edition, which will give you unlimited free access to a ~6 GB Spark 2.x local mode cluster. And in the process, you will learn how to create a cluster, navigate in Databricks, explore a couple of datasets, perform transformations and ETL, save your data as tables and parquet files, read from these sources, and analyze datasets using DataFrames/Datasets API and Spark SQL. Level: Beginner to intermediate, not for advanced Spark users. Prerequisite: You will need a laptop with Chrome or Firefox browser installed with at least 8 GB. Introductory or basic knowledge Scala or Python is required, since the Notebooks will be in Scala; Python is optional. Bio: Jules S. Damji is an Apache Spark Community Evangelist with Databricks. He is a hands-on developer with over 15 years of experience and has worked at leading companies, such as Sun Microsystems, Netscape, LoudCloud/Opsware, VeriSign, Scalix, and ProQuest, building large-scale distributed systems. Before joining Databricks, he was a Developer Advocate at Hortonworks.

Jump Start on Apache® Spark™ 2.x with Databricks

Databricks

Similar a AI at Scale (20)

An Insider’s Guide to Maximizing Spark SQL Performance

Fighting Fraud with Apache Spark

Spark + AI Summit 2020 イベント概要

Media_Entertainment_Veriticals

Apache Spark in Scientific Applications

Apache Spark in Scientific Applciations

IBM Strategy for Spark

What's New in Upcoming Apache Spark 2.3

2018 02-08-what's-new-in-apache-spark-2.3

End-to-End Data Pipelines with Apache Spark

Jump Start with Apache Spark 2.0 on Databricks

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

Databricks Meetup @ Los Angeles Apache Spark User Group

Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...

ASPgems - kappa architecture

Apache Spark - A High Level overview

Started with-apache-spark

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Apache spark 2.4 and beyond

Jump Start on Apache® Spark™ 2.x with Databricks

Más de Adi Polak

https://aka.ms/spark-architecture https://aka.ms/distributed-programming https://twitter.com/AdiPolak ------------------------------------------------------------------------------------------------- Spark is quickly gaining steam in the Big Data analytics world. However, it also has excellent features for Stream processing and machine learning development. Big Data systems are a new reality. Whether we're creating microservices-based solutions or machine learning-based products, we work with data, most often, big data. Apache Spark is your friend if you want to position yourself on the road for success as a Big Data Developer, Data Engineer, or Data Science with hands-on production experience. Come to this session to learn Apache Spark basics and how YOU can get starting with it.

Demystifying Apache Spark

Adi Polak

y running your workloads in Kubernetes, we can focus on designing and building your applications instead of managing the infrastructure that runs them. But wait! what about the cost?? With the Virtual Kubelet provider for AKS and Azure Container Instances, both Linux and Windows containers can be scheduled on a container instance as if it is a standard Kubernetes node. This configuration allows you to take advantage of both the capabilities of Kubernetes and the management value and cost-benefit of container instances. In this talk you will learn how to deploy an application to AKS and ACI with Virtual Kublet. While leveraging the scalability of Kubernetes and cost efficiency of ACI. https://dev.to/adipolak/kubernetes-and-virtual-kubelet-in-a-nutshell-gn4

Burst workloads Cutting costs with Kubernetes and Virtual Kubelet

Adi Polak

Breaking up a monolith or switching from client desktop to using the web in scale, require us to think of many factors, like the engineering team and the knowledge that the team already possess, technologies that exist, how to build the infrastructure right and much more. How can we use Kubernetes with Virtual Kubelet to cut costs and use the right service for the workload, whether it is a burst workload or a steady one

From desktop to the cloud, cutting costs with Virtual kubelet and ACI

Adi Polak

ETL – Everything you need to know

Adi Polak

People love Visual Studio Code for its superfast performance, lightweight nature and active open source community around it. All of these are now available for Java developers by extensions contributed by individuals and companies such as Red Hat, Microsoft, and Pivotal. By combining a couple of independent extensions, you can create a lightweight working environment just for your Java workload yet powerful enough to almost match the feature richness of existing IDEs. This sessions shows how you can use Visual Studio Code to develop your Java application. It guides you through all the available extensions so you can later explore then base on your own needs. The session will also introduce how to easily push your Java microservices to cloud within the editor itself.

Evolution of VS code Java ecosystem

Adi Polak

Make it clean - scala clean code

Adi Polak

Processing data at scale usually involves struggling with performance, strict SLA, limited hardware capabilities and more. After struggling with Spark SQL query run-time, I found the felon! In this lecture, I would like to share with you the change in perspective and process we had to go through in order to find the felon (and the solution!). Today in the world of Big Data and Spark we are processing high volume transactions. Catalyst is the Spark SQL query optimizer, in this talk, we will reveal how you can fully utilize Catalyst’s optimization power in order to make queries run as fast as possible, by pushing down actions and avoiding UDFs as much as possible, while still maximizing performance.

Spark UDFs are EviL, Catalyst to the rEsCue!

Adi Polak

Más de Adi Polak (7)

Demystifying Apache Spark

Burst workloads Cutting costs with Kubernetes and Virtual Kubelet

From desktop to the cloud, cutting costs with Virtual kubelet and ACI

ETL – Everything you need to know

Evolution of VS code Java ecosystem

Make it clean - scala clean code

Spark UDFs are EviL, Catalyst to the rEsCue!

Último

Roadmap to Membership of RICS - Pathways and Routes

M Maged Hegazy, LLM, MBA, CCP, P3O

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Chance Of Getting Into My Sexy Boobs? Booking Contact Details WhatsApp Chat: +91-8250192130 pune Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts pune understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide - 30-april-2024(v.n)

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

ranjana rawat

AKTU Computer Networks notes --- Unit 3.pdf

ankushspencer015

Result Management System Report for College Project The Result Management System is a crucial tool for educational institutions to efficiently manage and publish student results online. This project was developed as part of the 6th semester project for Pokhara University, with the aim of providing a user-friendly platform for students, faculty, and administrators to access and manage academic results. The system allows for easy input of student grades, calculation of final results, and generation of result reports. It also provides secure access to results for students through a login system, ensuring data privacy and confidentiality. To improve and increase the effectiveness of the Result Management System, several enhancements can be implemented. These include: 1. Integration of automated result processing algorithms to reduce manual data entry and calculation errors. 2. Implementation of a notification system to alert students and faculty when results are published or updated. 3. Addition of features such as result analysis tools, graphical representations of student performance, and trend analysis. 4. Enhancing the user interface to make it more intuitive and user-friendly for all stakeholders. 5. Incorporating a feedback mechanism to gather suggestions and improve the system based on user input. By incorporating these enhancements, the Result Management System can become a more robust and efficient tool for managing academic results online, benefiting both students and faculty alike.

result management system report for college project

Tonystark477637

Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Sex Service At Affordable Rate Booking Contact Details WhatsApp Chat: +91-6297143586 pune Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts pune understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide - 01-may-2024(v.n)

Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...

Call Girls in Nagpur High Profile

Call Girl Bhosari Indira Call Now: 8617697112 Bhosari Escorts Booking Contact Details WhatsApp Chat: +91-8617697112 Bhosari Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts Bhosari understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide –

(INDIRA) Call Girl Bhosari Call Now 8617697112 Bhosari Escorts 24x7

Call Girls in Nagpur High Profile Call Girls

Structural Analysis and Design of Foundations: A Comprehensive Handbook for Students and Professionals. Unlock the potential of foundation design with Dr. Costas Sachpazis’s enlightening handbook, a meticulously crafted guide poised to become an indispensable resource for both budding and seasoned civil engineers. This comprehensive manual illuminates the theoretical and practical aspects of structural analysis and design across various types of foundations and retaining walls. Within these pages, Dr. Sachpazis distills complex engineering principles into digestible, step-by-step processes, enhanced by detailed diagrams, case studies, and real-world examples that bridge the gap between academic study and professional application. From soil mechanics and load calculations to innovative design techniques and sustainability considerations, this book covers a vast landscape of structural engineering. Key Features: • In-Depth Analysis and Design: Explore thorough explanations of both shallow and deep foundation designs, supported by case studies that demonstrate their practical implementations. • Practical Guides: Benefit from detailed guides on site investigation, bearing capacity calculations, and settlement analysis, ensuring designs are both robust and reliable. • Innovative Techniques: Discover the latest advancements in foundation technology and retaining wall design, preparing you for future trends in civil engineering. • Educational Tools: Utilize this handbook as an educational tool, perfect for both classroom learning and professional development. Whether you're a student eager to learn the fundamentals or a professional seeking to deepen your expertise, Dr. Sachpazis’s handbook is designed to support and inspire excellence in the field of structural engineering. Embrace this opportunity to enhance your skills and contribute to building safer, more efficient structures.

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...

Dr.Costas Sachpazis

The Educational Administration: Theory and Practice publishes prominent empirical and conceptual articles focused on timely and critical leadership and policy issues of educational organizations. The journal embraces traditional and emergent research paradigms, methods, and issues. The journal particularly promotes the publication of rigorous and relevant scholarly work that enhances linkages among and utility for educational policy, practice, and research arenas. The goal of the editorial team and the journal’s editorial board is to promote sound scholarship and a clear and continuing dialogue among scholars and practitioners from a broad spectrum of education. Educational Administration: Theory and Practice presents prominent empirical and conceptual articles focused on timely and critical leadership and policy issues facing educational organizations. As an editorial team, we embrace traditional and emergent theoretical frameworks, research methods, and topics. We particularly promote the publication of rigorous and relevant scholarly work with utility for educational policy, practice, and research. The journal’s primary focus is on studies of educational leadership, organizations, leadership development, and policy as they relate to elementary and secondary levels of education. Examinations of leadership and policy that fall outside K-12 are considered insofar as there are meaningful connections to the K-12 arena (e.g., college pipeline). International comparative investigations are welcome to the extent they have implications for a broad audience.s.

Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...

Christo Ananth

VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to 25K High Profile Escorts In Pune Booking Now open +91- 8005736733 Why you Choose Us- +91- 8005736733 HOT⇄ 8005736733 Mr ashu ji Call Mr ashu Ji +91- 8005736733 (V030524]N) 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models • Foreigner Models • TV Actress and Celebrities • Receptionist • Air Hostess • Call Center Working Girls/Women • Hi-Tech Co. Girls/Women • Housewife

VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...

SUHANI PANDEY

International Journal of Intelligent Systems and Applications in Engineering (IJISAE) is an international and interdisciplinary journal for both invited and contributed peer reviewed articles that intelligent systems and applications in engineering at all levels. The journal publishes a broad range of papers covering theory and practice in order to facilitate future efforts of individuals and groups involved in the field. IJISAE, a peer-reviewed double-blind refereed journal, publishes original papers featuring innovative and practical technologies related to the design and development of intelligent systems in engineering. Its coverage also includes papers on intelligent systems applications in areas such as nanotechnology, renewable energy, medicine engineering, Aeronautics and Astronautics, mechatronics, industrial manufacturing, bioengineering, agriculture, services, intelligence based automation and appliances, medical robots and robotic rehabilitations, space exploration and etc.

Call for Papers - International Journal of Intelligent Systems and Applicatio...

Christo Ananth

Increased aeration of the soil; Stabilized soil structure; Higher and more diversified crop production; Better workability of the land; Earlier planting dates; Reduction of peak discharges by an increased temporary storage of water in the soil decomposition of organic matter; soil subsidence; reduced irrigation efficiency; increased risk of drought. excessive leaching of valuable nutrients from the soil; downstream environmental damage by salty or otherwise polluted drainage water; the presence of ditches, canals, and structures impending accessibility and interfering with other infrastructural elements of the land.

chapter 5.pptx: drainage and irrigation engineering

mulugeta48

Call Girl Aurangabad Indira Call Now: 8617697112 Aurangabad Escorts Booking Contact Details WhatsApp Chat: +91-8617697112 Aurangabad Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts Aurangabad understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide –

(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7

Call Girls in Nagpur High Profile Call Girls

Call Girl Meerut Indira Call Now: 8617697112 Meerut Escorts Booking Contact Details WhatsApp Chat: +91-8617697112 Meerut Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus they look fabulously elegant; making an impressionable. Independent Escorts Meerut understands the value of confidentiality and discretion - they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide –

(INDIRA) Call Girl Meerut Call Now 8617697112 Meerut Escorts 24x7

Call Girls in Nagpur High Profile Call Girls

Welcome to the April edition of WIPAC Monthly, the magazine brought to you by Water Industry Process Automation & Control. In this month's edition, along with the latest news from the industry we have articles on: The use of artificial intelligence and self-service platforms to improve water sustainability A feature article on measuring wastewater spills An article on the National Underground Asset Register Have a good month, Oliver

Water Industry Process Automation & Control Monthly - April 2024

Water Industry Process Automation & Control

data_management_and _data_science_cheat_sheet.pdf

JiananWang21

Thermal Engineering-R & A / C - unit - V

DineshKumar4165

Call Girls Walvekar Nagar Call Me 7737669865 Budget Friendly No Advance Booking Booking Now open +91- 7737669865 Why you Choose Us- +91- 7737669865 HOT⇄ 7737669865 Mr ashu ji Call Mr ashu Ji +91- 7737669865 (V020524]N) 𝐇𝐨𝐭𝐞𝐥 𝐑𝐨𝐨𝐦𝐬 𝐈𝐧𝐜𝐥𝐮𝐝𝐢𝐧𝐠 𝐑𝐚𝐭𝐞 𝐒𝐡𝐨𝐭𝐬/𝐇𝐨𝐮𝐫𝐲🆓 .█▬█⓿▀█▀ 𝐈𝐍𝐃𝐄𝐏𝐄𝐍𝐃𝐄𝐍𝐓 𝐆𝐈𝐑𝐋 𝐕𝐈𝐏 𝐄𝐒𝐂𝐎𝐑𝐓 Hello Guys ! High Profiles young Beauties and Good Looking standard Profiles Available , Enquire Now if you are interested in Hifi Service and want to get connect with someone who can understand your needs. Service offers you the most beautiful High Profile sexy independent female Escorts in genuine ✔✔✔ To enjoy with hot and sexy girls ✔✔✔ ★providing:- • Models • vip Models • Russian Models

Call Girls Walvekar Nagar Call Me 7737669865 Budget Friendly No Advance Booking

roncy bisnoi

PVC VS. FIBERGLASS (FRP) GRAVITY SEWER - UNI BELL

ManishPatel169454

UNIT-III FMM. DIMENSIONAL ANALYSIS

rknatarajan

GAS POWER CYCLES Cycles: Otto, Diesel, Dual, Brayton - Calculation of mean effective pressure - Air standard efficiency - Comparison of cycles INTERNAL COMBUSTION ENGINES Classification - Components and their function - valve timing diagram and port timing diagram - actual and theoretical p-v diagram of two stroke and four stroke engines – carburettor - diesel pump and injector system - battery and magneto ignition system - principles of combustion and detonation in CI engines - lubrication and cooling systems - performance parameters and calculations.

Thermal Engineering Unit - I & II . ppt

DineshKumar4165

AI at Scale

1. AI at Scale Adi Polak | Microsoft @adipolak

2. Start from the beginning • Massive datasets • Intelligent product @adipolak

3. • Apache Spark • Databricks for Production • Build machine learning models with Pipelines • Cognitive Services and Apache Spark • Data Engineer vs Data Scientist • MMLSpark Overview @adipolak

4. What is Apache Spark

5. What is Apache Spark? @adipolak

6. Motivation @adipolak

7. Apache Spark architecture @adipolak

8. Apache Spark submit job storage storage storage @adipolak

9. Framework libraries- Software Stracture @adipolak

10. Python, R & .NET for Apache Spark @adipolak

11. Embedded Service PySpark Spark Scala Process Pyspark Protocol Spark Worker @adipolak

12. Structured Streaming MLlib GraphFrame TensorFrames SQL SparkSession / DataFrame / Dataset APIs Data Source Connectors Catalyst Optimization & Tungsten Execution Spark Core (RDD APIs) SQL API More API @adipolak

13. Why Apache Spark is fast and efficient? CATALYST OPTIMIZER IN MEMORY COMPUTING TUNGSTEN @adipolak

14. CATALYST @adipolak

15. Fundamentals of Catalyst Optimizer SUB Attribute(x) SUB some_func(1) some_func(2) Tree Rules SUB Attribute(x) some_func(-1) @adipolak

16. Spark SQL Execution Plan Logical optimization –> Optimization rules • Constant folding • Predicate pushdown • Projection pruning • … Catalyst Frontend Backend @adipolak

17. Apache Spark Connectors

18. Spark Connector for SQL Server @adipolak

19. Spark connector for Cosmos DB @adipolak

20. Many more connectors… @adipolak

21. How to run Apache Spark on Azure? HDInsight Azure Databricks AKS

22. Azure Databricks

23. What is Azure Databricks @adipolak

24. Workspace and Notebooks

25. Build for Production • - Databricks limits and Azure limits • - Workspace isolation • - vnet • - Security and key vault • - Enable log analytics for monitoring

26. Databricks limits and Azure limits • Key Databricks limits ( per workspace): • - 150 concurrent jobs • - 150 max notebooks • - 1000 max hourly job submission Key Azure limits : - Storage accounts per region per subscription: 250 - VMs per subscription per region: 25,000 - Resource groups per subscription: 980 @adipolak

27. Workspace isolation @adipolak

28. Cluster modes and their characteristics @adipolak

29. Batch vs. Interactive workloads @adipolak

30. Data Scientists and Analysts – Hight concurrency mode @adipolak

31. Data Engineers @adipolak

32. Troubleshooting with cluster logs • Azure Databricks provides three kinds of logging of cluster-related activity: • Cluster event logs, which capture cluster lifecycle events, like creation, termination, configuration edits, and so on. • Apache Spark driver and worker logs, which you can use for debugging. • Cluster init-script logs, valuable for debugging init scripts. @adipolak

33. Cluster logs

34. Cluster Metric Report @adipolak

35. Log Analytics Workspace • Leverage Azure log analytics and connect it to the workspace • with init scripts and visualize it using Kibana or other tools. @adipolak

36. What About Machine Learning?

37. Gather Data ML Process / Basic Life Cycle Feature Extract, Clean and Normalize Select algorithm Evaluate model Data visualization 4 5 1 2 3 Repeat! @adipolak

38. H O W ? M L P i p e l i n e s ! @adipolak

39. What are pipelines?

40. Tools Spark Streaming Spark ML/Other Spark SQL MLflow @adipolak

41. @adipolak

42. Demo @adipolak

43. But in real life: Accuracy < 0.5 AUC L Results are far from being good

44. Building machine learning models from scratch is hard! @adipolak

45. We need a Data Science @adipolak

46. Our Data scientist needs tools @adipolak

47. Stand on the shoulder of giants @adipolak

48. Azure Cognitive Services @adipolak

49. Cognitive Services capabilities Infuse your apps, websites, and bots with human-like intelligence

50. A variety of real-world applications Vision Speech Intent: PlayCall Language Knowledge Search

51. Let’s Combine them + @adipolak

52. Apache Spark + Cognitive Services @adipolak

53. @adipolak

54. Data Engineer vs Data Scientist Advanced programming Distributed systems Data pipeline Basic analytics Data Engineer Advance math/statistics ML/AI Advanced analytics Basic programming Data Science Big Data constrains Both should understand: @adipolak

55. MMLSpark: Microsoft Machine Learning for Apache Spark @adipolak

56. aka.ms/mlflow_and_azure_ml aka.ms/azure_databricks aka.ms/twitter_sentiment_analysis Want to learn more? @adipolak Thank you! Dank u! Merci!

AI at Scale

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a AI at Scale

Similar a AI at Scale (20)

Más de Adi Polak

Más de Adi Polak (7)

Último

Último (20)

AI at Scale