Hands-on with Alluxio Structured Data Management

•

0 recomendaciones•1,618 vistas

Alluxio Online Office Hours January 14, 2020 We introduce the concepts and components of Alluxio Structured Data Management, and go through a demo with Presto.

Software

Alluxio Office Hour:
Alluxio Structured Data Management
Gene Pang | Alluxio

Motivation
Alluxio Structured Data Management
Developer Preview in Alluxio 2.1
Demo
Conclusion
Outline
2

Common Alluxio Use Cases
4
…
…
Unified Interface
Unified Namespace
Caching and Locality
SQL Engines are popular

5
Storage Systems SQL Frameworks
Files/Objects
Directories
Raw Bytes
Storage
Optimized
Tables
Schemas
Rows/Columns
Compute
Optimized
Impedance Mismatch
Further Expand Benefits!

Benefits of Alluxio Data Orchestration
6
Storage
Systems
SQL
Frameworks
Caching
Unified Interface/Namespace
Schema-Aware Optimizations
Compute-Optimized Formats
Physical Data Independence

Alluxio
Structured Data Management
Integrating SQL Engines with Alluxio

Provide Structured Data APIs
Focus on how frameworks interact with data
High-Level Philosophy
8
Cache Logical Data Access
Focus on caching what frameworks want

Alluxio Structured Data Management
Alluxio Structured Data Management
9
Storage
System
Transformation
Service
Structured Data
and Metadata
Logical Data
Access Layer
Structured
Data Client
SQL
Engine
Engine

Developer Preview in Alluxio 2.1
Try out initial components!

Target Environment
11
Presto
Hive
Connector
Hive
Metastore
Storage

Alluxio Structured Data Management
12
Presto
Alluxio Caching
Service
Alluxio Catalog
Service
AlluxioTransformation
Service
Hive
Connector
Alluxio
Connector
Hive
Metastore
Storage

Alluxio Catalog Service
13
Alluxio Catalog Service
Hive Metastore
Hive Under Database
Functionality
Manages metadata for structured data
Abstracts other database catalogs as
Under Database (UDB)
Benefits
Schema-aware optimizations
Simple deployment

Tighter integration with Presto
New plugin based on the Presto Hive connector
Available in Alluxio 2.1 distribution
In Progress: Merging connector into Presto codebase
Alluxio Presto Connector
14

Transformation Service
15
Transform data to be compute-optimized
independent from storage-optimized format
Coalesce Format Conversion
parquetcsv

2 isolated AWS 10-node clusters
Presto + Hive Metastore + S3 Data
Presto + Alluxio + Hive Metastore + S3 Data
TPCDS sample dataset on S3
~10,000 CSV files
Demo
17

Attached existing Hive database into Alluxio Catalog
Alluxio Catalog served table metadata for Presto
Transformed store_sales by coalescing and converting CSV to Parquet
Demo Summary
18
Presto Without
Alluxio
20s
Alluxio
Transformations
7s
AlluxioTransformations
With Caching
3s

User community feedback/collaboration is important!
Future projects
New UDB implementations (AWS Glue)
More conversion formats (json)
DDL/DML workloads (CREATETABLE, INSERT, etc.)
New Client APIs for structured data (Arrow)
Future Work
20

Try it out!
Documentation
Provide feedback
Feature requests and issues in Github Alluxio/alluxio
Developer Preview Available in Alluxio 2.1
21
ThankYou!

Más contenido relacionado

La actualidad más candente

Alluxio Community Office Hour October 20, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Data Orchestration for the Hybrid Cloud Era

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Webinar August 25, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Adit Madan Bin Fan With data lakes expanding from on-prem to the cloud as well as increasing use of new object data stores, data platform teams are challenged with providing consistent, high-throughput access to distributed data sources for analytics and AI/ML applications. In today’s hybrid cloud and multi-cloud era, data-intensive applications such as Presto, Spark, Hive, and Tensorflow are suffering more sluggish response times and increased complexity with the growing separation of data and compute. Join Alluxio’s distributed systems experts as they explore today’s data access challenges and open source data orchestration solutions for modernizing your data platform. In this tech talk, you'll learn: - How data access and throughput challenges are hindering large-scale analytics and AI/ML applications - How a data orchestration layer can simplify distributed data access and improve performance - Real-world production use cases and example journeys for architecting a modern data platform

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Alluxio, Inc.

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Alluxio, Inc.

Alluxio Community Office Hour February 23, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Global Online Meetup August 4, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Koen Michiels, Datasapiens Juraj Pohanka, Datasapiens Bin Fan, Alluxio Datasapiens is an international data-analytics startup based in Prague. We help our clients to uncover the value of their data and open up new revenue streams for them. We provide an end-to-end service that manages the data pipeline and automates the process of generating data insights. In this talk, we will describe how we have solved an issue with large S3 API costs incurred by Presto under several usage concurrency levels by implementing Alluxio as a data orchestration layer between S3 and Presto. Also, we will show the results of an experiment with estimating the per-query S3 API costs using the TPC-DS dataset. This talk will focus on: - The Hadoop ecosystem at Datasapiens - Drastic increase of S3 API costs during performance tests with Presto - S3 API costs tests with TPC-DS - Implications to the cloud data lake architecture

Reducing large S3 API costs using Alluxio at Datasapiens

Alluxio, Inc.

Introducing the Hub for Data Orchestration

Alluxio, Inc.

Alluxio on AWS EMR Fast Storage Access & Sharing for Spark

Alluxio, Inc.

Alluxio Community Office Hour July 14, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Calvin Jia, Alluxio Bin Fan, Alluxio Alluxio 2.3 was just released at the end of June 2020. Calvin and Bin will go over the new features and integrations available and share learnings from the community. Any questions about the release and on-going community feature development are welcome. In this Office Hour, we will go over: - Glue Under Database integration - Under Filesystem mount wizard - Tiered Storage Enhancements - Concurrent Metadata Sync - Delegated Journal Backups

What's New in Alluxio 2.3

Alluxio, Inc.

Iceberg + Alluxio for Fast Data Analytics

Alluxio, Inc.

Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017

Alluxio, Inc.

Spark Summit EU talk by Jiri Simsa

Spark Summit

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Alluxio, Inc.

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio, Inc.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

Alluxio Mesos Meetup - SMACK to SMAACK

Alluxio, Inc.

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio, Inc.

Embracing hybrid cloud for data-intensive analytic workloads

Alluxio, Inc.

Deep Learning and Gene Computing Acceleration with Alluxio in Kubernetes

Alluxio, Inc.

La actualidad más candente (20)

Data Orchestration for the Hybrid Cloud Era

Best Practices for Using Alluxio with Spark

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Accelerate Analytics and ML in the Hybrid Cloud Era

Best Practices for Using Alluxio with Spark

Reducing large S3 API costs using Alluxio at Datasapiens

Introducing the Hub for Data Orchestration

Alluxio on AWS EMR Fast Storage Access & Sharing for Spark

What's New in Alluxio 2.3

Iceberg + Alluxio for Fast Data Analytics

Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017

Spark Summit EU talk by Jiri Simsa

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio Mesos Meetup - SMACK to SMAACK

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Embracing hybrid cloud for data-intensive analytic workloads

Deep Learning and Gene Computing Acceleration with Alluxio in Kubernetes

Similar a Hands-on with Alluxio Structured Data Management

Alluxio Innovations for Structured Data

Alluxio, Inc.

Building the Perfect SharePoint 2010 Farm

Michael Noel

FHIR Server internals - sqlonfhir

Brian Postlethwaite

SharePoint 2010 High Availability - TechEd Brasil 2010

Michael Noel

Flask SQLite .pdf

SudhanshiBakre1

INFOGOV14 - Trusting Your KM & ECM Strategy to SharePoint

Jonathan Ralton

More Best Practices With Share Point Solutions

Alexander Meijers

MOSS 2007 Deployment Fundamentals -Part1

Information Technology

Tech Ed Africa Demystifying Backup Restore In Share Point 2007

Joel Oleson

Techedafricademystifyingbackuprestoreinsharepoint2007 090805103250 Phpapp02

malonzo

Alluxio: Unify Data at Memory Speed

Alluxio, Inc.

Large Scale SQL Considerations for SharePoint Deployments

Joel Oleson

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics - Apache Spark’s in memory capabilities catapulted it as the premier processing framework for Hadoop. Apache Ignite and Alluxio, both high-performance, integrated and distributed in-memory platform, takes Apache Spark to the next level by providing an even more powerful, faster and scalable platform to the most demanding data processing and analytic environments. Speaker Irfan Elahi, Consultant, Deloitte

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

DataWorks Summit

Alluxio Austin Meetup Aug 15, 2019 Speaker: Bin Fan Apache Spark and Alluxio are cousin open source projects that originated from UC Berkeley’s AMPLab. Running Spark with Alluxio is a popular stack particularly for hybrid environments. In this session, I will briefly introduce Apache Spark and Alluxio, share the top ten tips for performance tuning for real-world workloads, and demo Alluxio with Spark.

Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics

Alluxio, Inc.

Jonathan Ralton - Trusting Your KM & ECM Strategy To SharePoint

ARMA International

SQL Server 2016 provides for unprecedented high availability and disaster recovery options for SharePoint farms in the form of AlwaysOn Availability Groups. Using this new technology, SharePoint architects can provide for near-instant failover at the data tier, without the risk of any data loss. In addition, the latest version of this technology, available with SQL Server 2016, allows for replicas of SharePoint databases to be stored in the cloud in Microsoft’s Azure cloud offering. This technology, which will be demonstrated live, completely changes the data tier design options for SharePoint and revolutionises high availability options for a farm. This session covers in step-by-step detail the exact configuration required to enable this functionality for a SharePoint 2013 farm, based on the best practices, tips and tricks, and real-world experience of the presenter in deploying this technology in production. Understand the differences between SQL AlwaysOn options, and determine the requirements to deploy the technologies Examine how SQL Server 2016 AlwaysOn Availability Groups can provide aggressive Service Level Agreements (SLAs) with a Recovery Point Objective (RPO) of zero and a Recovery Time Objective (RTO) of a few seconds. See the exact steps required to enable SQL Server 2016 AlwaysOn Availability Groups for a SharePoint 2013 On-Premises environment, including options for storing replicas in Microsoft’s Azure cloud service.

AUDWC 2016 - Using SQL Server 20146 AlwaysOn Availability Groups for SharePoi...

Michael Noel

SPSPTCDC - SharePoint Admin 101 - SpeedMetal - PowerUser to Admin in 75 Minutes

Knowledge Management Associates, LLC

I/O & virtualization performance with a search engine based on an xml databa...

lucenerevolution

Spark Summit EU talk by Jiri Simsa

Alluxio, Inc.

Alluxio Tech Talk Mar 12, 2019 Speaker: Bin Fan, Alluxio Matt Fuller, Starburst As data analytic needs have increased with the explosion of data, the importance of the speed of analytics and the interactivity of queries has increased dramatically In this tech talk, we will introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn about: - The architecture of Presto, an open source distributed SQL engine, as well as innovations by Starburst like as it’s cost-based optimizer - How Presto can query data from cloud object storage like S3 at high performance and cost-effectively with Alluxio - How to achieve data locality and cross-job caching with Alluxio no matter where the data is persisted and reduce egress costs In addition, we’ll present some real world architectures & use cases from internet companies like JD.com and NetEase.com running the Presto and Alluxio stack at the scale of hundreds of nodes.

Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud

Alluxio, Inc.

Similar a Hands-on with Alluxio Structured Data Management (20)

Alluxio Innovations for Structured Data

Building the Perfect SharePoint 2010 Farm

FHIR Server internals - sqlonfhir

SharePoint 2010 High Availability - TechEd Brasil 2010

Flask SQLite .pdf

INFOGOV14 - Trusting Your KM & ECM Strategy to SharePoint

More Best Practices With Share Point Solutions

MOSS 2007 Deployment Fundamentals -Part1

Tech Ed Africa Demystifying Backup Restore In Share Point 2007

Techedafricademystifyingbackuprestoreinsharepoint2007 090805103250 Phpapp02

Alluxio: Unify Data at Memory Speed

Large Scale SQL Considerations for SharePoint Deployments

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics

Jonathan Ralton - Trusting Your KM & ECM Strategy To SharePoint

AUDWC 2016 - Using SQL Server 20146 AlwaysOn Availability Groups for SharePoi...

SPSPTCDC - SharePoint Admin 101 - SpeedMetal - PowerUser to Admin in 75 Minutes

I/O & virtualization performance with a search engine based on an xml databa...

Spark Summit EU talk by Jiri Simsa

Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud

Más de Alluxio, Inc.

As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

Alluxio, Inc.

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sally (Mihyoung) Lee (Senior Staff Engineer, TLM, @Uber) Machine learning models power Uber’s everyday business. However, developing and deploying a model is not a one-time event but a continuous process that requires careful planning, execution, and monitoring. This session will highlight Uber’s practice on the machine learning lifecycle to ensure high model quality.

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio, Inc.

Alluxio Monthly Webinar Oct. 3, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Greg Palmer (Lead Solutions Engineer) Model training requires extensive computational and GPU resources. When training models on AWS, loading data from S3 often becomes a major bottleneck, wasting valuable GPU cycles. Optimizing data loading can greatly reduce GPU idle time and increase GPU utilization. In this webinar, Greg Palmer will discuss best practices for efficient data loading during model training on AWS. He will demonstrate how to use Alluxio on EKS as a distributed cache to accelerate PyTorch training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs. What you will learn: - The challenges of feeding data-hungry GPUs in the cloud - How to accelerate model training by optimizing data loading on AWS - The reference architecture for running PyTorch jobs with Alluxio cache on EKS while reading data from S3, with benchmark results of training ResNet50 and BERT - How to use TensorBoard to identify bottlenecks in GPU utilization

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio, Inc.

Más de Alluxio, Inc. (20)

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Último

%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg

masabamasaba

%in Soweto+277-882-255-28 abortion pills for sale in soweto

masabamasaba

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal. Attraction Spells

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...

masabamasaba

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

%in Benoni+277-882-255-28 abortion pills for sale in Benoni

masabamasaba

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2

This presentation covers the following topics: What is logging? The purpose of logging: Debugging The purpose of logging: Security The purpose of logging: Stats & analytics Traditional logging Traditional logging: Advantages Traditional logging: Disadvantages The solution: Large-scale logging Large-scale logging: Core principles Large-scale logging: Solution types Large-scale logging: Cloud vs on-prem Large-scale logging: Operational complexity Large-scale logging: Security Large-scale logging: Costs Large-scale logging: On-prem comparison - Elasticsearch - Grafana Loki - VictoriaLogs On-prem comparison: Setup and operation On-prem comparison: Costs On-prem comparison: Full-text search support On-prem comparison: How to efficiently query 100TB of logs? On-prem comparison: Integration with CLI tools VictoriaLogs for large-scale logging VictoriaLogs demo instance - Ingestion rate: 3600 messages / minute - The number of log messages: 1.1 billion - Uncompressed log messages’ size: 1.5TB - Compressed log messages’ size: 23GB - Compression ratio: 47x - Memory usage: 150MB VictoriaLogs CLI integration demo - Which errors have occurred in all the apps during the last hour? - How many errors have occurred during the last hour? - Which apps generated the most of errors during the last hour? - The number of per-minute errors for the last 10 minutes - Status codes for the last hour - Non-200 status codes for the last week - Top client IPs for the last 4 weeks with 404 and 500 response status codes - Per-month stats for the given IP across all the logs Large-scale logging solution MUST provide excellent CLI integration VictoriaLogs: (temporary) drawbacks VictoriaLogs: Recap - Easy to setup and operate - The lowest RAM usage and disk space usage (up to 30x less than Elasticsearch and Grafana Loki) - Fast full-text search - Excellent integration with traditional command-line tools for log analysis - Accepts logs from popular log shippers (Filebeat, Fluentbit, Logstash, Vector, Promtail, Grafana Agent) - Open source and free to use!

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

VictoriaMetrics

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...

WSO2

WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...

WSO2

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Bert Jan Schrijver

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

WSO2CON2024 - It's time to go Platformless

WSO2

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

masabamasaba

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

WSO2

OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...

Shane Coughlan

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

Hands-on with Alluxio Structured Data Management

1. Alluxio Office Hour: Alluxio Structured Data Management Gene Pang | Alluxio

2. Motivation Alluxio Structured Data Management Developer Preview in Alluxio 2.1 Demo Conclusion Outline 2

3. Motivation

4. Common Alluxio Use Cases 4 … … Unified Interface Unified Namespace Caching and Locality SQL Engines are popular

5. 5 Storage Systems SQL Frameworks Files/Objects Directories Raw Bytes Storage Optimized Tables Schemas Rows/Columns Compute Optimized Impedance Mismatch Further Expand Benefits!

6. Benefits of Alluxio Data Orchestration 6 Storage Systems SQL Frameworks Caching Unified Interface/Namespace Schema-Aware Optimizations Compute-Optimized Formats Physical Data Independence

7. Alluxio Structured Data Management Integrating SQL Engines with Alluxio

8. Provide Structured Data APIs Focus on how frameworks interact with data High-Level Philosophy 8 Cache Logical Data Access Focus on caching what frameworks want

9. Alluxio Structured Data Management Alluxio Structured Data Management 9 Storage System Transformation Service Structured Data and Metadata Logical Data Access Layer Structured Data Client SQL Engine Engine

10. Developer Preview in Alluxio 2.1 Try out initial components!

11. Target Environment 11 Presto Hive Connector Hive Metastore Storage

12. Alluxio Structured Data Management 12 Presto Alluxio Caching Service Alluxio Catalog Service AlluxioTransformation Service Hive Connector Alluxio Connector Hive Metastore Storage

13. Alluxio Catalog Service 13 Alluxio Catalog Service Hive Metastore Hive Under Database Functionality Manages metadata for structured data Abstracts other database catalogs as Under Database (UDB) Benefits Schema-aware optimizations Simple deployment

14. Tighter integration with Presto New plugin based on the Presto Hive connector Available in Alluxio 2.1 distribution In Progress: Merging connector into Presto codebase Alluxio Presto Connector 14

15. Transformation Service 15 Transform data to be compute-optimized independent from storage-optimized format Coalesce Format Conversion parquetcsv

16. Demo!

17. 2 isolated AWS 10-node clusters Presto + Hive Metastore + S3 Data Presto + Alluxio + Hive Metastore + S3 Data TPCDS sample dataset on S3 ~10,000 CSV files Demo 17

18. Attached existing Hive database into Alluxio Catalog Alluxio Catalog served table metadata for Presto Transformed store_sales by coalescing and converting CSV to Parquet Demo Summary 18 Presto Without Alluxio 20s Alluxio Transformations 7s AlluxioTransformations With Caching 3s

19. Conclusion

20. User community feedback/collaboration is important! Future projects New UDB implementations (AWS Glue) More conversion formats (json) DDL/DML workloads (CREATETABLE, INSERT, etc.) New Client APIs for structured data (Arrow) Future Work 20

21. Try it out! Documentation Provide feedback Feature requests and issues in Github Alluxio/alluxio Developer Preview Available in Alluxio 2.1 21 ThankYou!

22. 22

Hands-on with Alluxio Structured Data Management

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Hands-on with Alluxio Structured Data Management

Similar a Hands-on with Alluxio Structured Data Management (20)

Más de Alluxio, Inc.

Más de Alluxio, Inc. (20)

Último

Último (20)

Hands-on with Alluxio Structured Data Management