Orchestrate a Data Symphony

•

1 like•1,208 views

Data Orchestration Summit www.alluxio.io/data-orchestration-summit-2019 November 7, 2019 Orchestrate a Data Symphony Speaker: Haoyuan Li, Alluxio For more Alluxio events: https://www.alluxio.io/events/

Software

DATA ORCHESTRATION SUMMIT
2019
Orchestrate a Data Symphony
Haoyuan (H.Y.) Li | Founder & CTO | haoyuan@alluxio.com | @haoyuan

The most valuable companies in the Data Era all depend on
Data Infrastructure

Data Infrastructure
drives these innovations

From Data to Value: Simple & Easy?
A Data Driven Application
ValueData

Building the right data infrastructure is really hard!

Explosionof
Compute
Frameworks
Storage
Innova9on
Life Cycle
Fast Moving Landscape
Journey to
Hybrid /
Multi-Cloud

Endless requirements
More opportunities come with more challenges
How to
modernize my
data infra to
Cloud?
Why can’t we
also support
Presto for
querying?
Why can’t I
train my model
on a public
cloud?
Job is taking
forever, can’t
you add more
resource?
How do I access
remote HDFS
data in Google
Dataproc?
…

HDFS
HIVE
HDFS
Spark
NFS
TENSOR
FLOW
DATA IN DISPARATE STORAGE SYSTEMS
OBJECT
STORE
PRESTO
COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS
WAN
HDFS
WAN
S3
Spark
AZURE
PRESTO
Data silos across data centers, regions, cloudsComplex. Error prone. Time consuming.

Vision: Orchestrate Data for Applications

Data Orchestration Requirements
Structured
Data Catalog
Intelligent
Caching
Data
Transformation
Data
Management
Global
Namespace

Structured
Data Catalog
Intelligent
Caching
Data
Transformation
Data
Management
Global
Namespace
Abstract data silos to make data elastic for data driven applications
Data Orchestra7on Characteris7cs

Structured
Data Catalog
Global
Namespace
Data
Transformation
Data
Management
Intelligent
Caching
Enable Data Locality for fast performance regardless where the data is stored
Data Orchestra9on Characteris9cs

Structured
Data Catalog
Intelligent
Caching
Data
Transformation
Data
Management
Seamless data movement across data lakes without app changes
Global
Namespace
Data Orchestra8on Characteris8cs

Structured
Data Catalog
Intelligent
Caching
Data
Transformation
Provide a structured data service to greatly optimize SQL engines
Global
Namespace
Data
Management
Data Orchestra;on Characteris;cs

Intelligent
Caching
Data
Transformation
Transform data to compute-optimized representations for easy consumption
Global
Namespace
Data
Management
Structured
Data Catalog
Data Orchestra7on Characteris7cs

A Data Orchestration Approach
HDFS
HIVE Spark
NFS
TENSOR
FLOW
DATA IN DISPARATE STORAGE SYSTEMS
PREST
O
COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS
S3
SPARK
DATA
ORCHESTRATION
DATA
ORCHESTRATION
DATA
ORCHESTRATION
DATA
ORCHESTRATION
DATA
ORCHESTRATION
ANY
DATA
APP
DATA
ORCHESTRATION

Alluxio – An Open Source Implementation of Data Orchestration
Intelligent
Caching
Data
Management
Global
Namespace

Structured
Data Catalog
Intelligent
Caching
Data
Transformation
Data
Management
Global
Namespace
Announcing Alluxio Structured Data Service

Data Orchestration Ecosystem
Many open source projects represented here today

Thank You & Welcome to the
Data Orchestra5on Summit

What's hot

Enabling big data & AI workloads on the object store at DBS

Alluxio, Inc.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

Introducing the Hub for Data Orchestration

Alluxio, Inc.

Alluxio Tech Talk January 21, 2020 Speakers: Matt Fuller, Starburst Dipti Borkar, Alluxio With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data. Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about: - The architecture of Presto, an open source distributed SQL engine - How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics - Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Alluxio, Inc.

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores

Alluxio, Inc.

Alluxio Community Office Hour October 20, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Data Orchestration for the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar September 22, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar August 25, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Adit Madan Bin Fan With data lakes expanding from on-prem to the cloud as well as increasing use of new object data stores, data platform teams are challenged with providing consistent, high-throughput access to distributed data sources for analytics and AI/ML applications. In today’s hybrid cloud and multi-cloud era, data-intensive applications such as Presto, Spark, Hive, and Tensorflow are suffering more sluggish response times and increased complexity with the growing separation of data and compute. Join Alluxio’s distributed systems experts as they explore today’s data access challenges and open source data orchestration solutions for modernizing your data platform. In this tech talk, you'll learn: - How data access and throughput challenges are hindering large-scale analytics and AI/ML applications - How a data orchestration layer can simplify distributed data access and improve performance - Real-world production use cases and example journeys for architecting a modern data platform

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Global Online Meetup August 4, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Koen Michiels, Datasapiens Juraj Pohanka, Datasapiens Bin Fan, Alluxio Datasapiens is an international data-analytics startup based in Prague. We help our clients to uncover the value of their data and open up new revenue streams for them. We provide an end-to-end service that manages the data pipeline and automates the process of generating data insights. In this talk, we will describe how we have solved an issue with large S3 API costs incurred by Presto under several usage concurrency levels by implementing Alluxio as a data orchestration layer between S3 and Presto. Also, we will show the results of an experiment with estimating the per-query S3 API costs using the TPC-DS dataset. This talk will focus on: - The Hadoop ecosystem at Datasapiens - Drastic increase of S3 API costs during performance tests with Presto - S3 API costs tests with TPC-DS - Implications to the cloud data lake architecture

Reducing large S3 API costs using Alluxio at Datasapiens

Alluxio, Inc.

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Alluxio, Inc.

Alluxio Webinar April 6, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio - Virtual Unified File System

Alluxio, Inc.

This talk was presented by Alluxio's top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup. This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.

Alluxio - Scalable Filesystem Metadata Services

Alluxio, Inc.

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

Alluxio Global Online Meetup August 25, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Abner Ferreira, Simbiose Ventures Caio Pavanelli, Simbiose Ventures Bin Fan, Alluxio Over the last few years, organizations have worked towards the separation of storage and compute for a number of benefits in the areas of cost, data duplication and data latency. Cloud resolves most of these issues but comes to the expense of needing a way to query data on remote storages. Alluxio and Presto are a powerful combination to address the compute problem, which is part of the strategy used by Simbiose Ventures to create a product called StorageQuery - A platform to query files in cloud storages with SQL. This talk will focus on: - How Alluxio fits StorageQuery's tech stack; - Advantages of using Alluxio as a cache layer and its unified filesystem; - Development of new under file system for Backblaze B2 and fine-grained code documentation; - ShannonDB remote storage mode.

StorageQuery: federated querying on object stores, powered by Alluxio and Presto

Alluxio, Inc.

Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid

Alluxio, Inc.

Alluxio Global Online Meetup Apr 23, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Jiao (Jennie) Wang, Intel Tsai Louie, Intel Bin Fan, Alluxio Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked. Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications. This talk, we will go over: - What is Analytics Zoo and how it works - How to run Analytics Zoo with Alluxio in deep learning applications - Initial performance benchmark results using the Analytics Zoo + Alluxio stack

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

Alluxio, Inc.

How to teach your data scientist to leverage an analytics cluster with Presto...

Alluxio, Inc.

Apache Iceberg - A Table Format for Hige Analytic Datasets

Alluxio, Inc.

What's hot (20)

Enabling big data & AI workloads on the object store at DBS

Alluxio Use Cases and Future Directions

Introducing the Hub for Data Orchestration

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores

Data Orchestration for the Hybrid Cloud Era

Accelerate Analytics and ML in the Hybrid Cloud Era

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Reducing large S3 API costs using Alluxio at Datasapiens

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio - Virtual Unified File System

Alluxio - Scalable Filesystem Metadata Services

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

StorageQuery: federated querying on object stores, powered by Alluxio and Presto

Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

How to teach your data scientist to leverage an analytics cluster with Presto...

Apache Iceberg - A Table Format for Hige Analytic Datasets

Similar to Orchestrate a Data Symphony

Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More

Alluxio, Inc.

Alluxio Tech Talk Feb 12, 2019 Speaker: Dipti Borkar, Alluxio The rise of compute intensive workloads and the adoption of the cloud has driven organizations to adopt a decoupled architecture for modern workloads – one in which compute scales independently from storage. While this enables scaling elasticity, it introduces new problems – how do you co-locate data with compute, how do you unify data across multiple remote clouds, how do you keep storage and I/O service costs down and many more. Enter Alluxio, a virtual unified file system, which sits between compute and storage that allows you to realize the benefits of a hybrid cloud architecture with the same performance and lower costs. In this webinar, we will discuss: - Why leading enterprises are adopting hybrid cloud architectures with compute and storage disaggregated - The new challenges that this new paradigm introduces - An introduction to Alluxio and the unified data solution it provides for hybrid environments

Achieving Separation of Compute and Storage in a Cloud World

Alluxio, Inc.

Presentation of my talk given in Phoenix Data Conference 2019. In this we will look at challenges with current Apache Hadoop ecosystem Apache Hadoop is still relevant but way of doing Hadoop and enterprise data architecture has to be re-looked as we enter Cognitive and Cloud Native Era We need  Architecture that is enabled by common run time layer across on premise and cloud  Architecture that can abstract away dependency and version conflicts with tons of open source machine learning out there. Yarn did not scale up in that aspect until one want to deal with multiple conda environment  Architecture that can enable real Hybrid Cloud and Multi Cloud portability  And many more challenges that one has to overcome to keep architecture simple, infrastructure agile and better utilized

Future of Data Platform in Cloud Native world

Srivatsan Srinivasan

Accenture-Cloud-Data-Migration-POV-Final.pdf

Rajvir Kaushal

Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

Watch full webinar here: https://bit.ly/2vN59VK What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics. Attend this session to learn: - What data virtualization really is. - How it differs from other enterprise data integration technologies. - Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations.

Data Virtualization: An Introduction

Denodo

Watch full webinar here: https://bit.ly/2JuD9NC Organizations are adopting cloud at a fast pace and migration of critical enterprise information resources could be a challenge when dealing with a complex and big data landscape. Building the right data services architecture can help alleviate the pain points, whereby data virtualization comes to the rescue by enabling the companies to gain maximum benefits from cloud initiatives in form of agility, cost savings, and more. In this webinar, you'll learn: - How Denodo Platform's multi-location architecture can simplify and accelerate cloud migration. - Best practices of deploying the Denodo Platform in the cloud. - Leverage Denodo's virtual data services layer to address and augment cloud solutions such as data warehouse modernization, data science, and data lakes in the cloud. - Watch a demo showcasing data virtualization and analytics in the cloud.

Accelerate Migration to the Cloud using Data Virtualization (APAC)

Denodo

SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101

Mithun T. Dhar

Research data management 1.5

John Martin

Alluxio Tech Talk Oct 17, 2019 Speaker: Alex Ma, Alluxio Want to leverage your existing investments in Hadoop with your data on-premise and still benefit from the elasticity of the cloud? Like other Hadoop users, you most likely experience very large and busy Hadoop clusters, particularly when it comes to compute capacity. Bursting HDFS data to the cloud can bring challenges – network latency impacts performance, copying data via DistCP means maintaining duplicate data, and you may have to make application changes to accomodate the use of S3. “Zero-copy” hybrid bursting with Alluxio keeps your data on-prem and syncs data to compute in the cloud so you can expand compute capacity, particularly for ephemeral Spark jobs.

From limited Hadoop compute capacity to increased data scientist efficiency

Alluxio, Inc.

Data Orchestration Platform for the Cloud

Alluxio, Inc.

In this presentation, we: 1. Look at the challenges and opportunities of the data era 2. Look at key challenges of the legacy data warehouses such as data diversity, complexity, cost, scalabilily, performance, management, ... 3. Look at how modern data warehouses in the cloud not only overcome most of these challenges but also how some of them bring additional technical innovations and capabilities such as pay as you go cloud-based services, decoupling of storage and compute, scaling up or down, effortless management, native support of semi-structured data ... 4. Show how capabilities brought by modern data warehouses in the cloud, help businesses, either new or existing ones, during the phases of their lifecycle such as launch, growth, maturity and renewal/decline. 5. Share a Near-Real-Time Data Warehousing use case built on Snowflake and give a live demo to showcase ease of use, fast provisioning, continuous data ingestion, support of JSON data ...

Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi

Slim Baltagi

Above the cloud joarder kamal

Joarder Kamal

In this webinar, we talk with experts from Integration Developer News about the SnapLogic Elastic Integration Platform and adoption trends for iPaaS in the enterprise. During the discussion, we address cloud application adoption challenges and 5 signs you need better cloud integration, including struggles with the "Integrator's Dilemma" and segregated integration. To learn more, visit: www.snaplogic.com/connect-faster

Cloud Con 2015 - Integration & Web APIs

SnapLogic

Enabling Apache Spark for Hybrid Cloud

Alluxio, Inc.

How Consistent Data Services Deliver Simplicity, Compatibility, And Lower Cost

Dana Gardner

Greg Dixon - 2011 ScanSource POS & Barcoding Partner Conference

ScanSource, Inc.

Achieving compute and storage independence for data-driven workloads

Alluxio, Inc.

In today’s world of exponentially growing big data, enterprises are becoming increasingly more aware of the business utility and necessity of harnessing, storing and analyzing this information. Apache Hadoop has rapidly evolved to become a leading platform for managing and processing big data, with the vital management, monitoring, metadata and integration services required by organizations to glean maximum business value and intelligence from their burgeoning amounts of information on customers, web trends, products and competitive markets. In this session, Hortonworks' Himanshu Bari will discuss the opportunities for deriving business value from big data by looking at how organizations utilize Hadoop to store, transform and refine large volumes of this multi-structured information. Connolly will also discuss the evolution of Apache Hadoop and where it is headed, the component requirements of a Hadoop-powered platform, as well as solution architectures that allow for Hadoop integration with existing data discovery and data warehouse platforms. In addition, he will look at real-world use cases where Hadoop has helped to produce more business value, augment productivity or identify new and potentially lucrative opportunities.

Apache Hadoop and its role in Big Data architecture - Himanshu Bari

jaxconf

2020 Cloud Data Lake Platforms Buyers Guide - White paper | Qubole

Vasu S

Similar to Orchestrate a Data Symphony (20)

Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More

Achieving Separation of Compute and Storage in a Cloud World

Future of Data Platform in Cloud Native world

Accenture-Cloud-Data-Migration-POV-Final.pdf

Data Orchestration for AI, Big Data, and Cloud

Data Virtualization: An Introduction

Accelerate Migration to the Cloud using Data Virtualization (APAC)

SeattleUniv-IntroductionToCloudComputing-WinsowsAzure101

Research data management 1.5

From limited Hadoop compute capacity to increased data scientist efficiency

Data Orchestration Platform for the Cloud

Modern-Data-Warehouses-In-The-Cloud-Use-Cases-Slim-Baltagi

Above the cloud joarder kamal

Cloud Con 2015 - Integration & Web APIs

Enabling Apache Spark for Hybrid Cloud

How Consistent Data Services Deliver Simplicity, Compatibility, And Lower Cost

Greg Dixon - 2011 ScanSource POS & Barcoding Partner Conference

Achieving compute and storage independence for data-driven workloads

Apache Hadoop and its role in Big Data architecture - Himanshu Bari

2020 Cloud Data Lake Platforms Buyers Guide - White paper | Qubole

More from Alluxio, Inc.

As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

Alluxio, Inc.

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sally (Mihyoung) Lee (Senior Staff Engineer, TLM, @Uber) Machine learning models power Uber’s everyday business. However, developing and deploying a model is not a one-time event but a continuous process that requires careful planning, execution, and monitoring. This session will highlight Uber’s practice on the machine learning lifecycle to ensure high model quality.

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio, Inc.

Alluxio Monthly Webinar Oct. 3, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Greg Palmer (Lead Solutions Engineer) Model training requires extensive computational and GPU resources. When training models on AWS, loading data from S3 often becomes a major bottleneck, wasting valuable GPU cycles. Optimizing data loading can greatly reduce GPU idle time and increase GPU utilization. In this webinar, Greg Palmer will discuss best practices for efficient data loading during model training on AWS. He will demonstrate how to use Alluxio on EKS as a distributed cache to accelerate PyTorch training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs. What you will learn: - The challenges of feeding data-hungry GPUs in the cloud - How to accelerate model training by optimizing data loading on AWS - The reference architecture for running PyTorch jobs with Alluxio cache on EKS while reading data from S3, with benchmark results of training ResNet50 and BERT - How to use TensorBoard to identify bottlenecks in GPU utilization

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio, Inc.

More from Alluxio, Inc. (20)

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Recently uploaded

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in Tembisa ● Abortion Pills For Sale in Tembisa ● Tembisa 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Investing in AI transformation today The modern business advantage: Uncovering deep insights with AI Organizations around the world have come to recognize AI as the transformative technology that enables them to gain real business advantage. AI’s ability to organize vast quantities of data allows those who implement it to uncover deep business insights, augment human expertise, drive operational efficiency, transform their products, and better serve their customers

Microsoft AI Transformation Partner Playbook.pdf

Willy Marroquin (WillyDevNET)

%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview

masabamasaba

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

8257 interfacing 2 in microprocessor for btech students

HimanshiGarg82

WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...

WSO2

In the past six months, the AI landscape has undergone a massive transformation, ushering in a new era of productivity with the latest in Large Language Models (LLMs) and AI technology. This deep dive unlocks how to: Create CustomGPT Models: No coding needed to tailor AI for your unique projects. Integrate your own data, including PDFs and Excel sheets, making information handling a breeze. Plus, discover how to call your own actions/integrations for even more personalized utility. Navigate Advanced Prompting: Overcome AI's memory limits and utilize Retrieval-Augmented Generation for accessing your personalized data, streamlining how you interact with AI. Stay Ahead with AI Trends: Peek into the evolving world of LLMs, featuring newcomers like Google Gemini, Anthropic Claude, Open Sora, and Twitter Grok, and understand what their advancements mean for your productivity. Witness Real-Life Transformations: Through examples and prompt demonstrations, see firsthand how these AI strategies revolutionize routine tasks, from data analysis to content creation. Learn to leverage image output and input for advanced practical use cases, adding a new dimension to your productivity toolkit. No previous coding or AI experience is needed for this talk. Stay ahead in the fast-evolving world of work. Embrace the AI revolution and transform your workflow with advanced LLM techniques. Join us to ensure you're not left behind in the productivity race.

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques

VictorSzoltysek

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

WSO2

The fast-paced world of Agile development just got a turbo boost! Dive into this engaging talk on "Innovation at Speed: ChatGPT & the AI Transformation in Agile Development." Over the last few months, AI, including notable tools like Google Bard and Microsoft Co-Pilot, has been leveraged to supercharge Agile workflows. In this session, attendees will get an overview of the current AI landscape and its impact on development. They will learn why AI is the game-changer needed to embrace in the Agile lifecycle, offering that elusive X factor to multiply productivity. Real-world examples will be presented, and the challenges of implementing AI tools like ChatGPT within corporate environments will be discussed.

Harnessing ChatGPT - Elevating Productivity in Today's Agile Environment

VictorSzoltysek

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

masabamasaba

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Bert Jan Schrijver

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

tonesoftg

lanshi9

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

Conference: Engage2024 in Antwerp Type: Workshop Speakers: Florian Vogler, Henning Kunz, Christoph Adler Title: Navigating the Future with The Hitchhiker's Guide to Notes and Domino 14 Abstract: Embark on an exhilarating journey with industry trailblazers Florian Vogler, Henning Kunz, and Christoph Adler in this not-to-be-missed workshop at the forefront of the tech universe. Get ready for a thrilling kick-off as we navigate the current state of the HCL universe, setting the stage for an exploration of the groundbreaking Notes and Domino 14. Discover the latest enhancements and revolutionary features that will redefine your experience. In this interactive session, unlock a treasure trove of tips and tricks to elevate your utilization of version 14, both with and without the game-changing panagenda MarvelClient. Brace yourself for also diving into Nomad, Nomad Web, and VoltMX, expanding your horizons in the expansive HCL landscape. Be a part of this exclusive opportunity to stay ahead in the ever-evolving world of HCL technologies. Your journey to mastering Notes and Domino 14 begins here. And remember, in the spirit of intergalactic exploration, don't forget to bring your towel!

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

panagenda

At the recent Microsoft Ignite 2023 conference, Microsoft unveiled a groundbreaking strategy that will redefine enterprise work management. The plan involves integrating Microsoft’s key planning tools, Microsoft To Do, Microsoft Planner, and Microsoft Project for the web into a unified experience called “Microsoft Planner.” What does this new strategy from Microsoft mean for current users? Join us and learn how best to take advantage of this announcement while gaining a clear path on how to elevate the current state of Microsoft Planner from a basic task manager to a comprehensive tool for Enterprise Work Management using OnePlan. Learn how OnePlan’s integration with Microsoft Planner allows for strategic alignment with business goals through advanced features like strategic planning, portfolio management, resource management, financial management, and more!

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

OnePlan Solutions

Model Call Girl Services in Delhi reach out to us at 🔝 9953056974 🔝✔️✔️ Our agency presents a selection of young, charming call girls available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives, College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Delhi ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 9953056974 🔝. for details Operating 24/7, we serve various locations in Delhi, including Green Park, Lajpat Nagar, Saket, and Hauz Khas near metro stations. For premium call girl services in Delhi 🔝 9953056974 🔝. Thank you for considering us!

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

9953056974 Low Rate Call Girls In Saket, Delhi NCR

ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.

Architecture decision records - How not to get lost in the past

Papp Krisztián

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SelfMade bd

Recently uploaded (20)

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Microsoft AI Transformation Partner Playbook.pdf

%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

8257 interfacing 2 in microprocessor for btech students

WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

Harnessing ChatGPT - Elevating Productivity in Today's Agile Environment

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

tonesoftg

Announcing Codolex 2.0 from GDK Software

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

Architecture decision records - How not to get lost in the past

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

Orchestrate a Data Symphony

1. DATA ORCHESTRATION SUMMIT 2019 Orchestrate a Data Symphony Haoyuan (H.Y.) Li | Founder & CTO | haoyuan@alluxio.com | @haoyuan

2. The most valuable companies in the Data Era all depend on Data Infrastructure

3. Data Infrastructure drives these innovations

4. From Data to Value: Simple & Easy? A Data Driven Application ValueData

5. The Reality

6. Building the right data infrastructure is really hard!

7. Explosionof Compute Frameworks Storage Innova9on Life Cycle Fast Moving Landscape Journey to Hybrid / Multi-Cloud

8. Endless requirements More opportunities come with more challenges How to modernize my data infra to Cloud? Why can’t we also support Presto for querying? Why can’t I train my model on a public cloud? Job is taking forever, can’t you add more resource? How do I access remote HDFS data in Google Dataproc? …

9. HDFS HIVE HDFS Spark NFS TENSOR FLOW DATA IN DISPARATE STORAGE SYSTEMS OBJECT STORE PRESTO COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS WAN HDFS WAN S3 Spark AZURE PRESTO Data silos across data centers, regions, cloudsComplex. Error prone. Time consuming.

10. Vision: Orchestrate Data for Applications

11. A Data Orchestra,on Approach

12. Data Orchestration Requirements Structured Data Catalog Intelligent Caching Data Transformation Data Management Global Namespace

13. Structured Data Catalog Intelligent Caching Data Transformation Data Management Global Namespace Abstract data silos to make data elastic for data driven applications Data Orchestra7on Characteris7cs

14. Structured Data Catalog Global Namespace Data Transformation Data Management Intelligent Caching Enable Data Locality for fast performance regardless where the data is stored Data Orchestra9on Characteris9cs

15. Structured Data Catalog Intelligent Caching Data Transformation Data Management Seamless data movement across data lakes without app changes Global Namespace Data Orchestra8on Characteris8cs

16. Structured Data Catalog Intelligent Caching Data Transformation Provide a structured data service to greatly optimize SQL engines Global Namespace Data Management Data Orchestra;on Characteris;cs

17. Intelligent Caching Data Transformation Transform data to compute-optimized representations for easy consumption Global Namespace Data Management Structured Data Catalog Data Orchestra7on Characteris7cs

18. A Data Orchestration Approach HDFS HIVE Spark NFS TENSOR FLOW DATA IN DISPARATE STORAGE SYSTEMS PREST O COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS S3 SPARK DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION ANY DATA APP DATA ORCHESTRATION

19. Alluxio – An Open Source Implementation of Data Orchestration Intelligent Caching Data Management Global Namespace

20. Structured Data Catalog Intelligent Caching Data Transformation Data Management Global Namespace Announcing Alluxio Structured Data Service

21. Data Orchestration Ecosystem Many open source projects represented here today

22. Thank You & Welcome to the Data Orchestra5on Summit

Orchestrate a Data Symphony

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Orchestrate a Data Symphony

Similar to Orchestrate a Data Symphony (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Orchestrate a Data Symphony