Hybrid is the New Normal

•

0 recomendaciones•242 vistas

It’s becoming clear that enterprises need more than one cloud. Hybrid enables enterprises to optimize how their business works – public cloud for elasticity and scale, multi-cloud for redundancy and choice, and on-premises for performance and privacy. Cloudera delivers a hybrid cloud solution that works where enterprises work, with the agility, security and governance enterprise IT needs, and the self-service analytics business people and enterprise data professionals demand. In this session, we will talk about how Cloudera helps deliver hybrid solutions for enterprises and will run a hands-on Cloudera PaaS demo to exhibit: - Altus Environment Setup - Configure Altus SDX - Spin-up transient clusters with Altus - Execute workload on Altus Data Engineering clusters - Run interactive queries on object store with Altus Data Warehouse - Job Analytics with Workload Experience Manager (WXM) Speaker: Junaid Rao, Senior Cloud Sales Engineer, Cloudera

Tecnología

Hybrid is the New Normal
Junaid Rao
Senior Cloud SE – APAC

2 © Cloudera, Inc. All rights reserved.
WHY HYBRID ?

3 © Cloudera, Inc. All rights reserved.
Three Types of Workload Lifecycles
1hr
SPIN UP SPIN
DOWN
24/7
24/7
1hr
SPIN UP SPIN
DOWN
Persistent
Transient
Elastic

4 © Cloudera, Inc. All rights reserved.
HOW CLOUDERA HELPS with HYBRID ?

5 © Cloudera, Inc. All rights reserved.
• The modern platform for machine
learning and analytics
• with multiple deployment options
• and one shared data experience

6 © Cloudera, Inc. All rights reserved.
SHARED DATA EXPERIENCEDEPLOYMENT OPTIONSMODERN PLATFORM
Amazon
S3
LOCATION
STORAGE
MANAGEABILITY
Microsoft
ADLS
HDFS KUDU
Data Center
Self Managed Managed Service
DATA
ENGINEERING
DATA
WAREHOUSE
DATA
SCIENCE
OPERATIONAL
DATABASE
SECURITY
GOVERNANCE
LIFECYCLE MANAGEMENT
DATA CATALOG
CLOUDERA VALUE PROPOSITION

7 © Cloudera, Inc. All rights reserved.
Big Data Infrastructure Evolution
Infrastructure traditionally
Each cluster is self-contained with compute,
data context, and data
Data context = HMS, Sentry, Navigator
Compute
Context
Data
Compute
Context
Data
Compute
Context
Data
• Compute, Context and Data together.
Designed for best performance.
• Highly Available, mission critical
• Multi-tenant, Secure and Governed
• Fixed Size, provisioned for peak capacity
• Low Utilization rates if only transient
workloads
• Not easy to Scale

8 © Cloudera, Inc. All rights reserved.
Big Data Infrastructure Evolution
Decoupled Infrastructure
Data is separate from compute (e.g., in
ADLS/S3), but context needs to be managed
redundantly in each cluster
Compute
Context
Compute
Context
Compute
Context
Data
• Decoupled data and compute
• Use Object Store
• Scale easily
• Workload Specific infrastructure
• Support Transient and Persistent workloads
• Compute and Context are still together
• Maintain schemas in application. You lose
Context when transient workload
completes
• No way to troubleshoot. You lose
Workload information (logs and statistics)
when the cluster goes away.

9 © Cloudera, Inc. All rights reserved.
Big Data Infrastructure Evolution
Modern Infrastructure with Cloudera
Compute clusters are launched as needed
Data and data context are stored externally and
are long-running
Workload Analytics stores your logs and job
statistics
ComputeCompute
Data
Cloudera SDX/WXM
Compute
• Decoupled data and compute
• Use Object Store
• Scale easily
• Workload Specific infrastructure
• Support Transient and Persistent workloads
• Persistent Context and Workload Analytics

10 © Cloudera, Inc. All rights reserved.
CLOUDERA
ALTUS
Flexible cloud
deployment options
including workload-
optimized Managed
Services with Cloudera
Shared Data
Experience (SDX)
DATA ENGINEERING DATA WAREHOUSE
MULTI
FUNCTION
CLOUD
STORAGE
DATA CATALOG
GOVERNANCESECURITY CONTROL
PLANE
LIFECYCLE
MANAGEMENT
Microsoft
ADLS
AWS
S3
DIRECTOR

11 © Cloudera, Inc. All rights reserved.
Cloudera Altus Architecture
Customer Cloud
Compute
Storage
CLI
Web
SDK
ALTUS DATA
WAREHOUSE
ALTUS DATA
ENGINEERING
ALTUS
CONTROL
PLANE

12 © Cloudera, Inc. All rights reserved.
On-Premises Cloud Bursting
Data Engineering Workflow:
Batch ML, Simulations, etc.
1+ Jobs
(e.g.Spark)
Transient Cluster(s):
Atlus Managed
1+ Jobs
(e.g. Spark)
On-Premises Cluster:
Bare Metal/Private
Cloud
HDFS
Data
Science
Data
Eng
Data
Warehouse
Continuous or
On-Demand
Persistent SDX
Altus Control PlaneSDX: Schema (HMS), Security (Sentry), Lineage/Metadata (Navigator)
Object Store (ADLS, S3)
Data
Warehouse
(Impala)
Elastic Cluster(s):
Atlus Managed
BI Tools
(e.g. Tableau)
SQL Editor
(e.g. Hue)

13 © Cloudera, Inc. All rights reserved.
Altus Demo
Data Engineering Workflow:
Batch ML, Simulations, etc.
1+ Jobs
(e.g.Spark)
Transient Cluster(s):
Atlus Managed
1+ Jobs
(e.g. Spark)
Persistent SDX
Altus Control PlaneSDX: Schema (HMS), Security (Sentry), Lineage/Metadata (Navigator)
Object Store (ADLS, S3)
Data
Warehouse
(Impala)
Elastic Cluster(s):
Atlus Managed
BI Tools
(e.g. Tableau)
SQL Editor
(e.g. Hue)
On-Premises Cluster:
Bare Metal/Private
Cloud
HDFS
Data
Science
Data
Eng
Data
Warehouse
Continuous or
On-Demand

Más contenido relacionado

La actualidad más candente

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

Cloudera, Inc.

IBM Power8 announce

Anna Landolfi

You like to use R, and you need to use big data. dplyr, one of the most popular packages for R, makes it easy to query large data sets in scalable processing engines like Apache Spark and Apache Impala. But there can be pitfalls: dplyr works differently with different data sources—and those differences can bite you if you don’t know what you’re doing. Ian Cook is a data scientist, an R contributor, and a curriculum developer at Cloudera University. In this webinar, Ian will show you exactly what you need to know about sparklyr (from RStudio) and the package implyr (from Cloudera). He will show you how to write dplyr code that works across these different interfaces. And, he will solve mysteries: Do I need to know SQL to use dplyr? When is a “tbl” not a “tibble”? Why is 1 not always equal to 1? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems? 3 things to learn: Do I need to know SQL to use dplyr? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems?

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Cloudera, Inc.

Oracle began with the belief that the foundation of IT was managing information. The Oracle Cloud Platform for Big Data is a natural extension of our belief in the power of data. Oracle’s Integrated Cloud is one cloud for the entire business, meeting everyone’s needs. It’s about Connecting people to information through tools which help you combine and aggregate data from any source. This session will explore how organizations can transition to the cloud by delivering fully managed and elastic Hadoop and Real-time Streaming cloud services to built robust offerings that provide measurable value to the business. We will explore key data management trends and dive deeper into pain points we are hearing about from our customer base.

Insights into Real-world Data Management Challenges

DataWorks Summit

Logical Data Warehouse: How to Build a Virtualized Data Services Layer

DataWorks Summit

Customers are preparing themselves to analyze and manage an increasing quantity of structured and unstructured data. Business leaders introduce new analytical workloads faster than what IT departments can handle. Legacy IT infrastructure needs to evolve to deliver operational improvements and cost containment, while increasing flexibility to meet future requirements. By providing HDP on IBM Power Systems, Hortonworks and IBM are giving customers have more choice in selecting the appropriate architectural platform that is right for them. In this webinar, we’ll discuss some of the challenges with deploying big data platforms, and how choosing solutions built with HDP on IBM Power Systems can offer tangible benefits and flexibility to accommodate changing needs.

Delivering a Flexible IT Infrastructure for Analytics on IBM Power Systems

Hortonworks

Open Innovation with Power Systems

IBM Power Systems

This white paper describes how BlueData enables virtualization of Hadoop and Spark workloads running on Intel architecture. Even as virtualization has spread throughout the data center, Apache Hadoop continues to be deployed almost exclusively on bare-metal physical servers. Processing overhead and I/O latency typically associated with virtualization have prevented big data architects from virtualizing Hadoop implementations. As a result, most Hadoop initiatives have been limited in terms of agility, with infrastructure changes such as provisioning a new server for Hadoop often taking weeks or even months. This infrastructure complexity continues to slow down adoption in enterprise deployments. Apache Spark is a relatively new big data technology, but interest is growing rapidly; many of these same deployment challenges apply to on-premises Spark implementations. The BlueData EPIC software platform addresses these limitations, enabling data center operators to accelerate Hadoop and Spark implementations on Intel architecture-based servers. For more information, visit intel.com/bigdata and bluedata.com

Hadoop Virtualization - Intel White Paper

BlueData, Inc.

Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop

DataWorks Summit/Hadoop Summit

The data center has gone through several inflection points in the past decades: adoption of Linux, migration from physical infrastructure to virtualization and Cloud, and now large-scale data analytics with Big Data and Hadoop. Please join us to learn about how Cloudera and Intel are jointly innovating through open source software to enable Hadoop to run best on IA (Intel Architecture) and to foster the evolution of a vibrant Big Data ecosystem.

Intel and Cloudera: Accelerating Enterprise Big Data Success

Cloudera, Inc.

Hadoop Everywhere & Cloudbreak

Sean Roberts

Back in 2014, our team set out to change the way the world exchanges and collaborates with data. Our vision was to build a single tenant environment for multiple organisations to securely share and consume data. And we did just that, leveraging multiple Hadoop technologies to help our infrastructure scale quickly and securely. Today Data Republic’s technology delivers a trusted platform for hundreds of enterprise level companies to securely exchange, commercialise and collaborate with large datasets. Join Head of Engineering, Juan Delard de Rigoulières and Senior Solutions Architect, Amin Abbaspour as they share key lessons from their team’s journey with Hadoop: * How a startup leveraged a clever combination of Hadoop technologies to build a secure data exchange platform * How Hadoop technologies helped us deliver key solutions around governance, security and controls of data and metadata * An evaluation on the maturity and usefulness of some Hadoop technologies in our environment: Hive, HDFS, Spark, Ranger, Atlas, Knox, Kylin: we've use them all extensively. * Our bold approach to expose APIs directly to end users; as well as the challenges, learning and code we created in the process * Learnings from the front-line: How our team coped with code changes, performance tuning, issues and solutions while building our data exchange Whether you’re an enterprise level business or a start-up looking to scale - this case study discussion offers behind-the-scenes lessons and key tips when using Hadoop technologies to manage data governance and collaboration in the cloud. Speakers: Juan Delard De Rigoulieres, Head of Engineering, Data Republic Pty Ltd Amin Abbaspour, Senior Solutions Architect, Data Republic

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

DataWorks Summit

Data Science and Machine Learning for the Enterprise

Cloudera, Inc.

The Department of Home Affairs’ initial big data use-cases were standard Hadoop fare: legacy system archival, log data retention and a scalable BLOB store. Along the way we discovered that a platform as open as HDP offered opportunities beyond those exposed by everyone’s favourite asparagus diagram. Here you’ll see how we’ve augmented our Hadoop stack with Solr for text searching, offloading significant work from our Teradata Warehouse in the process; and how we are currently implementing JanusGraph, improving the quality of models developed by our Data Scientists and providing richer data to our Intelligence Analysts – all backed by HDP! Speakers: Steven O'Neill, Director EDW platforms, Data Warehouse, Australian Government, Department of Home Affairs Dwane Hall, Hadoop Developer, Australian Government, Department of Home Affairs

Hadoop: The Unintended Benefits

DataWorks Summit

Presentation originated from live webinar... view the webinar here (https://www.brighttalk.com/webcast/10565/114179) During this webinar you will hear from Cloudera’s Director of Data Science, Sean Owen, as he discusses how Cloudera’s enterprise data hub allows data scientist to leverage libraries full of machine learning algorithms to analyze high dimensional, high volume data. Sean will also speak to common machine learning challenges and how Cloudera’s enterprise data hub can help eliminate these issues. Topics we will cover during the presentation will include: What is machine learning? Why should I use machine learning algorithms? What are the common challenges of machine learning? How does Cloudera’s enterprise data hub support machine learning?

Machine Learning Loves Hadoop

Cloudera, Inc.

Five Tips for Running Cloudera on AWS

Cloudera, Inc.

The Actian Analytics Platform™ Accelerate your transition to Big Data 2.0. Transform big data into business value with the Actian Analytics Platform. - Create transformational value along with many innovative companies - Deliver next generation analytics with our comprehensive platform - Accelerate your on-ramp for big data with disruptive price performance The Actian Analytics Platform-Hadoop SQL Edition accelerates Hadoop and makes it enterprise-grade by providing high-performance data enrichment, visual design and SQL analytics on Hadoop without the need for MapReduce skills. The platform edition uniquely enhances and extends Hadoop natively for both discovery and time-sensitive analytics using next generation capabilities: - Parallelized Pipelining – run high performance analytics natively on Hadoop 10X+ faster - Data Science Workbench – visually prep and analyze Hadoop data without coding - Ultrafast SQL on Hadoop – deliver industrialized access to Hadoop data to business users With Actian, you can run Hadoop at the speed and scale needed to help your organization transform big data into business value.

Actian Analytics Platform - Hadoop SQL Edition

Alessandro Salvatico

Actian Vector on Hadoop: First Industrial-strength DBMS to Truly Leverage Hadoop

DataWorks Summit

Oracle big data appliance and solutions

solarisyougood

LLAP: Sub-Second Analytical Queries in Hive

DataWorks Summit/Hadoop Summit

La actualidad más candente (20)

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the Cloud

IBM Power8 announce

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Insights into Real-world Data Management Challenges

Logical Data Warehouse: How to Build a Virtualized Data Services Layer

Delivering a Flexible IT Infrastructure for Analytics on IBM Power Systems

Open Innovation with Power Systems

Hadoop Virtualization - Intel White Paper

Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop

Intel and Cloudera: Accelerating Enterprise Big Data Success

Hadoop Everywhere & Cloudbreak

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...

Data Science and Machine Learning for the Enterprise

Hadoop: The Unintended Benefits

Machine Learning Loves Hadoop

Five Tips for Running Cloudera on AWS

Actian Analytics Platform - Hadoop SQL Edition

Actian Vector on Hadoop: First Industrial-strength DBMS to Truly Leverage Hadoop

Oracle big data appliance and solutions

LLAP: Sub-Second Analytical Queries in Hive

Similar a Hybrid is the New Normal

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera, Inc.

Cloudera SDX

Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19

Cloudera, Inc.

Multidisziplinäre Analyseanwendungen auf einer gemeinsamen Datenplattform ers...

Cloudera, Inc.

Cloudera GoDataFest Deploying Cloudera in the Cloud

GoDataDriven

Neueste Studien zeigen, dass Data Scientisten und Analysten bis zu 80% ihrer Zeit dafür nutzen, Daten zu reinigen und vorzubereiten. Eine ohnehin schon zeitaufwändige Aufgabe kann in der Cloud noch weiter erschwert werden, da das Cluster Management und Operations die Komplexität noch erhöhen. Nutzer wünschen sich daher, diese komplexen Workflows zu vereinheitlichen und zu vereinfachen. Um Big Data und Machine Learning Initiativen voranzutreiben, benötigen Unternehmen eine skalierbare und überall verfügbare Plattform. Diese muss Self-Service ermöglichen und Datensilos eliminieren.

Cloudera Altus: Big Data in der Cloud einfach gemacht

Cloudera, Inc.

Machine learning and analytics applications are exploding in the enterprise; driving use cases for preventative maintenance, delivering new desirable product offers to customers at the right time, and combating insider threats to your business. But each of these high-value use cases rely on a variety of data analysis capabilities working in concert to combine data from different sources into a single coherent picture. Cloudera SDX delivers a “shared data experience” that makes applications easier to develop, less expensive to deploy and more consistently secure. 3 things to learn: * Why multi-function applications are difficult to build and secure * How shared catalog, governance, management, and security applied consistently everywhere can deliver a “shared data experience” * How enterprise customers are building new, high-value applications with SDX

How to Build Multi-disciplinary Analytics Applications on a Shared Data Platform

Cloudera, Inc.

Cloud Data Warehousing with Cloudera Altus 7.24.18

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2

Cloudera, Inc.

Date: 14th November 2018 Location: Data-Driven Ldn Theatre Time: 12:30 - 13:00 Organisation: Cloudera About: The growth of public cloud is reinforcing the need to think more carefully about taking a consistent approach to data governance as technology teams build out a flexible and agile infrastructure to meet the demands of the business. Join this session to learn more about Cloudera's recommended approach for enterprise-grade security and governance and how to ensure a consistent framework across private, public and on-premises environments.

Big Data LDN 2018: CONSISTENT SECURITY, GOVERNANCE AND FLEXIBILITY FOR ALL WO...

Matt Stubbs

A deep dive into running data analytic workloads in the cloud

Cloudera, Inc.

Self-service Big Data Analytics on Microsoft Azure

Cloudera, Inc.

Cloudera - The Modern Platform for Analytics

Cloudera, Inc.

Big data journey to the cloud 5.30.18 asher bartch

Cloudera, Inc.

Comment développer une stratégie Big Data dans le cloud public avec l'offre P...

Cloudera, Inc.

Supercharge Splunk with Cloudera 

Cloudera, Inc.

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

Stefan Lipp

For self-service BI and exploratory analytic workloads, the cloud can provide a number of key benefits, but the move to the cloud isn’t all-or-nothing. Gartner predicts nearly 80 percent of businesses will adopt a hybrid strategy. Learn how a modern analytic database can power your business-critical workloads across multi-cloud and hybrid environments, while maintaining data portability. We'll also discuss how to best leverage the increased agility cloud provides, while maintaining peak performance.

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...

Cloudera, Inc.

Cloudera Altus makes it easier for data engineers, ETL developers, and anyone who regularly works with raw data to process that data in the cloud efficiently and cost effectively. In this webinar we introduce our new platform-as-a-service offering and explore challenges associated with data processing in the cloud today, how Altus abstracts cluster overhead to deliver easy, efficient data processing, and unique features and benefits of Cloudera Altus.

Cloudera Altus: Big Data in the Cloud Made Easy

Cloudera, Inc.

In this workshop, we will look outside the box and help expand the problem space to include issues you may not have thought were possible before Big Data. From Near Real Time (NRT) recommendation engines, loan applications to churn detection, Big Data is answering new questions and providing organisations with a competitive edge through revenue increase, cost savings and risk mitigation. We will take a special look at the role the Cloud can play in elevating your analytics environment. We will discuss real world examples of how Big Data answers these questions and does it at a lower cost outlay.

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Cloudera, Inc.

Similar a Hybrid is the New Normal (20)

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera SDX

Leveraging the cloud for analytics and machine learning 1.29.19

Multidisziplinäre Analyseanwendungen auf einer gemeinsamen Datenplattform ers...

Cloudera GoDataFest Deploying Cloudera in the Cloud

Cloudera Altus: Big Data in der Cloud einfach gemacht

How to Build Multi-disciplinary Analytics Applications on a Shared Data Platform

Cloud Data Warehousing with Cloudera Altus 7.24.18

Modern Data Warehouse Fundamentals Part 2

Big Data LDN 2018: CONSISTENT SECURITY, GOVERNANCE AND FLEXIBILITY FOR ALL WO...

A deep dive into running data analytic workloads in the cloud

Self-service Big Data Analytics on Microsoft Azure

Cloudera - The Modern Platform for Analytics

Big data journey to the cloud 5.30.18 asher bartch

Comment développer une stratégie Big Data dans le cloud public avec l'offre P...

Supercharge Splunk with Cloudera 

Cloudera Analytics and Machine Learning Platform - Optimized for Cloud

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...

Cloudera Altus: Big Data in the Cloud Made Easy

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Más de DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Data Science Crash Course

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

Managing the Dewey Decimal System

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Security Framework for Multitenant Architecture

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Computer Vision: Coming to a Store Near You

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

Más de DataWorks Summit (20)

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Último

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

Real Time Object Detection Using Open CV

Khem

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Developing An App To Navigate The Roads of Brazil

V3cube

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Hybrid is the New Normal

1. Hybrid is the New Normal Junaid Rao Senior Cloud SE – APAC

6. 6 © Cloudera, Inc. All rights reserved. SHARED DATA EXPERIENCEDEPLOYMENT OPTIONSMODERN PLATFORM Amazon S3 LOCATION STORAGE MANAGEABILITY Microsoft ADLS HDFS KUDU Data Center Self Managed Managed Service DATA ENGINEERING DATA WAREHOUSE DATA SCIENCE OPERATIONAL DATABASE SECURITY GOVERNANCE LIFECYCLE MANAGEMENT DATA CATALOG CLOUDERA VALUE PROPOSITION

7. 7 © Cloudera, Inc. All rights reserved. Big Data Infrastructure Evolution Infrastructure traditionally Each cluster is self-contained with compute, data context, and data Data context = HMS, Sentry, Navigator Compute Context Data Compute Context Data Compute Context Data • Compute, Context and Data together. Designed for best performance. • Highly Available, mission critical • Multi-tenant, Secure and Governed • Fixed Size, provisioned for peak capacity • Low Utilization rates if only transient workloads • Not easy to Scale

8. 8 © Cloudera, Inc. All rights reserved. Big Data Infrastructure Evolution Decoupled Infrastructure Data is separate from compute (e.g., in ADLS/S3), but context needs to be managed redundantly in each cluster Compute Context Compute Context Compute Context Data • Decoupled data and compute • Use Object Store • Scale easily • Workload Specific infrastructure • Support Transient and Persistent workloads • Compute and Context are still together • Maintain schemas in application. You lose Context when transient workload completes • No way to troubleshoot. You lose Workload information (logs and statistics) when the cluster goes away.

9. 9 © Cloudera, Inc. All rights reserved. Big Data Infrastructure Evolution Modern Infrastructure with Cloudera Compute clusters are launched as needed Data and data context are stored externally and are long-running Workload Analytics stores your logs and job statistics ComputeCompute Data Cloudera SDX/WXM Compute • Decoupled data and compute • Use Object Store • Scale easily • Workload Specific infrastructure • Support Transient and Persistent workloads • Persistent Context and Workload Analytics

10. 10 © Cloudera, Inc. All rights reserved. CLOUDERA ALTUS Flexible cloud deployment options including workload- optimized Managed Services with Cloudera Shared Data Experience (SDX) DATA ENGINEERING DATA WAREHOUSE MULTI FUNCTION CLOUD STORAGE DATA CATALOG GOVERNANCESECURITY CONTROL PLANE LIFECYCLE MANAGEMENT Microsoft ADLS AWS S3 DIRECTOR

12. 12 © Cloudera, Inc. All rights reserved. On-Premises Cloud Bursting Data Engineering Workflow: Batch ML, Simulations, etc. 1+ Jobs (e.g.Spark) Transient Cluster(s): Atlus Managed 1+ Jobs (e.g. Spark) On-Premises Cluster: Bare Metal/Private Cloud HDFS Data Science Data Eng Data Warehouse Continuous or On-Demand Persistent SDX Altus Control PlaneSDX: Schema (HMS), Security (Sentry), Lineage/Metadata (Navigator) Object Store (ADLS, S3) Data Warehouse (Impala) Elastic Cluster(s): Atlus Managed BI Tools (e.g. Tableau) SQL Editor (e.g. Hue)

13. 13 © Cloudera, Inc. All rights reserved. Altus Demo Data Engineering Workflow: Batch ML, Simulations, etc. 1+ Jobs (e.g.Spark) Transient Cluster(s): Atlus Managed 1+ Jobs (e.g. Spark) Persistent SDX Altus Control PlaneSDX: Schema (HMS), Security (Sentry), Lineage/Metadata (Navigator) Object Store (ADLS, S3) Data Warehouse (Impala) Elastic Cluster(s): Atlus Managed BI Tools (e.g. Tableau) SQL Editor (e.g. Hue) On-Premises Cluster: Bare Metal/Private Cloud HDFS Data Science Data Eng Data Warehouse Continuous or On-Demand

14. THANK YOUAltus - Demo

Hybrid is the New Normal

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Hybrid is the New Normal

Similar a Hybrid is the New Normal (20)

Más de DataWorks Summit

Más de DataWorks Summit (20)

Último

Último (20)

Hybrid is the New Normal