Cloudera Customer Success Story

At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.

Spark Summit EU 2015: Lessons from 300+ production users

Uber has real needs to provide faster, fresher data to data consumers & products, running hundreds of thousands of analytical queries everyday. Uber engineers will share the design, architecture & use-cases of the second generation of ‘Hudi’, a self contained Apache Spark library to build large scale analytical datasets designed to serve such needs and beyond. Hudi (formerly Hoodie) is created to effectively manage petabytes of analytical data on distributed storage, while supporting fast ingestion & queries. In this talk, we will discuss how we leveraged Spark as a general purpose distributed execution engine to build Hudi, detailing tradeoffs & operational experience. We will also show to ingest data into Hudi using Spark Datasource/Streaming APIs and build Notebooks/Dashboards on top using Spark SQL.

Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...

More and more organizations are moving their ETL workloads to a Hadoop based ELT grid architecture. Hadoop`s inherit capabilities, especially it`s ability to do late binding addresses some of the key challenges with traditional ETL platforms. In this presentation, attendees will learn the key factors, considerations and lessons around ETL for Hadoop. Areas such as pros and cons for different extract and load strategies, best ways to batch data, buffering and compression considerations, leveraging HCatalog, data transformation, integration with existing data transformations, advantages of different ways of exchanging data and leveraging Hadoop as a data integration layer. This is an extremely popular presentation around ETL and Hadoop.

A Reference Architecture for ETL 2.0

DataWorks Summit

Building an open data platform with apache iceberg

Alluxio, Inc.

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

We have covered the need for CDC and the benefits of building a CDC pipeline. We will compare various CDC streaming and reconciliation frameworks. We will also cover the architecture and the challenges we faced while running this system in the production. Finally, we will conclude the talk by covering Apache Hudi, Schema Registry and Debezium in detail and our contributions to the open-source community.

Building robust CDC pipeline with Apache Hudi and Debezium

Tathastu.ai

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies. The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.

Apache Kafka as Event Streaming Platform for Microservice Architectures

Kai Wähner

This Edureka Spark Tutorial will help you to understand all the basics of Apache Spark. This Spark tutorial is ideal for both beginners as well as professionals who want to learn or brush up Apache Spark concepts. Below are the topics covered in this tutorial: 1) Big Data Introduction 2) Batch vs Real Time Analytics 3) Why Apache Spark? 4) What is Apache Spark? 5) Using Spark with Hadoop 6) Apache Spark Features 7) Apache Spark Ecosystem 8) Demo: Earthquake Detection Using Apache Spark

Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...

Edureka!

Data Vault Modeling and Methodology introduction that I provided to a Montreal event in September 2011. It covers an introduction and overview of the Data Vault components for Business Intelligence and Data Warehousing. I am Dan Linstedt, the author and inventor of Data Vault Modeling and methodology. If you use the images anywhere in your presentations, please credit http://LearnDataVault.com as the source (me). Thank-you kindly, Daniel Linstedt

Data Vault Overview

Empowered Holdings, LLC

Oracle E-Business Suite R12.2.5 on Database 12c: Install, Patch and Administer

Andrejs Karpovs

Talend Big Data Capabilities Overview

Rajan Kanitkar

Data Warehousing - in the real world

ukc4

10 ways to improve your rman script

Maris Elsins

Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products. On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Intro to databricks delta lake

Mykola Zerniuk

Intro to Data Vault 2.0 on Snowflake

Kent Graziano

Using Databricks as an Analysis Platform

Learn more about InfluxDB’s new storage engine! The team developed a cloud-native, real-time, columnar database optimized for time series data. We built it all in Rust and it sits on top of Apache Arrow and DataFusion. We chose Apache Parquet as the persistent format, which is an open source columnar data file format. This new storage engine provides InfluxDB Cloud users with new functionality, including the removal of cardinality limits, so developers can bring in massive amounts of time series data at scale. In this webinar, Anais Dotis-Georgiou will dive into: Requirements for rebuilding InfluxDB’s core Key product features and timeline How Apache Arrow’s ecosystem is used to meet those requirements Stick around for a demo and live Q&A

Understanding InfluxDB’s New Storage Engine

InfluxData

While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.

Backup and Disaster Recovery in Hadoop

DataWorks Summit/Hadoop Summit

La actualidad más candente (20)

Moving to Databricks & Delta

Spark Summit EU 2015: Lessons from 300+ production users

Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...

A Reference Architecture for ETL 2.0

Building an open data platform with apache iceberg

Batch Processing at Scale with Flink & Iceberg

Building robust CDC pipeline with Apache Hudi and Debezium

Apache Kafka as Event Streaming Platform for Microservice Architectures

Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...

Data Vault Overview

Oracle E-Business Suite R12.2.5 on Database 12c: Install, Patch and Administer

Talend Big Data Capabilities Overview

Data Warehousing - in the real world

10 ways to improve your rman script

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Intro to databricks delta lake

Intro to Data Vault 2.0 on Snowflake

Using Databricks as an Analysis Platform

Understanding InfluxDB’s New Storage Engine

Backup and Disaster Recovery in Hadoop

Destacado

Non-geek's big data playbook - Hadoop & EDW - SAS Best Practices

Jyrki Määttä

Alternative data for capital markets, such as satellite imagery, logistics data, and social media feeds, has been getting a lot of attention recently. Like any trending topic, its uses and benefits can be hyped up a bit but if the right plumbing and creativity is in place, those benefits can be realized. 3 things to learn: * Examples of alt data use cases, sources, and recent market trends * Why a big data platform that facilitates self service and collaboration is critical in monetizing alternative data * How alternative data can be applied to enhance current processes (Demo)

Put Alternative Data to Use in Capital Markets 

Banks are rich in valuable data and can build and maintain a competitive advantage by identifying and executing on high-value machine learning projects leveraging the rich data available.This webinar will describe use cases fit for big data and machine learning in the banking sector (commercial, consumer, regulatory, and markets) and the impact they can have for your organization. 3 things to learn: * How to create a next generation data platform and why it is important * How to monetize big data using predictive modeling and machine learning * What is needed for automated machine learning as a sustainable, cost-effective, and efficient solution

Transform Banking with Big Data and Automated Machine Learning 9.12.17

New research shows that 57% of the buying cycle is completed before a prospect even speaks to a company. Marketers already know this, Ninety-six percent (96%) of organizations believe that email personalization can improve email marketing performance. But where do we get this increasingly personal direction? The answer is likely in your customer data. In order to understand your customer needs contextualized in the moment they feel the need to act you will require a platform that can leverage real-time data. Apache Kudu is a Cloudera component that makes dealing with quickly changing data fast and easy. Companies are leveraging next generation data stores like Kudu to build data applications that deliver smart promotions, real-time offers, and personalized marketing. Join us as we discuss modern approaches to real-time application development and highlight key Cloudera use cases being powered by Cloudera’s operational database.

The Big Picture: Real-time Data is Defining Intelligent Offers

With billions of new devices, IoT is transforming how businesses capitalize on data. Data driven organizations are using IoT as as a means to improve their customer experience, drive operational efficiencies, and enable new business models. However, without the right data management strategy and tools, investments in IoT can yield limited results. Join Cloudera and 451 Research for a joint webinar to learn more about some of the data management best practices and how organizations are using advanced analytics and machine learning to enable IoT use cases.

Large-Scale Data Science on Hadoop (Intel Big Data Day)

Uri Laserson

IoT - Data Management Trends, Best Practices, & Use Cases

Machine learning and analytics applications are exploding in the enterprise; driving use cases for preventative maintenance, delivering new desirable product offers to customers at the right time, and combating insider threats to your business. But each of these high-value use cases rely on a variety of data analysis capabilities working in concert to combine data from different sources into a single coherent picture. Cloudera SDX delivers a “shared data experience” that makes applications easier to develop, less expensive to deploy and more consistently secure. 3 things to learn: * Why multi-function applications are difficult to build and secure * How shared catalog, governance, management, and security applied consistently everywhere can deliver a “shared data experience” * How enterprise customers are building new, high-value applications with SDX

How to Build Multi-disciplinary Analytics Applications on a Shared Data Platform

Speaker: Hari Shreedharan Data Day Texas 2015 Apache Spark has emerged over the past year as the imminent successor to Hadoop MapReduce. Spark can process data in memory at very high speed, while still be able to spill to disk if required. Spark’s powerful, yet flexible API allows users to write complex applications very easily without worrying about the internal workings and how the data gets processed on the cluster. Spark comes with an extremely powerful Streaming API to process data as it is ingested. Spark Streaming integrates with popular data ingest systems like Apache Flume, Apache Kafka, Amazon Kinesis etc. allowing users to process data as it comes in. In this talk, Hari will discuss the basics of Spark Streaming, its API and its integration with Flume, Kafka and Kinesis. Hari will also discuss a real-world example of a Spark Streaming application, and how code can be shared between a Spark application and a Spark Streaming application. Each stage of the application execution will be presented, which can help understand practices while writing such an application. Hari will finally discuss how to write a custom application and a custom receiver to receive data from other systems.

Real Time Data Processing using Spark Streaming | Data Day Texas 2015

Unternehmen sind heutzutage in der Lage ihre Daten mit relativer Leichtigkeit aufzunehmen und zu verwalten. Die Herausforderung besteht nun darin, die verborgenen Muster in den Daten zu erkennen und diese zu verstehen, um einen Mehrwert zu generieren. Aufgrund der großen Datenmengen gelingt dies mit traditionelle Ansätzen zumeist nicht. Das Ergebnis: Organisationen kämpfen, um wirklich zu innovieren und sich zu differenzieren.

Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...

As enterprises continue to transition their technology organizations to operate at higher velocity and break down barriers with DevOps transformations, they need to be able to respond to service disruptions and incidents quickly, and with minimal impact to the productivity of their teams. Learn how ViaSat — a global provider of satellite and wireless communications — securely connects businesses, governments, and military organizations to the Internet using structure, automated toolsets and teams. Join Marty Jackson, Director, Office of the CTO, xMatters and Chris Crocco, ViaSat’s Network Solutions Engineer, to learn how the ViaSat team managed the integration of alerting, issue tracking and change and release management with JIRA Software, JIRA Service Desk, HipChat and xMatters into a complex IT ecosystem and drove the company’s transition from an Operations to a DevOps-centric model of IT incident communications. Vincent Wong, Product Manager, Atlassian Jared Sutherland, Product Manager, Intuit

Destacado (9)

Non-geek's big data playbook - Hadoop & EDW - SAS Best Practices

Put Alternative Data to Use in Capital Markets 

Transform Banking with Big Data and Automated Machine Learning 9.12.17

The Big Picture: Real-time Data is Defining Intelligent Offers

Large-Scale Data Science on Hadoop (Intel Big Data Day)

IoT - Data Management Trends, Best Practices, & Use Cases

How to Build Multi-disciplinary Analytics Applications on a Shared Data Platform

Real Time Data Processing using Spark Streaming | Data Day Texas 2015

Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...

Similar a Cloudera Customer Success Story

Next generation business automation with the red hat decision manager and red...

Masahiko Umeno

Modern Operations at Scale within Viasat – How to Structure Teams and Build A...

Atlassian

How Dealertrack Optimizes the DevOps Toolchain, FutureStack17

New Relic

Sabyasachee_Kar_cv

Sabyasachee Kar

MLOps - Build pipelines with Tensor Flow Extended & Kubeflow

Jan Kirenz

TaLend Online Training

Glory IT Technologies

Citihub IDC Event 2009 Cloud Mark Ellis

markellis

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

Safe Software

Gowthami_Resume

Gowthami Subramaniam

Resume1

Anantha Sai Yadlapalli

Maximizing Business Value: Optimizing Technology Investment

Teradata

Top 5 Things to Know About Integrating MongoDB into Your Data Warehouse

MongoDB

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Rittman Analytics

Sunny_Resume

Sunny Gupta

près de 60% de leur temps à essayer de comprendre comment fonctionnent les applications sur lesquelles ils travaillent ? CAST Imaging est un logiciel de Software Intelligence qui produit automatiquement la documentation/base de connaissances techniques de n’importe quelle application. Celle-ci prend la forme de blueprints interactifs – dont les données sont stockées dans Neo4j – qui cartographient en détail tous les éléments d’une application et toutes leurs dépendances. Peu importe qu’ils soient en train de développer, maintenir, moderniser ou simplement monter en compétence, les utilisateurs de CAST Imaging trouvent en quelques minutes les réponses aux questions qu’ils se posent sur le fonctionnement de l’application, sans passer des heures à fouiller dans le code !

CAST Imaging: Map & Master Your Software

Neo4j

Track c how do we break - jasper

Delphix

Sunny_Resume

CDS + Power Apps

TAUS Translation Technology Showcase Webinar - SmartMATE, 13 November 2013

TAUS - The Language Data Network

Similar a Cloudera Customer Success Story (20)

Next generation business automation with the red hat decision manager and red...

Modern Operations at Scale within Viasat – How to Structure Teams and Build A...

How Dealertrack Optimizes the DevOps Toolchain, FutureStack17

Sabyasachee_Kar_cv

MLOps - Build pipelines with Tensor Flow Extended & Kubeflow

TaLend Online Training

Citihub IDC Event 2009 Cloud Mark Ellis

The Zero-ETL Approach: Enhancing Data Agility and Insight

Gowthami_Resume

Resume1

Maximizing Business Value: Optimizing Technology Investment

Top 5 Things to Know About Integrating MongoDB into Your Data Warehouse

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Sunny_Resume

CAST Imaging: Map & Master Your Software

Track c how do we break - jasper

Delphix

Sunny_Resume

CDS + Power Apps

TAUS Translation Technology Showcase Webinar - SmartMATE, 13 November 2013

Más de Xpand IT

The Xray and Xporter Winter Tour kicked off last Wednesday with the Jira & Confluence Solutions. During Sérgio Freire’s (Xray Product Manager) presentation, he showed Jira as a Test Management tool and how to empower test teams to manage and deliver rock-solid software solutions with Xray. If you missed it or you want to know more about Testing in Jira, you can check it here. For more visit https://www.xpand-addons.com/

Xray & Xporter were in Austria: Jira & Confluence Solutions Day 2018

Using Xamarin for your Mobile+ Apps – Xamarin Experience London 2017

Xporter for Jira - Overview

Xray for Jira - How to automate your QA process

Xpand Addons - Addon Discovery Day 2017

Xray for Jira 3.0 - What's New?

Xray for Jira - Overview

Xporter for Jira - Advanced topics

Keynote - Xamarin Experience London 2017

Welcome & Introduction – Xamarin Experience London 2017

Gathering Customer Insights with Sitecore - Xamarin Experience London 2017

Why Speed Matters in Mobile Apps – Xamarin Experience London 2017

Mobile & Cognitive Services | Harnessing the Power of IoT – Xamarin Experienc...

Atlassian Tools in Practice: A Customer Success Story – Xpand IT & Atlassian ...

The Secret Sauce of Successful Teams - Xpand IT & Atlassian JAM Sessions 2017

Quality Assurance Made Easy in JIRA - Xpand IT & Atlassian JAM Sessions 2017

Improved Reporting with JIRA Add-ons - Xpand IT & Atlassian JAM Sessions 2017

How our Team Collaborates with Atlassian Tools - Xpand IT & Atlassian JAM Ses...

Welcome & Introduction - Xpand IT & Atlassian JAM Sessions 2017

The Real World with OpenShift - Red Hat DevOps & Microservices Conference 2017