Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applications on Spark?, Raj Babu, Co-Founder, Magnus Unum, David Levinger, Sr. Director, Paxata and Raymond Fu, Practice Architect, Trace3

•

2 recomendaciones•820 vistas

In this interactive panel discussion, you will hear from these Spark experts as to why they chose to go "all-in" on Spark, leveraging the rich core capabilities that make Spark so exciting, and committing to significant IP that turns Spark into a world-class enterprise data preparation engine. Raymond and David will explain specific cases where capabilities were built on top of core Spark to provide a true interactive data prep application experience. Innovations such as creating a Domain Specific Language (DSL), an optimizing compiler, a persistent columnar caching layer, application specific Resilient Distributed Datasets (RDDs), on-line aggregation operators to solve the core memory, pipelining and shuffling obstacles to produce a highly interactive application with the core user and data volume scale-out benefits of Spark.

Tecnología

Unleashing the power of Spark™
Big and small lessons learned on how to take Spark to the next level
Moderated by Raj Babu
Speakers: David Levinger (Paxata) and Raymond Fu (Trace3)

Quick introductions
• Founder of Agile Intelligent Services & Solutions) a Los
Angeles, CA based Cloud & Data ( BI, Big Data, Data
Integration ) engineering services company
• Co-founder of Magnus Unum a Big Data services
company
• Passion and deep technical expertise as Solution
Architect for Big Data/ Data Lake /BI and Analytics
solutions
• Successfully designed, implemented and deployed
over 100+ large implementations
August 12, 2016© Paxata, Inc. 2
Raj Babu, Founder and CEO at Agile iSS LLC

Quick introductions
• Runs infrastructure for Paxata, the pioneers of self-
service data preparation
• Supports 55+ enterprise customers who have
deployed Paxata on premise and in the cloud and 90
employees who design innovative solutions
• Optimizes Paxata's development operations and
technology across the entire Hadoop stack and Cloud
environments
August 12, 2016© Paxata, Inc. 3
David Levinger, VP of Operations at Paxata

Quick introductions
• Over 16 years of experience in IT specializing in big
data, business intelligence, and enterprise architecture
• Designs big data and analytic solutions within complex
customer environments
• Led many data integrations and warehousing
initiatives from mergers and acquisitions
• Builds, validates and improve business strategy and
tactics through managing, enriching and mining data
August 12, 2016© Paxata, Inc. 4
Raymond Fu, Big Data and Analytics Architect at Trace3

Big and small lessons learned on
how to take Spark to the next level
5

Más contenido relacionado

La actualidad más candente

Rethink Analytics with an Enterprise Data Hub

Cloudera, Inc.

In this talk, we’ll describe NoSQL (“not-only SQL”) and document-oriented databases and the value they provide for data science companies like Uptake. We will walk through the unique challenges such datastores pose for data science workflows. To make these challenges and lessons learned concrete, we’ll explore data science workflows through a discussion of the development efforts that led to “uptasticsearch”, an R package released by the Uptake Data Science team to reduce friction in interacting with a document store called Elasticsearch. The talk will conclude with a discussion of recent developments in NoSQL technologies and implications for data scientists.

The Proliferation of New Database Technologies and Implications for Data Scie...

Domino Data Lab

AURIN Data Hubs Supporting Smarter Cities - Phil Delaney, Locate14

Phillip Delaney

Building the Modern Data Hub: Beyond the Traditional Enterprise Data Warehouse

Formant

Mark Lewis, Senior MArketing Director EMEA, Cloudera. Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.

MongoDB IoT City Tour LONDON: Hadoop and the future of data management. By, M...

MongoDB

In this webinar, we talk to industry analyst, author and practitioner David Linthicum who provides a state-of-the-technology explanation of big data integration. David also provides 5 critical and lesser known data integration requirements, how to understand today's requirements, and guidance for choosing the right approaches and technology to solve these problems. To learn more, visit: www.snaplogic.com/big-data

Webinar: The 5 Most Critical Things to Understand About Modern Data Integration

SnapLogic

Beyond Batch: Is ETL still relevant in the API economy?

SnapLogic

In this webinar, we talk about the shift in data gravity as more and more business applications are moving to the cloud, and how the ability to deliver analytics in the cloud has evolved from idea to enterprise reality with new solutions being announced constantly that appeal to the need for speed, simplicity and access to insight on demand. Joining us in this webinar is David Glueck, Sr. Director of Data Science and Engineering at Bonobos. To learn more, visit: www.SnapLogic.com/salesforce-analytics

Webinar: BI in the Sky - The New Rules of Cloud Analytics

SnapLogic

Constant changes in the healthcare industry continue to drive innovation in technology and serve as a catalyst for cloud adoption. This trend will continue to evolve and accelerate in the coming years with the increasing need to store and analyze vast amounts of information for personal and population health initiatives. We joined guest speaker from HIMSS Analytics, James Gaston, to discuss the impact of the cloud on data architecture in healthcare. Topics included: -The benefits and risks of moving data and analytics environments to the cloud -Main healthcare use cases for cloud migration -Deep dive into two leading healthcare organizations’ cloud journeys including drivers, challenges, benefits, and lessons learned

Moving to the Cloud: Modernizing Data Architecture in Healthcare

Perficient, Inc.

Introduction to Cloud Applications

DataStax

Analytic workloads and the ability to determine “what happened” are some of the most common use cases across enterprises today - helping you understand and adapt based on changing trends. However, for most businesses today, they are only able to see a piece of the story. Analytics are limited by the amount of data able to be stored and ultimately accessed, it’s time-intensive to bring in new datasets or fit unstructured data into rigid schemas, and user access is constrained to a select few who must already know the questions they’re trying to answer. It’s no surprise that big data is disrupting this modus operandi for analytics. A modern, Hadoop-based platform is designed to help businesses break free of these analytic limitations, providing a new kind of adaptive, high-performance analytic database. The recent release of Cloudera 5.8 continues to advance Cloudera Enterprise as the foundation for these analytic workloads. Join Justin Erickson, Senior Director of Product Management at Cloudera, and Andy Frey, Chief Technology Officer at Marketing Associates, as they discuss: -What technology is needed to build a modern analytic database with Hadoop -What’s new with Cloudera 5.8 -How to align your teams around agile analytics -Real world success from Marketing Associates -What’s next for Cloudera Enterprise’s Analytic Database

Building a Modern Analytic Database with Cloudera 5.8

Cloudera, Inc.

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...

Databricks

Introduction: Architecting for Scale

DataStax

Webinar on the updated Payment Services Directive (PSD2) and how it will affect financial institutions when introduced in 2018. This webinar outlines the challenges facing financial institutions and how API connectivity can help them to not only survive but thrive in an increasingly competitive future. View webinar recording: https://youtu.be/K2bQ9qPgFL4 Explore all DataStax Webinars: http://www.datastax.com/resources/webinars

Webinar: Become PSD2 ready with DataStax

DataStax

This webinar follows the process of evaluating different big data platforms based on varying use cases and business requirements, and explains how big data professionals can choose the right technology to transform their business. During this session, Ooyala CTO, Sean Knapp will discuss why Ooyala selected DataStax as the big data platform powering their business, and how they provide real-time video analytics that help media companies create deeply personalized viewing experiences for more than 1/4 of all Internet video viewers each month.

Webinar: 2 Billion Data Points Each Day

DataStax

Apache Cassandra is the open source database technology that pioneered distributed data at scale. DataStax Enterprise, powered by the best distribution of Apache Cassandra, gives you up to 2x better compaction throughput, 3x better operational analytics performance, ease-of-use, and a secure, comprehensive multi-model data platform including search and operational analytics integrated with Cassandra to help you take on whatever challenges you might face along the way. View recording: https://youtu.be/qLJyFydE-uY Explore all DataStax webinars: http://www.datastax.com/resources/webinars

Webinar: Comparing DataStax Enterprise with Open Source Apache Cassandra

DataStax

How did Devon move from a traditional reporting and data warehouse approach to a modern data lake? What did it take to go from a slow and brittle technical landscape to an a flexible, scalable, and agile platform? In the past, Devon addressed data solutions in dozens of ways depending on the user and the requirements. Through a visionary program, driven by Databricks, Devon has begun a transformation of how it consumes data and enables engineers, analysts, and IT developers to deliver data driven solutions along all levels of the data analytics spectrum. We will share the vision, technical architecture, influential decisions, and lessons learned from our journey. Join us to hear the unique Databricks success story at Devon.

Transforming Devon’s Data Pipeline with an Open Source Data Hub—Built on Data...

Databricks

La actualidad más candente (17)

Rethink Analytics with an Enterprise Data Hub

The Proliferation of New Database Technologies and Implications for Data Scie...

AURIN Data Hubs Supporting Smarter Cities - Phil Delaney, Locate14

Building the Modern Data Hub: Beyond the Traditional Enterprise Data Warehouse

MongoDB IoT City Tour LONDON: Hadoop and the future of data management. By, M...

Webinar: The 5 Most Critical Things to Understand About Modern Data Integration

Beyond Batch: Is ETL still relevant in the API economy?

Webinar: BI in the Sky - The New Rules of Cloud Analytics

Moving to the Cloud: Modernizing Data Architecture in Healthcare

Introduction to Cloud Applications

Building a Modern Analytic Database with Cloudera 5.8

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...

Introduction: Architecting for Scale

Webinar: Become PSD2 ready with DataStax

Webinar: 2 Billion Data Points Each Day

Webinar: Comparing DataStax Enterprise with Open Source Apache Cassandra

Transforming Devon’s Data Pipeline with an Open Source Data Hub—Built on Data...

Destacado

(Presented at MapR's Big Data Everywhere event in Redwood City, CA in December 2016) The relationship between business teams and IT has changed as the complexity of data has increased. A traditional data pipeline designed for an IT-centered approach to information management is not designed for the data demands of today's business decisions. Designing a big data strategy requires modernizing previous approaches. Self-service data preparation in a collaborative, intuitive, governed, and secure environment is the key to a nimble and decisive business unit.

Reinventing the Modern Information Pipeline: Paxata and MapR

Lilia Gutnik

Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...

Data Con LA

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in...

Data Con LA

Born more than four decades ago from the partnership of two international NGOs in Brussels, the Encyclopedia of World Problems has hand-picked and refined profiles of tens of thousands of problems occurring around the world: from notorious global issues all the way down to very specific and peculiar ones. This talk presents an overview of the Encyclopedia and the interesting data science applications that have arisen from the Encyclopedia's body of work - notably, its database resources.

Big Data Day LA 2016/ Use Case Driven track - The Encyclopedia of World Probl...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Data Con LA

Today’s Software Defined environments attempt to remove the weakness of computing hardware from the operational equation. There is no doubt that this is a natural progress away from overpriced, proprietary compute and storage layers. However, even at the heart of any Software Defined universe is an underlying hardware stack that must be robust, reliable and cost effective. Our 20+ years experience delivering over 2000 clusters and clouds has taught us how to properly design and engineer the right hardware solution for Big Data, Cluster and Cloud environments. This presentation will share this knowledge allowing user to make better design decisions for any deployment.

Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...

Data Con LA

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Why is my Hadoop cluster s...

Data Con LA

Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...

Data Con LA

Big Data Day LA 2016/ NoSQL track - MongoDB 3.2 Goodness!!!, Mark Helmstetter...

Data Con LA

The advent of modern deep learning techniques has given organizations new tools to understand, query, and structure their data. However, maintaining complex pipelines, versioning models, and tracking accuracy regressions over time remain ongoing struggles of even the most advanced data engineering teams. This talk presents a simple architecture for deploying machine learning at scale and offer suggestions for how companies can get their feet wet with open source technologies they already deploy.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Deep Learning at Scale - A...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - How to Use Design Thinking to J...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - Data and Hollywood: "Je t'Aime ...

Data Con LA

The goal of this talk to lay out a framework for what algorithms work best in which situations, and why. Drawing on results of hundreds of crowd-sourced predictive modeling contests, this talk shows examples of how structure informs a choice in algorithm. As an illustration of these concepts, ZestFinance's work with China's retail giant, JD.com is used to describe how the right algorithms were applied to the right datasets to turn shopping data into credit data -- creating credit scores from scratch.

Big Data Day LA 2016/ Data Science Track - The Right Tool for the Job: Guidel...

Data Con LA

Abstract:- Data engineering at Dollar Shave Club has grown significantly over the last year. In that time, it has expanded in scope from conventional web-analytics and business intelligence to include real-time, big data and machine learning applications. We have bootstrapped a dedicated data engineering team in parallel with developing a new category of capabilities. And the business value that we delivered early on has allowed us to forge new roles for our data products and services in developing and carrying out business strategy. This progress was made possible, in large part, by adopting Apache Spark as an application framework. This talk describes what we have been able to accomplish using Spark at Dollar Shave Club. Bio:- Brett Bevers, Ph.D. Brett is a backend engineer and leads the data engineering team at Dollar Shave Club. More importantly, he is an ex-academic who is driven to understand and tackle hard problems. His latest challenge has been to develop tools powerful enough to support data-driven decision making in high value projects.

Joining the Club: Using Spark to Accelerate Big Data at Dollar Shave Club

Data Con LA

Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...

Data Con LA

Big Data Day LA 2016/ NoSQL track - Introduction to Graph Databases, Oren Gol...

Data Con LA

Prototypes are typically re-implemented in another language due to compatibility issues with R in the enterprise, but TIBCO Enterprise Runtime for R (TERR) allows the language to be run on several platforms. Enterprise-level scalability has been brought to the R language, enabling rapid iteration without the need to recode, re-implement and test. This presentation will delve further into these topics, highlighting specific use cases and the true value that can be gained from utilizing R. The session will be followed by a lively, open Q&A discussion.

Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...

Data Con LA

When you're the first data professional at the organization there are technical, process, and qualitative considerations for analytics and data science to address (A/DS). This talk is an overview of strategy, infrastructure, and tools for creating your first A/DS stacks. At this stage, the range of problems that you are able to solve relate to organization, operational, data engineering, business intelligence, and communication. Creating the optimal A/DS stack can seamlessly pave the way to big data and integrating the newest technologies in the future. Please share your stories and experience with us as well. Outline of talk, where sections intend to be interactive and get feedback from the audience: 1. So you're the first Data Scientist 2. Setting Their Expectations 3. Lay of the Land - Data requirements and organizational survey 4. Setting Your Expectations 5. Infrastructure - Your Stack Options 6. Resources: Get Help, Get a Team 7. Discussion

Big Data Day LA 2016/ Data Science Track - Backstage to a Data Driven Culture...

Data Con LA

Abstract: Data exploration often requires running aggregation/slice-dice queries on data sourced from disparate sources. You may want to identify distribution patterns, outliers, etc and aid the feature selection process as you train your predictive models. As you begin to understand your data, you want to ask ad-hoc questions expressed through your visualization tool (which typically translates to SQL queries), study the results and iteratively explore the data set through more queries. Unfortunately, even when data sets can be in-memory, large data set computations take time breaking the train of thought and increasing time to insight . We know Spark can be fast through its in-memory parallel processing. But, Spark 1.x isn’t quite there. Spark 2.0 promises to offer 10X better speed than its predecessor. Spark 2.0 ushers some impressive improvements to interactive query performance. We first explore these advances - compiling the query plan eliminating virtual function calls, and other improvements in the Catalyst engine. We compare the performance to other popular popular query processing engines by studying the spark query plans. We then go through SnappyData (an open source project that integrates Spark with a database that offers OLTP, OLAP and stream processing in a single cluster) where we use smarter data colocation and Synopses data (.e.g. Stratified sampling) to dramatically cut down on the memory requirements as well as the query latency. We explain the key concepts in summarizing data using structures like stratified sampling by walking through some examples in Apache Zeppelin notebooks (a open source visualization tool for spark) and demonstrate how we can explore massive data sets with just your laptop resources while achieving remarkable speeds. Bio: Jags is a founder and the CTO of SnappyData. Previously, Jags was the Chief Architect for “fast data” products at Pivotal and served in the extended leadership team of the company. At Pivotal and previously at VMWare, he led the technology direction for GemFire and other distributed in-memory Bio: Jags Ramnarayan is a founder and the CTO of SnappyData. Previously, Jags was the Chief Architect for “fast data” products at Pivotal and served in the extended leadership team of the company. At Pivotal and previously at VMWare, he led the technology direction for GemFire and other distributed in-memory products.

Explore big data at speed of thought with Spark 2.0 and Snappydata

Data Con LA

Abstract:- Tracking user events as they happen can challenge anyone providing real time user interaction. It can demand both huge scale and a lot of processing to support dynamic adjustment to targeting products and services. As the operational data store Couchbase data services are capable of processing tens of millions of updates a day. Streaming through systems such as Apache Spark and Kafka into Hadoop, information about these key events can be turned into deeper knowledge. We will review Lambda architectures deployed at sites like PayPal, Live Person and LinkedIn that leverage a Couchbase Data Pipeline. Bio:- Justin Michaels. With over 20 years experience in deploying mission critical systems, Justin Michaels industry experience covers capacity planning, architecture and industry vertical experience. Justin brings his passion for architecting, implementing and improving Couchbase to the community as a Solution Architect. His expertise involves both conventional application platforms as well as distributed data management systems. He regularly engages with existing and new Couchbase customers in performance reviews, architecture planning and best practice guidance.

Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couc...

Data Con LA

Destacado (20)