Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

•Descargar como PPTX, PDF•

1 recomendación•3,351 vistas

TERADATA VIEWPOINT
HADOOP INTEGRATION WITH AMBARI

Steve Ratay
Chief Architect, Teradata Viewpoint
April 2, 2013

Teradata Unified Data Architecture
• Combination of Teradata Database, Aster, and Hadoop in a
single appliance
• Connectors between the different systems
• Monitor the entire appliance with Teradata Viewpoint

DISCOVERY Aster Teradata INTEGRATED
PLATFORM DATA WAREHOUSE
Connector

SQL-H SQL-H

Aster Connector for Hadoop Teradata Connector for Hadoop

CAPTURE | STORE | REFINE

2 Copyright Teradata 2012

Viewpoint Architecture
TD 1
Viewpoint Managed Server

Data TD n
Viewpoint Postgres Collection
Portal Database
Service
Aster 1

ActiveMQ
Aster n

Alert
Service
HDP 1

Email SNMP Portlet HDP n

3 Copyright Teradata 2012

Monitoring Hadoop without Ambari
• Different technologies and formats:
> Ganglia
> Nagios
> JMX
> Screen scraping HTML

• Connection to multiple nodes:
> Connectivity to all nodes
> Access to several ports
> Location of each service, especially when using high availability

4 Copyright Teradata 2012

Ambari to the Rescue!
• RESTful APIs
• JSON response format
• Single server and port

5 Copyright Teradata 2012

Accessing Ambari APIs
• Viewpoint uses Java and Spring
• Used Spring’s RestTemplate class:
> Makes HTTP call to specified URL
> Binds JSON results to Java model objects
• Data is collected every minute and stored in Viewpoint’s
Postgres database
> Use of Rewind to view previous state of system
> Easy generation of trending charts and graphs

6 Copyright Teradata 2012

System Health

7 Copyright Teradata 2012

Metric Analysis

8 Copyright Teradata 2012

Node Monitor

9 Copyright Teradata 2012

Hadoop Services

10 Copyright Teradata 2012

Hadoop Alerts

11 Copyright Teradata 2012

Benefits of Using Ambari
• Viewpoint delivered comprehensive Hadoop monitoring
solution in a couple of months
• Java and web developers focused on collecting data and
building UI
• Didn’t waste time dealing with intricacies of Hadoop and
Hadoop monitoring

12 Copyright Teradata 2012

Más contenido relacionado

La actualidad más candente

As more applications are created using Apache Hadoop that derive value from the new types of data from sensors/machines, server logs, click-streams, and other sources, the enterprise "Data Lake" forms with Hadoop acting as a shared service. While these Data Lakes are important, a broader life-cycle needs to be considered that spans development, test, production, and archival and that is deployed across a hybrid cloud architecture. If you have already deployed Hadoop on-premise, this session will also provide an overview of the key scenarios and benefits of joining your on-premise Hadoop implementation with the cloud, by doing backup/archive, dev/test or bursting. Learn how you can get the benefits of an on-premise Hadoop that can seamlessly scale with the power of the cloud.

Data Lake for the Cloud: Extending your Hadoop Implementation

Data Lake for the Cloud: Extending your Hadoop Implementation

Data Lake for the Cloud: Extending your Hadoop Implementation

Data lake benefits

Data lake benefits

Data lake benefits

Mr. Slim Baltagi is a Systems Architect at Hortonworks, with over 4 years of Hadoop experience working on 9 Big Data projects: Advanced Customer Analytics, Supply Chain Analytics, Medical Coverage Discovery, Payment Plan Recommender, Research Driven Call List for Sales, Prime Reporting Platform, Customer Hub, Telematics, Historical Data Platform; with Fortune 100 clients and global companies from Financial Services, Insurance, Healthcare and Retail. Mr. Slim Baltagi has worked in various architecture, design, development and consulting roles at. Accenture, CME Group, TransUnion, Syntel, Allstate, TransAmerica, Credit Suisse, Chicago Board Options Exchange, Federal Reserve Bank of Chicago, CNA, Sears, USG, ACNielsen, Deutshe Bahn. Mr. Baltagi has also over 14 years of IT experience with an emphasis on full life cycle development of Enterprise Web applications using Java and Open-Source software. He holds a master’s degree in mathematics and is an ABD in computer science from Université Laval, Québec, Canada. Languages: Java, Python, JRuby, JEE , PHP, SQL, HTML, XML, XSLT, XQuery, JavaScript, UML, JSON Databases: Oracle, MS SQL Server, MYSQL, PostreSQL Software: Eclipse, IBM RAD, JUnit, JMeter, YourKit, PVCS, CVS, UltraEdit, Toad, ClearCase, Maven, iText, Visio, Japser Reports, Alfresco, Yslow, Terracotta, Toad, SoapUI, Dozer, Sonar, Git Frameworks: Spring, Struts, AppFuse, SiteMesh, Tiles, Hibernate, Axis, Selenium RC, DWR Ajax , Xstream Distributed Computing/Big Data: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, HBase, R, RHadoop, Cloudera CDH4, MapR M7, Hortonworks HDP 2.1

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Innovative Management Services

Presentation detailing how Elasticsearch is involved in Oncrawl, a SaaS solution for easy SEO monitoring. The presentation explains how the application is built, and how it integrates Elasticsearch, a powerful general purpose search engine. Oncrawl is data centric and elasticsearch is used as an analytics engine rather than a full text search engine. The application uses Apache Hadoop and Apache Nutch for the crawl pipeline and data analysis. Oncrawl is a Cogniteev solution.

Oncrawl elasticsearch meetup france #12

Oncrawl elasticsearch meetup france #12

Oncrawl elasticsearch meetup france #12

Why Data Lake should be the foundation of Enterprise Data Architecture

Why Data Lake should be the foundation of Enterprise Data Architecture

Why Data Lake should be the foundation of Enterprise Data Architecture

Agilisium Consulting

Hadoop Trends

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Hadoop adoption is a journey. Depending on the business the process can take weeks, months, or even years. Hadoop is a transformative technology so the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. There are challenges for companies who have lived with an application driven business for the last two decades to suddenly become data driven. Companies need to begin thinking less in terms of single, silo’d servers and more about “the cluster”. The concept of the cluster becomes the center of data gravity drawing all the applications to it. Companies, especially the IT organizations, embark on a process of understanding how to maintain and operationalize this environment and provide the data lake as a service to the businesses. They must empower the business by providing the resources for the use cases which drive both renovation and innovation. IT needs to adopt new technologies and new methodologies which enable the solutions. This is not technology for technology sake. Hadoop is a data platform servicing and enabling all facets of an organization. Building out and expanding this platform is the ongoing journey as word gets out to businesses that they can have any data they want and any time. Success is what drives the journey. The length of the journey varies from company to company. Sometimes the challenges are based on the size of the company but many times the challenges are based on the difficulty of unseating established IT processes companies have adopted without forethought for the past two decades. Companies must navigate through the noise. Sifting through the noise to find those solutions which bring real value takes time. As the platform matures and becomes mainstream, more and more companies are finding it easier to adopt Hadoop. Hundreds of companies have already taken many steps; hundreds more have already taken the first step. As the wave of successful Hadoop adoption continues, more and more companies will see the value in starting the journey and paving the way for others.

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

Hadoop data-lake-white-paper

Hadoop data-lake-white-paper

Hadoop data-lake-white-paper

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

Data is exponentially increasing in both types and volumes, creating opportunities for businesses. Watch this video and learn from three Big Data experts: John Kreisa, VP Strategic Marketing at Hortonworks, Imad Birouty, Director of Technical Product Marketing at Teradata and John Haddad, Senior Director of Product Marketing at Informatica. Multiple systems are needed to exploit the variety and volume of data sources, including a flexible data repository. Learn more about: - Apache Hadoop 2 and YARN - Data Lakes - Intelligent data management layers needed to manage metadata and usage patterns as well as track consumption across these data platforms.

Hadoop 2.0: YARN to Further Optimize Data Processing

Hadoop 2.0: YARN to Further Optimize Data Processing

Hadoop 2.0: YARN to Further Optimize Data Processing

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Joe Caserta, President at Caserta Concepts presented at the 3rd Annual Enterprise DATAVERSITY conference. The emphasis of this year's agenda is on the key strategies and architecture necessary to create a successful, modern data analytics organization. Joe Caserta presented Incorporating the Data Lake into Your Analytics Architecture. For more information on the services offered by Caserta Concepts, visit out website at http://casertaconcepts.com/.

Incorporating the Data Lake into Your Analytic Architecture

Incorporating the Data Lake into Your Analytic Architecture

Incorporating the Data Lake into Your Analytic Architecture

Planing and optimizing data lake architecture

Planing and optimizing data lake architecture

Planing and optimizing data lake architecture

Milos Milovanovic

Information Virtualization: Query Federation on Data Lakes

Information Virtualization: Query Federation on Data Lakes

Information Virtualization: Query Federation on Data Lakes

DataWorks Summit

Finance Data Lake objective is to create a centralized enterprise data repository for all Finance and Supply Chain data. It serves as the single source of truth. It enables a self-service discovery Analytics platform for business users to answer adhoc business questions and derive critical insights. The data lake is based on open source Hadoop big data platform and a very cost effective solution in breaking the ERP data silos and simplifying the data architecture in the enterprise. POCs were conducted on in-house Hortonworks Hadoop data platform to validate the cluster performance for Production volumes. Based on business priorities, an initial roadmap was defined using 3 data sources including 2 SAP ERPs and Peoplesoft (OLTP systems). Development environment was established in AWS Cloud for agile delivery. The near real time data ingestion architecture for the data lake was defined using replication tools and custom SQOOP based micro-batching framework and data persisted in Apache Hive DB in ORC format. Data and user security is implemented using Apache Ranger and sensitive data stored at rest in encryption zones. Business data sets were developed in Hive scripts and scheduled using Oozie. Multiple reporting tools connectivity including SQL tools, Excel and Tableau were enabled for Self-service Analytics. Upon successful implementation of the initial phase, a full roadmap is established to extend the Finance data lake to over 25 data sources and enhance data ingestion to scale as well as enable OLAP tools on Hadoop.

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

DataWorks Summit

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

DataWorks Summit

5 Steps for Architecting a Data Lake

5 Steps for Architecting a Data Lake

5 Steps for Architecting a Data Lake

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Oracle's BigData solutions consist of a number of new products and solutions to support customers looking to gain maximum business value from data sets such as weblogs, social media feeds, smart meters, sensors and other devices that generate massive volumes of data (commonly defined as ‘Big Data’) that isn’t readily accessible in enterprise data warehouses and business intelligence applications today.

Oracle's BigData solutions

Oracle's BigData solutions

Oracle's BigData solutions

Swiss Big Data User Group

La actualidad más candente (20)

Data Lake for the Cloud: Extending your Hadoop Implementation

Data Lake for the Cloud: Extending your Hadoop Implementation

Data Lake for the Cloud: Extending your Hadoop Implementation

Data lake benefits

Data lake benefits

Data lake benefits

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Oncrawl elasticsearch meetup france #12

Oncrawl elasticsearch meetup france #12

Oncrawl elasticsearch meetup france #12

Why Data Lake should be the foundation of Enterprise Data Architecture

Why Data Lake should be the foundation of Enterprise Data Architecture

Why Data Lake should be the foundation of Enterprise Data Architecture

Hadoop Trends

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

The Big Data Journey – How Companies Adopt Hadoop - StampedeCon 2016

Hadoop data-lake-white-paper

Hadoop data-lake-white-paper

Hadoop data-lake-white-paper

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

Hadoop 2.0: YARN to Further Optimize Data Processing

Hadoop 2.0: YARN to Further Optimize Data Processing

Hadoop 2.0: YARN to Further Optimize Data Processing

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...

Incorporating the Data Lake into Your Analytic Architecture

Incorporating the Data Lake into Your Analytic Architecture

Incorporating the Data Lake into Your Analytic Architecture

Planing and optimizing data lake architecture

Planing and optimizing data lake architecture

Planing and optimizing data lake architecture

Information Virtualization: Query Federation on Data Lakes

Information Virtualization: Query Federation on Data Lakes

Information Virtualization: Query Federation on Data Lakes

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

5 Steps for Architecting a Data Lake

5 Steps for Architecting a Data Lake

5 Steps for Architecting a Data Lake

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Big Data: Architecture and Performance Considerations in Logical Data Lakes

Oracle's BigData solutions

Oracle's BigData solutions

Oracle's BigData solutions

Similar a Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

Recent research has pointed out the complementary nature of Hadoop and other data management solutions and the importance of leveraging existing systems, SQL, engineering, and operational skills, as well as incorporating novel uses of MapReduce to improve analytic processing. Come to this session to learn how companies optimize the use of Hadoop with other enterprise systems to improve overall analytical throughput and build new data-driven products. This session covers: ways to achieve high-performance integration between Hadoop and relational-based systems; Hadoop+NoSQL vs Hadoop+SQL architectures; high-speed, massively parallel data transfer to analytical platforms that can aggregate web log data with granular fact data; and strategies for freeing up capacity for more explorative, iterative analytics and ad hoc queries.

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Cloud Computing and your Data Warehouse

Cloud Computing and your Data Warehouse

Cloud Computing and your Data Warehouse

Apache Apex Meetup at Cask

Apache Apex Meetup at Cask

Apache Apex Meetup at Cask

YugaByte + PKS CloudFoundry Meetup 10/15/2018

YugaByte + PKS CloudFoundry Meetup 10/15/2018

YugaByte + PKS CloudFoundry Meetup 10/15/2018

Real Time Analytics with Dse

Real Time Analytics with Dse

Real Time Analytics with Dse

DataStax Academy

DataTorrent Presentation @ Big Data Application Meetup

DataTorrent Presentation @ Big Data Application Meetup

DataTorrent Presentation @ Big Data Application Meetup

Cybersecurity is a broad topic and many commercial products are related to it. We demonstrate a fundamental concept in network analysis: re-construction and visualization of temporal networks. Furthermore, we apply the method to describe operational conditions of a Hadoop cluster. Our experiments provide first results and allow a classification of the cluster state related to current workloads. The temporal networks show significant differences for different operation modes. In reallity we would expect mixed workloads. If such workload parameters are known, we are able to handle a-typical events accordingly - which means, we are able to create alerts based on context information, rather than only the package content. We show an end-to-end example: (1) Data collection is done via python, using the sniffer script; (2) using Apache Hive and Apache Spark we analyze the network traffic data and create the temporary network. Finally, we are able to visualize the results using Gephi in step (3). In a next step, we plan to contribute to the Apache Spot project.

PCAP Graphs for Cybersecurity and System Tuning

PCAP Graphs for Cybersecurity and System Tuning

PCAP Graphs for Cybersecurity and System Tuning

Dr. Mirko Kämpf

Internet of Things (IoT) devices are becoming more ubiquitous in consumer, business and industrial landscapes. They are being widely used in applications ranging from home automation to the industrial internet. They pose a unique challenge in terms of the volume of data they produce, and the velocity with which they produce it, and the variety of sources they need to handle. The challenge is to ingest and process this data at the speed at which it is being produced in a real-time and fault tolerant fashion. Apache Apex is an industrial grade, scalable and fault tolerant big data processing platform that runs natively on Hadoop. In this deck, you will see how Apex is being used in IoT applications and also see how the enterprise features such as dimensional analytics, real-time dashboards and monitoring play a key role. Presented by Pramod Immaneni, Principal Architect at DataTorrent and PPMC member Apache Apex, on BrightTALK webinar on Apr 6th, 2016

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via a uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be expressed in a row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across your entire organization. This talk will be co-presented by Facebook and Teradata, the two largest contributors to Presto. The talk will focus on Presto’s ability to query virtually any data source via it’s connector interface. Facebook and Teradata will present some of their use cases of Presto querying various data sources, discuss the existing connectors in Presto, and describe the anatomy of a connector.

Presto: SQL-on-anything

Presto: SQL-on-anything

Presto: SQL-on-anything

DataWorks Summit

The community for Apache Atlas and Apache Ranger, which are foundational components for Security and Governance across the Hadoop stack, has spawned a robust partner ecosystem of tools and platforms. These partner solutions build upon the extensibility offered in these platforms via open and robust APIs via integration patterns to provide innovative “better together” capabilities. In this talk, we will showcase how three of Hortonworks partners Talend, Protegrity, and Arcadia Data have effectively extended Apache Ranger and Apache Atlas frameworks to provide value added security and governance features to complement the Hadoop ecosystem. The talk will showcase partner-led demonstrations that will include how to enhance Apache Atlas lineage and metadata to cover ETL operations, how to build Apache Ranger authorizations of custom objects such as visualizations and how to enhance Apache Ranger’s data protection capabilities for encryption and masking. We will also provide a short overview of Hortonworks Gov Ready and Sec Ready programs and how partners can benefit from the certification process as part of this program.

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

DataWorks Summit

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

How to Make Hadoop Easy, Dependable and Fast

How to Make Hadoop Easy, Dependable and Fast

How to Make Hadoop Easy, Dependable and Fast

MapR Technologies

Designing & Optimizing micro batch processing system to handle multi-billion events using 100+ nodes of Cassandra , spark and Kafka - Lessons learned from the trenches Designing and Optimizing 20+ billion operations a day presents a set of complex challenges especially when the SLA is near real-time. In this presentation we will walk through our experience in building large scale event processing pipeline using Cassandra , spark streaming and kafka using 100+ nodes. We will present the Design patterns, development steps and diagnostics setups at the technology level and application level that are needed to manage the application of this scale. We also aim to present some unique problems we encountered in optimizing and operationalizing these environments. About the Speakers Ananth Ram Senior Principal / Senior Manager, Accenture Ananth Ram is a Solution Architect with over 17 years of experience in Oracle database Architecture and designing large scale applications. He was with Oracle Corp for nine years before joining Accenture as Senior Principal . As a part of Accenture, Ananth has been working on many large scale Oracle and big data initiatives in the last four years. Rich Rein Solution Architect, DataStax Rich Rein is a Solutions Architect from DataStax on Accenture team with over 30+ years as an architect, manager, and consultant in Silicon Valley's computing industry. Rumeel Kazi, Accenture Federal Rumeel Kazi is a Senior Manager in the Accenture Health & Public Service (H&PS) practice. He has over 17 years of Systems Integration implementation experience involving Oracle, J2EE platforms, Enterprise Application Integration, Supply Chain, ETL and Business Rules Management Systems. Rumeel has been working on large scale Oracle and big data application solutions since the last 5 years.

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

SQL-H a new way to enable SQL analytics

SQL-H a new way to enable SQL analytics

SQL-H a new way to enable SQL analytics

DataWorks Summit

The Briefing Room with Dr. Robin Bloor and Teradata RainStor Live Webcast October 13, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=012bb2c290097165911872b1f241531d Hadoop data lakes are emerging as peers to corporate data warehouses. However, successful data management solutions require a fusion of all relevant data, new and old, which has proven challenging for many companies. With a data lake that’s been optimized for fast queries, solid governance and lifecycle management, users can take data management to a whole new level. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor as he discusses the relevance of data lakes in today’s information landscape. He’ll be briefed by Mark Cusack of Teradata, who will explain how his company’s archiving solution has developed into a storage point for raw data. He’ll show how the proven compression, scalability and governance of Teradata RainStor combined with Hadoop can enable an optimized data lake that serves as both reservoir for historical data and as a "system of record” for the enterprise. Visit InsideAnalysis.com for more information.

First in Class: Optimizing the Data Lake for Tighter Integration

First in Class: Optimizing the Data Lake for Tighter Integration

First in Class: Optimizing the Data Lake for Tighter Integration

Inside Analysis

Opower is a fast moving energy management SaaS company that collects sensor data from nearly all of the major utilities in the United States–meaning from more than 45 million American households–along with major utilities in 5 countries throughout Europe and AsiaPac. Opower manages more than 100 billion meter reads, ranging from high frequency power data (AMI), smart thermostats data, and weather data. Currently all data at Opower is stored in HBase or Hadoop (and is notably not security sensitive). This discussion will discuss Opower’s HBase architecture, highlight potential and current uses of data in HBase, share the vision of Opower’s future projects and directions, and reveal how Opower’s big data management has allowed the company to help its utility clients save enough energy to power a city of nearly 200,000 people and save utility customers more than $70 million since only 2008!

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

In a distributed system, components running on different Operating Systems and applications written in different programming languages all work together as one single reliable system. As the complexity of these systems increase, it becomes more important to gain visibility into your environment and expose potential problems before they jeopardize your production system. RTI Connext Tools offer a rich tools suite that helps you develop applications more efficiently. They enable you to monitor, analyze and debug your complete system in operation. And they provide facilities for recording and replaying data in real-time, and logging diagnostic system data for deeper analysis and archiving. This webinar explores the many tools available for development, debugging, and testing applications in your distributed system.

Discover problems in your distributed system before it's too late

Discover problems in your distributed system before it's too late

Discover problems in your distributed system before it's too late

Real-Time Innovations (RTI)

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Data Quality in the Data Hub with RedPointGlobal

Data Quality in the Data Hub with RedPointGlobal

Data Quality in the Data Hub with RedPointGlobal

Fluentd meetup #3

Fluentd meetup #3

Fluentd meetup #3

Treasure Data, Inc.

Similar a Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari (20)

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Hadoop World 2011: Big Data Architecture: Integrating Hadoop with Other Enter...

Cloud Computing and your Data Warehouse

Cloud Computing and your Data Warehouse

Cloud Computing and your Data Warehouse

Apache Apex Meetup at Cask

Apache Apex Meetup at Cask

Apache Apex Meetup at Cask

YugaByte + PKS CloudFoundry Meetup 10/15/2018

YugaByte + PKS CloudFoundry Meetup 10/15/2018

YugaByte + PKS CloudFoundry Meetup 10/15/2018

Real Time Analytics with Dse

Real Time Analytics with Dse

Real Time Analytics with Dse

DataTorrent Presentation @ Big Data Application Meetup

DataTorrent Presentation @ Big Data Application Meetup

DataTorrent Presentation @ Big Data Application Meetup

PCAP Graphs for Cybersecurity and System Tuning

PCAP Graphs for Cybersecurity and System Tuning

PCAP Graphs for Cybersecurity and System Tuning

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

IoT Ingestion & Analytics using Apache Apex - A Native Hadoop Platform

Presto: SQL-on-anything

Presto: SQL-on-anything

Presto: SQL-on-anything

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

Partner Ecosystem Showcase for Apache Ranger and Apache Atlas

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

Why Apache Spark is the Heir to MapReduce in the Hadoop Ecosystem

How to Make Hadoop Easy, Dependable and Fast

How to Make Hadoop Easy, Dependable and Fast

How to Make Hadoop Easy, Dependable and Fast

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

Designing & Optimizing Micro Batching Systems Using 100+ Nodes (Ananth Ram, R...

SQL-H a new way to enable SQL analytics

SQL-H a new way to enable SQL analytics

SQL-H a new way to enable SQL analytics

First in Class: Optimizing the Data Lake for Tighter Integration

First in Class: Optimizing the Data Lake for Tighter Integration

First in Class: Optimizing the Data Lake for Tighter Integration

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environmen...

Discover problems in your distributed system before it's too late

Discover problems in your distributed system before it's too late

Discover problems in your distributed system before it's too late

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Modernizing Global Shared Data Analytics Platform and our Alluxio Journey

Data Quality in the Data Hub with RedPointGlobal

Data Quality in the Data Hub with RedPointGlobal

Data Quality in the Data Hub with RedPointGlobal

Fluentd meetup #3

Fluentd meetup #3

Fluentd meetup #3

Más de Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Forrester forecasts* that direct spending on the Internet of Things (IoT) will exceed $400 Billion by 2023. From manufacturing and utilities, to oil & gas and transportation, IoT improves visibility, reduces downtime, and creates opportunities for entirely new business models. But successful IoT implementations require far more than simply connecting sensors to a network. The data generated by these devices must be collected, aggregated, cleaned, processed, interpreted, understood, and used. Data-driven decisions and actions must be taken, without which an IoT implementation is bound to fail. https://hortonworks.com/webinar/iot-predictions-2019-beyond-data-heart-iot-strategy/

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Johns Hopkins - Using Hadoop to Secure Access Log Events

Johns Hopkins - Using Hadoop to Secure Access Log Events

Johns Hopkins - Using Hadoop to Secure Access Log Events

Cybersecurity today is a big data problem. There’s a ton of data landing on you faster than you can load, let alone search it. In order to make sense of it, we need to act on data-in-motion, use both machine learning, and the most advanced pattern recognition system on the planet: your SOC analysts. Advanced visualization makes your analysts more efficient, helps them find the hidden gems, or bombs in masses of logs and packets. https://hortonworks.com/webinar/catch-hacker-real-time-live-visuals-bots-bad-guys/

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

HDF 3.2 - What's New

HDF 3.2 - What's New

HDF 3.2 - What's New

With the growth of Apache Kafka adoption in all major streaming initiatives across large organizations, the operational and visibility challenges associated with Kafka are on the rise as well. Kafka users want better visibility in understanding what is going on in the clusters as well as within the stream flows across producers, topics, brokers, and consumers. With no tools in the market that readily address the challenges of the Kafka Ops teams, the development teams, and the security/governance teams, Hortonworks Streams Messaging Manager is a game-changer. https://hortonworks.com/webinar/curing-kafka-blindness-hortonworks-streams-messaging-manager/

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

The healthcare industry—with its huge volumes of big data—is ripe for the application of analytics and machine learning. In this webinar, Hortonworks and Quanam present a tool that uses machine learning and natural language processing in the clinical classification of genomic variants to help identify mutations and determine clinical significance. Watch the webinar: https://hortonworks.com/webinar/interpretation-tool-genomic-sequencing-data-clinical-environments/

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

IBM+Hortonworks = Transformation of the Big Data Landscape

IBM+Hortonworks = Transformation of the Big Data Landscape

IBM+Hortonworks = Transformation of the Big Data Landscape

Premier Inside-Out: Apache Druid

Premier Inside-Out: Apache Druid

Premier Inside-Out: Apache Druid

Accelerating Data Science and Real Time Analytics at Scale

Accelerating Data Science and Real Time Analytics at Scale

Accelerating Data Science and Real Time Analytics at Scale

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Trimble Transportation Enterprise is a leading provider of enterprise software to over 2,000 transportation and logistics companies. They have designed an architecture that leverages Hortonworks Big Data solutions and Machine Learning models to power up multiple Blockchains, which improves operational efficiency, cuts down costs and enables building strategic partnerships. https://hortonworks.com/webinar/blockchain-with-machine-learning-powered-by-big-data-trimble-transportation-enterprise/

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

For years, the healthcare industry has had problems of data scarcity and latency. Clearsense solved the problem by building an open-source Hortonworks Data Platform (HDP) solution while providing decades worth of clinical expertise. Clearsense is delivering smart, real-time streaming data, to its healthcare customers enabling mission-critical data to feed clinical decisions. https://hortonworks.com/webinar/delivering-smart-real-time-streaming-data-healthcare-customers-clearsense/

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Making Enterprise Big Data Small with Ease

Making Enterprise Big Data Small with Ease

Making Enterprise Big Data Small with Ease

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Using your data smarter and faster than your peers could be the difference between dominating your market and merely surviving. Organizations are investing in IoT, big data, and data science to drive better customer experience and create new products, yet these projects often stall in ideation phase to a lack of global data management processes and technologies. Your new data architecture may be taking shape around you, but your goal of globally managing, governing, and securing your data across a hybrid, multi-cloud landscape can remain elusive. Learn how industry leaders are developing their global data management strategy to drive innovation and ROI. Presented at Gartner Data and Analytics Summit Speaker: Dinesh Chandrasekhar Director of Product Marketing, Hortonworks

Driving Digital Transformation Through Global Data Management

Driving Digital Transformation Through Global Data Management

Driving Digital Transformation Through Global Data Management

Hortonworks DataFlow (HDF) is the complete solution that addresses the most complex streaming architectures of today’s enterprises. More than 20 billion IoT devices are active on the planet today and thousands of use cases across IIOT, Healthcare and Manufacturing warrant capturing data-in-motion and delivering actionable intelligence right NOW. “Data decay” happens in a matter of seconds in today’s digital enterprises. To meet all the needs of such fast-moving businesses, we have made significant enhancements and new streaming features in HDF 3.1. https://hortonworks.com/webinar/series-hdf-3-1-technical-deep-dive-new-streaming-features/

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface. It’s being used across industries on large amounts of data that had stored in isolation which made collaboration and analysis difficult. Join industry experts from Hortonworks and Attunity as they explain how Apache NiFi and streaming CDC technology provides a distributed, resilient platform for unlocking the value of data in new ways.

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Más de Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Getting the Most Out of Your Data in the Cloud with Cloudbreak

Johns Hopkins - Using Hadoop to Secure Access Log Events

Johns Hopkins - Using Hadoop to Secure Access Log Events

Johns Hopkins - Using Hadoop to Secure Access Log Events

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys

HDF 3.2 - What's New

HDF 3.2 - What's New

HDF 3.2 - What's New

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Curing Kafka Blindness with Hortonworks Streams Messaging Manager

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

Interpretation Tool for Genomic Sequencing Data in Clinical Environments

IBM+Hortonworks = Transformation of the Big Data Landscape

IBM+Hortonworks = Transformation of the Big Data Landscape

IBM+Hortonworks = Transformation of the Big Data Landscape

Premier Inside-Out: Apache Druid

Premier Inside-Out: Apache Druid

Premier Inside-Out: Apache Druid

Accelerating Data Science and Real Time Analytics at Scale

Accelerating Data Science and Real Time Analytics at Scale

Accelerating Data Science and Real Time Analytics at Scale

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense

Making Enterprise Big Data Small with Ease

Making Enterprise Big Data Small with Ease

Making Enterprise Big Data Small with Ease

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Webinewbie to Webinerd in 30 Days - Webinar World Presentation

Driving Digital Transformation Through Global Data Management

Driving Digital Transformation Through Global Data Management

Driving Digital Transformation Through Global Data Management

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

1. TERADATA VIEWPOINT HADOOP INTEGRATION WITH AMBARI Steve Ratay Chief Architect, Teradata Viewpoint April 2, 2013

2. Teradata Unified Data Architecture • Combination of Teradata Database, Aster, and Hadoop in a single appliance • Connectors between the different systems • Monitor the entire appliance with Teradata Viewpoint DISCOVERY Aster Teradata INTEGRATED PLATFORM DATA WAREHOUSE Connector SQL-H SQL-H Aster Connector for Hadoop Teradata Connector for Hadoop CAPTURE | STORE | REFINE 2 Copyright Teradata 2012

3. Viewpoint Architecture TD 1 Viewpoint Managed Server Data TD n Viewpoint Postgres Collection Portal Database Service Aster 1 ActiveMQ Aster n Alert Service HDP 1 Email SNMP Portlet HDP n 3 Copyright Teradata 2012

4. Monitoring Hadoop without Ambari • Different technologies and formats: > Ganglia > Nagios > JMX > Screen scraping HTML • Connection to multiple nodes: > Connectivity to all nodes > Access to several ports > Location of each service, especially when using high availability 4 Copyright Teradata 2012

5. Ambari to the Rescue! • RESTful APIs • JSON response format • Single server and port 5 Copyright Teradata 2012

6. Accessing Ambari APIs • Viewpoint uses Java and Spring • Used Spring’s RestTemplate class: > Makes HTTP call to specified URL > Binds JSON results to Java model objects • Data is collected every minute and stored in Viewpoint’s Postgres database > Use of Rewind to view previous state of system > Easy generation of trending charts and graphs 6 Copyright Teradata 2012

7. System Health 7 Copyright Teradata 2012

8. Metric Analysis 8 Copyright Teradata 2012

9. Node Monitor 9 Copyright Teradata 2012

10. Hadoop Services 10 Copyright Teradata 2012

11. Hadoop Alerts 11 Copyright Teradata 2012

12. Benefits of Using Ambari • Viewpoint delivered comprehensive Hadoop monitoring solution in a couple of months • Java and web developers focused on collecting data and building UI • Didn’t waste time dealing with intricacies of Hadoop and Hadoop monitoring 12 Copyright Teradata 2012