Still on IBM BigInsights? We have the right path for you

IBM Cloud / Webinar: Dec 6th, 2018 / © 2018 IBM Corporation
IBM BigInsights Migration
Webinar
—
Jessica Lee Yau
Offering Manager, Big Data
jessicalee@us.ibm.com
Nagapriya (Priya) Tiruthani
Offering Manager, Big Data
ntiruth@us.ibm.com

Please
note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and
the workload processed. Therefore, no assurance can be given that an individual user will
achieve results similar to those stated here.

Your Tour Guides
Nagapriya (Priya) Tiruthani
Offering Manager
Jessica Lee Yau
Offering Manager

IBM and Hortonworks Deliver Data Science at Scale
Make our clients competitive in their markets using
advanced analytics faster and at scale
Provides Data Science
& Machine Learning
Provides Open Hadoop
Data Platform
+
Focus on extending data science and machine learning to
analyze the data in Apache Hadoop systems

+
• #1 Rank by Gartner
2017 Data Science Magic Quadrant
• Leader in SQL technology
for Hadoop
• Leader in data and analytics solutions
for hybrid cloud
• Leader in optimized infrastructure
for Big Data servers and storage
• Leader in Hadoop
Open Source Distribution
• 1000+ customers
and 2100+ ecosystem partners
• Hadoop original architects,
developers
employed by Hortonworks
Commitment to progressing advanced analytics
through open source
Leaders in Technology with Common Goals
Consumers get the best in class open technology

Today’s Objectives
 WHY NOW is the time to
migrate from BigInsights to
Hortonworks Data Platform?
 How will IBM support
migration efforts
6

Important Dates
Event Date
IBM-Hortonworks Partnership June 2017
HDP 3.0 Available September 2018
BigInsights End of Support June 30, 2019
Hortonworks driving
technical innovation in
Hadoop
No active BigInsights
development since 2017

Deprecated Services with Migration to HDP
IOP
• System ML
• Titan
• R4ML
BigInsights
• Text Analytics
• Big R
• Bigsheets
Following components will not be available in the Hadoop stack when
migration to HDP is completed

Upgrades and New Components
1. Migrate to start
using new
components such
as Atlas, Storm,
and Zeppelin!
2. Nearly all other
components have
been upgraded –
Migrate to use the
latest component
versions of Hive,
Ambari, Ranger,
Knox, and more!
Status IOP to HDP Component IOP 4.2.5 HDP 2.6.4 HDP 3.0.1
NEW Accumulo 1.7 1.7
NEW Atlas 0.8 1
NEW Calcite 1.2 1.16
NEW DataFu 1.3 1.3
NEW Falcon 0.1
NEW Hive2 Preview 2.1 3.1
NEW Mahout 0.9.0+
NEW Storm 1.1 1.2.1
NEW Tez 0.7 0.9.1
NEW Zeppelin 0.7.3 0.8.0
New in HDP 3.0 Livy 0.5
UPGRADED Ambari 2.4.2 2.5.2 2.7.1
UPGRADED Kafka 0.10.1 0.10.1.1 1.1.1
UPGRADED Knox 0.11 0.12.1 1.0.0
UPGRADED Ranger 0.6.2 0.7 1.1
UPGRADED Slider 0.91 0.92
UPGRADED Spark 2.1 2.2 2.3.1
Upgraded in 3.0 Hadoop, Yarn 2.7.3 2.7.3 3.1.1
Upgraded in 3.0 Hbase 1.2.4 1.1.2 2
Upgraded in 3.0 Oozie 4.3 4.2 4.3.1
Upgraded in 3.0 Phoenix 4.8.1 4.7 5.0.0
Upgraded in 3.0 Sqoop 1.4.6 1.4.6 1.4.7

Migration
WHAT:
Hortonworks and IBM have worked closely together to build a smooth
migration path for customers.
HOW:
Apache Ambari 2.5.2 is used to automate the upgrade from BigInsights
to HDP - only a few manual steps are involved. The data stored in
HDFS will be persisted and all metadata will be migrated as part of the
upgrade.
ADD’L HELP:
IBM Services can be engaged for additional migration assistance
10

1. Prepare
For a smooth upgrade process, cluster should be in a healthy state before
upgrading. Start with documentation HERE .
Key Steps:
 Must have IOP 4.2 or 4.2.5 installed
 All services must be started and not in maintenance mode
 Backup key databases and configurations
 Besides Ambari configuration, and Ambari, Hive, Ranger, Oozie and Big SQL
database backups, no other data backups are required for the migration.
The data stored in HDFS will be persisted and all metadata will be migrated
as part of the upgrade.

2. Upgrade Ambari
Ambari 2.5.2 will manage and automate
the rest of the migration process

3. Remove Value-adds
4. Remove Other Services and Components

5. Register and Install HDP
6. Upgrade to HDP
Register HDP HDP installed
side-by-side w/
existing IOP
Perform Cluster
Validation tests
Upgrade to
HDP Complete!
Ambari express wizard
auto updates config
changes and
packages

7. Upgrade and Finalize Db2 Big SQL
As a final step in the migration process, Big SQL must be upgraded.
From V5.0 onwards, Db2 Big SQL is installed using stack
extensions which delinks from Hadoop core components
Details on install/upgrade:
https://www.ibm.com/support/knowledgecenter/en/SSCRJT_5.0.1/com.ibm.swg.im.big
sql.install.doc/doc/hdp_valaddinst.html

Support

IBM Db2 Big SQL
IBM Db2 Big SQL, an advanced SQL engine on Hadoop,
supercharges your analytical workloads on data lakes with
no vendor lock-in.
The core capabilities of Db2 Big SQL focusses on data virtualization, SQL
compatibility, scalability, performance, and of course enterprise
security/governance, making it a desirable query engine to seek insights from
disparate data sources including Hadoop

Modern Data Warehouse Solution
Data Hub Design
Hortonworks HDP
(Compute and Storage Platform)
Db2 Big SQL
(High performance, Scalable, Complex queries, Data
virtualization, SQL compatibility, Spark integration )
Hive LLAP
(Fast and Scalable SQL)
HBase
(Key Value pair)
EDW
Weblog
Sensor
Clickstream
HDP/ HDF
Kafka - message
Sqoop - structure
Ni Fi - data flow
Storm - stream IGC , Big Quality, Big Match
(Data quality and governance for Hadoop and non-Hadoop data lake)
Data sources Ingestion Query processing with security Visualization
Ranger / Atlas
(Governance & Security)
Interactive BI
And
Cognitive tools
No SQL
Unstructured,
social media
RDBMSETL - IBM
BigIntegrate
Db2 Big SQL
[Load, Insert]

19IBM Cloud / © 2018 IBM Corporation
Right Tool for the Right SQL Workload
Hive Db2 Big
SQLFast ingest
ACID/MERGE capabilities
Complex queries
Streaming analytics
All open source file formats
Application portability
Data virtualization
Complex Queries with
high Concurrency
Query Hive & HBase
Great for exploratory
BI Data Analytics and streaming
analytics
Ideal for complex BI Data
Analytics and enterprise-level
production workloads
Hive & Db2 Big SQL can co-exist and complement each other
in a cluster

Value Delivered to Customers
Db2
Big SQL
No vendor
lock-in
Query siloed
data across
organization
Combine
streaming
data with
data at rest
Reporting
with BI tools
Operationali
ze ML
model with
SQL
Accelerated
reporting
using MQTs
for federated
data
Reuse
applications
& skills after
data offload
to Hadoop
• SQL compatibility with Oracle, Db2,
Netezza
• Applications / reports can be easily
ported to Hadoop
• Accelerated analytical
reports for historical data &
federated data
• Faster response for BI
• Invoke Big SQL directly from
Notebooks and make it easy for data
scientists to wrangle data
• Invoke Spark models directly from
Big SQL – make it easy for data
engineers to operationalize the
model
• BI tools (Tableau, Birst, etc.) have bad
performance when put directly on Hadoop
• Generate complex and star schema queries
• Enrich data lake with social media
data
• Add social sentiment data, click
stream data, log data or
unstructured data
• Overall provides a richer 360
degree view of customer
• Federation / Virtualization
• Avoid forced consolidation into Hadoop
• Make use of Hadoop infrastructure
• Centralized security & governance
• Separation of Compute from Hadoop storage (data is always
in Hadoop)
• Fully ANSI SQL engine – queries and skills can easily be
reused

RESOURCE UTILIZATION:
1.6x FEWER CPU CYCLES USED
PERFORMANCE: 6-streams
Db2 Big SQL 2.5X FASTER
HADOOP-DS @ 10TB
85 COMMON QUERIES
WORKING COMPLIANT QUERIES:
6-streams
WORKLOAD
SCALE FACTOR: 10 TB
FILE FORMAT: ORC (ZLIB
COMPRESSION)
CONCURRENCY: 6 STREAMS
QUERY SUBSET: 85 QUERIES
STACK
INTERESTING FACTS
FASTEST QUERY
2.6X FASTER (Db2 Big SQL: 3.1
SEC, HIVE: 8.1 SEC)
SLOWEST QUERY
1.9X FASTER (Db2 Big SQL: 6374
SEC, HIVE: 11830 SEC)
Db2 Big SQL FASTER FOR
84% OF QUERIES RUN
Query Performance at a Glance – Db2 Big SQL & Hive LLAP
HIVE:
HDP 2.6.1
HIVE 2.1 LLAP
BIG SQL:
HDP 2.6.4
Db2 Big SQL
5.0.3
PERFORMANCE: 1-stream
Db2 Big SQL 2.0X FASTER

Performance using MQTs on Star Schema Benchmark Queries
Quick
metric
queries
Product
insight
queries
Custome
r insight
queries
Using Scale Factor 1000, tested 13 queries that join 1 fact with 4 dimension tables6 Billion Lineitems &
30 Million Customers rows
Get sub-second response time for tables with Star Schema architecture
Response time in secs
Query performance
on non-MQT table

Recap: WHY and HOW
1 2 3 4 5 6 7
7 Easy Migration Steps:
Reasons to Migrate:
- Access the latest Innovations around Hadoop
- Centralized resource management
- Centralized Security and Governance
- Ability to bring AI and Machine Learning
- Empower SQL users and applications with IBM Db2 Big SQL

THINK 2019
February 12 – 15
San Francisco, CA
Register today
Agenda

Questions?

Thank you

© 2018 International Business Machines Corporation. No part of this
document may be reproduced or transmitted in any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed “as
is” without any warranty, either express or implied. In no event, shall IBM
be liable for any damage arising from the use of this information,
including but not limited to, loss of data, business interruption, loss of
profit or loss of opportunity. IBM products and services are warranted per
the terms and conditions of the agreements under which they are
provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled,
isolated environments. Customer examples are presented as illustrations of
how those
customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
References in this document to IBM products, programs, or services does
not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared
by independent session speakers, and do not necessarily reflect the views
of IBM. All materials and discussions are provided for informational
purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal
requirements and to obtain advice of competent legal counsel as to
the identification and interpretation of any relevant laws and regulatory
requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not
provide legal advice or represent or warrant that its services or products
will ensure that the customer follows any law.
Notices and disclaimers

Information concerning non-IBM products was
obtained from the suppliers of those products, their
published announcements or other publicly
available sources. IBM has not tested
those products about this publication and cannot
confirm the accuracy of performance, compatibility
or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM
products should be addressed to the suppliers of
those products. IBM does not warrant the quality of
any third-party products, or the ability of any such
third-party products to interoperate with IBM’s
products. IBM expressly disclaims all warranties,
expressed or implied, including but not limited to,
the implied warranties of merchantability and fitness
for a purpose.
The provision of the information contained herein is
not intended to, and does not, grant any right or
license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM products
and services used in the presentation] are trademarks of International
Business Machines Corporation, registered in many jurisdictions
worldwide. Other product and service names might be trademarks of IBM
or other companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml.
.
Notices and disclaimers continued

Still on IBM BigInsights? We have the right path for you

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Still on IBM BigInsights? We have the right path for you

Similar a Still on IBM BigInsights? We have the right path for you (20)

Más de ModusOptimum

Más de ModusOptimum (11)

Último

Último (20)

Still on IBM BigInsights? We have the right path for you