Think Big, A Teradata Company Rick Farnell, Co-Founder & SVP International

1. Think Big, A Teradata Company
Rick Farnell, Co-Founder & SVP International

2. 2
Open Data Platform Support
© 2015 Think Big, a Teradata Company

3. 3
What we learned over 5 years at Think Big
Teamwork is Critical Skills Matter Celebrate Success
© 2015 Think Big, a Teradata Company

4. 4
Our hunch was right…this market will be BIG
Source: www.Indeed.com April 8, 2015
© 2015 Think Big, a Teradata Company
http://wikibon.org/wiki/v/Big_Data_Vendor_Revenue_and_Market_Forecast_2013-2017

5. 5
• 100% Big Data Focus
• Founded in 2010 with over 100 engagements across 70 clients
• Unlock client value of big data with data science and data engineering services
• Proven vendor-neutral open source integration expertise
• Agile team-based development methodology
• Think Big Academy for skills and organizational development
• Global delivery model on-site services with near shore and off shore support
Who is Think Big?
© 2015 Think Big, a Teradata Company

6. 6
Think Big Services Engagement Model
STRATEGY IMPLEMENTATION SOLUTION SUPPORT
Think Big offers end-to-end Big Data strategy, implementation and support services focused on
helping customers quickly achieve ROI on their Big Data investments
Enterprise Data Lake Software Frameworks
Big Data
Roadmap
Establish
Data Lake
Analytic
Solutions
Managed
Services
Data Lake
Optimization
© 2015 Think Big, a Teradata Company

7. 7
5 Years of Big Data Services with Industry Leaders
Think Big Clients
eCommerce
2 of Global Top 5
Retail
2 of Global Top 5
Social Networking
Global #1
Banking
4 of Global Top 10
Credit Issuer
2 of Global Top 5
Financial Data Services
2 of Global Top 5
Financial Exchanges
Global #2
Brokerage & Mutual Funds
2 of Global Top 5
Asset Management
Global #1
Semiconductor
2 of Global Top 5
Data Storage Devices
3 of Global Top 5
Disk Drive Manufacturing
Global #1
Telecommunications
2 of Global Top 5
Media & Advertising
2 of Global Top 4
Internet Transaction Security
Global #1
© 2015 Think Big, a Teradata Company

8. 8
Top 5 Learnings over 5 Years in Big Data
1. Big Data is a journey not an event.
2. Open source is great for innovation but requires engineering
sophistication in design patterns & best practices to continually scale
analytics in production.
3. Importance of metadata, lineage, data management, data
governance, data quality is critical to successful enterprise Big Data
deployments.
4. A new data platform will not resolve organizational problems.
5. The Hadoop & opensource ecosystem is moving at the speed of light,
you must be agile. Test, build, learn, repeat.
© 2015 Think Big, a Teradata Company

9. Think Big Enterprise Data Lake Case Study:
Global Manufacturing Client

10. 10
Manufacturing Client Overview
Metrics:
• Collecting >2M manufacturing/testing binary files daily across Americas and APAC Facilities
• Collecting from ~500 tables across 6 databases  tens of millions of records daily
• Over 140 analytics users to date
• Over 150 attendees participated in Big Data Platform training
Business Goals for Investment in Enterprise Data Lake:
• Provide Enterprise-wide data access for timely analytics to product quality engineers
• Establish foundation for large scale proactive manufacturing and quality analytics
• Reduce $Millions in costs of scrap waste
• Reduce $Millions in costs of containment processes
• Reduce $Millions in costs of data search parties
• Increase revenue and market share by accelerating time to market of products
© 2015 Think Big, a Teradata Company

11. 11
Engineering and Ingestion Design are Critical Day 1
1. Ingestion1. Utilize a robust Ingestion
Design with a Buffer Server
Zone
2. Packaging
2. Robust packaging many
small flies into larger files
with metadata.
3. Parse on Demand
3. Parse on demand.
Increase velocity with right
level of parsing, parse and
refine in more detail as
needed.
4. Establish Zones for Control
4. Establish Zones for
comprehensive pipeline
control, buffer, landing,
ingest, core and publish.
© 2015 Think Big, a Teradata Company

12. 12
Think Globally
EDW
EDW
5. Replication on Ingestion
5. Replicate at ingestion
with collection of key
metadata information.
6. Data Treatment Facility
6. Utilize HDFS as giant file
system, all data lands but
some will continue in pipe
to other MPP/EDW systems
etc.
7. WAN Accelerators
7. WAN accelerates the
speed transfer of ASIA
to/from North America
8. Published Data
8. Processing and format is
customized for the
audience and
consumption pattern.
© 2015 Think Big, a Teradata Company

13. 13
Governance for Production Workloads
9. Design Zones for Enterprise Governance
9. Zones will have different
retention periods and
different access patterns
and in order to have
pipeline control within your
governance strategy you
must design for this upfront.
© 2015 Think Big, a Teradata Company

14. 14
Optimize Access Patterns of your Data in Hadoop
10. Relational access pattern
10. Optimized for known
queries. Equivalent of a
Ferrari moving a bag of
groceries.
11. Hadoop access pattern
11. Recast your problem.
Hadoop optimal for large
batches of work. Optimize
hierarchical queries with
transitive closure.
Equivalent of a freight train
moving multiple
warehouses of groceries.
© 2015 Think Big, a Teradata Company

15. 15
Pipeline Management, Monitoring & Control
HDFS
12. Data Pipeline
12. End to End Pipeline
visibility is extremely
important in an Enterprise
Data Lake. Metadata
Design and use of best
practices are key in the use
of this pattern.
© 2015 Think Big, a Teradata Company

16. 16
Detailed Ingestion Design Patterns
13. Source Systems
13. Hundreds of servers &
hundreds of data streams
require expert enterprise
engineering.
14. Utilities
14. Layer in logging,
monitoring and scheduling
for complete control of
your Enterprise Data Lake.
© 2015 Think Big, a Teradata Company

17. 17
ROI for Enterprise Data Lake Manufacturing Client
Est. at $11M year 1, $30M cumulative year 2
© 2015 Think Big, a Teradata Company
Operations Engineer: a recent production issue required detailed historical testing data. Our current
systems did not have the required retention for this request. The Big Data team was able to pull and
analyze all the required data from the Big Data Platform in minutes, as opposed to 3+ weeks that we
used to take to pull the data from multiple systems and off tape archive.
Legacy Systems Enterprise Data Lake
Retention 3-6 months scattered
DBs & tape archive.
100% data online in
enterprise data lake for 3+
years.
Coverage Summaries, samples
for several data sets.
100% parametric data
captured in raw form.
Analysis Reactive, missed
quality improvement
opportunities.
Daily dashboard with larger
data sets to support
proactive improvements.

18. 18 © 2015 Think Big, a Teradata Company
Enterprise Data Lake
Information Sources
Evaluate
Source Data
Ingest
Collect & Manage
Metadata
Apply
Structure
Sequence
Compress
Automate
Protect
Prepare Data
for Ingest
Prepare Source
Metadata
Perimeter-Authentication-Authorization
InfoSec
Downstream
Applications
Dashboard
Engine
Think Big Enterprise Data Lake
Industry Leading Assets, Services and Methodology

19. 19

20. 20 © 2015 Think Big, a Teradata Company
• Think Big is expanding, bringing its focus
on open source consulting to the
international region
• An office in the UK’s London Bridge
Business district will serve as its
international hub
• Think Big is aggressively hiring a team of
data engineers, data scientists,
technology project managers and sales
leaders
• Rick Farnell, Think Big co-founder and
SVP, International will lead the
international practice
London Bridge Business District
Photo credit: Duncan Harris. Courtesy of Flickr. Creative Commons
Think Big International Expansion

21. 21 © 2015 Think Big, a Teradata Company
Dublin, Ireland Munich, Germany Mumbai, India
Think Big International Expansion: Phase 1
Think Big International Hub
London, England
Photo credits: London (Duncan Harris). Dublin (Guiseppe Milo), Munich (John Morgan), Mumbai (McKay Savage).
Courtesy of Flickr. Creative Commons

22. 22

23. 23
Dashboard Engine for Hadoop
Fast Access for Comprehensive Historical Data Stored in Hadoop
“Right” time data
Latencies under a second
Scales easily for thousands of simultaneous users
Reporting, Visualization and
Analytics
© 2015 Think Big, a Teradata Company
Visit our Think Big Booth
to see our Dashboard Engine Demo.
Enterprise Data Lake
Information Sources
Downstream
Applications

24. 24 © 2015 Think Big, a Teradata Company
Thank you
Rick Farnell
Co-Founder and SVP International, Think Big
Rick.Farnell@thinkbiganalytics.com