Big Data insights

1. Big Data
Raji Gogulapati

2. Government
• Street bump
Mobile App –
City of Boston
initiatives
• City gets real
time information
on “bump” data
Car Insurance
• More granular in
pricing
• Address more in
depth questions
Recruiting
• How to hire
better
employees and
retain?
Simple solutions to Big problems
Source: Phil Simon’s Big Data, too big to ignore

3. Consumer
driven trends
and
technologies
Data Science
Infonomics
Platforms
Sandy,
Politics,
Social Media
Big Data
Influence

4. Describe Big Data
Incomplete
Fragmented
Not precise
Dynamic
External
Unmanageable
Democratic
???

5. Data Science
Ultimate Goal – Improve Decision making
principles
Frameworks
Data analytic
thinking
• Extract patterns
• Mining for useful knowledge
• Models
• Process
• Stages
• Assess how data can improve
performance
• Understand data science
• See data oriented competitive threats
• Question

6. Big Data Advantage – Analytics and decision
management
Decision
making
Data
deluge
Techniques Solutions
Rewards

7. Techniques
Statistical Visualization
Semantics Automation
Predictive
Analytics

8. Plunge
Issues
Problems

9. Source: Forbes.com, Cloud Predictive Analytics most used to gain customer insight, 10/24/2013
Data types in a Big Data context

10. Big Data Use Cases – Financial Sector
Source – HP Sponsored white paper, Case for Big Data in the Financial Services sector, IDC Financial Insights opinion 2012

11. Big Data Use Cases – Government
Source – IBM, “Accelerate Analytics and harness Big data within government”

12. Big Data Capabilities – health care
Source: McKinsey Report titled Big Data Revolution in health care, exhibit 9

13. Big Data Analytics capabilities – travel and transportation
Source: IBM, Big Data and Analytics in Travel and Transportation white paper, figure 4
Maintenance and Engineering – asset management data,
Spec sheets, product data
Capacity and Pricing optimization

14. Data Analytics for Software Assessment/ Evaluation
Source: http://blog.sei.cmu.edu/post.cfm/data-analytics-for-open-source-software-assessment
Context and challenges
Meeting milestones
Documentation
User base
growth over
time
bugs
Developer
involvement
over time

15. Analytics - Explained
Source: Analytics 3.0 by Thomas H Davenport, HBR, Dec 2013
Analytics 1.0
Era of business intelligence, go
beyond intuition, fact based
comprehension for decision making.
Era of enterprise data warehouse.
Dominant for about 50 years.
Analytics 2.0
From about 2005 onwards,
Internet based social network firms –
Google, eBay, LinkedIn..Not only
internal, externally sourced, sensors,
public data initiatives, multi media
recordings. Innovative technologies
NoSQL, Hadoop, machine learning.
Computational and analytical skills
Analytics
3.0
Data enriched offerings for every
industry. Driven by analytics, rooted in
enormous amounts of data.
Co-existence of traditional and new.

16. Information Providers Insight Providers
Companies Capitalizing on Analytics
Essence of Analytics 3.0:
“The resolve by a company’s management to compete on analytics not only in the traditional sense
(by improving internal business decisions) but also by creating more valuable products and services”
Analytics 3.0 by Thomas H. Davenport, Dec 2013, Harvard Business Review

17. Ability to handle new varieties of data – voice, text, log files, images,
video on a large scale
Sensors and operational data gathering devices in motion to optimize
Cost savings of storage – data base to database appliance to a Hadoop
cluster
Big companies always wrestled with the data volume issues. Bigness is
not new! Variety is new!
What is different from the past?
Source: Big Data in Big companies, May 2013:

18. Big Data Techniques – explained
Sources:
Data Science for Business, Chapter 2, Business Problems and Data Science solutions
Too Big to Ignore by Philip Simon, Chapter 3, elements of persuasion: Big Data Techniques
Techniques
Statistical – Regression, A/B Testing
Data visualization - Heat Maps, Time Series analysis
Automation – Machine learning, Sensors, Nano
technology, RFID and NFC
Semantics – natural language processing, text analytics, sentiment analysis
Predictive analytics
Collaborative Filtering
Business problems to Data Mining tasks

19. BI reporting Visualization
Functional
Applications
Industry
Applications
Predictive
Analytics
Content
Analytics
Analytics solutions
Source: IBM Big Data Application layer

20. Source: Information week, 16 top big data analytics platforms, 1/30/ 2014
Top 16 Big Data Analytics Platforms

21. Platform connections
Business
platforms, Gang
of four –
Amazon, Apple,
Google,
Facebook
More businesses
setting platform
trends – Industry
wide
transformation
Netflix, LinkedIn
Third Platform –
popularized by
IDC for social,
mobile, cloud,
Big data/
analytics and
emerging
markets

22. Mainframe,
Terminals
Level 1
platforms ‘70s
Tiered
architectures
(client server –
2 tier), (’80 -
90s)
Multi tiered
architectures
(2000+ )
Social, mobile,
Cloud, Big
data/ analytics
Convergence
(2010 +)
Value shifts for the enterprise

23. Big Data Optimizations – Concept
Distributed
optimization
Parallel
optimization
Large scale
optimizations

24. Optimizations and Challenges – know how for handling bottle necks
Computational challenges

25. Myths and Overlaps
Not just another hype of data related decisions and insights – requires a new mindset

26. People – roles
Data
Scientists
Statisticians
Business
Analysis
• Find story in a data set
• Experimental,
exploratory
• Data mining
• Statistical analysis
• Predictive model
development
•
• Multi dimensional
analysis
• Visual, data discovery
References:
Davenport, T. H., & Patil, D. J. (2012)]
Harvard Business Review, October 2012, pp 70- 76

27. Big Data and Analytics technologies – supplementing RDBMS’s
Scalable
MPP Data
warehouse
Hadoop
NewSQLGraph Database
NoSQL
Reference: WHITE PAPER
Discovering the Value of a Data Discovery Platform, Sponsored by: Teradata, Dan Vesset, September 2013

28. Impact on management
New skills and new management style
References:
McAfee, A., & Brynjolfsson, E. (2012). Big Data: The Management Revolution. (cover story).
Harvard Business Review, 90(10), 60-68.
Data driven companies, evidence based decisions
look for opportunities based on Big data in every business function
Leadership, talent, technology, Organizational culture
Experimental and exploratory

29. Introductory EMC Videos – Animated
http://www.youtube.com/watch?v=eEpxN0htRKI#t=67
Big Ideas – Simplifying cluster architectures
http://www.youtube.com/watch?v=4M3cROio9vU
Big Ideas - How big is Big data?
Big ideas – Why Big Data matters
http://www.youtube.com/watch?v=rTAn1bvy8vU
Big Ideas – Demystifying Hadoop
http://www.youtube.com/watch?v=xJHv5t8jcM8
EMC – Big Ideas videos
http://www.youtube.com/playlist?list=PLD298CBF8D0908E4C&
feature=view_all

30. Conclusions
Smart
Internet of things
Predictions
Big data
evolution
RFID,
sensors, NFC
Standards
ODaF
Conclusions