The past year was punctuated by significant advancements in Apache Hadoop and increasingly wider adoption of Hadoop technology across the enterprise. Companies are continuing to use Hadoop in exciting new ways to better serve their customers, inform product development and drive operational efficiency like never before. Join Mike Olson, founder and CEO of Cloudera, as he shares his twelve major predictions for Hadoop in 2012. He will also unveil predictions from key industry analysts.
Olson will discuss predictions for:
- Where new opportunities for Hadoop will be found within the enterprise
- How new projects being developed for and on Apache Hadoop will expand data analysis capabilities
- Ways that Apache Hadoop will help companies solve short term and long term business challenges
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Hadoop Twelve Predictions for 2012
1. Presenter: Mike Olson, CEO, Cloudera
Thursday, December 22, 2011
11:00 am PT, 2:00 pm ET
Hadoop: Twelve Predictions for 2012
2. Looking Back at Apache Hadoop in 2011
• 2011 was the year that Apache Hadoop became part of mainstream
dialogue, and industry interest in Hadoop exploded.
– VC firms and investors continued to make significant commitments to Hadoop,
like Accel Partners’ $100M Big Data Fund.
– Big corporate players such as EMC, IBM, Microsoft and Oracle announced their
entry into the Hadoop space. A handful of Hadoop-focused startups also
launched this year.
– Hadoop-related inquiries into major analyst firms moved from “What exactly is
Hadoop?” to “Which vendors offer robust Hadoop solutions?”
• Hadoop adoption increased within enterprises and new industry use
cases emerged.
• With the uptick of interest in Hadoop, there was a notable increase
in demand for skilled Hadoop workers and data scientists.
2
4. Industry Analysts Predict…
“We predict that through 2015, organizations integrating high-value,
diverse, new information types and sources into a coherent
information management infrastructure will outperform their industry
peers financially by more than 20%.”
— Merv Adrian, Research VP, Information Management, Gartner
4
5. Industry Analysts Predict…
“The core Hadoop platform will mature rapidly.
“While I question some of the Hadoop industry's marketing, I have
an optimistic view toward the community's plans for maturing core
Hadoop. There's low-hanging technical fruit. There's understanding
of the need. There's money. There are great engineers. There are
partners with market and technical understanding. The Hadoop
engineering stars are favorably aligned.”
— Curt Monash, Analyst, Monash Research
5
6. Industry Analysts Predict…
“1. Big Data was the term du jour for 2011. In 2012, we’ll move away from this term, as
ANY DATA will be of paramount importance for enterprises. At the same time, IT
resources are scarce, and quite frankly, most IT organizations are trying to simplify their
IT environments and look at it more strategically and holistically. Intelligent management
will be key, as enterprises look to analyze any data, at any given time (and as near real-
time as possible), and management tools will need to determine, in the most efficient
manner, where the HW/SW resources must come from, whether it’s for structured, semi-
structured, or unstructured…or any combination thereof. This demand will push the
ecosystem to continue to grow and deliver the necessary solutions to the market.
“ 2. Data protection, security are key areas that have not been adequately
addressed…and will become important to do so as enterprises think about leveraging
cloud for their analytics.
“3. ‘Big Data as a service’ the new term du jour. Enterprises will look to leverage both
their own resources for sensitive datasets and cloud for others…service providers will
become players in delivering these services.”
— Vanessa Alvarez, Analyst, Infrastructure & Operations, Forrester
6
7. Industry Analysts Predict…
“1. We will see a major shift of current experimental Hadoop users (the
bulk of current users) to mainstream production applications that will
ultimately drive the need for more enterprise-class, advanced data
management features. While these applications will be limited in scope
and moderate in cluster size, the business requirements will drive the
need for more options related to HA, advanced data protection and
remote-site failover.
“2. We will see more IT organizations initially considering Hadoop as an
alternative lower cost online storage for Big Data – essentially a
cheaper data store (versus in relational databases with associated license
and SAN attached storage) in preparation for future data processing and
analytics tasks.”
— Julie Lockner, Senior Analyst and Vice President, Data
Management Solutions, ESG
7
8. Industry Analysts Predict…
“In 2011, the federal markets with the most dramatic growth in
Hadoop were associated with counter terrorism missions and law
enforcement. Fraud detection was also an important market, as was
the use of Hadoop-centric solutions for search needs.
“We expect 2012 will result in growth in federal heath missions and
in military logistics. Research and interest by agencies indicates
health related solutions will leverage Hadoop for predicting
treatment outcomes based on patient data and historical medical
use. We expect military logistics use cases to provide DoD with new
efficiencies in optimizing supply chains and predicting needs before
they arise.”
— Bob Gourley, Founder and CTO, Crucial Point LLC
8
9. Industry Analysts Predict…
“We believe that during 2012, enterprise distributions of Hadoop will
mature enough that enterprises will accelerate production
deployments and begin to yield tangible organizational value.”
— Ben Woo, VP, Storage and Big Data, IDC
9
11. #12 The Open Source Platform Expands
• New contributions will go mostly to existing and new packages that
complement Hadoop. In 2011, nearly 70% of all new work was
outside the core project.
• New and incubator projects like Flume, Bigtop and Crunch will
graduate to critical pieces of the Hadoop platform. This is the
continued “Linuxification” of Hadoop.
11
2006 2007 2008 2009 2010 2011
Core
Hadoop
as % of
New
Contribs
Nothing but
Hadoop
… plus 11… plus 8…plus 6Hadoop
plus 3
100% 100%
37%
58%
37% 31%
12. #11 Vendor Investment Accelerates
• Every significant server, storage and data infrastructure vendor will
have a clearly-articulated Hadoop strategy by end of year.
• Product development dollars will go to:
– Contributions to the open source platform
– Integration of the open source platform with existing IT infrastructure
• Best-of-breed, rather than single-vendor, deployments will dominate
in 2012 (and for at least three years following).
12
The Cloudera Connect partner program
has added at least one new member every
day since its launch in 3Q11.
13. #10 The Platform Grows Up
• Key issues that worry enterprise users will be addressed.
– High availability guaranteed by improvements to storage infrastructure
• No single point of failure!
– Dramatic performance improvements for storage and analytical workloads
• Integration with existing data center infrastructure will become complete.
• Computer science PhD will no longer be required to operate Hadoop in
production.
13
14. #9 Hadoop Stops Being Cool
• Fascination with the technology ebbs and platform novelty subsides.
• New focus: What problems can Hadoop solve for me, and how do
I put together a complete suite of products – stack, analytics,
visualization – that I can deploy and use easily?
14
(This is a lie. Hadoop will remain cool.)
15. #8 It’s All About Apps
• Existing BI and analytics products will connect to Hadoop, unlocking
it to a large audience of business and technical professionals.
• New offerings from established vendors and emerging companies
will be designed from the ground up to take advantage of rich data,
powerful algorithms and new techniques for data visualization. This
represents a major expansion of BI and analytics into new data and
problem domains.
15
16. #7 Chasm? What Chasm?
• Hadoop adoption by mainstream enterprises will accelerate.
• Smaller companies are seeing the value of Big Data, too, and will
need the tools to analyze it.
• Deployments expand beyond single use cases inside departments to
the broader enterprise.
• It's not how much data you have (PB, TB) but what kind of data you
have and what you do with it.
16
Financial ServicesTelecommunications Life SciencesRetailGovernment
Platform maturity and app proliferation will drive broad adoption.
17. #6 Here Comes Everybody
• Small and emerging companies,
not widely viewed as having Big
Data problems, will discover
ways to capture and analyze
information in new ways, just like
their larger counterparts.
• Cloud deployment of Hadoop will
accelerate as a result of this
trend – small companies and
startups have little sunk
infrastructure cost and are cash-
conscious, so will deploy on
hosted platforms faster than will
their peers at large enterprises.
17
• With the creation of the Accel Big Data Fund and other VC firms’
interest in Big Data and Hadoop, we will see the launch of numerous
Hadoop-focused startups in 2012.
18. #5 Opportunity Expands: Security, Storage, Efficiency
• As use of Hadoop in core workloads – storage, data processing and
analytics – matures inside enterprises, they will begin to look at it as
a more general-purpose platform able to attack other business
problems.
• Improvements in platform security, access control and audit logging
means that some compliance workloads will begin to migrate to
Hadoop in the second half of the year.
• Improvements in the open source storage layer mean that storage
gets cheaper still, making Hadoop attractive as an online archive for
data previously written to tape and stored in caves.
• Better measuring and management of resource consumption will
lead to improvements in computational efficiency.
• The variety of problems that Hadoop addresses will expand.
18
19. #4 Skilled People Get Easier to Find
• Professional training courses aimed at developers, operators and
users will be widely available and very popular in 2012 and beyond.
– Finding people who can deploy, configure and use Hadoop has been
difficult in 2011.
• Systems integrators and consulting firms will establish Hadoop-
focused practices around the world.
19
20. #3 Data Science Gets Hotter
• Widespread use of Hadoop in new industries, aimed at previously-
intractable problems, will create a new role in the organization:
“Data Scientist.”
– Mathematical and statistical expertise
– Programming skill
– A passion for data and an understanding of the business
• Industry and academia will collaborate to produce this new class
of expert.
20
21. #2 The Demise of the “Hadoop-Killer”
• Second and third quarters of 2011 saw several vendors announce
“Hadoop killers” – legacy products, deployed in niche markets, that
they wanted to use to capitalize on the opportunity that Big Data
presents.
• In the fourth quarter of 2011, Microsoft announced it was
discontinuing its investment in Dryad. Dryad was, simply, the most
interesting alternative to Hadoop in the market. Instead, the
company is investing in Hadoop.
• The vendors offering alternatives to Hadoop will follow Microsoft’s
lead. These products have no real traction in the Big Data market
and will quietly disappear by the end of 2012.
21
22. #1 Forks Begin to Fade
• The open source Hadoop platform will be robust, scalable and reliable
enough to take on any enterprise Big Data workload.
• As a result, proprietary forks – MapReduce grafted onto legacy-vendor SQL
databases, appliances that replace the storage layer with proprietary file
systems and others – will see their differentiation erode or disappear.
• Buyers will elect to deploy the open source platform once it demonstrates
feature parity or superiority with these forked products.
22
Commercial opportunities
will constrict for vendors of
forked products.
It will take several years
for these offerings to
disappear altogether.
23. #0 Bonus Prediction! M&A Consolidation Begins
• Large vendors lacking product and expertise in Hadoop will survey
the market for teams and technologies that can close those gaps.
• Specialist vendors, especially those that have struggled to grow or
with positioning and differentiation challenges, will hire investment
bankers to seek acquirers.
• These early, small acquisitions, combined with an expanding market
and accelerated demand for big data solutions, will drive valuations
for new and existing independent companies.
Buyer meets seller. Nature follows its course.
23
24. 24
We appreciate your time
and interest.
For additional information:
cloudera.com
+1 (888) 789-1488
sales@cloudera.com
twitter.com/
cloudera
facebook.com/
cloudera
Notas del editor
Hadoop-related inquiries into major analyst firms moved from “What exactly is Hadoop?” to “Which vendors offer robust Hadoop solutions?”: (This noted in a blog post by Forrester analyst James Kobielus)