Hadoop World 2011 Keynote: The State of the Apache Hadoop Ecosystem
•
3 likes•2,723 views
Doug Cutting, co-creator of Apache Hadoop, will discuss the current state of the Apache Hadoop open source ecosystem and its trajectory for the future. Doug will highlight trends and changes happening throughout the platform, including new additions and important improvements. He will address the implications of how different components of the stack work together to form a coherent and efficient platform. He will draw particular attention to Big Top, a project initiated by Cloudera to build a community around the packaging and interoperability testing of Hadoop-related projects with the goal of providing a consistent and interoperable framework. Cutting will also discuss the latest additions in CDH4 and the platform roadmap for CDH.
Recommended
More Related Content
Viewers also liked
Viewers also liked (13)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
Hadoop World 2011 Keynote: The State of the Apache Hadoop Ecosystem
- 1. The State of the Apache Hadoop Ecosystem Doug Cutting Cloudera & Apache
- 2. Outline ● the ecosystem ○ why we need it ○ what it is ○ why its strong ○ how it can evolve ● highlights ○ current ○ next ● wrap up
- 3. Why are we here? Hardware has improved ● exponentially for decades ● both storage and compute We can now store and process much more! ○ yet have been slow to leverage Analyzing more data makes us smarter. ○ Norvig's Unreasonable Effectiveness of Data
- 4. The Ecosystem is the System ● Hadoop has become the kernel ○ of the distributed operating system for Big Data ○ a de-facto industry standard ● No one uses the kernel alone ● A collection of projects at Apache
- 5. Strengths of Apache Mandates diversity & transparency ○ you control your fate Insures against vendor lock-in ○ can't buy the ASF Allows competing projects ○ survival of the fittest Ecosystem as loose federation ○ lets platform evolve
- 6. What's new? ● Apache Hadoop 0.20.205 ○ append ○ security ● CDH3 ○ Mahout included ○ Avro support across components
- 7. What's next? ● Apache Hadoop 0.23 ○ HDFS ■ performance ■ scalability (federation) ■ availability (HA) ○ MR2 ● CDH4 ○ includes Hadoop 0.23 ○ BigTop-based ● S4, Giraph, Crunch, Blur, ...
- 8. Apache BigTop (incubating) Ecosystem as a project ○ integration tests Includes: ○ compatible versions ● Hadoop ○ common packaging ● HBase ○ release is a set ● Zookeeper ● Avro ● Hive Basis for CDH ● Pig ○ like Fedora is for RHEL ● Oozie ● Flume ● Mahout Community driven ● ...
- 9. Join the community Hadoop and Big Data are still young. Hardware trends will continue. Hadoop started with just two developers. Now it has hundreds. You can be the next. What do you need?