Let's Talk Operations! (Hadoop Summit 2014)

•

0 likes•1,768 views

These are the introductory slides I used (in some form or another) for the Let's Talk Operations! sessions for the 2014 Hadoop Summits. No video for this one!

Technology

Let’s Talk Operations!
Allen Wittenauer!

Twitter: @_a__w_
Email: aw @ apache.org!

How many individual
grids should I have?

One big
grid
Grid per
project
• Pros!
• Lower ops overhead!
• One location for all data!
• Cons !
• Dev and Prod on one
system
• Pros!
• Capacity planning per project!
• Cons !
• More headcount to maintain!
• Multiple copies of data!
• Data ingress is a mess

ETL
Dev Prod
Base ETL Pull
Event Feeds
Database Feeds
Base ETL Pull
Base ETL Pull
Post-Processed
Data

How do I solve some
common distcp issues?

• Common issues!
• Version incompatibilities!
• Network bandwidth consumption!
!
• Some tricks!
• Use WebHDFS!
• All modern versions support it!
• Read and write in both directions!
• Create a separate queue with hard limits!
• Pull from larger, push from smaller

Q&A
Allen
Wittenauer

Twitter:
@_a__w_
Email:
aw
@
apache.org

20 GB /, ... 200 GB task space (rest) HDFS
• root partitioning
!
!
!
!
!
• non-root partitioning
5 GB
swap
200 GB task space (rest) HDFS

What's hot

You sit on a big pile of data and want to know how to leverage it in your company? Interested in use-cases, examples and practical demos about the full Hadoop stack? Looking for big-data inspiration? In this talk we will cover: - Use-cases how implementing a Hadoop stack in TheNewMotion drastically helped us, software engineers, with our everyday challenges. And how Hadoop enables our management team, marketing and operations to become more data-driven. - Practical introduction into our data warehouse, analytical and visualization stack: Apache Pig, Impala, Hue, Apache Spark, IPython notebook and Angular with D3.js. - Easy deployment of the Hadoop stack to the cloud. - Hermes - our homegrown command-line tool which helps us automate data-related tasks. - Examples of exciting machine learning challenges that we are currently tackling - Hadoop with Azure and Microsoft stack.

Hadoop in Practice (SDN Conference, Dec 2014)

Marcel Krcah

Hadoop and HBase make it easy to store terabytes of data, but how do you scale your search mechanism to sift through these mountains of bits and retrieve large result sets in a matter of milliseconds? The Solr search server, based on Lucene, provides a scalable querying capability that nicely complements HBase. In this webinar, Rod Cope uses OpenLogic's production Solr and Hadoop environment as a case study on how you can handle rapid fire queries against terabytes of data, primarily through a combination of index sharding and fault-tolerant load balancing.

Real-time searching of big data with Solr and Hadoop

Rogue Wave Software

Hadoop, HBase, and friends are built from the ground up to support big data, but that doesn't make them easy. Just like with any other relatively new and complex technologies, there are some rough edges and growing pains to manage. We've learned some hard lessons while deploying HBase tables containing billions of rows and dozens of terabytes on OpenLogic's Hadoop infrastructure. In this webinar, Rod Cope discusses some of the "gotchas" you might run into when deploying Hadoop and HBase in your own private cloud and how to avoid them.

Top 10 lessons learned from deploying hadoop in a private cloud

Rogue Wave Software

Have you ever thought that your SSD storage just doesn't seem anywhere near as fast as advertised? What if I told you that a couple of small changes to your Linux servers had the potential to more than double the amount of throughput your MySQL servers could handle while simultaneously reducing query response time by around 50 percent? It's not a dream, Neo. If you're still in a traditional data center and/or your MySQL footprint lives on spinning rust, then you might want to take the blue pill, attend a different session, and believe whatever you want to believe. However, if your server footprint runs inside Amazon Web Services or if you use SSDs in any fashion, I invite you to take the red pill and come with me on a journey down the rabbit hole, where we'll discuss such things as the Linux block IO driver, CPU starvation, and interrupt handling, using real-world query performance data from the primary MySQL data stores at Pinterest to illustrate the effect.

All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...

Ernie Souhrada

Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014

NoSQLmatters

File Context

Hadoop User Group

This talk will start with an overview of Go, then dive into some examples of using it to work with Postgres. We'll show basics like running queries, then demonstrate how Go makes difficult things easy, such as inspecting Postgres's TCP wire protocol and providing retry mechanisms and monitoring for restores (including fun stories, tips and tricks). Join us and see why Go - with it's powerful concurrency primitives and unbeatable performance - can be one of the most powerful tools in your toolbox.

Gophers Riding Elephants: Writing PostgreSQL tools in Go

AJ Bahnken

The next major version - 0.96- of Apache HBase have several new features. The "Singularity", because you will have to start and stop your cluster to upgrade to 0.96. 0.96 requires Apache Hadoop 1.0.0 at least, and supported on Hadoop 2.0.0 as well. 0.96 uses protobufs all the time. All of its serializations to ZooKeeper, to the filesystem, and over rpc are protobufs. It runs on JDK7. Metrics have been edited and converted to use Hadoop Metrics2. It has HBase Snapshots and PrefixTreeCompression, etc. This presentation captures a high-level overview of what's new in HBase 0.96.

October 2013 HUG: HBase 0.96

Yahoo Developer Network

DrupalCon 2011 Highlight

Supakit Kiatrungrit

The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we’ve encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating played song events for financial reports, to our current 900 node cluster that plays a large role in many features that you see in our application today. Two members of Spotify’s Hadoop ‘squad’ will weave in war stories, failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and talk about how that architecture has evolved. We’ll talk about how and why we use a number of tools, including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive w/ Tez to build features and provide analytics; and Snakebite and Luigi, two in-house tools created to overcome common frustrations.

The Evolution of Hadoop at Spotify - Through Failures and Pain

Rafał Wojdyła

Flexible and Real-Time Stream Processing with Apache Flink

DataWorks Summit

NYC HUG - Application Architectures with Apache Hadoop

markgrover

Application architectures with Hadoop – Big Data TechCon 2014

hadooparchbook

Using Morphlines for On-the-Fly ETL

Cloudera, Inc.

An over-ambitious introduction to Spark programming, test and deployment. This slide tries to cover most core technologies and design patterns used in SpookyStuff, the fastest query engine for data collection/mashup from the deep web. For more information please follow: https://github.com/tribbloid/spookystuff A bug in PowerPoint used to cause transparent background color not being rendered properly. This has been fixed in a recent upload.

How to build your query engine in spark

Peng Cheng

Spark is by its nature very fault tolerant. However, faults, and application failures, can and do happen, in production at scale. In this talk, we’ll discuss the nuts and bolts of fault tolerance in Spark. We will begin with a brief overview of the sorts of fault tolerance offered, and lead into a deep dive of the internals of fault tolerance. This will include a discussion of Spark on YARN, scheduling, and resource allocation. We will then spend some time on a case study and discussing some tools used to find and verify fault tolerance issues. Our case study comes from a customer who experienced an application outage that was root caused to a scheduler bug. We discuss the analysis we did to reach this conclusion and the work that we did to reproduce it locally. We highlight some of the techniques used to simulate faults and find bugs. At the end, we’ll discuss some future directions for fault tolerance improvements in Spark, such as scheduler and checkpointing changes.

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...

Spark Summit

Coscup

Giivee The

Hadoop operations-2015-hadoop-summit-san-jose-v5

Chris Nauroth

Kafka and Hadoop at LinkedIn Meetup

Gwen (Chen) Shapira

Simplified Cluster Operation & Troubleshooting

DataWorks Summit/Hadoop Summit

What's hot (20)

Hadoop in Practice (SDN Conference, Dec 2014)

Real-time searching of big data with Solr and Hadoop

Top 10 lessons learned from deploying hadoop in a private cloud

All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...

Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014

File Context

Gophers Riding Elephants: Writing PostgreSQL tools in Go

October 2013 HUG: HBase 0.96

DrupalCon 2011 Highlight

The Evolution of Hadoop at Spotify - Through Failures and Pain

Flexible and Real-Time Stream Processing with Apache Flink

NYC HUG - Application Architectures with Apache Hadoop

Application architectures with Hadoop – Big Data TechCon 2014

Using Morphlines for On-the-Fly ETL

How to build your query engine in spark

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...

Coscup

Hadoop operations-2015-hadoop-summit-san-jose-v5

Kafka and Hadoop at LinkedIn Meetup

Simplified Cluster Operation & Troubleshooting

Viewers also liked

Apache Hadoop Shell Rewrite

Allen Wittenauer

Apache Hadoop for System Administrators

Allen Wittenauer

Apache Yetus: Intro to Precommit for HBase Contributors

Allen Wittenauer

Presentation given at Apache: Big Data and ApacheCon North America 2016. "In this time of rapidly growing software projects and software capabilities, where it is expected for “software to eat the world,” there is still a huge challenge going from source code to a tested, fully functional release. This is the “last mile problem,” ensuring that vision and coding become real, deployable software. To help address this problem, members of the extended Apache Hadoop/”big data” ecosystem have joined forces to create tools that reduce the burden of pre-commit testing, release note compilation and interface documentation. In this talk, Allen Wittenauer, a PMC member of the Apache Yetus project, will discuss the various components that make up the Yetus toolset, as well as how Apache Hadoop and other projects are using Apache Yetus to improve release quality. "

Apache Yetus: Helping Solve the Last Mile Problem

Allen Wittenauer

Hadoop Operations at LinkedIn

Allen Wittenauer

Hadoop Performance at LinkedIn

Allen Wittenauer

Viewers also liked (6)

Apache Hadoop Shell Rewrite

Apache Hadoop for System Administrators

Apache Yetus: Intro to Precommit for HBase Contributors

Apache Yetus: Helping Solve the Last Mile Problem

Hadoop Operations at LinkedIn

Hadoop Performance at LinkedIn

Similar to Let's Talk Operations! (Hadoop Summit 2014)

VoltDB and Erlang: two very promising beasts, made for the new parallel world, but still lingering in the wings. Not only are they addressing todays challenges but they are using parallel architectures as corner stone of their new and surprising approach to be faster and more productive. What are they good for? Why are we working to team them up? Erlang promises faster implementation, way better maintenance and 4 times shorter code. VoltDB claims to be two orders of magnitude faster than its competitors. The two share many similarities: both are the result of scientific research and designed from scratch to address the new reality of parallel architectures with full force. This talk presents the case for Erlang as server language, where it shines, how it looks, and how to get started. It details Erlang's secret sauce: microprocesses, actors, atoms, immutable variables, message passing and pattern matching. (Note: for a longer version of this treatment of Erlang only see: Why Erlang? http://www.slideshare.net/eonblast/why-erlang-gdc-online-2012) VoltDB's inner workings are explained to understand why it can be so incredibly fast and still better than its NoSQL competitors. The well publicized Node.js benchmark clocking in at 695,000 transactions per second is described and the simple steps to get VoltDB up and running to see the prodigy from up close. Source examples are presented that show Erlang and VoltDB in action. The speaker is creator and maintainer of the Erlang VoltDB driver Erlvolt.

VoltDB and Erlang - Tech planet 2012

Eonblast

Introduction to Cassandra and CQL for Java developers

Julien Anguenot

Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...

Ceph Community

Fixing twitter

Roger Xia

Fixing_Twitter

liujianrong

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...

smallerror

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...

xlight

Big Data Developers Moscow Meetup 1 - sql on hadoop

bddmoscow

Liferay & Big Data Dev Con 2014

Miguel Pastor

What ya gonna do?

CQD

PhegData X - High Performance EBS

Hanson Dong

Brian Brownlow is an experienced senior analyst programmer for Mayo Clinic. He is made a workshop presentation at the 2014 BDPA Technology Conference on the topic, 'Big Data Implementation - Mayo Clinic Case Study'. This presentation will show part of the Mayo Clinic story on the embarking of an exploration of the application of `Big Data' technologies. `Big Data' is seen as one set of tools that can be used to enhance medical research, medical education and practice management. Mayo Clinic is always searching for better, faster and cheaper ways to use its data to improve patient care and sustain financial outcomes in a challenging reimbursement environment. Our approach uses several components that are open source and combines them with data from various sources to provide information to decision makers in near real time. We have created a center of `Big Data' excellence using in-house staff and vendor engagements. `Big Data' is one element of our Enterprise Data Trust framework.

A Mayo Clinic Big Data Implementation

BDPA Education and Technology Foundation

Inside Wordnik's Architecture

Tony Tam

Five Years of EC2 Distilled

Grig Gheorghiu

SQL Server Konferenz 2014 - SSIS & HDInsight

Tillmann Eitelberg

Elixir Programming Language 101

Around25

Petabyte scale on commodity infrastructure

elliando dias

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...

BIWUG

Why ruby and rails

Reuven Lerner

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Bob Pusateri

Similar to Let's Talk Operations! (Hadoop Summit 2014) (20)

VoltDB and Erlang - Tech planet 2012

Introduction to Cassandra and CQL for Java developers

Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...

Fixing twitter

Fixing_Twitter

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...

Big Data Developers Moscow Meetup 1 - sql on hadoop

Liferay & Big Data Dev Con 2014

What ya gonna do?

PhegData X - High Performance EBS

A Mayo Clinic Big Data Implementation

Inside Wordnik's Architecture

Five Years of EC2 Distilled

SQL Server Konferenz 2014 - SSIS & HDInsight

Elixir Programming Language 101

Petabyte scale on commodity infrastructure

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...

Why ruby and rails

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Recently uploaded

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

How to convert PDF to text with Nanonets

naman860154

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Choosing the right accounts payable services provider is a strategic decision that can significantly impact your business's financial performance and operational efficiency. By considering factors such as expertise, range of services, technology infrastructure, scalability, cost, and reputation, businesses can make informed decisions and select a provider that aligns with their unique needs and objectives. Partnering with the right provider can streamline accounts payable processes, drive cost savings, and position your business for long-term success. https://katprotech.com/accounts-payable-and-purchase-order-automation/

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Katpro Technologies

A Call to Action for Generative AI in 2024

Results

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Slack Application Development 101 Slides

praypatel2

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Microsoft's Threat Matrix for Kubernetes helps organizations understand the attack surface a Kubernetes deployment introduces to their environments. This ensures that adequate detections and mitigations are in place. By covering over 40 different attacker techniques, defenders can learn about Kubernetes-specific mitigations and controls to deploy to their environments. In this session, we will explore the MS-TA9013 Host Path Mount technique, which is commonly used by attackers to perform privilege escalation in a Kubernetes cluster. Attendees will learn how attackers and defenders can: * Escape the container's host volume mount to gain persistence on an underlying node * Move laterally from the underlying node into the customer's cloud environment * Analyze Kubernetes audit logs to detect pods deployed with a hostPath mount * Deploy an admission controller that prevents new pods from using a hostPath mount

Breaking the Kubernetes Kill Chain: Host Path Mount

Puma Security, LLC

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Recently uploaded (20)

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Tata AIG General Insurance Company - Insurer Innovation Award 2024

How to convert PDF to text with Nanonets

Driving Behavioral Change for Information Management through Data-Driven Gree...

Boost Fertility New Invention Ups Success Rates.pdf

Handwritten Text Recognition for manuscripts and early printed texts

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

A Call to Action for Generative AI in 2024

Data Cloud, More than a CDP by Matt Robison

Slack Application Development 101 Slides

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Presentation on how to chat with PDF using ChatGPT code interpreter

Exploring the Future Potential of AI-Enabled Smartphone Processors

Breaking the Kubernetes Kill Chain: Host Path Mount

A Domino Admins Adventures (Engage 2024)

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

A Year of the Servo Reboot: Where Are We Now?

Let's Talk Operations! (Hadoop Summit 2014)

1. Let’s Talk Operations! Allen Wittenauer!

3. Twitter: @_a__w_ Email: aw @ apache.org!

4. How many individual grids should I have?

5. One big grid Grid per project • Pros! • Lower ops overhead! • One location for all data! • Cons ! • Dev and Prod on one system • Pros! • Capacity planning per project! • Cons ! • More headcount to maintain! • Multiple copies of data! • Data ingress is a mess

6. Data Center Production ETL Development

7. ETL Dev Prod Base ETL Pull Event Feeds Database Feeds Base ETL Pull Base ETL Pull Post-Processed Data

8. DC2DC1 Production ETL Development

9. How do I solve some common distcp issues?

10. • Common issues! • Version incompatibilities! • Network bandwidth consumption! ! • Some tricks! • Use WebHDFS! • All modern versions support it! • Read and write in both directions! • Create a separate queue with hard limits! • Pull from larger, push from smaller

11. Q&A Allen Wittenauer Twitter: @_a__w_ Email: aw @ apache.org

12. Bonus Slide!

13. 20 GB /, ... 200 GB task space (rest) HDFS • root partitioning ! ! ! ! ! • non-root partitioning 5 GB swap 200 GB task space (rest) HDFS

Let's Talk Operations! (Hadoop Summit 2014)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Let's Talk Operations! (Hadoop Summit 2014)

Similar to Let's Talk Operations! (Hadoop Summit 2014) (20)

Recently uploaded

Recently uploaded (20)

Let's Talk Operations! (Hadoop Summit 2014)