Latency: The Silent Monitoring System Killer

•

2 recomendaciones•1,761 vistas

Lindsay Holmwood

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

Tecnología Empresariales

LATENCY
THE SILENT
MONITORING SYSTEM
KILLER

Saturday, 21 January 2012

#MONITORINGSUCKS

Saturday, 21 January 2012

execute large volumes of monitoring
checks under a variety of conditions
(good + bad) with a consistent throughput

Saturday, 21 January 2012

CONSISTENT
THROUGHPUT

Saturday, 21 January 2012

SHORT CHECK
EXECUTION TIME

Saturday, 21 January 2012

WHAT
INTRODUCES
VARIABILITY?

Saturday, 21 January 2012

LATENCY INTRODUCED
c SYNCHRONOUS CALLS

Saturday, 21 January 2012

= 1 monitoring check

Saturday, 21 January 2012

150 monitoring checks
each executed every 300 seconds
each takes 1 second
checks are executed serially

Saturday, 21 January 2012

all checks executed in 150 seconds
monitoring system at 50% capacity

Saturday, 21 January 2012

DOUBLE THE CHECKS

Saturday, 21 January 2012

all checks executed in 300 seconds
monitoring system at 100% capacity

Saturday, 21 January 2012

DOUBLE THE
EXECUTION TIME

Saturday, 21 January 2012

all checks executed in 600 seconds
monitoring system at 200% capacity
only 50% of the checks are “on time”

Saturday, 21 January 2012

CHECK LATENCY

Saturday, 21 January 2012

HOW DO
WE FIX THIS!?

Saturday, 21 January 2012

PERFORMANCE
ANALYSIS!

Saturday, 21 January 2012

MONITORING CHECK
==
“ACTION” ON MVC WEB APP

Saturday, 21 January 2012

SEPARATE DATA
COLLECTION
FROM

THRESHOLDING
f NOTIFICATIONS

Saturday, 21 January 2012

THIS SHIFTS
LATENCY

Saturday, 21 January 2012

IT DOES NOT
ELIMINATE IT!

Saturday, 21 January 2012

RRDTOOL IS EVIL

Saturday, 21 January 2012

USE SOMETHING
BETTER!

Saturday, 21 January 2012

USE
OPENTSDB
OR

GANGLIA
WITH
CHECK_TSDB
OR

CHECK_GMOND
Saturday, 21 January 2012

A DIFFERENT SET a
PROBLEMS

Saturday, 21 January 2012

STORAGE WILL
GO AWAY

Saturday, 21 January 2012

CHAOS WILL ENSUE

Saturday, 21 January 2012

PAGERS WILL MELT

Saturday, 21 January 2012

SET UP “META-
PARENTING”

Saturday, 21 January 2012

BUILD A KILL
SWITCH

Saturday, 21 January 2012

READ MORE
ABOUT THIS:

Saturday, 21 January 2012

bit.ly/yN4mdy

Saturday, 21 January 2012

Más contenido relacionado

Destacado

Your own (little) gem: building an online business with Ruby

Lindsay Holmwood

On January 31, 2000, Alaska Airlines Flight 261 plunged into the Pacific ocean in an extreme "nose down" position, killing all 88 crew and passengers on board. The NTSB concluded AA261's horizontal stabiliser trim system's jackscrew was inadequately maintained, causing the pilots to lose all control of the plane. There are striking parallels with the problems we face daily in IT operations & software development, and the 30 years of give and take between the aircraft manufacturer's engineers, airline maintenance staff, and federal regulators that preceded AA261's simple mechanical failure. In this talk, Lindsay looks at the complex interplay between the parties in the AA261 crash through a DevOps lens, investigating the collaborative approach to maintenance and operation of the MD-83 aircraft, and relating the complexities back to the complex IT systems we build and maintain.

AA261: DevOps lessons in collaborative maintenance

Lindsay Holmwood

Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well. In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.

Islands: Puppet at Bulletproof Networks

Lindsay Holmwood

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems. Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand. While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated. In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

Escalating complexity: DevOps learnings from Air France 447

Lindsay Holmwood

We all know that load testing is important, but it's all too common that it's left to the very end of a project and it's invariably the first thing that gets dropped when budgets and timeframes get cut. Furthermore, most of us don't know where or how to start implementing effective load tests, let alone how to analyse the results. Lindsay Holmwood, Software Manager at Bulletproof Networks, will be talking about integrating performance testing into your application development + deploy cycle from the very beginning, using inexpensive and easy to use SaaS tools. There will be a hands on demonstration of the Blitz load + performance testing tool, coupled with a brief dive into the Blitz API internals to retrieve and analyse advanced reporting information.

Load testing with Blitz

Lindsay Holmwood

Writing checks for your monitoring system is boring. You end up writing the same checks again and again, and it can be difficult to verify behavior instead of availability. Wouldn't it be useful to have a standard library of checks you could reuse across your infrastructure? it lets you write reusable behavioral tests in human-readable language.Say hello to cucumber-nagios - it lets you write reusable behavioral tests in human-readable language. As cucumber-nagios output the test results in the Nagios plugin format you can run your checks from any monitoring system that understands the format.

Behaviour Driven Monitoring with cucumber-nagios

Lindsay Holmwood

Destacado (6)

Your own (little) gem: building an online business with Ruby

AA261: DevOps lessons in collaborative maintenance

Islands: Puppet at Bulletproof Networks

Escalating complexity: DevOps learnings from Air France 447

Load testing with Blitz

Behaviour Driven Monitoring with cucumber-nagios

Más de Lindsay Holmwood

We’re all familiar with the Puppet manifest development cycle: “make your change, commit your change, push your change, run Puppet, debug the output, then go back to the beginning to refactor or fix a bug”. What if you could could shorten the cycle to “make a change, run puppet, commit change”? Enter Rump, a tool for doing Puppet runs locally from a Git checkout. Rump encourages a Puppet workflow where you quickly + iteratively develop your Puppet manifests on a single machine, then push your changes up to a repository to deploy to the rest of your infrastructure. Lindsay will be demonstrating how to setup Rump, showing off some awesome workflows for doing local iterative development, and exposing super cool features like testing your manifests against new versions of Puppet in a single command.

Rump - making Puppetmaster-less Puppet meaty

Lindsay Holmwood

Does Behaviour Driven Development have a role in the infrastructure world? Enter Behaviour Driven Infrastructure where systems administrators can apply some simple open source tools and BDD principles to make infrastructure management more powerful, more insightful and deliver more value to their customers. The typical enterprise monitoring and configuration management set-up for a website is: - Is the Apache package installed and the appropriate version? - Is the Apache service running? - Can I connect to the HTTP port and is HTML returned? - Multiply this by a few hundred iterations of hosts and types of services and you’re probably looking at your typical Nagios, Puppet, Cfengine, Hyperic set-up. All this monitoring misses something critical though – we’re not actually monitoring that the service does what it should. Yes, it matters whether Apache is installed, the Apache service is running, and you can connect to HTTP but does this actually prove anything about the availability of the service we’re managing and providing for our customers? Nope… We need to demonstrate more than just a check that says the Apache server is up. We need to demonstrate that the service delivered by that infrastructure was available to our customers AND functioning as intended. Enter Behaviour Driven Infrastructure or BDI which applies the principles of Behavioural Driven Development to the management of infrastructure. In this presentation you’ll learn: - How Behaviour Driven Development works - What makes a behavioural test - How to install and use Cucumber to perform BDI - Practical examples of behavioural tests in Cucumber, and - How to integrate BDI into your work flow and your enterprise monitoring and configuration management frameworks.

Behaviour driven infrastructure

Lindsay Holmwood

HA websites are where the rubber meets the road - at 200km/h. Traditional separation of dev and ops just doesn't cut it. Everything is related to everything. Code relies on performant and resilient infrastructure, but highly performant infrastructure will only get a poorly written application so far. Worse still, root cause analysis in HA sites will more often than not identify problems that don't clearly belong to either devs or ops. The two options are collaborate or die. This talk will introduce 3 core principles for improving collaboration between operations and development teams: consistency, repeatability, and visibility. These principles will be investigated with real world case studies and associated technologies audience members can start using now. In particular, there will be a focus on: - fast provisioning of test environments with configuration management - reliable and repeatable automated deployments - application and infrastructure visibility with statistics collection, logging, and visualisation

Burn down the silos! Helping dev and ops gel on high availability websites

Lindsay Holmwood

Flapjack: rethinking monitoring for the cloud

Lindsay Holmwood

Setting up monitoring for web applications can be complicated - tests tend to lack expressiveness, or and quite often they don't even test the right problem in the first place. cucumber-nagios lets a sysadmin write behavioural tests for their web apps in plain English, and outputs the test results in the Nagios plugin format, allowing a sysadmin to be notified by Nagios when their production apps aren't behaving.

Monitoring web application behaviour with cucumber-nagios

Lindsay Holmwood

Deploying Merb

Lindsay Holmwood

Más de Lindsay Holmwood (6)

Rump - making Puppetmaster-less Puppet meaty

Behaviour driven infrastructure

Burn down the silos! Helping dev and ops gel on high availability websites

Flapjack: rethinking monitoring for the cloud

Monitoring web application behaviour with cucumber-nagios

Deploying Merb

Último

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Navi Mumbai Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Navi Mumbai Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Navi Mumbai Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

ICT role in 21st century education and its challenges

rafiqahmad00786416

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Modernizing Securities Finance: The cloud-native prime brokerage platform transforming capital markets. Madhu Subbu, Managing Director, Head of Securities Finance Engineering Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

apidays

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Latency: The Silent Monitoring System Killer

1. LATENCY THE SILENT MONITORING SYSTEM KILLER Saturday, 21 January 2012

2. #MONITORINGSUCKS Saturday, 21 January 2012

3. SCALABILITY Saturday, 21 January 2012

4. execute large volumes of monitoring checks under a variety of conditions (good + bad) with a consistent throughput Saturday, 21 January 2012

5. CONSISTENT THROUGHPUT Saturday, 21 January 2012

6. SHORT CHECK EXECUTION TIME Saturday, 21 January 2012

7. WHAT INTRODUCES VARIABILITY? Saturday, 21 January 2012

8. LATENCY INTRODUCED c SYNCHRONOUS CALLS Saturday, 21 January 2012

9. Saturday, 21 January 2012

10. = 1 monitoring check Saturday, 21 January 2012

11. Saturday, 21 January 2012

12. 150 monitoring checks each executed every 300 seconds each takes 1 second checks are executed serially Saturday, 21 January 2012

13. all checks executed in 150 seconds monitoring system at 50% capacity Saturday, 21 January 2012

14. DOUBLE THE CHECKS Saturday, 21 January 2012

15. Saturday, 21 January 2012

16. all checks executed in 300 seconds monitoring system at 100% capacity Saturday, 21 January 2012

17. DOUBLE THE EXECUTION TIME Saturday, 21 January 2012

18. Saturday, 21 January 2012

19. all checks executed in 600 seconds monitoring system at 200% capacity only 50% of the checks are “on time” Saturday, 21 January 2012

20. CHECK LATENCY Saturday, 21 January 2012

21. Saturday, 21 January 2012

22. HOW DO WE FIX THIS!? Saturday, 21 January 2012

23. PERFORMANCE ANALYSIS! Saturday, 21 January 2012

24. AN ANALOGY Saturday, 21 January 2012

25. MONITORING CHECK == “ACTION” ON MVC WEB APP Saturday, 21 January 2012

26. Saturday, 21 January 2012

27. Saturday, 21 January 2012

28. SEPARATE DATA COLLECTION FROM THRESHOLDING f NOTIFICATIONS Saturday, 21 January 2012

29. Saturday, 21 January 2012

30. THIS SHIFTS LATENCY Saturday, 21 January 2012

31. IT DOES NOT ELIMINATE IT! Saturday, 21 January 2012

32. Saturday, 21 January 2012

33. RRDTOOL IS EVIL Saturday, 21 January 2012

34. USE SOMETHING BETTER! Saturday, 21 January 2012

35. USE OPENTSDB OR GANGLIA WITH CHECK_TSDB OR CHECK_GMOND Saturday, 21 January 2012

36. Saturday, 21 January 2012

37. A DIFFERENT SET a PROBLEMS Saturday, 21 January 2012

38. STORAGE WILL GO AWAY Saturday, 21 January 2012

39. CHAOS WILL ENSUE Saturday, 21 January 2012

40. PAGERS WILL MELT Saturday, 21 January 2012

41. SET UP “META- PARENTING” Saturday, 21 January 2012

42. BUILD A KILL SWITCH Saturday, 21 January 2012

43. Saturday, 21 January 2012

44. READ MORE ABOUT THIS: Saturday, 21 January 2012

45. bit.ly/yN4mdy Saturday, 21 January 2012

Latency: The Silent Monitoring System Killer

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (6)

Más de Lindsay Holmwood

Más de Lindsay Holmwood (6)

Último

Último (20)

Latency: The Silent Monitoring System Killer