Slides from my talk at AWS ComSum 2022 in Manchester.
If you are writing code in Python which communicates with AWS APIs you are more than likely using the boto3 library.
boto3 isn't extended via inheritance or callbacks, instead it offers an event system which allows you to intercept and modify calls at different stages in their lifecycle.
This is a very powerful mechanism but it can be hard to write code for very rare events.
Using a tool like mitmproxy you can rewrite responses boto3 receives and simulate rarer events to aid writing code to handle them.
In this talk I'll show:
- How to write basic code to listen to events
- An example of some events you might see in typical S3 requests
- How to use mitmproxy to intercept, understand and ultimately rewrite HTTP requests between boto3 and AWS to simulate different scenarios
- An example of the events you'll see during request retries
- An example putting this together to insert monitoring into the boto3 retry mechanism to diagnose network issues
Full source here: https://github.com/micktwomey/exploring-boto3-events-with-mitmproxy
The document discusses three different buffer overflow exploits against three users (superuser, hyperuser, and masteruser) on a Linux machine. The superuser's program can be exploited through a simple buffer overflow. The hyperuser's program uses a canary but can still be exploited by overwriting a vulnerable function pointer. The masteruser's program must be exploited by overwriting the virtual pointer table. Code examples and steps are provided for each exploit.
This document discusses hacking serverless runtime environments like AWS Lambda, Azure Functions, and Auth0 WebTask. It begins by introducing the presenters and what will be covered. The document then explores how different vendors implement sandbox isolation and common attack techniques like persistence and data exfiltration. It examines specific runtimes like AWS Lambda in depth, investigating how to profile the environment, persist code, and escalate privileges. The document emphasizes that detection is difficult in serverless environments and provides examples of potential indicators of compromise. Overall, the document provides an overview of attacking and defending serverless architectures.
Probabilistic Data Structures (Edmonton Data Science Meetup, March 2018)Kyle Davis
Let's explore how Redis (and Redis Enterprise) can be used to store data in not only deterministic structures but also probabilistic structures like Bloom filters, HyperLogLog, Count Min Sketch and Cuckoo filters. We examine both usage and briefly summarize the algorithms that back these structures. Also we review the use-cases and applications for probabilistic structures.
Approaches for application request throttling - dotNetCologneMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Infrastructure as code might be literally impossible part 2ice799
The document discusses various issues with infrastructure as code including complexities that arise from software licenses, bugs, and inconsistencies across tools and platforms. Specific examples covered include problems with SSL and APT package management on Debian/Ubuntu, Linux networking configuration difficulties, and inconsistencies in Python packaging related to naming conventions for packages containing hyphens, underscores, or periods. Potential causes discussed include legacy code, lack of time for thorough testing and bug fixing, and economic pressures against developing fully working software systems.
The Chromium browser is developing very fast. When we checked the solution for the first time in 2011, it included 473 projects. Now it includes 1169 projects. We were curious to know if Google developers had managed to keep the highest quality of their code with Chromium developing at such a fast rate. Well, they had.
The document discusses three different buffer overflow exploits against three users (superuser, hyperuser, and masteruser) on a Linux machine. The superuser's program can be exploited through a simple buffer overflow. The hyperuser's program uses a canary but can still be exploited by overwriting a vulnerable function pointer. The masteruser's program must be exploited by overwriting the virtual pointer table. Code examples and steps are provided for each exploit.
This document discusses hacking serverless runtime environments like AWS Lambda, Azure Functions, and Auth0 WebTask. It begins by introducing the presenters and what will be covered. The document then explores how different vendors implement sandbox isolation and common attack techniques like persistence and data exfiltration. It examines specific runtimes like AWS Lambda in depth, investigating how to profile the environment, persist code, and escalate privileges. The document emphasizes that detection is difficult in serverless environments and provides examples of potential indicators of compromise. Overall, the document provides an overview of attacking and defending serverless architectures.
Probabilistic Data Structures (Edmonton Data Science Meetup, March 2018)Kyle Davis
Let's explore how Redis (and Redis Enterprise) can be used to store data in not only deterministic structures but also probabilistic structures like Bloom filters, HyperLogLog, Count Min Sketch and Cuckoo filters. We examine both usage and briefly summarize the algorithms that back these structures. Also we review the use-cases and applications for probabilistic structures.
Approaches for application request throttling - dotNetCologneMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Infrastructure as code might be literally impossible part 2ice799
The document discusses various issues with infrastructure as code including complexities that arise from software licenses, bugs, and inconsistencies across tools and platforms. Specific examples covered include problems with SSL and APT package management on Debian/Ubuntu, Linux networking configuration difficulties, and inconsistencies in Python packaging related to naming conventions for packages containing hyphens, underscores, or periods. Potential causes discussed include legacy code, lack of time for thorough testing and bug fixing, and economic pressures against developing fully working software systems.
The Chromium browser is developing very fast. When we checked the solution for the first time in 2011, it included 473 projects. Now it includes 1169 projects. We were curious to know if Google developers had managed to keep the highest quality of their code with Chromium developing at such a fast rate. Well, they had.
How to Meta-Sumo - Using Logs for Agile Monitoring of Production ServicesChristian Beedgen
Christian Beedgen discusses how to use application logs for monitoring production services. He provides examples of important information to include in logs like timestamps, log levels, and context. Beedgen also describes how Sumo Logic uses logs from its own production system to monitor and manage it through a "shadow system". He gives examples of log searches and aggregations that can provide insight into issues like errors, API performance, and long running operations.
Speaking from experience building MyGet.org: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Approaches for application request throttling - Cloud Developer Days PolandMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
ConFoo Montreal - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Let's face it, the web can be a dangerous place. So how do you protect your users and yourself? Tony Amoyal answers that and more as he shows how Rails can help protect against miscreants.
6 ways to hack your JavaScript application by Viktor Turskyi OdessaJS Conf
This will be 6 live hacking demos. We will not do theory, but will see in practice how small and not always obvious errors lead to significant vulnerabilities in your JavaScript application.
VISUG - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Join this video course on Udemy. Click the below link
https://www.udemy.com/mastering-rtos-hands-on-with-freertos-arduino-and-stm32fx/?couponCode=SLIDESHARE
>> The Complete FreeRTOS Course with Programming and Debugging <<
"The Biggest objective of this course is to demystifying RTOS practically using FreeRTOS and STM32 MCUs"
STEP-by-STEP guide to port/run FreeRTOS using development setup which includes,
1) Eclipse + STM32F4xx + FreeRTOS + SEGGER SystemView
2) FreeRTOS+Simulator (For windows)
Demystifying the complete Architecture (ARM Cortex M) related code of FreeRTOS which will massively help you to put this kernel on any target hardware of your choice.
The document discusses 5 ways that code can still be stuck in the 1990s and provides solutions to modernize it. Specifically, it addresses issues with 1) implicit messaging between systems, 2) complex threading code, 3) overuse of remote procedure calls, 4) abuse of garbage collection, and 5) over-reliance on logging without metrics. The solutions proposed are to use explicit contracts between systems via message passing, break operations into stateless and stateful "actors", embrace failure of remote systems, understand garbage collection behavior, and instrument code with metrics for monitoring.
100 bugs in Open Source C/C++ projects Andrey Karpov
This article demonstrates capabilities of the static code analysis methodology. The readers are offered to study the samples of one hundred errors found in open-source projects in C/C++.
This document summarizes a presentation about Rails 3 and OAuth. It discusses:
1. The history and key components of Rails, including ActiveRecord, ActionController, ActionView, and Railties.
2. New features in Rails 3, such as ActiveModel, ActionController::Responder, CSRF protection, unobtrusive JavaScript, and removing scripts/*.
3. What OAuth is and how it allows secure authorization for API access while limiting client applications' access.
4. The basic flow of an OAuth authorization, including requesting tokens from providers and exchanging them for access tokens.
Covers Performance improvements with the Symfony web framework for PHP.
- Google cares about user happiness, Google owns your search traffic ...so Google put page speed in PageRank (and crawl speed)
- Your site is more trustworthy and less frustrating
- Increase page views and ad impressions
- Increase conversions and revenue! It pays for itself!
- Bonus: run less app servers
Это будет 6 живых демо взлома. Идея не обсудить сухую теория, а увидеть на практике, как не всегда очевидные ошибки являются источником серьезных уязвимостей в твоем JavScript приложении.
MongoDB 4.2 comes GA soon delivering some amazing new features on multiple areas. In this talk, we will focus on changes related to sharded clusters. We are going to cover distributed transactions & mutable shard keys providing examples that will reveal the internals of those new features. We will provide best practices around the new sharding features and we will cover other minor changes related to it.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUANatan Silnitsky
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
BlackHat EU 2012 - Zhenhua Liu - Breeding Sandworms: How To Fuzz Your Way Out...MindShare_kk
Adobe's interpretation of sandboxing is called Adobe Reader X Protected Mode. Inspired by Microsoft's Practical Windows Sandboxing techniques, it was introduced in July 2010. So far, it had been doing a good job at limiting the impact of exploitable bugs in Adobe Reader X, as escaping the sandbox after successful exploitation turned to be particularly challenging, and hasn't been witnessed in the wild, yet.
This paper exposes how we did just this: By leveraging some broker APIs, a policy flaw, and a little more, we were able to break free from Adobe's sandbox.
The particular vulnerability we used was patched by Adobe in September 2011 (CVE-2011-1353), as a result of our responsible disclosure action; yet, this demonstrates that Adobe's sandbox cannot be considered a panacea against security flaws exploitation in Adobe Reader X, and paves the way toward further interesting discoveries for security researchers.
Indeed, beyond this particular vulnerability, this paper dives deep into the sandbox implementation of Adobe Reader X, and debates ways to audit its broker APIs, which, to our minds, offer a major attack surface. In particular, the paper details how we configured an open-source fuzzing tool to audit them through the IPC Framework.
Java Hurdling: Obstacles and Techniques in Java Client Penetration-TestingTal Melamed
Testing Java client applications is not always straightforward as testing web applications. Even under experienced hands, there might be obstacles coming your way; what if you cannot use a proxy? How do you MitM? What if you just can't? How do you modify the app to your benefit?
Fortunately, Java is still Java. This lecture is based on a true story, and will follow an interesting case of pen-testing a known product; what tools and techniques can be used in order to jump over hurdles, all the way to the finish line.
The lecture aims to enrich the pentester's toolbox as well as mind, when facing Java client applications; MitM-ing, run-time manipulations and patching the code are only some of the discussed cases.
In addition, a newly developed proxy for intercepting and tampering with TCP communication over TLS/SSL and bypassing certificate-pinning protections, will be introduced during the lecture.
This document discusses the Machine Learning Security Evasion Competition 2020 organized by Hyrum Anderson and Zoltan Balazs. It provided an overview of the competition which challenged participants to modify malware samples to evade detection by machine learning models in an offensive track, or to develop their own machine learning models for submission in a defensive track. The document outlines various approaches used by winners to evade models, such as appending extra data to executables. It concludes by providing the names of the competition winners and statistics on participant numbers and model checks during the competition.
The document provides an overview of topics that may be covered in DevOps and cloud engineering interviews. It includes questions on Linux, Kubernetes, Docker, shell scripting, the Amazon interview process, and networking. Sample questions are provided for each topic along with an example of Amazon's interview structure and common principles assessed. Key components, configurations, and commands are outlined for areas like containers, orchestration, configuration management, and continuous delivery.
The document summarizes Dan Kaminsky's talk at Black Hat 2007 about exploiting the DNS rebinding vulnerability to bypass firewalls and access internal networks from external web browsers. It describes how DNS rebinding works by abusing the same-origin policy to treat websites with different domain names but the same IP address as coming from the same origin. It then outlines several ways an attacker can force a domain to resolve to different IP addresses and use this to tunnel network traffic over the browser.
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
How to Meta-Sumo - Using Logs for Agile Monitoring of Production ServicesChristian Beedgen
Christian Beedgen discusses how to use application logs for monitoring production services. He provides examples of important information to include in logs like timestamps, log levels, and context. Beedgen also describes how Sumo Logic uses logs from its own production system to monitor and manage it through a "shadow system". He gives examples of log searches and aggregations that can provide insight into issues like errors, API performance, and long running operations.
Speaking from experience building MyGet.org: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Approaches for application request throttling - Cloud Developer Days PolandMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
ConFoo Montreal - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Let's face it, the web can be a dangerous place. So how do you protect your users and yourself? Tony Amoyal answers that and more as he shows how Rails can help protect against miscreants.
6 ways to hack your JavaScript application by Viktor Turskyi OdessaJS Conf
This will be 6 live hacking demos. We will not do theory, but will see in practice how small and not always obvious errors lead to significant vulnerabilities in your JavaScript application.
VISUG - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Join this video course on Udemy. Click the below link
https://www.udemy.com/mastering-rtos-hands-on-with-freertos-arduino-and-stm32fx/?couponCode=SLIDESHARE
>> The Complete FreeRTOS Course with Programming and Debugging <<
"The Biggest objective of this course is to demystifying RTOS practically using FreeRTOS and STM32 MCUs"
STEP-by-STEP guide to port/run FreeRTOS using development setup which includes,
1) Eclipse + STM32F4xx + FreeRTOS + SEGGER SystemView
2) FreeRTOS+Simulator (For windows)
Demystifying the complete Architecture (ARM Cortex M) related code of FreeRTOS which will massively help you to put this kernel on any target hardware of your choice.
The document discusses 5 ways that code can still be stuck in the 1990s and provides solutions to modernize it. Specifically, it addresses issues with 1) implicit messaging between systems, 2) complex threading code, 3) overuse of remote procedure calls, 4) abuse of garbage collection, and 5) over-reliance on logging without metrics. The solutions proposed are to use explicit contracts between systems via message passing, break operations into stateless and stateful "actors", embrace failure of remote systems, understand garbage collection behavior, and instrument code with metrics for monitoring.
100 bugs in Open Source C/C++ projects Andrey Karpov
This article demonstrates capabilities of the static code analysis methodology. The readers are offered to study the samples of one hundred errors found in open-source projects in C/C++.
This document summarizes a presentation about Rails 3 and OAuth. It discusses:
1. The history and key components of Rails, including ActiveRecord, ActionController, ActionView, and Railties.
2. New features in Rails 3, such as ActiveModel, ActionController::Responder, CSRF protection, unobtrusive JavaScript, and removing scripts/*.
3. What OAuth is and how it allows secure authorization for API access while limiting client applications' access.
4. The basic flow of an OAuth authorization, including requesting tokens from providers and exchanging them for access tokens.
Covers Performance improvements with the Symfony web framework for PHP.
- Google cares about user happiness, Google owns your search traffic ...so Google put page speed in PageRank (and crawl speed)
- Your site is more trustworthy and less frustrating
- Increase page views and ad impressions
- Increase conversions and revenue! It pays for itself!
- Bonus: run less app servers
Это будет 6 живых демо взлома. Идея не обсудить сухую теория, а увидеть на практике, как не всегда очевидные ошибки являются источником серьезных уязвимостей в твоем JavScript приложении.
MongoDB 4.2 comes GA soon delivering some amazing new features on multiple areas. In this talk, we will focus on changes related to sharded clusters. We are going to cover distributed transactions & mutable shard keys providing examples that will reveal the internals of those new features. We will provide best practices around the new sharding features and we will cover other minor changes related to it.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUANatan Silnitsky
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
BlackHat EU 2012 - Zhenhua Liu - Breeding Sandworms: How To Fuzz Your Way Out...MindShare_kk
Adobe's interpretation of sandboxing is called Adobe Reader X Protected Mode. Inspired by Microsoft's Practical Windows Sandboxing techniques, it was introduced in July 2010. So far, it had been doing a good job at limiting the impact of exploitable bugs in Adobe Reader X, as escaping the sandbox after successful exploitation turned to be particularly challenging, and hasn't been witnessed in the wild, yet.
This paper exposes how we did just this: By leveraging some broker APIs, a policy flaw, and a little more, we were able to break free from Adobe's sandbox.
The particular vulnerability we used was patched by Adobe in September 2011 (CVE-2011-1353), as a result of our responsible disclosure action; yet, this demonstrates that Adobe's sandbox cannot be considered a panacea against security flaws exploitation in Adobe Reader X, and paves the way toward further interesting discoveries for security researchers.
Indeed, beyond this particular vulnerability, this paper dives deep into the sandbox implementation of Adobe Reader X, and debates ways to audit its broker APIs, which, to our minds, offer a major attack surface. In particular, the paper details how we configured an open-source fuzzing tool to audit them through the IPC Framework.
Java Hurdling: Obstacles and Techniques in Java Client Penetration-TestingTal Melamed
Testing Java client applications is not always straightforward as testing web applications. Even under experienced hands, there might be obstacles coming your way; what if you cannot use a proxy? How do you MitM? What if you just can't? How do you modify the app to your benefit?
Fortunately, Java is still Java. This lecture is based on a true story, and will follow an interesting case of pen-testing a known product; what tools and techniques can be used in order to jump over hurdles, all the way to the finish line.
The lecture aims to enrich the pentester's toolbox as well as mind, when facing Java client applications; MitM-ing, run-time manipulations and patching the code are only some of the discussed cases.
In addition, a newly developed proxy for intercepting and tampering with TCP communication over TLS/SSL and bypassing certificate-pinning protections, will be introduced during the lecture.
This document discusses the Machine Learning Security Evasion Competition 2020 organized by Hyrum Anderson and Zoltan Balazs. It provided an overview of the competition which challenged participants to modify malware samples to evade detection by machine learning models in an offensive track, or to develop their own machine learning models for submission in a defensive track. The document outlines various approaches used by winners to evade models, such as appending extra data to executables. It concludes by providing the names of the competition winners and statistics on participant numbers and model checks during the competition.
The document provides an overview of topics that may be covered in DevOps and cloud engineering interviews. It includes questions on Linux, Kubernetes, Docker, shell scripting, the Amazon interview process, and networking. Sample questions are provided for each topic along with an example of Amazon's interview structure and common principles assessed. Key components, configurations, and commands are outlined for areas like containers, orchestration, configuration management, and continuous delivery.
The document summarizes Dan Kaminsky's talk at Black Hat 2007 about exploiting the DNS rebinding vulnerability to bypass firewalls and access internal networks from external web browsers. It describes how DNS rebinding works by abusing the same-origin policy to treat websites with different domain names but the same IP address as coming from the same origin. It then outlines several ways an attacker can force a domain to resolve to different IP addresses and use this to tunnel network traffic over the browser.
Similar a Exploring Boto3 Events With Mitmproxy (20)
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Skybuffer SAM4U tool for SAP license adoptionTatiana Kojar
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Webinar: Designing a schema for a Data WarehouseFederico Razzoli
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 6
Exploring Boto3 Events With Mitmproxy
1. Exploring Boto3 Events with MitmProxy
AWS Community Summit, September 22 2022
Michael Twomey
@micktwomey / michael.twomey@fourtheorem.com
1
2. About Me
— Hi!
!
I'm Michael Twomey
"
— Started my career in Sun
Microsystems working on the Solaris
OS in 1999 (when Y2K was a thing)
—
#
Been coding in Python for over 20
years
— Started kicking the tyres of AWS back
when it was just S3, EC2 and SQS
—
☁
Senior Cloud Architect at
fourTheorem
https://fourtheorem.com/
Reach out to us at
hello@fourTheorem.com
2
3. What I'll be Talking About
— Going to go through a problem from beginning
to end
— Show what issues I ran into and how I solved
them
— Will try to give just enough explanation of
everything I use
— There are many ways to achieve what I wanted,
this is just one path!
PS: I've a bunch of DALL-E credits to burn so expect
silly images
3
4. What I'll be Talking About
— A dash of AWS APIs
— Some boto3
— A bit of Python
— A pinch of HTTP
— A tiny bit of TLS
— A portion of mitmproxy
4
5. The Setup
Code base using Python and the boto3 library to talk
to AWS
The core of the system runs a huge amount of
computations spread over a large amount of jobs in
either Lambda or Fargate containers 1
It wouldn't be unusual to have thousands of
containers running many compute jobs per second.
1
For more details check out the post "A serverless architecture for high performance
financial modelling" - https://aws.amazon.com/blogs/hpc/a-serverless-architecture-for-
high-performance-financial-modelling/
5
6. The Problem
During very large job runs we would occassionally
see inexplicable slow downs and sometimes rate
limit errors
This prompted the question:
"Are we triggering a lot of S3 request retries?"
6
7. Request Retries?
AWS has rate limits on their APIs (sensible!)
S3 PUT object might have a rate limit of 3,500 requests per second 2
When you hit this you might get back a HTTP 429 or HTTP 503
boto3 attempts to handle this invisibly via retries3
to minimize impact on your application
3
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#standard-retry-mode
2
https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
7
8. Retry Mechanism
boto3's default retry handler4
implements the classic "retry with jitter" approach5
:
1. For a known set of errors catch them
2. Keep count of the number of times we've tried
3. If we've hit a maximum retry count fail and allow the error to bubble up
4. Otherwise take the count and multiply by some random number and some scale factor
5. Sleep for that long
6. Retry the call
5
https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
4
https://github.com/boto/botocore/blob/develop/botocore/retryhandler.py
8
9. Retry Sleep Formula
# From https://github.com/boto/botocore/blob/develop/botocore/retryhandler.py
base * (growth_factor ** (attempts - 1))
base = random.random() # random float between 0.0 and 1.0
growth_factor = 2
attempts = 2
random.random() * (2 ** (2 - 1))
0.75 * 2 = 1.5
attempt 1 = 1 second max
attempt 2 = 2 second max
attempt 3 = 8 second max
attempt 4 = 16 second max
attempt 5 = 32 second max
# Default of 5 retries
32 + 16 + 8 + 2 + 1 = 59 seconds max sleep total, with 5x requests
9
10. The Impact
Lots lots of calls per second * sleeping for a bunch
of time = a big pile up
As more calls bunch up and sleep, we encounter
more rate limits, leading to more calls...
Could this account for our stalls?
10
11. How Can We Figure Out The
Cause?
Could use logging at DEBUG level
logging.basicConfig(level=logging.DEBUG)
This is super verbose and logs an overwhelming
level of detail
What we want is some kind of hook to increment a
count or emit a metric on retry
Does boto3 offer any hooks?
!
11
12. boto3 Events
Events6
are an extension mechanism for boto3
6
boto3 event docs over at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
12
13. boto3 Events
You register a function to be called when
an event matching a pattern happens.
Wildcards (*) are also allowed for
patterns.
"provide-client-params.s3.ListObjects"
"provide-client-params.s3.*"
"provide-client-params.*"
"*"
s3 = boto3.client("s3")
s3.meta.events.register("needs-retry.*", my_function)
13
18. Some Observations
— That's a lot of different inputs for
different events!
— The list of events isn't explicitly
documented
— The args each event can receive isn't
explicitly documented
=> It's hard to guess what code you'll
need to implement without triggering the
behaviour you want
provide-client-params.s3.ListBuckets
{
'params': {},
'model': OperationModel(name=ListBuckets),
'context': {
'client_region': 'eu-west-1',
'client_config': <botocore.config.Config object at 0x1078b8d90>,
'has_streaming_input': False,
'auth_type': None
}
}
request-created.s3.ListBuckets
{
'request': <botocore.awsrequest.AWSRequest object at 0x1078bac20>,
'operation_name': 'ListBuckets'
}
needs-retry.s3.ListBuckets
{
'response': (
<botocore.awsrequest.AWSResponse object at 0x107abb970>,
{
'ResponseMetadata': {
'RequestId': 'QZV9EWHJMR4T8VQ9',
...
'endpoint': s3(https://s3.eu-west-1.amazonaws.com),
'operation': OperationModel(name=ListBuckets),
'attempts': 1,
'caught_exception': None,
'request_dict': {
'url_path': '/',
'query_string': '',
'method': 'GET',
...
}
18
19. Side track: Extending Libraries in Python
(Mick Complains About Lack of Autocomplete)
There are a few "classic" approaches to extending code in Python:
1. Inheritence
2. Callbacks
3. Events
19
20. Inheritence
class MyLibrary:
def do_something(self, arg1: int, arg2: float):
... library does something here ...
class MyModifiedLibrary(MyLibrary):
def do_something(self, arg1: int, arg2: float):
... your stuff happens ...
# call the original code too:
super().do_something(arg1, arg2)
— Works best with libraries written as a
bunch of classes
— Can be very clunky and hard to
predict how code will interact (hello
mixins!)
— Usually needs explicit hooks for
cleanly overriding functionality
20
21. Callbacks
def add_handler(handler: Callable[[int, float], str]):
pass
def my_handler(arg1: int, arg2: float) -> str:
pass
def my_broken_handler(arg1: str, arg2: str) -> str:
pass
add_handler(my_handler)
# error: Argument 1 to "add_handler"
# has incompatible type "Callable[[str, str], str]";
# expected "Callable[[int, float], str]"
add_handler(my_broken_handler)
— Generally add_handler keeps a lis of
functions to call somewhere
— This approach allows for typing hints
to guide the developer
— Generally easy to document (code
signatures do half the work)
21
22. Events
def add_handler(event: str, handler: Callable):
pass
def my_x_handler(arg1: int, arg2: float):
pass
def my_y_handler(arg1: str):
pass
add_handler("x.some_event", my_x_handler)
add_handler("y.rare_important_event", my_y_handler)
# This will probably break at runtime
add_handler("y.rare_important_event", my_x_handler)
— Events usually used for generic
hooks in libraries
— Having a consistent set of args for
your handlers makes life easier
— Requires more documentation to
guide the programmer
22
23. boto3 Uses Events
Generic event hooks much easier to integrate to library, especially when dynamically
generated like boto3
Drawback: can be very hard for the developer to know what events exist and how they
behave
Solution: Lets watch them play out!
!
Now, how do we trigger rate limits?
23
24. Triggering a rate limit
There are many ways to trigger rate limits:
— Hacking the library
!
— Hacking Python
"
— Hacking the OS
#
— Hacking the network
$
— Triggering the rate limit for real
%
I chose to mess with the network
!
Why? This is close to what would be seen in real life and cheaper than calling for real!
24
25. HTTP
We can mess with the HTTP responses boto3 gets
In particular:
- For a rate limit I'm betting boto3 looks at the HTTP response code
- I'm also betting it'll be HTTP 429
- I'm also betting the code doesn't care about the payload too much once it's a 429
- Finally I'm betting boto3 doesn't verify a response checksum
=> Lets change the response code!
25
26. From this
HTTP/1.1 200 OK
Content-Length: 34202
Content-Type: application/json
...
{
...
}
To this
HTTP/1.1 429 Rate limit exceeded
Content-Length: 34202
Content-Type: application/json;
...
{
...
}
26
27. How do we achieve this?
One way to mess with HTTP is using a HTTP proxy
One tool which implements this is mitmproxy
27
28. mitmproxy
https://mitmproxy.org
mitmproxy is a free and open source interactive
HTTPS proxy.
What?
Let's you mess with the HTTP requests and
responses from programs
Bit like Chrome Dev Tools for all your HTTP
speaking commands
28
29. Basic usage
1. Run mitmproxy (or mitmweb for a
fancier web interface)
2. Set the HTTP proxy settings to
mitmproxy's (defaults to http://
localhost:8080)
3. Run your program
4. Watch in mitmproxy
Easy right?
export http_proxy=localhost:8080
export https_proxy=localhost:8080
python my_app.py
29
31. curl
❯ https_proxy=localhost:8080 curl -I https://www.fourtheorem.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Not so easy after all!
31
34. What's Going Wrong?
What's happening?
1. We tell curl to connect via mitmproxy to www.fourtheorem.com using HTTPS
2. curl connects to mitmproxy and tries to verify the TLS certificate
3. curl decides the certificate in the proxy isn't to be trusted and rejects the connection
34
35. MITM
curl (and TLS) is doing its job: preventing someone from injecting themselves into the
HTTP connection and intercepting traffic.
A man in the middle attack (or MITM) was prevented!
Unfortunately that's what we want to do!
35
36. mitmproxy has an answer
Luckily mitmproxy generates TLS certificates for you to use:
❯ ls -l ~/.mitmproxy/
total 48
-rw-r--r-- 1 mick staff 1172 Sep 4 19:26 mitmproxy-ca-cert.cer
-rw-r--r-- 1 mick staff 1035 Sep 4 19:26 mitmproxy-ca-cert.p12
-rw-r--r-- 1 mick staff 1172 Sep 4 19:26 mitmproxy-ca-cert.pem
-rw------- 1 mick staff 2411 Sep 4 19:26 mitmproxy-ca.p12
-rw------- 1 mick staff 2847 Sep 4 19:26 mitmproxy-ca.pem
-rw-r--r-- 1 mick staff 770 Sep 4 19:26 mitmproxy-dhparam.pem
If you can somehow tell your command to trust these it will talk via mitmproxy!
36
37. ⚠
Danger!
⚠
Here be
Dragons!
To work mitmproxy requires clients to trust these
certificates
This potentially opens up a massive security hole on
your machine depending how this is set up
Recommendation: if possible restrict to one off
command line invocations rather than install system
wide
Luckily we can override on a per invocation basis in
curl and boto3
Full guide: https://docs.mitmproxy.org/stable/
concepts-certificates/
37
38. Overriding the cert bundle
curl offers a simple way to trust a cert: --cacert
❯ https_proxy=localhost:8080 curl --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem -I https://www.fourtheorem.com
HTTP/1.1 200 Connection established
HTTP/1.1 200 OK
Server: openresty
Date: Sun, 04 Sep 2022 20:09:03 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 520810
Connection: keep-alive
...
38
42. boto3
We can do something similar with boto3:
❯ https_proxy=localhost:8080
AWS_CA_BUNDLE=$HOME/.mitmproxy/mitmproxy-ca-cert.pem
python examples/print_events.py
S3:
provide-client-params.s3.ListBuckets
before-parameter-build.s3.ListBuckets
before-call.s3.ListBuckets
request-created.s3.ListBuckets
...
We tell boto3 to use a different cert bundle (AWS_CA_BUNDLE)
42
44. What Were We Trying to Do
Again?
We can now:
1. Run some requests from boto3 to AWS
2. Intercept and inspect these requests in mitmproxy
How does this help us?
44
45. More than a HTTP debugger
mitmproxy offers the ability to intercept and change HTTP requests
- https://docs.mitmproxy.org/stable/mitmproxytutorial-interceptrequests/
- https://docs.mitmproxy.org/stable/mitmproxytutorial-modifyrequests/
45
46. Intercepting
1. Hit i to create an intercept
2. ~d s3.eu-west-1.amazonaws.com & ~s
— ~d match on domain, ~s match on server response
3. Run the command
4. In the UI go into the response and hit e
5. Change the response code to 429
6. Hit a to allow the request to continue
7. Watch what happens in the command
46
47. Modified code to focus on retry mechanism for brevity
import boto3
from rich import print
import time
def print_event(event_name: str, attempts: int, operation, response, request_dict, **_):
print(
event_name,
operation,
attempts,
response[1]["ResponseMetadata"]["HTTPStatusCode"],
request_dict["context"]["retries"],
)
s3 = boto3.client("s3")
s3.meta.events.register("needs-retry.s3.ListBuckets", print_event)
s3.list_buckets()
47
52. So What Was the Point of All
That?
fields @timestamp, event_name
| filter ispresent(event_name)
| filter event_name = 'needs-retry.s3.PutObject'
| filter attempts > 1
| sort by @timestamp asc
| stats count() by bin(1m)
The graph shows over 250K retry attempts at the
peak!
It also shows some kind of oscillation, possibly due
to so many connections sleeping at the same time.
52
53. What We Covered
— AWS API limits
— https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-
performance.html
— boto3's event system
— https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
— How request retries behave
— https://boto3.amazonaws.com/v1/documentation/api/latest/guide/
retries.html#standard-retry-mode
— mitmproxy
— https://mitmproxy.org
53