Distributed systems in practice, in theory

•

5 recomendaciones•2,127 vistas

Modern systems in production rely on decades of computer science research. Over time, new architectural patterns emerge that enable more resilient and robust systems. In this talk, we'll discuss some of these patterns from systems I've worked on at Google and the related work that provide insights into the motivations behind them.

Software

Confidential + ProprietaryConfidential + Proprietary
Distributed Systems in Practice, in Theory
Aysylu Greenberg
March 8th, 2016

Confidential + Proprietary
Aysylu Greenberg
@aysylu22

Confidential + Proprietary
Towards Distributed Build System

Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

Confidential + Proprietary
Papers We Love SF

Confidential + Proprietary
meetup.com/Papers-We-Love-London

Confidential + Proprietary
Today
● Staged Event-Driven Architecture

Confidential + Proprietary
Today
● Staged Event-Driven Architecture
● Leases

Confidential + Proprietary
Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

Confidential + ProprietaryConfidential + Proprietary
Computer Science Research
In
Distributed Systems Industry

Confidential + Proprietary
Operating systems research

Confidential + Proprietary
Operating systems research
Concurrency

Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore

Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore
Processes execute at
different speeds

Confidential + Proprietary
Time in distributed systems

Confidential + Proprietary
Time in distributed systems
Pipelining

Confidential + Proprietary
Internet
Distributed consensus

Confidential + Proprietary
Internet
Distributed consensus
Paxos

Confidential + Proprietary
Reconsider large systems

Confidential + Proprietary
Reconsider large systems
Platform as a service

Confidential + Proprietary
CS Research is Timeless
Inform decisions
Mitigate technical risk

Confidential + ProprietaryConfidential + Proprietary
Staged Event
Driven
Architecture
&
Deep Pipelines
2001

Confidential + Proprietary
Hardware Pipelines

Confidential + Proprietary
Data Pipelines
https://en.wikipedia.org/wiki/Graphics_pipeline

Confidential + Proprietary
Search Indexing Pipelines

Confidential + Proprietary
Staged Event Driven Architecture

Confidential + Proprietary
Single-machine pipeline
generalizes to distributed pipelines
Staged Event Driven Architecture

Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers

Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
+
-

Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
+
-

Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
■ event batching +
-

Confidential + ProprietaryConfidential + Proprietary
Leases
as Heart Beat in
Distributed
Systems 1989

Confidential + Proprietary
Leases
● Distributed locking

Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short

Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short
■ delay from client and server failures
minimized
■ reduced false write-sharing

Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long

Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
■ more efficient for frequently accessed
data
■ minimized lease extension overhead on
server and client

Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
● Use of leases in modern applications
○ Leader election TTL (in etcd)

Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
Dita Roque-Gourary

Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Dita Roque-Gourary

Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Dita Roque-Gourary

Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Dita Roque-Gourary

Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Build is finished/fini.
Dita Roque-Gourary
Waiting for the results, merci.

Confidential + Proprietary
Leases in Build System

Confidential + ProprietaryConfidential + Proprietary
Inaccurate
Computations
&
Serving Search
Results

Confidential + Proprietary
From Accurate to
"Good Enough"

Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition

Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness

Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness
3. Criticality Testing

Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness
3. Criticality Testing
4. Distortion and timing models

Confidential + Proprietary
Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

Confidential + Proprietary
Robust & scaleable pipelines

Confidential + Proprietary
Robust & scaleable pipelines
Leases for sharing & heartbeat

Confidential + Proprietary
Robust & scaleable pipelines
Leases for sharing & heartbeat
Trade off inaccuracy for
resilience & performance

Confidential + Proprietary
Robust, scaleable pipelines
Leases for sharing & heartbeat
Trade off inaccurate for
resilient
CS research is
timeless:
use it to mitigate risk

Confidential + Proprietary
Thanks
Ines Sombra
David Greenberg
Alex Hutcheson
Scott Zawalski
Karan Parikh

Confidential + Proprietary
References
● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All"
● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant
Computations that Discard Tasks"
● F. Corbato, M. Daggett, R. Daley "An Experimental Time-
Sharing System"
● E. Dijkstra "Cooperating Sequential Processes"
● L. Lamport "Time, Clocks, and the Ordering of Events in a
Distributed System"

Confidential + Proprietary
References
● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy
Method to Support Highly-Available Distributed Systems"
● L. Lamport "The Part-Time Parliament"
● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-
Conditioned, Scalable Internet Services"
● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant
Mechanism for Distributed File Cache Consistency"
● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded
Errors and Bounded Response Times on Very Large Data"

Confidential + Proprietary
Should I read papers?
YES

Más contenido relacionado

La actualidad más candente

C3P: Context-Aware Crowdsourced Cloud Privacy (at PETS 2014)

Hamza Harkous

This talk will introduce Zero-Knowledge Proofs (ZKPs) and explain why they are a key element in a growing number of privacy-preserving, digital-identity platforms. Clare will provide basic illustrations of ZKPs and leave the necessary mathematics foundations to the readers. After this talk you will understand that there is a variety of ZKPs, it’s still early days, and why ZKP is such a perfect tool for digital identity platforms. This talk includes significant updates from the newly-organized ZKProof Standardization organization plus a signal of maturity: one of the first known ZKP vulnerabilities. Clare will explain why ZKPs are so powerful, and why they are building blocks for a range of applications including privacy-preserving cryptocurrency such as Zcash, Ethereum, Artificial Intelligence, and older versions of Trusted Platform Modules (TPMs). The presentation includes many backup slides for future learning and researching, including four slides of references.

Zero-Knowledge Proofs: Privacy-Preserving Digital Identity with Clare Nelson

SSIMeetup

PGP Basic Lecture 01

Qaisar Ayub

Share Orlando Ulf Mattsson session 9353 2011

Ulf Mattsson

Cook your KV

Gianluca Padovani

As an incident responder, have you ever thought about how much easier an investigation would be if you had the C2 server in your possession? In this talk, we are going to deep dive a rare investigation in which Mandiant obtained a forensic copy of an attacker C2 system. You will learn about the initial compromise of the C2 server, the tools and tactics used by the attacker, and the investigative steps taken to identify the full scope of the attack. In addition, you will learn about the specific challenges involved with the analysis, the tool I developed to carve all PostGreSQL rows from a forensic image, and some unique lessons learned from performing this investigation.

Honey, I Stole Your C2 Server: A Dive into Attacker Infrastructure

Shakacon

La actualidad más candente (6)

C3P: Context-Aware Crowdsourced Cloud Privacy (at PETS 2014)

Zero-Knowledge Proofs: Privacy-Preserving Digital Identity with Clare Nelson

PGP Basic Lecture 01

Share Orlando Ulf Mattsson session 9353 2011

Cook your KV

Honey, I Stole Your C2 Server: A Dive into Attacker Infrastructure

Destacado

Capacity Management for Web Operations

John Allspaw

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/29v2Ick. Aysylu Greenberg discusses some of the new architectural patterns from systems she has worked on at Google and the related work that provides insights into the motivations behind them. Filmed at qconlondon.com. Aysylu Greenberg works at Google on a distributed build system. In her spare time, she ponders the design of systems that deal with inaccuracies.

Distributed Systems in Practice, in Theory

C4Media

Once a system becomes successful, releasing fixes and improvements to its backends without affecting user productivity becomes more challenging. Over time the need arises to re-think the architecture of the system and release its implementation to better support the most popular (and potentially unanticipated) use cases and growth. In globally distributed systems, like the distributed build system at Google which serves millions of requests per day, the luxury of downtime is not an option. In this talk, we’ll look at the general patterns that allow us to replace the previous production system with a new architecture, with no downtime or user visible effects.

MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Aysylu Greenberg

CS6601 DISTRIBUTED SYSTEMS

Kathirvel Ayyaswamy

Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency. About the Speaker Abhishek Verma Software Engineer, Uber Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.

Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...

DataStax

Learn how to build Reactive Systems with Akka. Examples in both Java 8 and Scala. Abstract: The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand. We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will introduce you to Akka and discuss how it can help you deliver on the four key traits of Reactive; Responsive, Resilient, Elastic and Message-Driven. We will start with the basics of Akka and work our way towards some of its more advanced modules such as Akka Cluster and Akka Persistence—all driven through code and practical examples.

Building Reactive Systems with Akka (in Java 8 or Scala)

Jonas Bonér

Distributed system notes unit I

NANDINI SHARMA

Distributed systems in practice, in theory (ScaleConf Colombia)

Aysylu Greenberg

Introduction to Search Systems - ScaleConf Colombia 2017

Toria Gibbs

This presentation will explore some of the teachings from the young field of behavioral game theory, which empirically measures how humans behave in games, as an improvement upon prior discussions involving traditional Game Theory models in which humans are considered perfectly rational. I will use behavioral game theory to examine how people’s natural cognitive biases lead to sub-optimal behavior in their decision-making processes in adversarial games – and specifically processes related to playing defense in the information security “game.” I will detail various sorts of games in which this sub-optimal performance manifests, how humans cognitively approach these games and touch on some of the algorithms, such as self-tuning EWAs, that help predict how people will behave in certain defender-attacker-defender (DAD) games. Finally, I will explore what sort of strategies and counter-measures can be implemented to improve defense’s performance in DAD games, incorporating techniques such as belief prompting, improved incorporation of information and decision trees.

Volatile Memory: Behavioral Game Theory in Defensive Security

Kelly Shortridge

Distributed Systems

Rupsee

Destacado (11)

Capacity Management for Web Operations

Distributed Systems in Practice, in Theory

MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

CS6601 DISTRIBUTED SYSTEMS

Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...

Building Reactive Systems with Akka (in Java 8 or Scala)

Distributed system notes unit I

Distributed systems in practice, in theory (ScaleConf Colombia)

Introduction to Search Systems - ScaleConf Colombia 2017

Volatile Memory: Behavioral Game Theory in Defensive Security

Distributed Systems

Similar a Distributed systems in practice, in theory

Talk @FH Hagenberg - Data viz in a collaborative mixed reality space

Stefan Wasserbauer

Microservices Workshop All Topics Deck 2016

Adrian Cockcroft

The Internet You Want

US-Ignite

It’s hard to imagine a modern developer workflow without a sufficiently advanced build system: Make, Gradle, Maven, Rake, and many others. In this talk, we’ll discuss the evolution of build systems that leads to distributed build systems, like Google's BuildRabbit. Then, we’ll dive into how we can build a scalable system that is fast and resilient, with examples from Google. We’ll conclude with the discussion of general challenges of migrating systems from one architecture to another.

Building a Distributed Build System at Google Scale

Aysylu Greenberg

DLT analytics and AI workshop 13 march 2019

Stavros Zervoudakis

Beyond the Google Search Appliance with Lucidworks Fusion

MC+A

Internet of Things (IOT) Cloud Security by Dr. Anton Ravindran

GSTF

ProtectWise and Ixia make it easy to secure workloads in the cloud, including multi-cloud environments, in addition to traditional on-premises networks. Running entirely from the cloud, the ProtectWise Grid delivers pervasive visibility via automatic real-time and retrospective threat detection, long-term retention of full-fidelity packet capture (PCAP) and unlimited forensic exploration. Ixia CloudLens™ simplifies capture of network traffic from public cloud, virtual, or physical environments and delivers it to The ProtectWise Grid seamlessly. Through this integration, ProtectWise and Ixia give security teams a single solution for end-to-end visibility across heterogenous public cloud, enterprise, and hybrid environments. Attend this webinar and learn how easy it is to secure the cloud with the joint ProtectWise-Ixia solution and: Why security in mixed- and multi-cloud environments can be difficult How moving to the cloud limits visibility for legacy security solutions How a modern approach to security provides pervasive visibility

3 Things to Know When Securing Mixed, Multi-Cloud Environments

ProtectWise

Using Blockchain to Increase Supply Chain Transparency

Horea Porutiu

This talk gives a technical and innovative overview of how companies can face the challenge of protecting the data and services that are in their data-centric platform, focusing on three main aspects: implementing network segmentation, managing AAA and securing data processing. https://www.bigdataspain.org/2017/talk/big-data-security-facing-the-challenge Big Data Spain 2017 16th - 17th November Kinépolis Madrid

Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Big Data Spain

Fast Delivery DevOps Israel

Adrian Cockcroft

Traditional security tools like security information and event managers (SIEMs) are struggling to keep up with the terabytes of event data (250M to 2B events) being generated each day from an ever-growing number of devices. Cybersecurity has become a data problem, and enterprises need to reply with scalable solutions to enable effective hunting and combat evolving attacks. Rethinking the cybersecurity problem as a data-centric problem led Accenture Labs’s Cybersecurity team to use emerging big data tools along with new approaches such as graph databases and analysis to exploit the connected nature of the data to its advantage. Joshua Patterson, Michael Wendt, and Keith Kraus explain how Accenture Labs’s Cybersecurity team is using Apache Kafka, Spark, and Flink to stream data into Blazegraph and Datastax Graph to accelerate cyber defense. Leveraging Datastax Graph and Blazegraph allows Accenture Labs to greatly accelerate query and analysis performance compared to traditional security tools like SIEM. Josh, Michael, and Keith share the challenges of fitting cybersecurity data into each of the graph structures, as well as the ways they exploited the connectedness of events to discover new threats that would have been missed in traditional SIEM tools. In addition, they explain how they use GPUs to accelerate graph analysis by using Blazegraph DASL. Josh, Michael, and Keith end by demonstrating how to efficiently and effectively stream data into these graph databases using best-in-breed technologies such as Apache Kafka, Spark, and Flink and touch on why Kudu is becoming an integral part of Accenture’s technology stack. Utilizing these technologies, clients have supercharged their security analysts’ cyber-hunting abilities and are uncovering threats faster.

Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph an...

Keith Kraus

Webinar: Why evasive zero day attacks are killing traditional sandboxing

Cyren, Inc

apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...

apidays

Looker Studio Data Contracts - Data.Monks.pptx

Doug Hall

2017 12 lab informatics summit

Chris Dwan

PinTrace Advanced AWS meetup

Suman Karumuri

2017 bio it world

Chris Dwan

2019 BioIt World - Post cloud legacy edition

Chris Dwan

Real World Threat Hunting Security threats have grown from network annoyances to attacks on sensitive infrastructure; penetrating network perimeters, moving laterally within networks, breaching new device types, and cloaking movements. This presentation will share techniques utilized by Cisco to detect and investigate sophisticated, embedded threats. The speaker, who has conducted monitoring and investigations on customer networks, will review recent real attacks observed on customer networks, from discovery to remediation, and provide lessons learned. These interactive case examples will highlight how to identify these threats using security intelligence, expert staff, and the Cisco OpenSOC platform. Examples of attacks and illustrations: * Sophisticated phishing attacks targeted at customer environments. * Breaches and data exfiltration resulting from the high-profile HeartBleed and Shellshock vulnerabilities. * Sophisticated malware targeting financial institutions with the goal of data theft. * Use of full packet capture to identify data exfiltration.

CONFidence2015: Real World Threat Hunting - Martin Nystrom

PROIDEA

Similar a Distributed systems in practice, in theory (20)

Talk @FH Hagenberg - Data viz in a collaborative mixed reality space

Microservices Workshop All Topics Deck 2016

The Internet You Want

Building a Distributed Build System at Google Scale

DLT analytics and AI workshop 13 march 2019

Beyond the Google Search Appliance with Lucidworks Fusion

Internet of Things (IOT) Cloud Security by Dr. Anton Ravindran

3 Things to Know When Securing Mixed, Multi-Cloud Environments

Using Blockchain to Increase Supply Chain Transparency

Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Fast Delivery DevOps Israel

Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph an...

Webinar: Why evasive zero day attacks are killing traditional sandboxing

apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...

Looker Studio Data Contracts - Data.Monks.pptx

2017 12 lab informatics summit

PinTrace Advanced AWS meetup

2017 bio it world

2019 BioIt World - Post cloud legacy edition

CONFidence2015: Real World Threat Hunting - Martin Nystrom

Más de Aysylu Greenberg

Several recent high-profile security incidents were due to compromised software supply chains. Software Supply Chain is a collective term used to describe the stages of software lifecycle from source to deployment through CI/CD pipelines, and all the static and dynamic analyses in between. In the world of microservices and cloud computing, trust in your company’s supply chain is critical, as most of the tooling and dependencies are from open source and vendor projects. When the code hits production, it’s essential to have enough observability to detect and investigate the problem and get to the root cause and mitigation as quickly as possible. With software supply chain attacks, not only is the newly deployed code under suspicion, but also all the tooling used to produce it becomes a potential attack vector, so an efficient and effective way to verify the integrity of the supply chain is paramount. This talk will discuss what information needs to be collected to allow DevOps to inspect and verify the integrity of the supply chain, the challenges of having the right level of detail to reduce mean-time-to-detection and mean-time-to-understanding, some of the existing solutions and open problems in this space.

Software Supply Chains for DevOps @ InfoQ Live 2021

Aysylu Greenberg

Kritis is an open-source solution for securing your software supply chain for Kubernetes applications. Kritis enforces deploy-time security policies that ensures only trusted container images are deployed on kubernetes to your cluster. With Kritis, you can require images to be signed by trusted authorities during the development process and then enforce signature validation when deploying. Kritis enables tighter control over your container environment by ensuring only verified images are integrated into production. Talk outline: - Introduction to the concept of binary authorization - Live demo of using Kritis and Grafeas for deploying images with confidence in Kubernetes - Grafeas and Kritis roadmap At the end, attendees will gain solid understanding on the process of binary authorization and how to incorporate it in their build and deployment pipelines

Binary Authorization in Kubernetes

Aysylu Greenberg

Software Supply Chain is a collective term used to describe the continuous integration and delivery pipelines. In addition, it refers to the observability tools that track what happens to a piece of code from the moment it’s in the source code to when it gets deployed, and everywhere in between. Grafeas is an open-source artifact metadata API to audit and govern your software supply chain. It's built as an industry standard for storing and retrieving metadata about software resources. Kritis is an open-source solution for securing your software supply chain for Kubernetes applications. It enforces deploy-time security policies using Grafeas. This talk will discuss the goals for each of the two open source projects, dive into the examples of how they can be used to secure your company's software supply chain, and conclude with the details of current and future development.

Software Supply Chain Management with Grafeas and Kritis

Aysylu Greenberg

Software Supply Chain is a collective term used to describe the continuous integration and delivery pipelines. In addition, it refers to the observability tools that track what happens to a piece of code from the moment it’s in the source code to when it gets deployed, and everywhere in between. Grafeas (https://grafeas.io/) is an open-source artifact metadata API to audit and govern your software supply chain. It's built as an industry standard for storing and retrieving metadata about software resources. Kritis (https://github.com/grafeas/kritis) is an open-source solution for securing your software supply chain for Kubernetes applications. It enforces deploy-time security policies using Grafeas. This talk will discuss the goals for each of the two open source projects, dive into the examples of how they can be used to secure your company's software supply chain, and conclude with the details of current and future development.

Software Supply Chain Observability with Grafeas and Kritis

Aysylu Greenberg

Software Supply Chain Management with Grafeas and Kritis

Aysylu Greenberg

Zero Downtime Migrations at Scale

Aysylu Greenberg

Zero Downtime Migration

Aysylu Greenberg

The paper describes an interesting approach to data replication which allows for finer control over the probability of data loss occurrence and the amount of data loss during such an event. In addition, we'll discuss a technique for moving randomization from runtime to initialization to achieve the same benefits. After the discussion of the paper's contributions, we'll turn to pragmatic aspects of this approach.

PWL Denver: Copysets

Aysylu Greenberg

Distributed systems in practice, in theory (JAX London)

Aysylu Greenberg

It's hard to imagine a modern developer workflow without a sufficiently advanced build system: Make, Gradle, Maven, Rake, and many others. In this talk, we'll discuss the evolution of build systems that leads to distributed build systems. Then, we'll dive into how we can build a scalable system that is fast and resilient, with examples from Google. We'll conclude with the discussion of general challenges of migrating systems from one architecture to another.

Building A Distributed Build System at Google Scale (StrangeLoop 2016)

Aysylu Greenberg

QCon NYC: Distributed systems in practice, in theory

Aysylu Greenberg

Loom is an open-source Clojure library that provides many graph algorithms and visualizations. Loom's graph API focuses on generality and simplicity of integration, which enables other graph representations to be ported to Loom. In this talk, we'll look at how Loom's API and graph implementations evolved in the past 2 years since being presented at Clojure/West 2014. We'll also discuss complexities of maintaining an open-source library.

(+ Loom (years 2))

Aysylu Greenberg

Aysylu Greenberg presents the Probabilistic Accuracy Bounds for Fault-Tolerant Computations that Discard Tasks paper (http://people.csail.mit.edu/rinard/paper/ics06.pdf ) Aysylu tells us "As our systems get more complex and expensive to operate, tradeoffs between accuracy and performance gains become more relevant. The paper demonstrates a new approach to analyzing programs where we can train statistical models to bound the error as tasks fail. This allows us to be more resilient in the face of system failures in many applications that can tolerate "good enough" results. This area of research is particularly dear to my heart as I was first exposed to it while taking a compiler engineering course at MIT which the author, Prof. Martin Rinard, taught. The probabilistic high-performance computing captured my interest because it challenges the widely accepted expectation that for-loops are deterministic."

Probabilistic Accuracy Bounds @ Papers We Love SF

Aysylu Greenberg

Knowledge of how to set up good benchmarks is invaluable in understanding performance of the system. Writing correct and useful benchmarks is hard, and verification of the results is difficult and prone to errors. When done right, benchmarks guide teams to improve the performance of their systems. When done wrong, hours of effort may result in a worse performing application, upset customers or worse! In this talk, we will discuss what you need to know to write better benchmarks. We will look at examples of bad benchmarks and learn about what biases can invalidate the measurements, in the hope of correctly applying our new-found skills and avoiding such pitfalls in the future.

Benchmarking (JAXLondon 2015)

Aysylu Greenberg

Graphs are ubiquitous data structures, and the algorithms for analyzing them are fascinating. Loom is an open-source Clojure library that provides many graph algorithms and visualizations. We will discuss how graphs are represented in a functional world, bridge the gap between procedural description of algorithms and their functional implementation, and learn about the way Loom integrates with other graph representations.

Loom & Functional Graphs in Clojure @ LambdaConf 2015

Aysylu Greenberg

Benchmarking (DevNexus 2015)

Aysylu Greenberg

Knowledge of how to set up good benchmarks is invaluable in understanding performance of the system. Writing correct and useful benchmarks is hard, and verification of the results is difficult and prone to errors. When done right, benchmarks guide teams to improve the performance of their systems. When done wrong, hours of effort may result in a worse performing application, upset customers or worse! In this talk, we will discuss what you need to know to write better benchmarks for distributed systems. We will look at examples of bad benchmarks and learn about what biases can invalidate the measurements, in the hope of correctly applying our new-found skills and avoiding such pitfalls in the future.

Benchmarking (RICON 2014)

Aysylu Greenberg

Benchmarking: You're Doing It Wrong (StrangeLoop 2014)

Aysylu Greenberg

The paper explains how you can write an interpreter and get an optimizing just-in-time (JIT) compiler for free. This enables language designers to focus on features without worrying about the complexities of compiler optimizations and code generation. This paper presents a Java Virtual Machine (JVM) that allows the application to control the JIT compiler behavior at runtime. We'll discuss how various programming languages can take advantage of this framework. To intrigue compiler aficionados, the authors show how combining AST node rewriting during interpretation, optimization, and deoptimization produces high performance code from the interpreter without a language-specific compiler. In addition, they present how features of a variety of programming languages, such as JavaScript, Ruby, Python, R and others, map on the framework.

PWL: One VM to Rule Them All

Aysylu Greenberg

Loom at Clojure/West

Aysylu Greenberg

Más de Aysylu Greenberg (20)

Software Supply Chains for DevOps @ InfoQ Live 2021

Binary Authorization in Kubernetes

Software Supply Chain Management with Grafeas and Kritis

Software Supply Chain Observability with Grafeas and Kritis

Software Supply Chain Management with Grafeas and Kritis

Zero Downtime Migrations at Scale

Zero Downtime Migration

PWL Denver: Copysets

Distributed systems in practice, in theory (JAX London)

Building A Distributed Build System at Google Scale (StrangeLoop 2016)

QCon NYC: Distributed systems in practice, in theory

(+ Loom (years 2))

Probabilistic Accuracy Bounds @ Papers We Love SF

Benchmarking (JAXLondon 2015)

Loom & Functional Graphs in Clojure @ LambdaConf 2015

Benchmarking (DevNexus 2015)

Benchmarking (RICON 2014)

Benchmarking: You're Doing It Wrong (StrangeLoop 2014)

PWL: One VM to Rule Them All

Loom at Clojure/West

Último

At TECUNIQUE, we're a stable and steadily growing Indian software services company with over 14 years of industry experience. Specializing in offshore software development and quality assurance services, we've built a reputation for delivering unique and effective solutions to start-ups, software development companies, enterprises, and digital agencies. We pride ourselves on our commitment to excellence and innovation. By blending insightful business domain knowledge with exceptional technical prowess, we craft tailor-made solutions that meet the unique needs of our clients. Our dedicated teams are adept in specific technologies, ensuring seamless integration of skills and delivering reliable, scalable, and high-quality software solutions aligned with our clients' preferences. Bespoke Dedicated Teams: Crafted to meet your specific needs and technology preferences, our dedicated teams are committed to delivering top-notch software solutions. Offshore Software Development: Accelerate your software development and scale up quickly with our 12+ years of expertise in offshore development. Quality Assurance Services: Ensure the quality of your software products with our dedicated teams of experienced QA professionals. IT Staff Augmentation: Overcome skill gaps with our client-centric software team, offering staff augmentation services. Expert Software Services: Unlock our capabilities in custom software development, product development, and quality assurance. Mission and Vision: Our mission at TECUNIQUE is to be the catalyst for our clients' success in the dynamic domain of software development. Rooted in our core values of respect, authenticity, and responsibility, we strive to ease the software outsourcing experience, reducing both time and cost to market for our clients. We envision ourselves as the leading Indian software services company, renowned for our unwavering commitment to excellence and innovation. www.tecunique.com

TECUNIQUE: Success Stories: IT Service provider

mohitmore19

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

masabamasaba

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

10 Trends Likely to Shape Enterprise Technology in 2024

Mind IT Systems

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

Looking for an efficient way to manage your finances? Look no further than our money management app. With easy-to-use features, you can track your expenses, create budgets, and monitor your savings goals all in one place. Our app provides real-time updates on your spending habits and helps you make smarter financial decisions. Take control of your finances today with our user-friendly money management app.

Right Money Management App For Your Financial Goals

Jhone kinadey

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

Model Call Girl Services in Delhi reach out to us at 🔝 9953056974 🔝✔️✔️ Our agency presents a selection of young, charming call girls available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives, College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Delhi ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 9953056974 🔝. for details Operating 24/7, we serve various locations in Delhi, including Green Park, Lajpat Nagar, Saket, and Hauz Khas near metro stations. For premium call girl services in Delhi 🔝 9953056974 🔝. Thank you for considering us!

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

9953056974 Low Rate Call Girls In Saket, Delhi NCR

Azure Native Qumulo scales elastically for common High Performance Compute (HPC) workloads based on application requirements for: Financial Services, Automotive, Genomics / Life Sciences, Media and Entertainment, Energy, Oil and Gas, etc. Performance can be dialed UP (and back down) much higher than the examples shown here. These slides offer a glimpse into ANQ's HPC capabilities, although at a smaller scale. We invite YOU to do your own testing (with a free ANQ trial) and work with us to test your HPC workloads in Azure.

Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf

ryanfarris8

HR Software Buyers Guide in 2024 - HRSoftware.com

Fatema Valibhai

Many specialized tools cater to distinct stages within the software development lifecycle (SDLC). These tools target various aspects of development, delivery, and operations, each with its unique strengths. Uniting these diverse testing needs into a single continuous testing platform presents several challenges. Such a platform must seamlessly integrate with various development tools and environments, accommodate different testing methodologies, and remain flexible to adapt to organizational processes and quality standards.

The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...

kalichargn70th171

Define the academic and professional writing..pdf

PearlKirahMaeRagusta1

LEVEL 5 - SESSION 1 2023 (1).pptx - PDF 123456

KiaraTiradoMicha

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Jittipong Loespradit

introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf

VishalKumarJha10

MakeMyPass" Online Bus Pass Management System illustrates the flow of activities and actions that occur within the system to accomplish specific tasks or use cases. This type of diagram focuses on representing the sequence of activities and decision points involved in a particular process. Below is an example outline and description of key elements that could be included in an Activity Diagram for the system:

BUS PASS MANGEMENT SYSTEM USING PHP.pptx

alwaysnagaraju26

Unlocking the Future of AI Agents with Large Language Models

aagamshah0812

Pharm-D Biostatistics and Research methodology

Anusha Are

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

Software Quality Assurance Interview Questions

Arshad QA

Distributed systems in practice, in theory

1. Confidential + ProprietaryConfidential + Proprietary Distributed Systems in Practice, in Theory Aysylu Greenberg March 8th, 2016

2. Confidential + Proprietary

3. Confidential + Proprietary

4. Confidential + Proprietary Aysylu Greenberg @aysylu22

5. Confidential + Proprietary Towards Distributed Build System

6. Confidential + Proprietary Towards Distributed Build System

7. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

8. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

9. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

10. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

11. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

12. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

13. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

14. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

15. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

16. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

17. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

18. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

19. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

20. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

21. Confidential + Proprietary http://opencv.org/ Towards Distributed Build System

22. Confidential + Proprietary

23. Confidential + Proprietary Papers We Love SF

24. Confidential + Proprietary Aysylu Greenberg @aysylu22

25. Confidential + Proprietary meetup.com/Papers-We-Love-London

26. Confidential + Proprietary Today ● Staged Event-Driven Architecture

27. Confidential + Proprietary Today ● Staged Event-Driven Architecture ● Leases

28. Confidential + Proprietary Today ● Staged Event-Driven Architecture ● Leases ● Inaccurate Computations

29. Confidential + ProprietaryConfidential + Proprietary Computer Science Research In Distributed Systems Industry

30. Confidential + Proprietary Operating systems research

31. Confidential + Proprietary Operating systems research

32. Confidential + Proprietary Operating systems research Concurrency

33. Confidential + Proprietary Operating systems research Concurrency Concurrency primitives: mutex & semaphore

34. Confidential + Proprietary Operating systems research Concurrency Concurrency primitives: mutex & semaphore Processes execute at different speeds

35. Confidential + Proprietary Time in distributed systems

36. Confidential + Proprietary Time in distributed systems

37. Confidential + Proprietary Time in distributed systems Pipelining

38. Confidential + Proprietary Time in distributed systems Pipelining

39. Confidential + Proprietary Internet

40. Confidential + Proprietary Internet Distributed consensus

41. Confidential + Proprietary Internet Distributed consensus

42. Confidential + Proprietary Internet Distributed consensus

43. Confidential + Proprietary Internet Distributed consensus Paxos

44. Confidential + Proprietary Reconsider large systems

45. Confidential + Proprietary Reconsider large systems Platform as a service

46. Confidential + Proprietary CS Research is Timeless Inform decisions Mitigate technical risk

47. Confidential + ProprietaryConfidential + Proprietary Staged Event Driven Architecture & Deep Pipelines 2001

48. Confidential + Proprietary Hardware Pipelines

49. Confidential + Proprietary Data Pipelines https://en.wikipedia.org/wiki/Graphics_pipeline

50. Confidential + Proprietary Search Indexing Pipelines

51. Confidential + Proprietary Search Indexing Pipelines

52. Confidential + Proprietary Search Indexing Pipelines

53. Confidential + Proprietary Search Indexing Pipelines

54. Confidential + Proprietary Search Indexing Pipelines

55. Confidential + Proprietary Search Indexing Pipelines

56. Confidential + Proprietary Search Indexing Pipelines

57. Confidential + Proprietary Search Indexing Pipelines

58. Confidential + Proprietary Search Indexing Pipelines

59. Confidential + Proprietary Search Indexing Pipelines

60. Confidential + Proprietary Search Indexing Pipelines

61. Confidential + Proprietary

62. Confidential + Proprietary Staged Event Driven Architecture

63. Confidential + Proprietary Single-machine pipeline generalizes to distributed pipelines Staged Event Driven Architecture

64. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers

65. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers + -

66. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning + -

67. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing + -

68. Confidential + Proprietary Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching + -

69. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue

70. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue ■ backpressure

71. Confidential + Proprietary + - Staged Event Driven Architecture ● Dynamic resource controllers ○ automatic tuning ■ thread pool sizing ■ event batching ○ load shedding via queue ■ backpressure ■ threshold, filter, reorder, aggregate

72. Confidential + ProprietaryConfidential + Proprietary Leases as Heart Beat in Distributed Systems 1989

73. Confidential + Proprietary

74. Confidential + Proprietary Leases ● Distributed locking

75. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short

76. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short ■ delay from client and server failures minimized ■ reduced false write-sharing

77. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long

78. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ■ more efficient for frequently accessed data ■ minimized lease extension overhead on server and client

79. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd)

80. Confidential + Proprietary Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd) ○ Liveness detection

81. Confidential + Proprietary

82. Confidential + Proprietary Do the build, s'il vous plaît! Build System Dita Roque-Gourary

83. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Dita Roque-Gourary

84. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Dita Roque-Gourary

85. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Dita Roque-Gourary

86. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Dita Roque-Gourary Waiting for the results, merci.

87. Confidential + Proprietary Do the build, s'il vous plaît! Build System OK/D'Accord! Waiting for the results, merci. Build is in progress Build is finished/fini. Dita Roque-Gourary Waiting for the results, merci.

88. Confidential + Proprietary Leases in Build System

89. Confidential + ProprietaryConfidential + Proprietary Inaccurate Computations & Serving Search Results

90. Confidential + Proprietary From Accurate to "Good Enough"

91. Confidential + Proprietary

92. Confidential + Proprietary

93. Confidential + Proprietary

94. Confidential + Proprietary

95. Confidential + Proprietary

96. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition

97. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness

98. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing

99. Confidential + Proprietary Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing 4. Distortion and timing models

100. Confidential + Proprietary Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

101. Confidential + Proprietary Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

102. Confidential + Proprietary

103. Confidential + Proprietary

104. Confidential + Proprietary Robust & scaleable pipelines

105. Confidential + Proprietary Robust & scaleable pipelines Leases for sharing & heartbeat

106. Confidential + Proprietary Robust & scaleable pipelines Leases for sharing & heartbeat Trade off inaccuracy for resilience & performance

107. Confidential + Proprietary Robust, scaleable pipelines Leases for sharing & heartbeat Trade off inaccurate for resilient CS research is timeless: use it to mitigate risk

108. Confidential + Proprietary Thanks Ines Sombra David Greenberg Alex Hutcheson Scott Zawalski Karan Parikh

109. Confidential + Proprietary References ● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All" ● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant Computations that Discard Tasks" ● F. Corbato, M. Daggett, R. Daley "An Experimental Time- Sharing System" ● E. Dijkstra "Cooperating Sequential Processes" ● L. Lamport "Time, Clocks, and the Ordering of Events in a Distributed System"

110. Confidential + Proprietary References ● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems" ● L. Lamport "The Part-Time Parliament" ● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well- Conditioned, Scalable Internet Services" ● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency" ● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data"

111. Confidential + Proprietary Should I read papers? YES

Distributed systems in practice, in theory

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (6)

Destacado

Destacado (11)

Similar a Distributed systems in practice, in theory

Similar a Distributed systems in practice, in theory (20)

Más de Aysylu Greenberg

Más de Aysylu Greenberg (20)

Último

Último (20)

Distributed systems in practice, in theory