SlideShare una empresa de Scribd logo
1 de 55
Descargar para leer sin conexión
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
• built at Staples-SparX
• one box serving all Staples’s
experimentations
• 8 GB of data per day
• 5 million sessions a day
• 500 requests per second
• SLA of 99.9th percentile at 10ms
what we built
• values of different experiments setup
• how to efficiently use traffic
• some nice things about clojure
• building assembly lines using core.async
• putting a complex system under simulation testing
what you will learn
1. explaining experimentation
2. implementation
3. simulation testing
structure of the talk
explaining
experimentation
experimentation is the step in the scientific
method that helps people decide between two
or more competing explanations – or
hypotheses.
the experimental method
experimentation in
business
• a process for where business ideas can be
evaluated at scale, analyzed scientifically and in
a consistent manner
• data driven decisions
hypotheses
• “a red button will be more compelling than a
blue button”
• algorithms, navigation flows
• measurement of overall performance of an
entire product
treatment
• values for the variables in the system under
investigation
• control (no treatment) vs test (some treatment)
• red/blue/green
coverage
• effect of external factors (business rules,
integration bug, etc.)
• fundamental in ensuring a precise
measurement
• design: not covered by default
sequence of interactions
experiment
infrastructure
A/B
traffic is split
A/B/C
no limitation in the number of treatments you can
associate to an experiment
messy
testing orthogonal hypotheses
precise
testing non-orthogonal hypotheses
messy/precise
first version of experiment infrastructure
traffic is precious
nested
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
shared bucket
A/A
null hypothesis test
why build ep?
• capacity to run a lot of experiments in parallel
• eCommerce opinionated
• low latency (synchronous)
• real time reports
• controlled ramp-ups
• layered experiments
• statistically sound (needs to be auditable by data
scientists, CxOs, etc.)
• deeper integration
परन्तु
• the domain is quite complex
• significant investment of time, effort and
maintenance (takes years to build correctly)
• you might not need to build this if your
requirements can be met with existing 3rd
party services.
implementation
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
postgres cluster
• data centered domain
• data integrity
• quick failover mechanism
• no out of the box postgres cluster management solution
• built it ourselves using repmgr
• multiple lines of defense
• repmgr pushes
• applications poll
• zfs - mirror and incremental snapshots
reporting on postgres
• sweet spot of a medium sized warehouse
• optimized for large reads
• streams data from master (real time reports)
• crazy postgres optimizations
• maintenance (size, bloat) is non trivial
• freenode#postgresql rocks!
real OLAP solution
• reporting on historical data (older than 6 months)
• reporting across multiple systems’ data
• tried greenplum
• loading, reporting was pretty fast
• has a ‘merge’/upsert strategy for loading data
• not hosted, high ops cost
• leveraged existing ETL service built for Redshift
• assembly line built using core.async
Building an Experimentation Platform in Clojure
why clojure?
• lets us focus on the actual problem
• expressiveness (examples ahead)
• jvm: low latency, debugging, profiling
• established language of choice among the teams
• java, scala, go, haskell, rust, c++
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
Building an Experimentation Platform in Clojure
परन्तु
realize your lazy seqs!
simulation
testing
why
• top of the test pyramid
• generating confidence that your system will
behave as expected during runtime
• humans can't possibly think of all the test cases
• simulation testing is the extension of property
based testing to whole systems
• testing a system or a collection of systems as a
whole
tools
• simulant - library and schema for developing
simulation-based tests
• causatum - library designed to generate streams
of timed events based on stochastic state
machines
• datomic - data store
Building an Experimentation Platform in Clojure
state machine to create streams of actions
run the simulation, record the data
setting up and teardown of target system
validate the recorded data
examples of validations
• are all our requests are returning non-500
responses under the given SLA.
• invalidity checks for sessions, like no
conflicting treatments were assigned
• traffic distribution
• the reports match
running diagnostics
• all the data is recorded
• you can create a timeline for a specific session
from the data recorded for diagnostics purposes
Building an Experimentation Platform in Clojure
परन्तु
• requires dedicated time and effort
• was difficult to for us to put into CI
• many moving parts
conclusions
• traffic is precious, take it account when
you are designing your experiments
• ETL as assembly line work amazingly well
• test your system from the outside
• use simulation testing
• use clojure ;)
• Overlapping Experiment Infrastructure
• More, Better, Faster Experimentation (Google)
• A/B testing @ Internet Scale
• LinkedIn, Bing, Google
• Controlled experiments on the web
• survey and practical guide
• D. Cox and N. Reid
• The theory of the design of experiments, 2000
• Netflix Experimentation Platform
• Online Experimentation at Microsoft
• Practical Guide to Controlled Experiments on the Web:
Listen to Your Customers not to the HiPPO (Microsoft)
Great Material on
Experiment Infrastructure
Building an Experimentation Platform in Clojure

Más contenido relacionado

La actualidad más candente

An Introduction to Prometheus
An Introduction to PrometheusAn Introduction to Prometheus
An Introduction to PrometheusEvgeny Shmarnev
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Brian Brazil
 
Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P) Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P) Elasticsearch
 
Cassandra Essentials Day Cambridge
Cassandra Essentials Day CambridgeCassandra Essentials Day Cambridge
Cassandra Essentials Day CambridgeMarc Fielding
 
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...InfluxData
 
An introduction to Workload Modelling for Cloud Applications
An introduction to Workload Modelling for Cloud ApplicationsAn introduction to Workload Modelling for Cloud Applications
An introduction to Workload Modelling for Cloud ApplicationsRavi Yogesh
 
Hibernate performance tuning
Hibernate performance tuningHibernate performance tuning
Hibernate performance tuningIgor Dmitriev
 
Test Automation for NoSQL Databases
Test Automation for NoSQL DatabasesTest Automation for NoSQL Databases
Test Automation for NoSQL DatabasesTobias Trelle
 
Introduction to Prometheus and Cortex (WOUG)
Introduction to Prometheus and Cortex (WOUG)Introduction to Prometheus and Cortex (WOUG)
Introduction to Prometheus and Cortex (WOUG)Weaveworks
 
Entity framework advanced
Entity framework advancedEntity framework advanced
Entity framework advancedUsama Nada
 
Presto meetup 2015-03-19 @Facebook
Presto meetup 2015-03-19 @FacebookPresto meetup 2015-03-19 @Facebook
Presto meetup 2015-03-19 @FacebookTreasure Data, Inc.
 
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016ShuttleCloud
 
Populate your Search index, NEST 2016-01
Populate your Search index, NEST 2016-01Populate your Search index, NEST 2016-01
Populate your Search index, NEST 2016-01David Smiley
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...Hakka Labs
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemAccumulo Summit
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with PrometheusQAware GmbH
 
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민NAVER D2
 
Sql azure cluster dashboard public.ppt
Sql azure cluster dashboard public.pptSql azure cluster dashboard public.ppt
Sql azure cluster dashboard public.pptQingsong Yao
 

La actualidad más candente (20)

An Introduction to Prometheus
An Introduction to PrometheusAn Introduction to Prometheus
An Introduction to Prometheus
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)
 
Reactive Streams
Reactive StreamsReactive Streams
Reactive Streams
 
Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P) Architectural Best Practices to Master + Pitfalls to Avoid (P)
Architectural Best Practices to Master + Pitfalls to Avoid (P)
 
Cassandra Essentials Day Cambridge
Cassandra Essentials Day CambridgeCassandra Essentials Day Cambridge
Cassandra Essentials Day Cambridge
 
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...
Monitoring and Alerting with InfluxDB 2.0 | Deniz Kusefoglu & Nate Isley | In...
 
An introduction to Workload Modelling for Cloud Applications
An introduction to Workload Modelling for Cloud ApplicationsAn introduction to Workload Modelling for Cloud Applications
An introduction to Workload Modelling for Cloud Applications
 
Hibernate performance tuning
Hibernate performance tuningHibernate performance tuning
Hibernate performance tuning
 
Hui 3.0
Hui 3.0Hui 3.0
Hui 3.0
 
Test Automation for NoSQL Databases
Test Automation for NoSQL DatabasesTest Automation for NoSQL Databases
Test Automation for NoSQL Databases
 
Introduction to Prometheus and Cortex (WOUG)
Introduction to Prometheus and Cortex (WOUG)Introduction to Prometheus and Cortex (WOUG)
Introduction to Prometheus and Cortex (WOUG)
 
Entity framework advanced
Entity framework advancedEntity framework advanced
Entity framework advanced
 
Presto meetup 2015-03-19 @Facebook
Presto meetup 2015-03-19 @FacebookPresto meetup 2015-03-19 @Facebook
Presto meetup 2015-03-19 @Facebook
 
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
 
Populate your Search index, NEST 2016-01
Populate your Search index, NEST 2016-01Populate your Search index, NEST 2016-01
Populate your Search index, NEST 2016-01
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with Prometheus
 
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
Elastic v5.0.0 Update uptoalpha3 v0.2 - 김종민
 
Sql azure cluster dashboard public.ppt
Sql azure cluster dashboard public.pptSql azure cluster dashboard public.ppt
Sql azure cluster dashboard public.ppt
 

Similar a Building an Experimentation Platform in Clojure

Performance tuning Grails applications
 Performance tuning Grails applications Performance tuning Grails applications
Performance tuning Grails applicationsGR8Conf
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Lari Hotari
 
05. performance-concepts-26-slides
05. performance-concepts-26-slides05. performance-concepts-26-slides
05. performance-concepts-26-slidesMuhammad Ahad
 
Performance tuning Grails applications SpringOne 2GX 2014
Performance tuning Grails applications SpringOne 2GX 2014Performance tuning Grails applications SpringOne 2GX 2014
Performance tuning Grails applications SpringOne 2GX 2014Lari Hotari
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Databricks
 
Mtc learnings from isv & enterprise (dated - Dec -2014)
Mtc learnings from isv & enterprise (dated - Dec -2014)Mtc learnings from isv & enterprise (dated - Dec -2014)
Mtc learnings from isv & enterprise (dated - Dec -2014)Govind Kanshi
 
Mtc learnings from isv & enterprise interaction
Mtc learnings from isv & enterprise  interactionMtc learnings from isv & enterprise  interaction
Mtc learnings from isv & enterprise interactionGovind Kanshi
 
Building data intensive applications
Building data intensive applicationsBuilding data intensive applications
Building data intensive applicationsAmit Kejriwal
 
A data driven etl test framework sqlsat madison
A data driven etl test framework sqlsat madisonA data driven etl test framework sqlsat madison
A data driven etl test framework sqlsat madisonTerry Bunio
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools Mickey Boxell
 
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!Richard Robinson
 
Observability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyObservability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyTimetrix
 
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability -  The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Observability -  The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Aleksandr Tavgen
 
Cloud patterns at Carleton University
Cloud patterns at Carleton UniversityCloud patterns at Carleton University
Cloud patterns at Carleton UniversityTaswar Bhatti
 
QA Team Goes to Agile and Continuous integration
QA Team Goes to Agile and Continuous integrationQA Team Goes to Agile and Continuous integration
QA Team Goes to Agile and Continuous integrationSujit Ghosh
 
Cloud Design Patterns - Hong Kong Codeaholics
Cloud Design Patterns - Hong Kong CodeaholicsCloud Design Patterns - Hong Kong Codeaholics
Cloud Design Patterns - Hong Kong CodeaholicsTaswar Bhatti
 
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...InfluxData
 
Machine learning systems for engineers
Machine learning systems for engineersMachine learning systems for engineers
Machine learning systems for engineersCameron Joannidis
 
Discrete event simulation
Discrete event simulationDiscrete event simulation
Discrete event simulationssusera970cc
 

Similar a Building an Experimentation Platform in Clojure (20)

Performance tuning Grails applications
 Performance tuning Grails applications Performance tuning Grails applications
Performance tuning Grails applications
 
Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014Performance tuning Grails Applications GR8Conf US 2014
Performance tuning Grails Applications GR8Conf US 2014
 
05. performance-concepts-26-slides
05. performance-concepts-26-slides05. performance-concepts-26-slides
05. performance-concepts-26-slides
 
Performance tuning Grails applications SpringOne 2GX 2014
Performance tuning Grails applications SpringOne 2GX 2014Performance tuning Grails applications SpringOne 2GX 2014
Performance tuning Grails applications SpringOne 2GX 2014
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
 
Mtc learnings from isv & enterprise (dated - Dec -2014)
Mtc learnings from isv & enterprise (dated - Dec -2014)Mtc learnings from isv & enterprise (dated - Dec -2014)
Mtc learnings from isv & enterprise (dated - Dec -2014)
 
Mtc learnings from isv & enterprise interaction
Mtc learnings from isv & enterprise  interactionMtc learnings from isv & enterprise  interaction
Mtc learnings from isv & enterprise interaction
 
Building data intensive applications
Building data intensive applicationsBuilding data intensive applications
Building data intensive applications
 
A data driven etl test framework sqlsat madison
A data driven etl test framework sqlsat madisonA data driven etl test framework sqlsat madison
A data driven etl test framework sqlsat madison
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools
 
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
 
Performance Testing Overview
Performance Testing OverviewPerformance Testing Overview
Performance Testing Overview
 
Observability – the good, the bad, and the ugly
Observability – the good, the bad, and the uglyObservability – the good, the bad, and the ugly
Observability – the good, the bad, and the ugly
 
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability -  The good, the bad and the ugly Xp Days 2019 Kiev Ukraine Observability -  The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
Observability - The good, the bad and the ugly Xp Days 2019 Kiev Ukraine
 
Cloud patterns at Carleton University
Cloud patterns at Carleton UniversityCloud patterns at Carleton University
Cloud patterns at Carleton University
 
QA Team Goes to Agile and Continuous integration
QA Team Goes to Agile and Continuous integrationQA Team Goes to Agile and Continuous integration
QA Team Goes to Agile and Continuous integration
 
Cloud Design Patterns - Hong Kong Codeaholics
Cloud Design Patterns - Hong Kong CodeaholicsCloud Design Patterns - Hong Kong Codeaholics
Cloud Design Patterns - Hong Kong Codeaholics
 
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
Using InfluxDB for Full Observability of a SaaS Platform by Aleksandr Tavgen,...
 
Machine learning systems for engineers
Machine learning systems for engineersMachine learning systems for engineers
Machine learning systems for engineers
 
Discrete event simulation
Discrete event simulationDiscrete event simulation
Discrete event simulation
 

Más de Srihari Sriraman

REPL Driven Mobile Development with Clojure(script)
REPL Driven Mobile Development with Clojure(script)REPL Driven Mobile Development with Clojure(script)
REPL Driven Mobile Development with Clojure(script)Srihari Sriraman
 
Practical Generative Testing Patterns
Practical Generative Testing PatternsPractical Generative Testing Patterns
Practical Generative Testing PatternsSrihari Sriraman
 
Making machines that make music
Making machines that make musicMaking machines that make music
Making machines that make musicSrihari Sriraman
 
On The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL ClusterOn The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL ClusterSrihari Sriraman
 
Overlapping Experiments Infrastructure
Overlapping Experiments InfrastructureOverlapping Experiments Infrastructure
Overlapping Experiments InfrastructureSrihari Sriraman
 
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014Srihari Sriraman
 

Más de Srihari Sriraman (9)

REPL Driven Mobile Development with Clojure(script)
REPL Driven Mobile Development with Clojure(script)REPL Driven Mobile Development with Clojure(script)
REPL Driven Mobile Development with Clojure(script)
 
Practical Generative Testing Patterns
Practical Generative Testing PatternsPractical Generative Testing Patterns
Practical Generative Testing Patterns
 
Making machines that make music
Making machines that make musicMaking machines that make music
Making machines that make music
 
On The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL ClusterOn The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL Cluster
 
Overlapping Experiments Infrastructure
Overlapping Experiments InfrastructureOverlapping Experiments Infrastructure
Overlapping Experiments Infrastructure
 
Carnatic Music Synthesis
Carnatic Music SynthesisCarnatic Music Synthesis
Carnatic Music Synthesis
 
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014
Making an impact in Rural India with Ruby - Lightning talk at GCRC 2014
 
Rails on Mobile
Rails on MobileRails on Mobile
Rails on Mobile
 
Carnatic Music App
Carnatic Music AppCarnatic Music App
Carnatic Music App
 

Último

Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 

Último (20)

Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 

Building an Experimentation Platform in Clojure

  • 3. • built at Staples-SparX • one box serving all Staples’s experimentations • 8 GB of data per day • 5 million sessions a day • 500 requests per second • SLA of 99.9th percentile at 10ms what we built
  • 4. • values of different experiments setup • how to efficiently use traffic • some nice things about clojure • building assembly lines using core.async • putting a complex system under simulation testing what you will learn
  • 5. 1. explaining experimentation 2. implementation 3. simulation testing structure of the talk
  • 7. experimentation is the step in the scientific method that helps people decide between two or more competing explanations – or hypotheses. the experimental method
  • 8. experimentation in business • a process for where business ideas can be evaluated at scale, analyzed scientifically and in a consistent manner • data driven decisions
  • 9. hypotheses • “a red button will be more compelling than a blue button” • algorithms, navigation flows • measurement of overall performance of an entire product
  • 10. treatment • values for the variables in the system under investigation • control (no treatment) vs test (some treatment) • red/blue/green
  • 11. coverage • effect of external factors (business rules, integration bug, etc.) • fundamental in ensuring a precise measurement • design: not covered by default
  • 15. A/B/C no limitation in the number of treatments you can associate to an experiment
  • 18. messy/precise first version of experiment infrastructure
  • 25. why build ep? • capacity to run a lot of experiments in parallel • eCommerce opinionated • low latency (synchronous) • real time reports • controlled ramp-ups • layered experiments • statistically sound (needs to be auditable by data scientists, CxOs, etc.) • deeper integration
  • 26. परन्तु • the domain is quite complex • significant investment of time, effort and maintenance (takes years to build correctly) • you might not need to build this if your requirements can be met with existing 3rd party services.
  • 30. postgres cluster • data centered domain • data integrity • quick failover mechanism • no out of the box postgres cluster management solution • built it ourselves using repmgr • multiple lines of defense • repmgr pushes • applications poll • zfs - mirror and incremental snapshots
  • 31. reporting on postgres • sweet spot of a medium sized warehouse • optimized for large reads • streams data from master (real time reports) • crazy postgres optimizations • maintenance (size, bloat) is non trivial • freenode#postgresql rocks!
  • 32. real OLAP solution • reporting on historical data (older than 6 months) • reporting across multiple systems’ data • tried greenplum • loading, reporting was pretty fast • has a ‘merge’/upsert strategy for loading data • not hosted, high ops cost • leveraged existing ETL service built for Redshift • assembly line built using core.async
  • 34. why clojure? • lets us focus on the actual problem • expressiveness (examples ahead) • jvm: low latency, debugging, profiling • established language of choice among the teams • java, scala, go, haskell, rust, c++
  • 42. why • top of the test pyramid • generating confidence that your system will behave as expected during runtime • humans can't possibly think of all the test cases • simulation testing is the extension of property based testing to whole systems • testing a system or a collection of systems as a whole
  • 43. tools • simulant - library and schema for developing simulation-based tests • causatum - library designed to generate streams of timed events based on stochastic state machines • datomic - data store
  • 45. state machine to create streams of actions
  • 46. run the simulation, record the data
  • 47. setting up and teardown of target system
  • 49. examples of validations • are all our requests are returning non-500 responses under the given SLA. • invalidity checks for sessions, like no conflicting treatments were assigned • traffic distribution • the reports match
  • 50. running diagnostics • all the data is recorded • you can create a timeline for a specific session from the data recorded for diagnostics purposes
  • 52. परन्तु • requires dedicated time and effort • was difficult to for us to put into CI • many moving parts
  • 53. conclusions • traffic is precious, take it account when you are designing your experiments • ETL as assembly line work amazingly well • test your system from the outside • use simulation testing • use clojure ;)
  • 54. • Overlapping Experiment Infrastructure • More, Better, Faster Experimentation (Google) • A/B testing @ Internet Scale • LinkedIn, Bing, Google • Controlled experiments on the web • survey and practical guide • D. Cox and N. Reid • The theory of the design of experiments, 2000 • Netflix Experimentation Platform • Online Experimentation at Microsoft • Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO (Microsoft) Great Material on Experiment Infrastructure