SlideShare una empresa de Scribd logo
1 de 55
Descargar para leer sin conexión
riak@ideeli
Aaron Brown, Lead Systems Engineer
abrown@ideeli.com
@aaronbbrown777
Thursday, February 14, 13
What is an ideeli?
Thursday, February 14, 13
Flash Sales
http://www.flickr.com/photos/moonlightbulb/4064060773/
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Flash Sales
Thursday, February 14, 13
Obligatory Stats
• 2,000,000 visitors/week
• 80,000 front end requests/minute
• 120 production servers
• 300 deploys a year
• 40ms front end response time
Thursday, February 14, 13
Obligatory Riak Stats
• 1.2MM keys
• 4 virtualized nodes
• 4ms response
• 40,000 GETs/min
• 2,000 PUTs/min
Thursday, February 14, 13
The Spike
Thursday, February 14, 13
The Stack
Thursday, February 14, 13
New Toys Are Fun
(9AM - 5PM)
Thursday, February 14, 13
What Counts
1AM - 7AM
Thursday, February 14, 13
K.I.S.S.
Keep It Simple Stupid
Thursday, February 14, 13
K.I.S.S.
Keep It Simple Stupid
Kan It Sell Shoes?
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
• User Favorites
Thursday, February 14, 13
Cache Warming
Thursday, February 14, 13
The Spike
Thursday, February 14, 13
Cache Warming
Pseudo-code:
every.30.minutes do
  all_urls.each do |url|
    curl "#{url}?force_cache=true"
  end
end
Thursday, February 14, 13
Cache Warming
Thursday, February 14, 13
Response times
Thursday, February 14, 13
History
Thursday, February 14, 13
memcached
Thursday, February 14, 13
memcached
~40ms
Thursday, February 14, 13
memcached
5/6 requests ~ 40ms
1/6 requests ~ 3000ms
=
~500ms average
Thursday, February 14, 13
Domino Effect
http://www.flickr.com/photos/soham_pablo/309379628/
Thursday, February 14, 13
Manual Intervention
Thursday, February 14, 13
membase
(couchbase)
Thursday, February 14, 13
Riak
All requests ~40ms
Thursday, February 14, 13
Riak
All requests ~40ms
Thursday, February 14, 13
LABS
• Feature Testing Framework
• Key based on user id
• ValueYAML string w/ list of experiments
Thursday, February 14, 13
Favorites
Thursday, February 14, 13
Favorites
Thursday, February 14, 13
Favorites
• Partially normalized
• user_#{id} → array of products
product_#{id} → metadata about product
Thursday, February 14, 13
Rails Integration
• Modified riak-client gem for Rails 2.3
• Load balancing built in
• Multiple primary and secondary servers
• Custom ActiveSupport::Cache store
Thursday, February 14, 13
Rails Integration
module ActiveSupport
  module Cache
    class RailsStore < Store
      def read
      end
      def write
      end
      def delete
      end
  end
end
config.after_initialize do
  ActionController::Base.cache_store = Rails.rails_store.fragments
end
Thursday, February 14, 13
Monitoring
Thursday, February 14, 13
Graphite
Thursday, February 14, 13
New Relic
Thursday, February 14, 13
Alerting
• Compute random string
• PUT value into Riak
• GET value out of Riak
• alert if value_in != value_out
• alert if operation took too long
Thursday, February 14, 13
Tuning
Thursday, February 14, 13
Bitcask Merges
Thursday, February 14, 13
Disk Usage
Thursday, February 14, 13
Heavy I/O
Thursday, February 14, 13
expiry_grace_time
Thursday, February 14, 13
Offset Merge Windows
Thursday, February 14, 13
Puppet
class ideeli_riak ( $merge_window_start, $merge_window_end ) {
...
file { '/etc/riak/app.config' :
content => template("ideeli_riak/app.config.erb"),
}
...
}
class riak-node {
$merge_window_start = inline_template(
‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>'
)
class { 'ideeli_riak' :
merge_window_start => $merge_window_start,
merge_window_end => 6
}
}
Thursday, February 14, 13
Riak Loves Disk
Thursday, February 14, 13
New Disk Array
Thursday, February 14, 13
New Disk Array
Thursday, February 14, 13
Improvement!
Thursday, February 14, 13
We’re Hiring
http://ideelicareers.com/
Thursday, February 14, 13
Questions?
Thursday, February 14, 13

Más contenido relacionado

Similar a Riak at ideeli

Stanford session
Stanford sessionStanford session
Stanford session
Ty Smith
 
Riak seattle-meetup-august
Riak seattle-meetup-augustRiak seattle-meetup-august
Riak seattle-meetup-august
pharkmillups
 

Similar a Riak at ideeli (20)

Treating Infrastructure as Garbage
Treating Infrastructure as GarbageTreating Infrastructure as Garbage
Treating Infrastructure as Garbage
 
SF Hadoop Users Group August 2014 Meetup Slides
SF Hadoop Users Group August 2014 Meetup SlidesSF Hadoop Users Group August 2014 Meetup Slides
SF Hadoop Users Group August 2014 Meetup Slides
 
dots. Conference Spring 2016 大規模Webサービスを支える技術 (mercari)
dots. Conference Spring 2016 大規模Webサービスを支える技術 (mercari)dots. Conference Spring 2016 大規模Webサービスを支える技術 (mercari)
dots. Conference Spring 2016 大規模Webサービスを支える技術 (mercari)
 
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services 2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
2013 - Matías Paterlini: Escalando PHP con sharding y Amazon Web Services
 
Escalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP ConferenceEscalando una PHP App con DB sharding - PHP Conference
Escalando una PHP App con DB sharding - PHP Conference
 
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and SparkCassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
 
State of Puppet
State of PuppetState of Puppet
State of Puppet
 
Mysql
MysqlMysql
Mysql
 
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at Ooyala
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at OoyalaCassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at Ooyala
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at Ooyala
 
TRAP (transient detection pipeline) status update
TRAP (transient detection pipeline) status updateTRAP (transient detection pipeline) status update
TRAP (transient detection pipeline) status update
 
Getting 100B Metrics to Disk
Getting 100B Metrics to DiskGetting 100B Metrics to Disk
Getting 100B Metrics to Disk
 
Feeding Cassandra with Spark-Streaming and Kafka
Feeding Cassandra with Spark-Streaming and KafkaFeeding Cassandra with Spark-Streaming and Kafka
Feeding Cassandra with Spark-Streaming and Kafka
 
AppEngine Performance Tuning
AppEngine Performance TuningAppEngine Performance Tuning
AppEngine Performance Tuning
 
Deep dive into enterprise data lake through Impala
Deep dive into enterprise data lake through ImpalaDeep dive into enterprise data lake through Impala
Deep dive into enterprise data lake through Impala
 
RailsAdmin - Overview and Best practices
RailsAdmin - Overview and Best practicesRailsAdmin - Overview and Best practices
RailsAdmin - Overview and Best practices
 
Stanford session
Stanford sessionStanford session
Stanford session
 
Unleashing the Rails Asset Pipeline
Unleashing the Rails Asset PipelineUnleashing the Rails Asset Pipeline
Unleashing the Rails Asset Pipeline
 
In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)
In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)
In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)
 
The Rise of BaaS A Utopia for Client-Side Developers
The Rise of BaaS A Utopia for Client-Side DevelopersThe Rise of BaaS A Utopia for Client-Side Developers
The Rise of BaaS A Utopia for Client-Side Developers
 
Riak seattle-meetup-august
Riak seattle-meetup-augustRiak seattle-meetup-august
Riak seattle-meetup-august
 

Último

CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Último (20)

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 

Riak at ideeli