RAILS APPRAILS APP PERFORMANCEPERFORMANCE
AT THEAT THE LIMITLIMIT
NOVEMBER 2018NOVEMBER 2018
BOGDAN GUSIEVBOGDAN GUSIEV

BOGDAN G.BOGDAN G.
is 10 years in IT
8 years with Ruby and Rails
Long Run Rails Contributor
Active Speaker

SOME OF MY GEMSSOME OF MY GEMS
http://github.com/bogdan
datagrid
js-routes
accepts_values_for
furi

THE SCOPE OF THIS PRESENTATIONTHE SCOPE OF THIS PRESENTATION
App server optimization
Only technologies compatible with rails
Relay on rails tools if we can

THE PLANTHE PLAN
1. General optimization
2. When to optimize specifically?
3. Identify slow parts
4. Targeted Optimization Methods

8 years old startup
A lot of code
Rails version from 2.0 to 5.1
35_000 RPM
2TB sharded database

IS RAILS SLOW?IS RAILS SLOW?
A MINUTE OF SEVERE REALITYA MINUTE OF SEVERE REALITY

HOW SLOW IS ACTION CONTROLLER & ACTION DISPATCH?HOW SLOW IS ACTION CONTROLLER & ACTION DISPATCH?
Everyone's favorite hello-world benchmark
Source:
Framework Reqs/sec % from best
-----------------------------------
rack 15839.64 100.00%
sinatra 3977.30 25.11%
grape 2937.03 18.54%
rails-api 1211.33 7.65%
bench-micro

HOW SLOW ISHOW SLOW IS
ACTIVERECORD?ACTIVERECORD?
sql = "select * from users where id = 1"
# No ActiveRecord
Mysql2::Client~query(sql)
# Connection Query
AR::Base.connection.execute(sql)
# ActiveRecord::Base
User.find(1)
Benchmark Gist

DEV ENVIRONMENT:DEV ENVIRONMENT:
QUERY FROM THE SAME MACHINE AS THE DBQUERY FROM THE SAME MACHINE AS THE DB
AWS RDS EC2 INSTANCE:AWS RDS EC2 INSTANCE:
QUERY FROM A DIFFERENT MACHINE THAN THE DBQUERY FROM A DIFFERENT MACHINE THAN THE DB
No ActiveRecord: 7034.8 i/s
Connection query: 6825.3 i/s - same-ish
ActiveRecord::Base: 1244.8 i/s - 5.65x slower
No ActiveRecord: 3204.2 i/s
Connection query: 2762.6 i/s - 1.16x slower
ActiveRecord::Base: 781.3 i/s - 4.10x slower

NO ACTIVERECORD IMPACTNO ACTIVERECORD IMPACT
Up to 4 times faster
Ugliest API
No code reuse tools
The dev process will be slowed down from 4 times

HOW SLOW IS ACTIONVIEW?HOW SLOW IS ACTIONVIEW?
It is really hard to evaluate.
Hello-world benchmark will not show anything
Main things that impact performance:
Advanced layout structure
Render partials
Helper method calls
Using ActiveRecord inside views

GOOD NEWS ABOUTGOOD NEWS ABOUT
ACTIONVIEWACTIONVIEW
Easy to replace with different technology
Client side rendering is available

OPTIMIZATIONOPTIMIZATION ANTI-PATTERNANTI-PATTERN
Our App is slow, can we make it fast?
Sure!

WHAT ARE YOU GOING TO IMPROVE WITH?WHAT ARE YOU GOING TO IMPROVE WITH?
App Server Container
Changing RDBMS
Ruby interpreter

GENERAL OPTIMIZATIONGENERAL OPTIMIZATION
Worms are not made to move fast

EFFECTIVE DATABASE STRUCTUREEFFECTIVE DATABASE STRUCTURE
IS THE ONLY ONE GENERAL OPTIMIZATIONIS THE ONLY ONE GENERAL OPTIMIZATION
TECHNIQUE WE FOUND USEFULTECHNIQUE WE FOUND USEFUL IN 8 YEARSIN 8 YEARS

GREAT DATABASE SCHEMAGREAT DATABASE SCHEMA
Allows all controllers to do their work efficiently
Reduces the operations using ruby to minimum
Reduces SQL complexity

GOLDEN RULESGOLDEN RULES
Optimize data storage for reading not for writing
Business Rules define the database schema
There is usually only one way that "feels" right
Design efficiently for today

CACHE COLUMNSCACHE COLUMNS
FOREIGN KEY CACHE EXAMPLEFOREIGN KEY CACHE EXAMPLE
Poll.has_many :options
Option.belongs_to :poll
Option.has_many :votes
Vote.belongs_to :option, counter_cache: true
Vote.belongs_to :user
Poll.has_many :votes
Vote.belongs_to :poll, counter_cache: true
Vote.validates_uniqueness_of :poll_id, scope: [:user_id]

JOIN AVOIDANCEJOIN AVOIDANCE
Site.has_many :campaigns
Campaign.has_many :offers
Offer.has_many :offer_shares
OfferShare.has_many :visitor_offers
VisitorOffer.has_many :referrals
Referral.scope :by_site, -> (id) {
joins(visitor_offer: {offer_share: {offer: :campaign}}).
where(campaigns: {site_id: id})
}
add_column :referrals, :site_id
Referral.before_validation do
self.site_id = visitor_offer.offer_share.offer.campaign.site_id
end

REAL EXAMPLE OFREAL EXAMPLE OF
CACHE COLUMNSCACHE COLUMNS
id :integer
referred_origin_id :integer
visitor_offer_id :integer
status :string(20)
webhook_status :string(10)
track_method :string(20)
processed_by :integer
created_at :datetime
updated_at :datetime
processed_at :datetime
offer_id :integer
site_id :integer
campaign_id :integer
advocate_visitor_id :integer
friend_timing :decimal
referred_subtotal :decimal
qa_generated :boolean
ad_rewarded :boolean

CACHE COLUMNSCACHE COLUMNS
BEST PRACTICESBEST PRACTICES
Mostly for read-only data
Remember what is the source and what is cache
Watch the disk space
It is worth it!

SPECIFIC OPTIMIZATIONSPECIFIC OPTIMIZATION
Applied only to problematic pieces
Makes sense for the used functionality
Makes sense only when the functionality is stable
Can be faster than switch to faster technology in general

HOW BUSINESS VIEWS THE OPTIMIZATION?HOW BUSINESS VIEWS THE OPTIMIZATION?
You: Lets Optimize!
Business:
Business says yes when:
Functionality is stable
Feature is being used
Company is making money with it

HOW TO IDENTIFYHOW TO IDENTIFY
THE OPTIMIZATION AREASTHE OPTIMIZATION AREAS

THROUGHPUT BY ACTIONTHROUGHPUT BY ACTION

RESOURCES CONSUMEDRESOURCES CONSUMED

OPTIMIZATIONOPTIMIZATION
MEASUREMENTMEASUREMENT
Average response time
Consumed resources
(pseudocode)
select controller, action, http_method,
sum(duration), average(duration), count(*)
from app_server_requests

TOOLSTOOLS
HTTP access.log
Rails production.log
Kibana
New Relic
Your variant?

SPECIFIC OPTIMIZATIONSPECIFIC OPTIMIZATION
PLANPLAN
1. Define Optimization Metric
2. Measure the metric
3. Zoom in to see optimizable parts
4. Choose optimized fragment and strategy

OPTIMIZATION METHODSOPTIMIZATION METHODS

AVOIDING ACTIVERECORDAVOIDING ACTIVERECORD
Looks awesome on the first glance
Less valuable with effective DB schema
More work with more DB queries
Unreusable code
Caching can be superior

AVOIDING ACTIVERECORD EXAMPLESAVOIDING ACTIVERECORD EXAMPLES
class Campaign
has_many :tags
def tag_names
tags.pluck(:name)
end
end
def offers_created_at_range
scope = offers.reorder(created_at: :asc).limit(1)
scope.pluck(:created_at).first..
scope.reverse_order.pluck(:created_at).first
end

BOGDAN ❤ ACTIVERECORDBOGDAN ❤ ACTIVERECORD
WHY DON'T ALL PEOPLE LOVE ACTIVERECORD?WHY DON'T ALL PEOPLE LOVE ACTIVERECORD?
They don't optimize their schema
They don't realize it does the hardest work
They don't appreaciate the feature-set/performance trade off
ALL MY EXAMPLES ARE CONSIDEREDALL MY EXAMPLES ARE CONSIDERED
MICRO OPTIMIZATIONMICRO OPTIMIZATION

CONDITIONAL GETCONDITIONAL GET
Client GET /products/1827.json
Server Response /products/1827.json
Etag: "2018-10-29 16:36"
Client GET /products/1827.json
If-None-Match: "2018-10-29 16:36"
Server STATUS 304 Not Modified
OR
Response /products/1827.json
Etag: "2018-10-29 16:37"

CONDITIONAL GET TOOLSCONDITIONAL GET TOOLS
class Api::ProductsController < ApplicationController
def show
@product = Product.find(params[:id])
if stale?(last_modified: @product.updated_at,
etag: @product.cache_key)
render json: { product: @product }
end
end
end
Rails Conditional GET Guide

CONDITIONAL GETCONDITIONAL GET
IS GREAT WHENIS GREAT WHEN
A single page is viewed multiple times by the same browser
Lightweight actions
Actions returning JSON

REWRITING WITH RACKREWRITING WITH RACK
1. The action is already under 500ms
2. Heavily loaded action
3. It is painful
4. Looks better than rewriting on Sinatra

CONDITIONAL GET HELD IN RACKCONDITIONAL GET HELD IN RACK
class OffersController < ApplicationController
def show
@offer = Offer.find(params[:id])
digest = SecureRandom.uuid
data = { offer_id: offer.id, cached_at: Time.zone.now, }
Rails.cache.write(digest, data,
expires_in: CACHE_PERIOD.from_now)
response.headers['Cache-Control'] =
'max-age=0, private, must-revalidate'
response.headers['ETag'] = %(W/"#{digest}")
end
end

RACK MIDDLEWARERACK MIDDLEWARE
def call(env)
if fresh?(env['HTTP_IF_NONE_MATCH'])
return [304, {}, ['Not Modified']]
end
@app.call(env)
end
def fresh?(etag_header)
return unless data = Rails.cache.read(digest)
site_id, timestamp =
Offer.where(id: data[:offer_id]).pluck(:site_id, :updated_at)
SettingsChange.where(site_id: site_id, created_at: lookup_period)
!((data[:cached_at]..Time.zone.now).include?(timestamp))
end

INTRODUCE CACHINGINTRODUCE CACHING
1. Find suitable code fragment
2. Measure Cache Hit
3. Use expiration by key
4. Always expire by timeout
5. Expire on deploy

CACHE HITCACHE HIT
How many cache entries will be made?
What is the average size of cache entry?
How many times each entry will be used?
What % of requests will use cache?
How much memory would it take?

CACHE HIT EXAMPLECACHE HIT EXAMPLE
class Campaign
has_many :offers
def cached_offers_count
@offers_count ||= if active?
offers.count
else
# Caching only inactive campaigns
# because they can not get additional offers
Rails.cache.fetch(
["campaign_offers_count", id], expires_in: 1.month) d
offers.count
end
end
end
end

EXPIRATION BY KEYEXPIRATION BY KEY
Manual expiration example
def show
response = Rails.cache.fetch(["products", @product.id]) do
@product.to_json
end
render json: response
end
def update
@product.update!(params[:product])
Rails.cache.delete(["products", @product.id])
end

EXPIRATION BY KEY EXAMPLEEXPIRATION BY KEY EXAMPLE
class ViewStylesheet
def css
# Combining many cache keys here
# to expire whenever any object is updated
cache_key = ['view_setup_css', campaign, self, account]
Rails.cache.fetch(cache_key,
force: template_changed?, expires_in: 3.days) do
Sass.compile(template)
end
end
end
Key-based Expiration from DHH

EXPIRATION BEST PRACTICESEXPIRATION BEST PRACTICES
config.cache_store = :redis_store,
namespace: Deploy.identifier, expires_in: 1.day

RAILS MAGIC ON WORKING WITH CACHERAILS MAGIC ON WORKING WITH CACHE
Variables usage inside ActionView is implicit
Magic is always slow
- cache @projects do
- @projects.each do |project| %>
= render partial: 'projects/project', project
%a{href: project_path(product)}= project.name
.star{class: current_user.bookmarked?(project) ? 'enabled' : ''}

OPTIMIZABLE REQUESTSOPTIMIZABLE REQUESTS
Lightweight
GET method
Used data is explicit
Return JSON but not HTML
Do not use ActionView

WHAT OPTIMIZABLE PAGE SHOULD LOOK LIKE?WHAT OPTIMIZABLE PAGE SHOULD LOOK LIKE?

OPTIMIZATION ❤ CLIENT SIDE RENDERINGOPTIMIZATION ❤ CLIENT SIDE RENDERING
Saves server resources
Parallel load
Makes used data explicit

OPTIMIZATION BYOPTIMIZATION BY
CODE STRUCTURE CHANGECODE STRUCTURE CHANGE
Trivial
Always considered first
Significant
If it gives a huge performance boost
Radical
If you re-think the business process

TRIVIAL CODE STRUCTURE CHANGETRIVIAL CODE STRUCTURE CHANGE
def render_main_template
- view_setup.render_liquid(:main_template, translator, options)
+ template = rendering_cache do
+ view_setup.render_liquid(:main_template, translator, options)
+ end
end
def rendering_cache
# 100 lines of code
end

RESULTS OF GOOD OPTIMIZATIONRESULTS OF GOOD OPTIMIZATION
Throughput 35_000 RPM
Infrastructure Cost $16_000/Month

AWS SETUPAWS SETUP
Instance Type c4.2xlarge
vCPU 8
ECU 31
Memory 15 GiB
Requests 120/second
Per CPU 15/second

THE STRATEGYTHE STRATEGY
1. Generic Optimization
Have the schema always optimized
Add cache columns
2. Ensure specific optimization is needed
Functionality is stable
The performance is measured
3. Apply specific optimization
Use Conditional Get
Rewrite with Rack
Introduce caching
Use direct SQL
ETC