Performance Tuning 101

Problem Approach Nature of ﬁxes War Stories

Performance Tuning 101
An introduction to performance tuning an application in managed
environment

Pavan KS1 Janmejay Singh2

1 mail: itspanzi@gmail.com

blog: http://itspanzi.blogspot.com
2 mail: singh.janmejay@gmail.com

blog: http://codehunk.wordpress.com

Rootconf Bangalore, May 2012


Gist
Don’t be out there with a gun looking for problems.


Gist
Do not go overboard, know your end goal.


Gist
Intution tends to work very poorly. Belive proof, not gut feel.


Gist
Understand your environment. Know what to measure.


Gist
Round pegs for round holes. Use appropriate tools.


Gist
Jumps places, unlike functional bugs, look beyond local
impact.


Gist
impact.
Requires mechanical approach, attack worst problem ﬁrst.


Gist
impact.
Prefer correctness over performance.


Gist
impact.
Reasoning every ﬁx (of paramount importance in
multithreaded app).


Gist
impact.
multithreaded app).
Do one ﬁx at a time, allows easy reasoning and
veriﬁcation, can be reverted.


Gist
impact.
multithreaded app).
Don’t jump in the lake without knowing whats waiting
inside.


Context

App in reference: ThoughtWorks Studios - Go (CI/CD
suite)


Context

suite)
Java (backend) and JRuby on Rails (frontend) webapp.


Context

suite)
Managed environment.


Context

suite)
GNU/Linux


Context

suite)
GNU/Linux
Approach and learning is independent of tech-stack.


Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of ﬁxes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching


Don’t go looking for performance issues

Doing it proactively is a wild goose chase!

Varying parameters and running into a real problem is improbable, if
not impossible.


Remember...


Its easier to reproduce and ﬁx a performance issue when you have
seen it happen in a real setup. That way, you also know for sure, it is
real.


My app is slow, it doesn’t respond...

In case of Go, this could mean,
The dashboard is slow



Javascript is slow



Javascript is slow
Builds taking too long to schedule



Javascript is slow
Worker processes picking up work too late



Javascript is slow
SCM changes reﬂect after a long time



Javascript is slow
SCM changes reﬂect after a long time
Process is hogging resources(requires 16 cores, 8 gig ram
for a basic setup)


Benchmark

How slow is slow?


Benchmark

How slow is slow?
How fast is fast?


Benchmark

How slow is slow?
How fast is fast?
Benchmark using realistic setup.


Benchmark

How slow is slow?
How fast is fast?
Benchmark using realistic setup.
Use a repeatable and automated setup.


Remember...

Do not go overboard.

Know your end goal.


Trust in testing

What if I start at ﬁx?


Trust in testing

What if I stop at ﬁx?


Trust in testing

Why do I need to iterate?


Trust in testing

Why do I need to iterate?
How long should I iterate?


Remember...

Intution tends to work very poorly.

Belive proof, not gut feel.


What to measure

Easy and obvious:
Page response time


What to measure

Easy and obvious:
Page response time
Not so easy:
App throughput
Enqueue/Dequeue time for message queues
CPU/IO churn
Database performance
Lock contention
Managed environment speciﬁc parameters(Memory usage,
GC churn etc).
Environmental issues(too many processes, too low
memory, very slow IO).


Remember...

Understand your environment.

Know what to measure.


Tools to measure

User load
ab
htperf


Tools to measure

User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts


Tools to measure

User load
ab
htperf
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.


Tools to measure

User load
ab
htperf
CPU/IO churn: Load Avg, iostat, klogd, inotify.


Tools to measure

User load
ab
htperf
Database performance: Proﬁler.


Tools to measure

User load
ab
htperf
Lock contention: Proﬁler.


Tools to measure

User load
ab
htperf
Managed env speciﬁc params: GC logs, GC proﬁling.


Tools to measure

User load
ab
htperf
Managed env speciﬁc params: GC logs, GC proﬁling.
Environmental issues: ps, dmesg, /proc/<pid>/status,
/proc/net/dev, syslog, messages.


Remember...

Round pegs for round holes.

Use appropriate tools. Conﬁgure them correctly with least possible
overhead.


How to carry out ﬁxes

Understand libraries: read ﬁne print in documentation, or
better yet, read code.



Be mechanical.



Be mechanical.
Use exhaggerated load to maginify the problem.



Be mechanical.
Use the same load proﬁle to test your ﬁx.



Be mechanical.
Be rigorous.



Be mechanical.
Be rigorous.
Mob pair on complicated ﬁxes, especially ﬁxes involving
data mutation across threads.



Be mechanical.
Be rigorous.
Attack one and only one problem.



Be mechanical.
Be rigorous.
Attack one and only one problem.
Having said all that, correctness always comes ﬁrst.


Remember...

Requires mechanical approach

Attack worst problem first

Reason every fix

Reasoning is of paramount importance in multithreaded app

Do one fix at a time

Allows easy reasoning and verification, can be reverted

Prefer correctness over performance


Lock granularity and lockless

When you ﬁx lock contention somewhere, ensure it hasn’t
moved elsewhere, making overall performance worse.



More than one shared variable mandates synchronization
usually, watch out!



More than one shared variable mandates synchronization
usually, watch out!
Atomic operations and globaly unique objects(like interned
symbols) are your friends.


Remember...

Jumps places, unlike functional bugs

Look beyond local impact of a ﬁx


Garbage Collection

When you hear unresponsiveness is intermittent, be
alarmed!
It probably is GC
It is likely not easy to ﬁx
Root cause may be you or a library


Garbage Collection

alarmed!
It probably is GC
Scan GC logs


Garbage Collection

alarmed!
It probably is GC
Scan GC logs
Proﬁle to see if majority of allocations are from just a
handful of calls
If its several calls with each contributing very little to the pie,
don’t even try ﬁxing it.


Garbage Collection

alarmed!
It probably is GC
Scan GC logs
Proﬁle to see if majority of allocations are from just a
handful of calls
If its several calls with each contributing very little to the pie,
don’t even try ﬁxing it.
Understand GC ergonomics and each algorithm(if you
have alternatives) in detail.
Tweek your max/min permitted allocation judiciously,
malloc/free are expensive system calls.


SQL Database

Optimize connection pool size.
Expand is very expensive.
Is dispatched on caller thread(unless your connection pool
is super-human).


SQL Database

is super-human).
Watch out for slow queries, it can snowball.


SQL Database

is super-human).
Understand your DB settings, MVCC, table-locking
scenarios, row-locking scenarios, transaction isolation
tradeoffs.


SQL Database

is super-human).
Understand your DB settings, MVCC, table-locking
scenarios, row-locking scenarios, transaction isolation
tradeoffs.
Contd...


SQL Database contd...

Explain and Analyze are your friends.



Understanding implications of ﬁle-system IO helps realize
table scans can be horrible
DB runs with ﬁxed page cache.
Table scan can wipe it all cold.
Use indexes to avoid table scan.
Indexes are order sensitive.
Tweek inner queries to reduce selectivity.



Every DB may be subtly different(joins can be more
expensive than materialzed views(if you have em))



Every DB may be subtly different(joins can be more
expensive than materialzed views(if you have em))
Denormalization can help crack tough nuts.


Context Switch Thrashing

Too few cores, too many processes will obviously hurt.



Too many locks will cause too many sleeps, hence too
much ctx switching.



much ctx switching.
Ctx switch may schedule a different process, causing CPU
caches and TLB to get cold.



much ctx switching.
Too frequent IO will cause io-wait, hence switch.
Use of swap can completely kill performance. Swap is for
emergency, not for regular usage.
Over logging can hurt, be careful!
Concept of buffered IO exists for a reason, use it!
Use memory mapped IO if it makes sense.



much ctx switching.
Too frequent IO will cause io-wait, hence switch.
Use of swap can completely kill performance. Swap is for
emergency, not for regular usage.
Over logging can hurt, be careful!
Concept of buffered IO exists for a reason, use it!
Use memory mapped IO if it makes sense.
Assigning very low max memory limit can cause frequent
GCs, allocate enough memory so its your functional thread
that works and not GC threads.


Common Pitfalls

Prejudice
SSL is bad
Using asynchronicity to avoid ﬁxing root cause


Common Pitfalls

Prejudice
SSL is bad
Lack of testing: In Go context, myth of 150 agent limit.


Common Pitfalls

Prejudice
SSL is bad
Lack of setup understanding: Misconception that 64 bit
JVM is always bad.


Common Pitfalls

Prejudice
SSL is bad
JVM is always bad.


Common Pitfalls

Prejudice
SSL is bad
JVM is always bad.
Unchartred waters
ERB garbage churn
Strange JVM allocations
Single threaded database
JRuby class hirarchy modiﬁcation(runtime level) locks.


Know what you are getting into...

Know what you are getting into...


Gist, to reiterate...

impact.
multithreaded app).
Don’t jump in the lake without knowing whats waiting
inside.


Garbage churn issues

Debug log statements



ERB rendering



ERB rendering
Pooling objects with signiﬁcant memory footprint
XML parser factory
Buffered reader



ERB rendering
XML parser factory
Buffered reader
Stream being dereferenced into a string



ERB rendering
XML parser factory
Buffered reader
Stream being dereferenced into a string
Thread locals to the rescue(used for caching big reusable
objects)


Contention and reasoning issues

Locking on interned strings generated for context sensitive
locking



locking
Death of synchronized methods



locking
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)



locking
Use read/write lock when appropriate



locking
Use atomic operations for lockess concurrency



locking
Concurrent data-structure != thread-safety



locking
Take difﬁcult design tradeoffs to keep code requiring a
class of lock in one place



locking
Take difﬁcult design tradeoffs to keep code requiring a
class of lock in one place
Double check locking can be very helpful in frequently
called code


Caching

Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)


Caching

Testing cache with boundry conditions
Test to ensure cache-hit
Ensure boundry conditions are well understood, overﬂow is
judiciously deﬁned/used


Caching

LRU eviction is the right(only?) solution sometimes


Caching

Double check cache invalidation ﬂows, its hard, sometimes
bleeds across classes.


Caching

Double check cache invalidation ﬂows, its hard, sometimes
bleeds across classes.
Be vigilant when using hirarchical caches(no
associative-arrays or maps should be cached), its easy to
run memory out with this.


Caching contd...

Do not cache uncommited data or invalidate cache before
commit(transactional caches may come in handy for this).


Caching contd...

Ensure transactions use cached data iff no other
transaction modiﬁes it or ensure transactionally consistent
invalidation.


Caching contd...

invalidation.
Understand your caching library, too low overﬂow limit may
cause IO issues.


Caching contd...

invalidation.
Understand your caching library, too low overﬂow limit may
cause IO issues.
Use one and only one cache instance(its easy to end up in
a memory runout with this). We had view fragments, urls,
domain objects and view models cached in the same
instance.


Javascript

JS dom frequent offset/size calls hurt


Javascript

Framework provided foreach loops are bad, use native for
loops.


Javascript

loops.
Using beautiful framework provided methods like ﬁlter, ﬁnd
and select can have detrimental consequences.


Javascript

loops.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.


Javascript

loops.
Avoid CSS selectors in frequently called code.


Javascript

loops.
Avoid too much DOM churn.


Javascript

loops.
Use tools like Chromium dev toolset to nail leaks and
proﬁle methods.


Javascript

loops.
Use tools like Chromium dev toolset to nail leaks and
proﬁle methods.
Auto-refreshing dashboards should refresh only visible and
changed area.


Questions?

Questions?

Performance Tuning 101

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to Performance Tuning 101

Similar to Performance Tuning 101 (6)

Recently uploaded

Recently uploaded (20)

Performance Tuning 101