Alfresco DevCon 2019 Performance Tools of the Trade

Performance Tools of the Trade
Luis Colorado, Zia Consulting

Learn. Connect. Collaborate.
Happy New
Alfresco User

But time
passes…
• Slow searches
• Slow pages
• Slow everything!
• Crashes!
What
to do?

Caveat
Emptor
• Not covering:
– Performance techniques
– Architecture
– Development techniques
– (Okay, okay… we will cover a bit of it)
• Focus on Alfresco on-premise
– But it also applies Cloud deployments

Slowness:
level 1
• Things are slower than demo
• We have only loaded like a million documents!
• Users are complaining that Alfresco is very
slow

Slowness:
level 2
• Sometimes pages or searches take more than
30 seconds to load
• Searches are terrible slow
• Users start to click two or three times on links
• Some pages will just not load

Slowness:
level 3
• Alfresco Servers starts to crash once on a
while
• But they start crashing more and more often

Finding the
Bottleneck
• But it feels like finding a needle in a haystack
or ….

Step Zero
Follow or Set …
… before Troubleshooting Performance.

Step 0.1

Step 0.2
In Virtual Environments (e.g., VMWare)
Make sure that
Memory and
CPU are
Pinned/Allocated to Alfresco

Possible
Performance
Bottlenecks
Alfresco
Memory
Database
Repository
CPU
Virtualization
Storage

Cutting the
Performance
Gordian Knot

Slicing the
Performance
Problem
Low
CPU
CPU
Usage
High
CPU
I/O,
Database,
Repo,
Other Service
Alfresco
Other
Process

First Half of the Gordian Knot:
What is Killing the CPU?

Who is killing
the CPU?
• Java (Alfresco)?
• LibreOffice?
• Other Process?

CPU
Utilization
Tools
(System)
• Linux
– htop
– top
– sar (part of sysstat)
• Task Manager (Windows)

High Java
CPU usage
Find that hot thread!
• Support Tools
• Hot Threads
• jVisualvm or jConsole
• Application Performance Monitor (APM)
Gotcha Note!
– Docker/Kubernetes may not report CPU usage
correctly

High Java
CPU usage
• Support Tools for the Admin Console
– Enterprise:
• 4.2 – 5.1: Support Tools AMP
(https://github.com/Alfresco/alfresco-
support-tools)
• 5.2+: Integrated
– Community:
• Order of the Bee
(https://community.alfresco.com/docs/DOC-
7941-order-of-the-bee-support-tools)
• Pros: integrated, easy to use, non-intrusive
• Cons: may require 5.2+ or AMP installation,
not persistent

High Java
CPU usage
• Hot Threads
– Pros: some tools provide it, very light,
straightforward analysis, free, command line
(https://web.archive.org/web/20111225025057/h
ttp://weblogs.java.net/blog/brucechapman/archiv
e/2008/03/hot_threads.html), minimal
installation
– Cons: no UI, only instant snapshots

High Java
CPU usage
• jVisualVM, included with JDK
– Pros: low/medium overhead, persistency for
hours or days, thread analysis, free
– Cons: connecting not always straightforward or
even possible, needs to set java options in
Alfresco start script

High Java
CPU usage
• YourKit, jvisualvm on steroids, “Back to the
Future”
– Pros: database analysis, monitoring and
profiling, reviewing previous, best for random
crashes
– Cons: not free, requires installing agent, may be
heavy on CPU, complex configuration

High Java
CPU usage
• Application Performance Monitor (APM)
– Pros: some Open Source, constant connection,
ideal for constant monitoring
– Cons: some commercial (not free), shop
carefully (capabilities vary widely)

“The Gordian Knot: Part II”
Low CPU Usage

Low CPU
usage, now
what?
Alfresco
Memory
Database
Repository
CPU
Virtualization
Storage

Low Java
CPU usage
• Low CPU usage (but Alfresco is slow!)
• The problem is related to I/O issues or other
systems:
– Database issues
– Storage issues (too slow, too full)
– Problematic repository (too many children in a
folder, lots of sites, lots of users or permissions)
– Remove services (e.g., Solr) are slow

Database
(latency,
maintenance)
• ping is your friend (but not always)
• If your DBA is not your friend, it should be!
• Long-running queries
• Refreshing statistics

Memory
Problems
• Java
– Small heap size
– Garbage Collection (too much time, improper
algorithm)
– Small New/Eden space
• Operating System
– 35% to 25% should available

Tools to
Diagnose
Memory
Problems
(Java)
• jstat – JDK utility
• Use -verbose:gc command line option
– Requires visualgc
• jconsole
• jvisualvm
• Support Tools

Memory
Problems:
Pro Tips
• Java Heap Size
– < 75% OS Memory (Linux)
– < 60% Os Memory (Windows)
• Understand “free” OS memory in Linux
• Java:
– (Actually) Used <= Committed (allocated)
• Watch the Java Large Garbage Collections
• Try to use G1C for Java:
-XX:+UseG1GC
• G1C uses more Memory

Low Java
CPU usage:
Diagnostic
Tools
• If DB, storage and repo are okay, let’s look for
stuck/slow threads
– Obtaining Thread dumps:
• jstack
• jconsole
• Support Tools
– Spotify (https://spotify.github.io/threaddump-
analyzer/) thread analyzer: most useful
• Pros: needs only one thread dump
• Cons: will not find stuck
– Thread Dump Analyzers: no silver bullet
• Multiple thread dumps needed at even
intervals

Storage
• Content Store doesn’t have to be super fast
• Solr and tomcat/temp should use the fastest
storage available

Monitoring is
critical (if your
environment is
critical)

Application
Performance
Management
(APM)
• Commercial or Open Source?
• Commercial
– AppDynamics
– Wily
– Dynatrace
– New Relic
– More…
• Open Source
– Pinpoint
– MoSKito
– Glowroot

Conclusions
• Most of the times the tools to troubleshoot
performance problems are free, but
commercial tools may be necessary.
• Be ready to collect information in a crisis.
• It’s a good idea to use APM tools to monitor
24x7 to prevent crashes and performance
degradation.
• Keep your mind open.

Alfresco DevCon 2019 Performance Tools of the Trade

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Alfresco DevCon 2019 Performance Tools of the Trade

Similar a Alfresco DevCon 2019 Performance Tools of the Trade (20)

Último

Último (20)

Alfresco DevCon 2019 Performance Tools of the Trade

Notas del editor