En savoir plus sur www.opensourceschool.fr Ce support est diffusé sous licence Creative Commons (CC BY-SA 3.0 FR) Attribution - Partage dans les Mêmes Conditions 3.0 France Plan : 1. Reverse proxies 2. Varnish overview 3. VCL 4. CLI tools

1. Varnish 4.0
Advanced and high-performance HTTP caching

2. 1
La 1ère école 100 % dédiée à l'open source
Open Source School est fondée à l'initiative de Smile, leader de
l'intégration et de l'infogérance open source, et de l'EPSI,établissement
privé pionnier de l’enseignement supérieur en informatique.
Dans le cadre du Programme d’Investissements d’Avenir (PIA), le
gouvernement français a décidé de soutenir la création de cette école en
lui attribuant une première aide de 1,4M€ et confirme sa volonté de
soutenir la filière du Logiciel Libre actuellement en plein développement.
Avec une croissance annuelle de plus de 10%, et 4 000
postes vacants chaque année dans le secteur du Logiciel
Libre, OSS entend répondre à la pénurie de compétences du
secteur en mobilisant l’ensemble de l’écosystème et en
proposant la plus vaste offre en matière de formation aux
technologies open source tant en formation initiale qu'en
formation continue.

3. 2
Les formations du plein emploi !
 Formation Continue
Open Source School "Executive Education" est un organisme
de formation qui propose un catalogue de plus de 200
formations professionnelles et différents dispositifs de
reconversion permettant le retour à l’emploi (POE) ou une
meilleure employabilité pour de nombreux professionnels de
l’informatique.
 Pour vos demandes : formations@opensourceschool.fr
 Formation Initiale
100% logiciels libres et 100% alternance, le cursus Open
Source School s’appuie sur le référentiel des blocs de
compétences de l’EPSI.
Il est sanctionné par un titre de niveau I RNCP, Bac+5.
Le programme est proposé dans 6 campus à Bordeaux, Lille,
Lyon, Montpellier, Nantes, Paris.

4. 3
Nos domaines de formations

5. Reverse proxies Varnish overview VCL CLI tools
Plan
1 Reverse proxies
2 Varnish overview
3 VCL
4 CLI tools
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 2/36

6. Reverse proxies Varnish overview VCL CLI tools
Reverse proxies
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 3/36

7. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
What is a reverse proxy?
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 4/36

8. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
Proxies: forward, reverse
Cache headers can increase performance when used by browser
mainly on the loading time (from the second page)
But the cache is multiplied on the users browser
Small impact on the server load
Solution : reverse proxy
HTTP server and HTTP client at the same time
No effect if no cache headers
Difference Proxy / Reverse proxy
Proxy : on the user side
Reverse proxy : on the server side
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 5/36

9. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
Architecture
A reverse proxy is used to lower the load on the web server, not to
decrease bandwidth
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 6/36

10. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
HTTP protocol
(RFC7234)
The Cache-Control header is used to control caching
max-age: How long to keep the object in the cache
s-max-age: How long to keep the object in a shared cache
(reverse proxy)
must-revalidate: After the max age has been reached,
revalidate the object
no-cache: Do not reuse the cache without validation
(If-Modified-Since or If-None-Match)
no-store: Do not store the object into the cache
stale-while-revalidate : serve old versions of the objets
for this amount of time while it’s being refreshed
Expires: not necessary, choose between TTL and date
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 7/36

11. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
Benefits of using a reverse proxy
Performance !
Cache shared by all clients: far more efficient, decrease the
load on the server
Decrease the processing latency
Multiple architecture possible
Load balancing with reverse proxy
Apache + mod php specificity
PHP embedded in all Apache processes
Can consume a lot of resources
Static files in the reverse proxy cache = important gain
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 8/36

12. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
Drawbacks
The cache is shared by all users:
In case of page custom per user (ex: account page)
Cache will break the application
Rule of caching: never cache something that is only seen by
one user
Hybrid page: mix between user-specific content and common
content
Separate per-user and public blocks
AJAX loading for user-specific part
Cut the page in fragments and use ESI
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 9/36

13. Reverse proxies Varnish overview VCL CLI tools
What is a reverse proxy?
Drawbacks
Complexity :
Additional level of cache (application, browser)
Hybrid configurations (Varnish)
Config split between application and caching server
Need good understanding of both HTTP protocol and Varnish
configuration language
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 10/36

14. Reverse proxies Varnish overview VCL CLI tools
Varnish overview
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 11/36

15. Reverse proxies Varnish overview VCL CLI tools
What is Varnish?
What is Varnish?
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 12/36

16. Reverse proxies Varnish overview VCL CLI tools
What is Varnish?
What is Varnish
High performance HTTP Cache
First release in 2006
Lead dev : Poul-Henning Kamp
(opinionated FreeBSD kernel developer)
BSD license
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 13/36

17. Reverse proxies Varnish overview VCL CLI tools
What is Varnish?
Varnish : performance
“Just” a reverse proxy
No SSL 1
But load balancing is supported
Very deep work on the performance side
Optimization on the system calls when processing a request
Absence of performance-damaging features
Grace time
when a resource is expired on the cache server, Varnish can
send the expired resource while getting the new resource on
the web server
avoid impact on the user when a new page is generated
Several thousands request per second without problem
1
use HAproxy/nginx
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 14/36

18. Reverse proxies Varnish overview VCL CLI tools
What is Varnish?
Configuration
Two files :
Daemon settings (ports, config file, memory limits)
/etc/defaults/varnish
/etc/sysconfig/varnish
Configuration
/etc/varnish/default.vcl
copy it and change the daemon config to myproject.vcl
Can be reloaded on the fly
Implement subroutines to change the Varnish behaviour
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 15/36

19. Reverse proxies Varnish overview VCL CLI tools
What is Varnish?
Version History
Do not use :
2006 : Varnish 1.0
2008 : Varnish 2.0
Current versions :
2011 : Varnish 3.0
VMODs, Transparent gzip compression. Lots of VCL changes
since 2.1
2014 : Varnish 4.0
Backend/frontend separation. Lots of VCL changes since 3.0
Different major versions of Varnish usually have completely
incompatible configurations. Varnish 2 is still used on some
projects !
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 16/36

20. Reverse proxies Varnish overview VCL CLI tools
VCL
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 17/36

21. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
VCL configuration
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 18/36

22. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
VCL Configuration
VCL is a simplified programming language
Looks like a blend of perl and C
C code can be included, don’t do it
Two parts
Backend declaration : address, ports, timeout, load balancing
Subroutines
Called at a specific point in Varnish processing
Very flexible
Can return a label (change workflow)
Can alter objects (request, response. . . )
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 19/36

23. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
Main subroutines
We usually focus on three subroutines:
vcl recv: beginning of request processing, can decide to look
up the object in the cache or skip the cache entirely, make
alterations to the request, or do access control
vcl backend response: after getting the response for the
backend, can decide to store the object in the cache or
remember not to look up the cache for this object, can change
TTLs or alter the object we are about to store in the cache
vcl deliver: just before sending the response to the client,
whether or not the cache was used, for any last-minute
modifications, including diagnostics or cleanup
Sometimes, we use vcl hash to manipulate the cache key
(examples later). But it’s usually better to use a Vary header.
There are a few other seldom used subroutines : vcl hit,
vcl pass, etc
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 20/36

24. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
Varnish workflow example
In this example, we are
asking for a cacheable
object
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 21/36

25. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
Varnish workflow example
In this example, the
servers tells us about a
non-cacheable object
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 22/36

26. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
Varnish workflow example
In this example, we skip
the cache altogether for
certain URLs
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 23/36

27. Reverse proxies Varnish overview VCL CLI tools
VCL configuration
Configuration approach
Approach 1 : recommended
The application decides how long each page has to be cached
HTTP headers
Standards-compliant, minimal changes to the Varnish config
Approach 2 : Magento
Application assumes every page can NOT be cached
Force Varnish to cache some of them
Force Varnish to skip the cache for others
You need to know the URL plan of your application
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 24/36

28. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
VCL crash course
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 25/36

29. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Returning
Returning is how you you change the workflow. If you do not
return, the default subroutine will be executed after your own.
everywhere
error code [msg]
vcl recv
pass : disable the cache for this request, but not other type of
processing. When using Keepalive, further requests will go
through vcl recv again
pipe : disable varnish for the whole connection, including any
further requests
hash : look inside the cache for this object
purge : purge the current URL from the cache 2
2
usually when using a custom HTTP method such as PURGE
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 26/36

30. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Returning
vcl backend response
deliver : continue processing, object will be inserted in the
cache
abandon : give up on the backend fetch and yield a 503
restart : retry the backend fetch
vcl deliver
deliver : send the request to the client
restart
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 27/36

31. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Synthetic pages
Varnish is able to serve HTTP pages without fetching them from a
backend.
The synth(code, message) value can be returned by almost any
function
Upon returning a synth value, the vcl synth subroutine is called.
The subroutine may use the synthetic() function to inline a
HTML page, or redirect to a themed error page on another
platform (to avoid redirection loops).
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 28/36

32. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Variables
There are many variables, here are the most useful :
req : the request
req.url
req.http.something : HTTP headers
resp : the response that will be sent to the client
resp.status
resp.http.something
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 29/36

33. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Variables
There are many variables, here are the most useful :
beresp : backend response
beresp.status : HTTP status code
beresp.ttl : TTL infered from Cache-Control headers
beresp.uncacheable : Set this to trigger a pass next time
the object is requested
obj : the object from the cache
obj.ttl
obj.http.something
obj.hits : how many time the object has been used
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 30/36

34. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Variable scope
Not all variables are available or useful everywhere :
req is mostly accessed in vcl recv
beresp is mostly used in vcl backend response
obj and resp are mostly used in vcl deliver
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 31/36

35. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Setting variables
Changing an existing variable is simple
set beresp.ttl = 600s;
The best way to create a custom variable is to set a HTTP header
in the request :
set req.http.myvar = "something";
You can remove a header like this
unset resp.http.Server;
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 32/36

36. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
Tests
You can use regexps or C-like operators
if (req.url ~ "/admin")
if (req.http.User-Agent ~ "MSIE" )
if (beresp.ttl <= 0s)
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 33/36

37. Reverse proxies Varnish overview VCL CLI tools
VCL crash course
That’s all
look at man vcl and the Smile Wiki for the rest
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 34/36

38. Reverse proxies Varnish overview VCL CLI tools
CLI tools
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 35/36

39. Reverse proxies Varnish overview VCL CLI tools
varnishlog
varnishlog is the swiss army knife of Varnish debugging
varnishlog dumps all varnish activity in real time, use it to
follow each step of a request
-c : client side
-b : backend side
-O : do not group (keep client/backend sequence in order)
Interesting things to watch for
SessionOpen/SessionClose : beginning/end of request
Rx : receiving from Varnish POV, can either be from the client
(c) or from the backend (b)
Tx : transmitting from Varnish POV, can either be to the
client or to the backend
VCL call/VCL return : subroutine calls and their results
www.opensourceschool.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 36/36