Learn from Fastly veteran Cassandra Dixon on some of the most common customer issues we see — such as why things aren’t caching, misconfigured origins, issues with intermediary proxies, and VCL snafus — and the best ways to resolve them. We’ll also discuss our unique approach to debugging — using seemingly mundane tools to diagnose issues in creative ways — and how you can apply these methods to your own organization to get the most out of Fastly’s offerings.

1. Debugging Fastly
VCL: 101
Cassandra Dixon
Solutions Architect | Fastly
Steven Chuob
Technical Account Manager | Fastly

2. Discussion
● Common Fastly VCL Debugging
● More Technical Fastly VCL
Debugging

3. Introduction
● Purpose of this training
● What will you learn
● Aiden’s Journey

4. When Contacting Fastly Support
What to provide:
● Tell us who you are
● Tell us the issue
● Be specific
● Steps to reproduce
● How did you discover it?

6. Aiden’s Journey
through Fastly

7. FOR PARTICIPANTS:
ALL CODE IN THIS PRESENTATION IS BROKEN.
DO NOT USE IN PRODUCTION.

8. Fastly’s Shielding Service

9. Fastly’s Shielding Service (VCL Flow)
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetch
User
restart

10. • vcl_recv - This is the first
subroutine the request goes
through
• Handles backend/shield
selections
• Modifications to hostname, url
and other req.headers
vcl_recv

11. • vcl_fetch - This subroutine runs
after the object has been
fetched from the origin/shield
• Does not run on a hit. Only for
miss, pass requests
• Caching logic and TTLs
vcl_fetch

12. ● vcl_deliver - This is one of the
last subroutine that handles the
response.
● This subroutine sends the object
and its headers to a client
making a request.
● Handles modifications to the
response headers before
delivery
vcl_deliver

13. ● vcl_log - Like vcl_deliver this is
one of the last subroutine to be
run.
● If you configure log, it is called in
this subroutine
vcl_log

14. • vcl_recv - This is the first subroutine the request goes
through, backend/shield selections happens in this routine.
• vcl_deliver - This is one of the last subroutine that
handles the response from the origin. This subroutine sends
the object and its headers to a client making a request.
• vcl_log - Like vcl_deliver this is one of the last subroutine
to be runned. This will send a log to a logging endpoint
after the request has been handled.
• vcl_error - This subroutine handles any error handling
when the origin responds with an error.
• vcl_miss - This subroutine runs once the object is
determined to not be in cache.
• vcl_hit - This subroutine runs once the object is
determined to be in cache.
• vcl_pass - This subroutine will either be runned after
vcl_recv or vcl_fetch.
• vcl_fetch - This subroutine runs right after the object has
been fetched from the origin and its caching TTL set (based
on what’s being sent from the origin).
Cluster Edge Cluster Node

15. Clustering

16. The more you know...

17. Months 1-6
● Setting up an anonymous cloud
hosting origin
● Enabling Fastly’s Shielding
Service

18. backend default {
.host = “174.138.66.5”;
.port = “80”;
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
backend shield_ssl_cache_lcyxxxx_LAX {
.is_shield = true;
.connect_timeout = 2s;
.port = "80";
.host = "23.235.47.20";
.max_connections = 1000;
.share_key = "fastlyshield";
} ...
Aiden’s VCL

19. sub vcl_recv {
# custom vcl set_hostname
# setting host header to match backend
set req.http.host = "hostname expected by origin";
set req.backend = F_addr_xxx;
#do shield here
{ if (req.backend == F_addr_xxx && req.restarts == 0) {
if (server.identity !~ "-LCY$" &&
req.http.Fastly-FF !~ "-LCY") {
set req.backend = ssl_shield_lax_ca_us;}
if (!req.backend.healthy) {
# the shield datacenter is broken so don’t go to it
set req.backend = F_addr_xxx;
}}}
if (req.request != "HEAD" && req.request != "GET" &&
req.request != "FASTLYPURGE") {
return(pass);
}
return(lookup);}
Aiden’s VCL

20. Aiden’s Testing
● Test URL
● Check Aiden’s VCL
● Testing Tools
○ watch
○ curl
○ Logging

21. Logging

22. curl -svo /dev/null
http://altitude-stage.global.ssl.fastly.net/shielding
< HTTP/1.1 200 OK
< Server: Apache/2.4.7 (Ubuntu)
< Content-Type: text/html
< Fastly-Restarts: 1
< Cache-Control: max-age=86400
< Content-Length: 14405
< Accept-Ranges: bytes
< Date: Sun, 18 Jun 2017 16:22:53 GMT
< Via: 1.1 varnish
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-sjc3635-SJC
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1497802973.177423,VS0,VE171
< Vary: Accept-Encoding
Test URL

23. ● Hits to Origin multiple times for
the same object
● Fastly-Restart > 0
● Served-By header shows only
one POP
● Age is randomly coming back
as 0.

24. sub vcl_recv {
# custom vcl set_hostname
# setting host header to match backend
set req.http.host = "hostname expected by origin";
set req.backend = F_addr_xxx;
#do shield here
{ if (req.backend == F_addr_xxx && req.restarts == 0) {
if (server.identity !~ "-LCY$" &&
req.http.Fastly-FF !~ "-LCY") {
set req.backend = ssl_shield_sjc_ca_us;}
if (!req.backend.healthy) {
# the shield datacenter is broken so don’t go to it
set req.backend = F_addr_xxx;
}}}
if (req.request != "HEAD" && req.request != "GET" &&
req.request != "FASTLYPURGE") {
return(pass);
}
return(lookup);}
Reviewing: Aiden’s VCL

25. Fastly’s Shielding Service (VCL Flow)
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetch
User
restart
http://altitude-stage.global.ssl.fastly.net/req.http.host == altitude-stage.global.ssl.fastly.net
req.url == / vcl_recv
req.http.host == altitude-debug-workshop.dixonkin.com
req.url == /
vcl_recv
Delivers a 500 internal domain not found to the Edge POP
vcl_deliver
vcl_fetch
A restart is triggered due to the 500 error.
Cause: Edge thinks the shield is
unhealthy and thus down
Origin
vcl_deliver

26. Easiest Solution

27. # OPTION 1 - Create 2 snippet
# vcl_miss & vcl_pass
## Setting a hostname
if (!req.backend.is_shield) {
set bereq.http.host = "hostname expected by
origin";
}
# OPTION 2 - Create a vcl snippet with
# type “init routine”
# create snippets for vcl_miss & vcl_pass
sub set_hostname {
## Setting a hostname
if (!req.backend.is_shield) {
set bereq.http.host = "hostname expected by
origin";
}
}
Better Solution
# OPTION 2 - Cont’d
# Update Custom VCL file
sub vcl_miss {
#FASTLY miss
call set_hostname;
return(fetch);
}
sub vcl_pass {
#FASTLY pass
call set_hostname;
}

28. sub vcl_recv {
#FASTLY recv
## Setting a hostname
if (!req.http.Fastly-FF) {
set req.http.host = "hostname expected by origin";
}
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
return(pass);
}
return(lookup);
}
What about Fastly-FF?

29. • Aiden creates a login page
• Set Cache-control: max-age=0
• He implemented serving stale
Month 8-10

30. Fastly Serve Stale
Client
Fastly Origin
Is content cached?
Yes No
Serve Stale Serve Error

31. Fastly Testing
● Test URL
● Testing Tools
○ watch
○ curl
○ logging
○ Fastly-Debug

32. curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/caching/time
< HTTP/1.1 200 OK
< Server: Apache/2.4.7 (Ubuntu)
< Cache-Control: max-age=0, public
< Via: 1.1 varnish
< Content-Length: 1533
< Accept-Ranges: bytes
< Date: Tue, 20 Jun 2017 21:48:21 GMT
< Via: 1.1 varnish
< Age: 8
< Connection: keep-alive
< Fastly-Debug-Digest: 6d34439e4502f525f4dee9255bb9fb1f3b4943b4a9b792c1b07f09f4d31b7c38
< X-Served-By: cache-lax8635-LAX, cache-sjc3640-SJC
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1497995301.171881,VS0,VE0
Test URL

33. Logging

34. ● Hits in logging
● Cache-Control: max-age=0
● Age is greater than 0.

35. ● Includes log line with a HIT
response
● Includes curl responses;
showing Age=0 and sometimes
Age=6
Aiden’s Support Ticket
Jun 20 14:48:19 Fastly_logs
AltitudeDebugWorkshop-Staging:
2017-06-20T21:48:18+0000 SJC 8.18.217.202 "-" "-"
[20/Jun/2017:21:48:18 +0000] "GET /caching/time
HTTP/1.1" 200 1533 "" "curl/7.48.0"
recv[shield_lax_ca_us:/caching/time] hit
deliver[HIT-STALE:hits=1:age=1] HIT-STALE

36. sub vcl_recv {
# setting host header to match backend
set req.http.host = "hostname expected by origin";
# Fix encoding issues in urls
set req.url = regsub(req.url, "&amp;", "&");
set req.http.tmpLog = if (req.restarts == 0, "recv", req.http.tmpLog "; recv");
set req.backend = F_addr_xxx;
#do shield here ...
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
return(pass);
}
# Strip out the service ID for readability
set req.http.tmpLog = req.http.tmpLog "[" regsub(req.backend, "^[a-zA-Z0-9].*--", "") ":"
req.url "]";
return(lookup);
}
Aiden’s VCL - vcl_recv

37. sub vcl_miss {
set req.http.tmpLog = req.http.tmpLog " miss";
#--FASTLY MISS START
...
# Snippet set_hostname_miss : 100
if (!req.backend.is_shield) {
set bereq.http.host = "altitude-debug-workshop.dixonkin.com";
}
#--FASTLY MISS END
return(fetch);
}
Aiden’s VCL - vcl_miss

38. sub vcl_fetch {
set beresp.do_stream = true;
#if (beresp.status >= 500 && beresp.status < 600) {
# /* deliver stale if the object is available */
# if (stale.exists) {
# return(deliver);
# }
...
#set stale_if_error and stale_while_revalidate (customize these values)
set beresp.stale_if_error = 86400s;
set beresp.stale_while_revalidate = 10s;
if (req.http.X-Debug) {
set beresp.http.Backend-Name = beresp.backend.name;
}
...
}
Aiden’s VCL - vcl_fetch

39. ● Cache-control: max-age=0 is
not equivalent to a pass object
● TTL of zero is cacheable- serve
stale applies
Aiden: No Regrets,
Lessons Learned

40. Solutions
OPTION 1 - Update
cache-control header on your
origin
OPTION 2 - Create a cache setting
on Fastly with a condition on the
path
OPTION 3 - Custom VCL in vcl_recv after
the #FASTLY recv macros.

41. Break

42. • New Photo Gallery released on
website
• TTL needed for static pictures
• Aiden’s origin is not sending the
cache control header for this
content.
• He adds a cache header on
Fastly
Month 14 -15

43. sub vcl_fetch {
set beresp.do_stream = true;
...
#set stale_if_error and stale_while_revalidate
(customize these values)
set beresp.stale_if_error = 86400s;
set beresp.stale_while_revalidate = 10s;
if (req.http.X-Debug) {
set beresp.http.Backend-Name =
beresp.backend.name;
}
...
#--FASTLY FETCH BEGIN
...
# priority: 10
if ( req.url == "/redirect1" ) {
# Header rewrite Static Page Cache : 10
set beresp.http.Cache-Control = "max-age=3600";
}
...
}
Aiden’s VCL

44. Fastly Testing
● Test URL
● Testing Tools
○ watch
○ curl
○ logging
○ Fastly-Debug

45. curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/redirect1 -H 'Fastly-Debug:1'
< HTTP/1.1 200 OK
< Server: Apache/2.4.7 (Ubuntu)
< Cache-Control: max-age=3600
< Content-Length: 1917
< Accept-Ranges: bytes
< Date: Wed, 21 Jun 2017 20:24:02 GMT
< Via: 1.1 varnish
< Age: 1
< Connection: keep-alive
< CDN-Process-Log: recv[shield_lax_ca_us:/redirect1] hit deliver[HIT:hits=1:age=1]
< Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC 1498076642)
< Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)
< Fastly-Debug-Digest: cfb378e8d0413b6345893969c58d8ed23b6db487be9197c0d24d8b5833df8970
< X-Served-By: cache-sjc3121-SJC
< X-Cache: HIT
< X-Cache-Hits: 1
< X-Timer: S1498076643.983655,VS0,VE0
Curl Test

46. Logging

47. Wait - What is Fastly-Debug showing?
Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC
1498076642)
Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)
The first number is the TTL
remaining for the object
The third number is current age
of the item in cache
The second number refers to
the grace

48. ● Object is expiring after 120
seconds
● Fastly-Debug is saying the
object TTL is 120 seconds

49. sub vcl_fetch {
set beresp.do_stream = true;
...
#set stale_if_error and stale_while_revalidate
(customize these values)
set beresp.stale_if_error = 86400s;
set beresp.stale_while_revalidate = 10s;
...
#FASTLY fetch
...
if (beresp.http.Expires ||
beresp.http.Surrogate-Control ~ "max-age" ||
beresp.http.Cache-Control ~"(s-maxage|max-age)") {
# keep the ttl here
} else if (beresp.cacheable) {
# apply the default ttl
set beresp.ttl = 3701s;
}
}
Aiden’s VCL
Important VCL Logic

50. TTL Flow
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetch
User
vcl_recv vcl_recv
Origin TTL = 0
TTL = 120
Cache-Control:
max-age=3600
vcl_fetch
vcl_deliver
vcl_fetch
vcl_deliver
TTL = 3600

51. ● There is a default TTL of 120
that is used when the object
has no TTL or Cache-Control
headers from the origin.
● The starting TTL is set before
vcl_fetch runs on a service
Lessons Learned

52. Solution
A Cache-control header created
in vcl_fetch does not reset the
TTL
To set a TTL, modify the variable
beresp.ttl in vcl_fetch
Therefore create a cache setting on Fastly
to actually change and set the beresp.ttl

53. • A starting point for Debugging
• Streaming Logs
• Basic understanding on VCL
subroutines
Recap

54. Questions?

55. Thank you
Cassandra & Steven