Fastly delivers more than a million log events per second. Our Real-Time Log Streaming is easy to set up, but there are many features you might not be using to their full extent. This workshop will cover setting up logging to various endpoints, dealing with structured data, and getting real-time insights into your customers’ behavior.

1. presents
Logging at
the Edge
Chris Jackel | Sr. Solutions Architect
Seth Vargo | Cloud Advocate at Google

2. Introduction
Logging at the Edge

5. ● Why to log
● How to log
● Where to log
● What to log
● How to log better
● How to query your logs in a really big way
What are we doing?

6. Why Logging?
Logging at the Edge

7. Logging at the Edge

8. Logging at the Edge

9. “Don't tell me the moon is
shining; show me the glint of
light on broken glass."”
—Anton Chekhov

10. How to Log
to GCS
Talk Title

11. Workshop Notes
● Two accounts needed:
○ Fastly Portal
■ https://manage.fastly.com
■ logging.workshop@gmail.com : altitude2018
■ Your service is your number
○ Google Cloud Platform
■ https://console.cloud.google.com
■ <random slip of paper on your desk
■ Incognito mode

12. ● Create a GCP project
● Configure GCS credentials
● Create a GCS bucket
● Configure Fastly logging to the bucket
Exercise 1 - Footloose

13. Configure GCS here
Logging at the Edge

14. Configure GCS here
Logging at the Edge

15. Configure GCS here
Logging at the Edge

16. Configure GCS here
Logging at the Edge

17. Logging at the Edge

18. Logging at the Edge

19. Logging at the Edge

20. Logging at the Edge

21. Logging at the Edge
Hey look - Logs!

22. Logging at the Edge

23. ● Create a GCP project
● Configure GCS credentials
● Create a GCS bucket
● Configure Fastly logging to the bucket
Exercise 1 - Footloose

24. What just
happened?
Talk Title

26. Get / Get /

30. Logging at the Edge

31. Logging at the Edge
159847 TxHeader c Connection: keep-alive
159847 TxHeader c X-Served-By: cache-jfk8125-JFK
159847 TxHeader c X-Cache: HIT
159847 TxHeader c X-Cache-Hits: 2
159847 TxHeader c X-Timer: S1489171719.109755,VS0,VE0
159847 VCL_call c log deliver
159847 VCL_call c log
159847 VCL_Log c syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test ::
{"client.ip":"207.38.219.230","req.request":"GET","req.http.host":"altitude-fastly.s3-website-us-east-
1.amazonaws.com","req.request":"GET","req.url":"/","req.bytes_read":94,"resp.status":200,"resp.bytes
_written":545,"resp.http.X-
Cache":"HIT","fastly_info.state":"HIT","time.start.usec":1489171719109755,"time.start.iso8601":"2017-
03-10 18:48:39","time.end.usec":1489171719109876,"time.elapsed.usec":121}
159847 VCL_return c deliver

32. Logging at the Edge
Control Plane
Portal
API

33. Logging at the Edge
Control Plane
Portal
API
Cache Nodes
Log Aggregators

34. Portal
API
Cache Nodes
Log Aggregators
Log string
Credentials

35. Cache Nodes
Log Aggregators
Log string /
Service ID
Credentials
Get /
Log string

36. Logging at the Edge

39. ● We do not store your logs!
● We prioritize speed of delivery
over reliability!

40. What to log
Talk Title

41. Variables
client.geo.city
client.geo.country
client.as.number
client.as.name

42. More
Variables if(fastly_info.state ~"^(HIT|MISS)(?:-|$)", "true",
"false")

43. Yet More
Variables if(fastly_info.is_h2, "true", "false")
if(fastly_info.h2.is_push, "true", "false")

44. Network
Variables
client.socket.nexthop
client.socket.tcpi_rtt
client.socket.ploss

45. Stop already
https://docs.fastly.com/guides/streaming-logs/useful-
variables-to-log

46. Try out some variables!
https://fiddle.fastlydemo.net/fiddle/ad937a59
Logging at the Edge

47. A note about log formats:
In the UI:
Logging at the Edge
In VCL:
log {"syslog 4uMLyO87CfYmzDIjpvfyPo loggy log log :: "} "CLIENT_IP="
req.http.Fastly-Client-IP " CLIENT_AS_NUMBER=" client.as.number "

48. Fun fact:
Kenny Loggins turned down the offer to co-star
with Barbra Streisand in A Star Is Born
10 Minute Break

49. How to log
Part 2: This is It!
Working with variables
Logging at the Edge

51. Classic:
<134>2017-03-19T13:13:33Z cache-jfk8147
foobarlogs2[453796]:
{"client_ip":"207.237.138.218","req_url":"/"}
Blank:
{"client_ip":"207.237.138.218","req_url":"/"}
Prefixes

52. declare local var.client-ip-redacted STRING;
set var.client-ip-redacted = regsub(client.ip, "[0-9]+$", "0");
log "syslog " req.service_id " loggerName :: " var.client-ip-redacted;
54.225.41.123 -> 54.225.41.0
Logging at the Edge

53. Put in some safe defaults
set var.user_agent = json.escape(if(req.http.User-
Agent,
req.http.User-
Agent, "-"));
Logging at the Edge

54. To log or not to log:
if (randombool(1,100) {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test :: "} var.logstr;
}
Logging at the Edge

55. To (dynamically) log or not to log:
if (randombool(std.atoi(table.lookup(logging, "percentage"))), 100) {
log {"syslog 4rqBj4oy1YChTPgdapiWS4 gcs-test :: "} var.logstr;
}
Logging at the Edge

56. < There will be a fully worked example in the handout >
Logging at the Edge

57. ● Change the logging prefix to ‘blank’
● Declare some variables
● Experiment
Exercise 2 - Cut loose

58. This is @sethvargo
He advocates for Google
He wrote the Golang fastly bindings
He used to advocate for HashiCorp
He used to advocate for Chef
He likes bacon

59. Google BigQuery: An Overview
Analytics service in the cloud for big data

60. Origins of BigQuery
How did this thing come to exist?

61. Origins of BigQuery
How do you find slow running servers from billions of log
entries across hundreds of data centers.... in seconds?

62. Origins of BigQuery
SELECT
COUNT(*) AS count,
source_machine AS machine
FROM ...
WHERE elapsed_time > 4000
GROUP BY source_machine
ORDER BY count DESC

63. Origins of BigQuery

64. What is BigQuery?
I'm intrigued...

65. What is BigQuery?
Scalable Storage
Google
Spreadsheets
App Engine
App
Co-Workers
Adwords
DCLK
Analytics
Corporate data
3rd party data
API
Analyze interactively
Mash it up
Securely Share/
distribute the results
Store all your data
in the cloud
SQL
Other BI
Tools

66. What is BigQuery?
Interactive analysis on large data sets is a requirement
Data mashups when scale and ease is important
Elasticity of environment is important
Minimal administration is required
Empowering Analysts or Data scientist with self service capabilities

67. Why is BigQuery Special?
What makes this thing so great?

68. Why is BigQuery Special?
Zero administration with performance and scale
No complex data architecture required (e.g. CSV, JSON, HTTP requests)
Just works with what you know (SQL, BI/ETL, REST API)
Powerful engine enables things like Machine Learning (BQML)

69. Why is BigQuery Special?
Nested Field Support
● Analyze details in single SQL statement
● Ingest Web data (like JSON) directly into BigQuery
Query Result Caching at Scale
● Cache query results at scale
Real-time Data Ingest & Reporting
● Ingest at 100K rows/sec
● Support for real-time to near real-time query/reporting

70. BigQuery ML
Machine Learning in SQL

71. BigQuery ML
SELECT
country, SUM(predicted_label) AS pred_visits
FROM
ML.PREDICT(MODEL `company.requests_model`, (
SELECT totals.pageviews,geoNetwork.country
FROM `company.fastly_requests_*`
WHERE _TABLE_SUFFIX BETWEEN '20180701' AND '20180801'
)
)
GROUP BY country
ORDER BY pred_visits DESC
LIMIT 10

72. BigQuery ML
+----------------+-------------+
| country | pred_visits |
+----------------+-------------+
| United States | 1104 |
| Canada | 54 |
| India | 22 |
| Japan | 2 |
| Indonesia | 1 |
| United Kingdom | 1 |
+----------------+-------------+

73. Big Query
Talk Title

74. - Create a data set and table
- Create a schema
- Enable the BigQuery API
- Configure Fastly to send the logs formatted to match the schema
Big Query - Setup

78. {
"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
"time_elapsed":%{time.elapsed.usec}V,
"is_tls":%{if(req.is_ssl, "true", "false")}V,
"client_ip":"%{req.http.Fastly-Client-IP}V",
"geo_city":"%{client.geo.city}V",
"geo_country_code":"%{client.geo.country_code}V",
"request":"%{req.request}V",
"host":"%{req.http.Fastly-Orig-Host}V",
"url":"%{json.escape(req.url)}V",
"request_referer":"%{json.escape(req.http.Referer)}V",
"request_user_agent":"%{json.escape(req.http.User-Agent)}V",
"request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
"request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
"cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "23") }V"
}
Fastly Log String

79. {
"timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
"time_elapsed":%{time.elapsed.usec}V,
"is_tls":%{if(req.is_ssl, "true", "false")}V,
"client_ip":"%{req.http.Fastly-Client-IP}V",
"geo_city":"%{client.geo.city}V",
"geo_country_code":"%{client.geo.country_code}V",
"request":"%{req.request}V",
"host":"%{req.http.Fastly-Orig-Host}V",
"url":"%{json.escape(req.url)}V",
"request_referer":"%{json.escape(req.http.Referer)}V",
"request_user_agent":"%{json.escape(req.http.User-Agent)}V",
"request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
"request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
"cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "23") }V"
}
Logging at the Edge
Better Fastly Log String
Yes, I know this is the same as the last slide,
TODO

80. <sample queries in the demo>

81. ● Create a data set and table
● Create a schema
● Enable the BigQuery API
● Configure Fastly to send the logs formatted to match the schema
Exercise 3 - Kick off those Sunday Shoes

82. Logging at the Edge
Wrapping Up

83. Adam Tomason
Travis Bischel
Ines Sombra
Dom Fee
Simon Winstow
… and SRE and NetEng and on and on
People

84. Scale!
Error surfacing!
More Endpoints!
What’s
Next

85. Final
thing

86. Questions?