In the SEO industry, we obsess on everything Google says, from John Mueller dropping a hint in a Webmaster Hangout, to the ranking data we spend $1000s to gather. Yet we ignore the data Google throws at us every day, the crawling data. For the longest time, site crawls, traffic data, and rankings have been the pillars of SEO data gathering. Log files should join them as something everyone is doing. We'll go through how to get everything set-up, look at some of the tools to make it easy and repeatable and go through the kinds of analysis you can do to get insights from the data.

1. 2009

4. God it’s bad.

16. -$1.5 Billion

18. Why hasn’t Google seen the changes on my page?

19. Has Google noticed my pages are broken?

20. Why isn’t my page appearing in Google?

21. Does Google think this page is important?

28. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

31. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
IP Address

32. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Timestamp

33. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Request type

34. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Homepage

35. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Protocol

36. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Status Code

37. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
Size of the page (in bytes)

38. What does a log look like?
123.65.150.10 - - [23/Aug/2010:03:50:59 +0000] "GET /my_homepage
HTTP/1.1" 200 2262 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html))"
User Agent

39. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

40. 5 things
2 3 4 51

41. 5 ½ things
2 3 4 51

42. 1 Diagnose crawling &
indexation issues
2 3 4 51

45. Number of
requests
Five folders Googlebot crawled the most

46. Five folders Googlebot crawled the most
Number of
requests

47. % of Organic sessions VS % of crawl budget
Sessions Crawl budget

48. 2 Prioritisation
2 3 4 51

50. example.com/article

51. Prioritizing
1
Full Print

52. example.com/article/full

53. example.com/article/print

54. Prioritizing
2

55. example.com/article/pdf

56. Prioritizing
3

57. Prioritizing
1
Full Print

58. 3 Spot bugs &
view site health
2 3 4 51

61. Delayed errors with a limit of 1000

63. 4 How important does Google
see parts of your site?
2 3 4 51

64. My SEO was as bad as my design

65. But at least my hair was better

66. teflsearch.com

67. teflsearch.com/job-results

68. teflsearch.com/job-results/country/china

69. teflsearch.com/jobadvert3455

70. Average number of times Googlebot crawled a template

71. 1. teflsearch.com
2. teflsearch.com/job-results
3. teflsearch.com/job-results/country/china
4. teflsearch.com/job-advert3455

72. 1. teflsearch.com
2. teflsearch.com/job-results
3. teflsearch.com/job-results/country/china
4. teflsearch.com/job-advert3455

73. teflsearch.com/job-results

74. Average number of times Googlebot crawled a template
35%

75. 5 How fresh does it think your
content is?
2 3 4 51

76. bit.ly/moz-fresh

77. Average number of times a page template is crawled by
Googlebot

78. ●Improve our internal linking
●Build trust with last modified date in
sitemap

79. 5 ½ Spotting trends
2 3 4 51

80. Google desktop vs mobile user agent crawlers
Desktop Mobile

81. 2 3 4 51

82. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

83. Are all the logs in one place?

84. Talk to a developer
and ask for
information

85. Hi x
I’m {x} from {y} and we’ve been asked to do some log analysis to understand better how Google is behaving on the website and I was hoping you could help with some questions
about the log set-up (as well as with getting the logs!).
What we’d ideally like is 3-6 months of historical logs for the website. Our goal is look at all the different pages search engines are crawling on our website, discover where they’re
spending their time, the status code errors they’re finding etc.
There are also some things that are really helpful for us to know when getting logs.
Do the logs have any personal informationin?
We’re just concerned about the various search crawler bots like Google and Bing, we don’t need any logs from users, so any logs with emails, or telephone numbers etc. can be
removed.
Do you have any sort of caching which would create separate sets of logs?
If there is anything like Varnish running on the server, or a CDN which might create logs in different location to the rest of your server? If so then we will need those logs as well
as just those from the server. (Although we’re only concerned about a CDN if it’s caching pages, or serving from the same hostname; if you’re just using Cloudflare for example
to cache external images then we don’t need it).
Are there any sub parts of your site which log to a different place?
Have you got anything like an embedded Wordpress blog which logs to a different location? If so then we’ll need those logs as well.
Do you log hostname?
It’s really useful for us to be able to see hostname in the logs. By default a lot of common server logging set-ups don’t log hostname, so if it’s not turned on, then it would be very
useful to have that turned on now for any future analysis.
Is there anything else we should know?
Best,
{x}
Email for a developer

86. So we might have something that looks like this

87. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

91. BigQuery

93. BigQuery

94. Google’s online database for
data analysis.

95. 1. Ask powerful questions
2. Repeatable
3. Scaleable
4. Combine with crawl data
5. Easy to set-up
6. Easy to learn
What do we want from analysing our logs?

101. 9,000,000 rows of data for 2
months.
400 - 800 queries

103. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

104. Format the logs so we can import them into
BigQuery
Separate the Googlebot logs from all the
other logs

105. Screaming Frog Log
Analyser
Code something

106. Screaming Frog Log Analyser

108. Code something

109. bit.ly/logs-code

110. What can you do with
logs?
PART 1: THE WHY
Getting logs
Analysing Logs
Processing Logs
PART 2: THE HOW

111. Our data in BQ

112. We make sure we got what we
wanted

113. THE QUESTION:
What is the total number of requests
Googlebot makes each day to our site?

114. Our first SQL query
SELECT
timestamp
FROM
[mydata.log_analysis]

115. Our first SQL query
SELECT
timestamp
FROM
[mydata.log_analysis]

116. Our first SQL query
SELECT
DATE(timestamp)
FROM
[mydata.log_analysis]

117. Our first SQL query
SELECT
DATE(timestamp)
FROM
[mydata.log_analysis]

118. Our first SQL query
SELECT
DATE(timestamp) as date
FROM
[mydata.log_analysis]

119. Our first SQL query
SELECT
DATE(timestamp) as date
FROM
[mydata.log_analysis]

120. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*)
FROM
[mydata.log_analysis]

121. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*)
FROM
[mydata.log_analysis]
GROUP BY
date

122. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*) as number_of_requests
FROM
[mydata.log_analysis]
GROUP BY
date

123. Our first SQL query
SELECT
DATE(timestamp) as date,
count(*) as number_of_requests
FROM
[mydata.log_analysis]
GROUP BY
date

124. Comparing logs to GSC crawl volume
Number of
requests

125. Run queries
Find something weird
Go look at crawl & website

126. Our data in BQ

127. 1 Diagnose crawling &
indexation issues

128. 2 Prioritisation

129. 3 Spot bugs &
view site health

130. 4 How important does Google
see parts of your site?

131. 5 How fresh does it think
your content is?

132. 1 Diagnose crawling &
indexation issues
4 How important does Google
see parts of your site?

133. What are the top 20 URLs crawled by
Google over our logs?

134. Login is my top crawled page and then search?

135. What are the top 20 page_path_1 folders
crawled by Google over our logs?

136. Location folders are taking more than 70% of my budget

137. Getting data by the day
Page Number of Googlebot Requests
page1 200,000
page2 120,000

138. Number of Googlebot requests day by day

139. 3 Spot bugs &
view site health

140. How many of each status code does
Google find per day over our logs?

141. Number of Googlebot requests day by day

142. What are most requested 404 URLs by
Googlebot over the past 30 days?

143. Boy does it want that ad-tech snippet

144. 5 How fresh does it think your
content is?

145. How many times on average is each page
in a page template crawled a day?

146. Average number of times a page template is crawled by
Googlebot

147. How long does it take for a page to be discovered after being published?

148. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?

149. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?

150. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?

151. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?

152. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?

153. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?

154. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?

155. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?

156. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?
What percentage of a directory was crawled over the past 30 days?

157. How long does it take for a page to be discovered after being published?
What are the top 20 combinations of page_path_1 & path_path_2 folders
crawled by Google over the time period of our logs?
Which pages have requests from Googlebot, which don’t appear in our crawl?
What are the top non-canonical pages being crawled?
Which are most crawled parameters on the website?
How often are the most visited parameters crawled each day?
Which directories have the most 301 & 404 error codes?
Which pages are crawled with parameters and without parameters?
Which pages are only partly downloaded?
How many hits does each section get, when the sections are classified in an
external dataset?
What percentage of a directory was crawled over the past 30 days?
What are the total number of requests across two different time periods?

158. That’s a lot of questions

159. bit.ly/logs-resource

160. bit.ly/logs-resource

161. bit.ly/logs-resource

162. bit.ly/logs-resource

163. In Summary

164. This is the thing you’re probably not doing

168. bit.ly/logs-resource
@dom_woodman