This presentation is an integral part of SEMrush webinar

1. Make Log File
Analysis Great Again!

4. Good
▷ 200 OK
▷ 304 Not Modified

5. 304 - Not Modified

6. Bad
▷ 301 Moved Permanently
▷ 302 Found (temporary redirect)
▷ 404 Not Found
▷ 503 Service Unavailable

7. Ugly
▷ 307/308 Redirects
▷ 401 Unauthorized
▷ 403 Forbidden
▷ 410 Gone
▷ 500 Internal Server Error

8. Help
▷ Curl
○ curl -vvvvv http://domain.tld
○ curl -I http://domain.tld
○ curl -I -A “googlebot” http://domain.tld
○ curl -I -H 'Referer: http://ilovevitaly.ua/'
http://domain.tld
○ curl -I --insecure https://domain.tld/
○ curl -I https://domain.tld/

9. Log File Analysis
▷ awk -F" '{print $2}' combined_log |
sort | uniq -c | sort -fr
○ url - 2; referrer - 4; user agent - 6
▷ awk '{print $1}' combined_log | sort
| uniq -c | sort -fr
○ ip - 1; status code - 9; size - 10
▷ grep “Googlebot/2” combined_log
▷ tail -n 100 combined_log

10. Result
▷ “URLs that are more popular
on the Internet tend to be
crawled more often to keep
them fresher in our index.”
▷ “Yes, Internal links do impact
crawl budget through crawl
demand”
http://www.thesempost.com/google-crawl-budget-seos-need-know-googlebot-crawling/