This document provides an overview of 11 advanced uses for the SEO tool Screaming Frog. It begins with an introduction to Screaming Frog and its history of updates. It then discusses using Screaming Frog to crawl tricky sites like those with JavaScript, large sites, or password protection. Other sections cover scheduling crawls, integrating APIs like Google Analytics and Ahrefs, and performing post-crawl analysis of things like pagination, Hreflang, and XML sitemaps. Later sections discuss visualizations, auditing structured data and page speed, and using Screaming Frog for content marketing tasks like scraping news sites. The document concludes with some bonus uses like reviving old Google Search Console reports and scraping SERP features.

1. 11 Advanced Uses For
Screaming Frog
OLIVER BRETT, SEO MANAGER, SCREAMING FROG

6. Can you make it
pop?

7. Can you make it
pop?

8. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

9. http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV

10. @LordOfTheSERPs

11. @LordOfTheSERPs

12. @LordOfTheSERPs

13. @LordOfTheSERPs

14. I Accept Fan
Submissions

15. John Muller
Follows Me
;-)

26. 3 Rules For Today:

27. 1. This isn’t a sales
pitch.

33. 2. This isn’t a
walkthrough.

37. Links will be provided
for in-depth guides.
Don’t panic.

38. 3. Just ask if you
need help!

44. Part One: What is Screaming Frog?

45. Why is it called
‘Screaming Frog’?

47. Screaming Frog is
built by SEOs, for
SEOs.

56. Screaming Frog
(usually) runs locally.
But that’s a good thing.

61. “The Swiss
Army Knife
of SEO”
(IT’S CUTTING EDGE…)

62. Screaming
Frog Can…

63. https://www.seerinteractive.com/blog/screaming-frog-guide/

64. https://www.seerinteractive.com/blog/screaming-frog-guide/

65. https://www.seerinteractive.com/blog/screaming-frog-guide/

66. https://www.seerinteractive.com/blog/screaming-frog-guide/

68. We don’t have the
answers.
We have the data.

71. Screaming Frog
keeps getting bigger

72. Timeline:
V.1(2011)-V.11(2019)

73. V1.0 (2011)
LAUNCH

74. V2.0 (2012)
Word Counts
URL Rewrites
Crawl Control

75. No big updates
(2013/2014)

76. V3.0 (2015)
Tree View
Insecure Content Audits
Image Sitemaps + Enhanced XML Sitemaps

77. V4.0 (2015)
Google Analytics Integration
Custom Extraction

78. V5.0 (2015)
Webmaster Tools (Google
Search Console) integration
Better Robots.txt
Auditing/Control

79. http://taylortomita.com/seo-graph-portfolio/

80. http://taylortomita.com/seo-graph-portfolio/

81. V6.0 (2016)
JavaScript Rendering
Rendered Crawling
Crawl Indexed Sitemaps
SERP Emulator

82. V7.0 (2016)
Fetch and Render
Custom Robots.txt Rules
Hreflang
Web Form Authentication

83. V8.0 (2017)
Made it look nice(r)
More APIs
Custom Profiles

84. V9.0 (2018)
Database Storage
Store HTML and Rendered HTML
Better Search Functions

85. V10.0 (2018)
Scheduling
Visualisations
CLI (Headless Mode)

86. V11.0 (2019)
Structured Data- Audit,
Validate and Expor t

87. Lots of Updates…
V1.0 -2011- Launch
V2.0-2012- Word Counts, URL Rewrites, Crawl
Control
V3.0- 2015- Tree View, Insecure Content, Image
Sitemaps
V4.0- 2015- GA Integration, Custom Extraction
V5.0- 2015- GSC Integration, Better Robots.txt
V6.0- 2016- Rendered Crawling, Crawl Indexed
Sitemaps, SERP Emulator
V7.0- 2016- Fetch and Render, Custom Robots.txt,
hreflang, Web Form Authentication
V8.0- 2017- Made it look nice(r), More APIs,
Custom Profiles
V9.0- 2018- Database Storage, Store and Render
HTML, Better Search
V10.0- 2018- Scheduling, Visualisations,
Headless Mode
V11.0- 2019- Structured Data- Audit, Validate and
Export

88. Lots of Updates…
V1.0 -2011- Launch
V2.0-2012- Word Counts, URL Rewrites, Crawl
Control
V3.0- 2015- Tree View, Insecure Content, Image
Sitemaps
V4.0- 2015- GA Integration, Custom Extraction
V5.0- 2015- GSC Integration, Better Robots.txt
V6.0- 2016- Rendered Crawling, Crawl Indexed
Sitemaps, SERP Emulator
V7.0- 2016- Fetch and Render, Custom Robots.txt,
hreflang, Web Form Authentication
V8.0- 2017- Made it look nice(r), More APIs,
Custom Profiles
V9.0- 2018- Database Storage, Store and Render
HTML, Better Search
V10.0- 2018- Scheduling, Visualisations,
Headless Mode
V11.0- 2019- Structured Data- Audit, Validate and
Export

90. V12.0 (2019)…
literally last
week!
PageSpeed Insights
Integration
Autosave
Faster Crawling/Loading
More Extractors

91. 11 Advanced Uses For
Screaming Frog
OLIVER BRETT, SEO MANAGER, SCREAMING FROG

92. 11 Advanced Uses For
Screaming Frog
OLIVER BRETT, SEO MANAGER, SCREAMING FROG

93. 11 Advanced Uses For
Screaming Frog
OLIVER BRETT, SEO MANAGER, SCREAMING FROG

94. Part Two: Setup and Data

95. 1. Crawling tricky
sites

96. The number one
question our support
guys get asked:

97. Why won’t this Logfile
Analyser crawl my
site?

100. Why won’t this Logfile
Analyser crawl my
site?

101. The number one
question our support
team get asked:

102. Q: Why isn’t the
spider crawling more
than one page?

106. A: JAVASCRIPT!!

107. Tricky site 1:
JavaScript websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWL-
JAVASCRIPT-SEO/

111. Tricky site 2:
big websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/HOW-TO-CRAWL-
LARGE-WEBSITES/

112. What do we mean by
big?

115. Do you really need to
crawl the whole
damn site?

116. That’s gonna be a big
Excel doc…

117. Crawl in sections,
subdomains, or
subfolders.

118. Use the exclude
function to avoid
faceted navigation.

119. https://www.johnlew
is.com/browse/men/
mens-
trousers/adidas/alls
aints/gant-
rugger/hymn/kin/sel
ected-
femme/homme/size
=36r/_/N-
ebiZ1z13yvxZ1z0g0g6
Z1z04nruZ1z0s0laZ1z
0vl67Z1z01kl3Z1z0sw
k1

120. https://www.johnlew
is.com/browse/men/
mens-
trousers/adidas/alls
aints/gant-
rugger/hymn/kin/sel
ected-
femme/homme/size
=36r/_/N-
ebiZ1z13yvxZ1z0g0g6
Z1z04nruZ1z0s0laZ1z
0vl67Z1z01kl3Z1z0sw
k1

121. Database Storage
Mode
CONFIGURATION > STORAGE > DATABASE STORAGE MODE

122. Spider saves crawl
data to an SSD rather
than to the RAM.

132. Tricky site 3:
password protected
websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWLING-
PASSWORD-PROTECTED-WEBSITES/

133. Disclaimer:

134. Disclaimer:
You need to know what
the password is.

135. Disclaimer:
You need to know what
the password is.
We can’t help you hack it…

136. (Maybe like a dev site
for example…)

137. Forms Based
Authentication
CONFIGURE > AUTHENTICATION

139. Disclaimer 2:

140. Disclaimer 2:
This will click every link
on the page.

141. Disclaimer 2:
This will click every link
on the page.
Yes, really…

153. 2. Scheduling

154. How to schedule
crawls
HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-
SPIDER/USER-GUIDE/GENERAL/#SCHEDULING

156. CLI (Headless Mode)

157. CLI (Headless Mode)

162. You can use a cloud
service for scheduled
crawls. (You don’t have
to though.)

163. https://online.marketing/guide/screaming-frog-in-google-cloud/

164. https://ipullrank.com/how-to-run-screaming-frog-and-url-profiler-on-amazon-web-services/

165. https://linki.cz/postavte-si-seo-stroj-amazon-cloudu-aws/

166. 3. Integrating
APIs

167. Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.
UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#GOOGLE-
ANALYTICS-INTEGRATION

168. Google Analytics
Sessions
% New Sessions
New Users
Bounce Rate
Page Views Per Session
Avg Session Duration
Page Value
Goal Conversion Rate
Goal Completions All
Goal Value All
Sessions Above 0
Bounce Rate Above 70%
No GA Data
Non-Indexable with GA Data
Orphan URLs

169. Web Master
Tools
H T T P S :/ / W W W.S C R E A M I N G F RO
G .C O.U K / S E O - S P I DE R / U SE R -
G U I DE / C ON F I GU R AT I ON / # G OO
G L E - S E A RCH - C ON SOL E -
I N T E G R AT I ON

170. Google Search
Console
Web Master
Tools
HTTPS://WWW.SCREAMINGFROG.CO.
UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#GOOGLE-
SEARCH-CONSOLE-INTEGRATION

171. Google Search Console
Clicks
Impressions
CTR
Position
Clicks above 0
No GSC Data
Non-Indexible with GSC Data
Orphan URLs

172. ahrefs
HTTPS://WWW.SCREAMINGFR
OG.CO.UK/SEO -
SPIDER/USER -
GUIDE/CONFIGURATION/#AH
REFS

173. ahrefs
Backlinks
Referring Domains
URL Rating
Social Shares etc.
Referring Pages
No/Do Follow
Gov/Edu

175. Majestic
HTTPS://WWW.SCREAMINGFR
OG.CO.UK/SEO -
SPIDER/USER -
GUIDE/CONFIGURATION/#MA
JESTIC

176. Majestic
External Backlinks
Referring Domains
Trust Flow
Citation Flow
Edu/Gov Links
Anchor Text
Historic/Fresh Datasets

177. Moz
HTTPS://WWW.SCREAMINGFRO
G.CO.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#MOZ

179. Moz
Page Authority
MozRank
Time Last Crawled
Total Links
Domain Authority

180. Get Ya DAs
Done
1. Put the spider into list
mode
2. Connect Moz API, enable
DA
3. Paste in list of coverage
URLs
4. Crawl
5. Export list of URLs + DAs
6. Make link builders happy

181. Get Ya DAs
Done
1. Put the spider into list
mode
2. Connect Moz API, enable
DA
3. Paste in list of coverage
URLs
4. Crawl
5. Export list of URLs + DAs
6. Make link builders happy

182. PageSpeed
Insights
HTTPS://WWW.SCREAMINGFROG.CO.
UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#PAGESPE
ED-INSIGHTS-INTEGRATION

184. More on PageSpeed
Insights Soon….

185. 4. Post Crawl
Analysis

187. Link Score

188. A good way to estimate internal
PageRank.

191. Link score calculates the
relative value of a page based
off its internal links.

192. Link score uses a relative point
scale of 0-100, and it takes into
account redirects, canonicals,
nofollow links, and more.

193. The tool needs to find all your
pages before it can assign link
score values to them.

194. Pagination
UNLINKED PAGINATION URLS, PAGINATION LOOPS

195. Pagination
UNLINKED PAGINATION URLS, PAGINATION LOOPS

196. Hreflang
UNLINKED/MISSING HREFLANG URLS

197. AMP
MISSING <HTML AMP> TAG, NON-200 RESPONSE

198. Sitemaps
URLS IN/OUT OF SITEMAPS, ORPHAN URLS, NON-INDEXABLE
URLS IN SITEMAP, URLS IN MULTIPLE SITEMAPS

199. Analytics
+ Search Console
ORPHAN PAGES

200. Part Three: Reports and Jobs

201. 5. Visualisations

202. HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-SPIDER-10/
HTTPS://WWW.SCREAMINGFROG.CO.UK/SITE-ARCHITECTURE-CRAWL-
VISUALISATIONS/

203. Force-Directed Crawl
Diagram
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK

205. Crawl Tree Graph
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK

207. Force-Directed
Directory Tree
Diagram

209. Directory Tree Graph
FOLLOWS URL STRUCTURE

211. Inlink Anchor Text
Word Cloud
WHAT TEXT ARE YOU LINKING WITH?

213. Body Text Word Cloud
WHAT IS YOUR CONTENT ABOUT?

215. 6. Creating/auditing
Sitemaps

216. How to create XML
Sitemaps:
HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-SPIDER/USER-
GUIDE/GENERAL/#XML-SITEMAP-CREATION

219. How to Audit XML
Sitemaps:
HTTPS://WWW.SCREAMINGFROG.CO.UK/HOW-TO-AUDIT-
XML-SITEMAPS/

220. Or, paste
your
sitemap
URLs into
list mode.

221. Sitemap Reports
URLs In Sitemap
URLs Not In Sitemap
Orphan URLs
Non-Indexable URLs in Sitemap
URLs In Multiple Sitemaps
XML Sitemap With Over 50k URLs
XML Sitemap With Over 50mb

222. 7. Structured Data
Audits

223. How to audit
structured data:
HTTPS://WWW.SCREAMINGFROG.CO.UK/STRUCTURED-
DATA-TESTING-VALIDATION/

225. Structured
Data Issues
Contains Structured Data
Missing Structured Data
Validation Errors
Validation Warnings
Parse Errors
Microdata/JSON-LD/RDFa
URLs

228. 8. Page Speed
Insights with SF

229. PSI (Lighthouse) ft.
CrUX
HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-SPIDER-12/

233. 9. Audit Redirects
in List Mode

234. How to audit
redirects using list
mode:
HTTPS://WWW.SCREAMINGFROG.CO.UK/AUDIT-
REDIRECTS/

238. Part Four: Fun Extra Stuff

239. 10. Reviving old
search console
reports

243. @RichLawther
SCREAMING FROG
TECHNICAL SEO
MANAGER

244. Reviving old Search
Console reports
using SF
HTTPS://WWW.SCREAMINGFROG.CO.UK/REVIVING-
SEARCH-CONSOLE/

245. I miss the old
structured data
report…

248. I miss the old HTML
Improvements tab…

252. I miss the old
International
Targeting…

255. I miss Blocked
Resources…

258. I miss robots.txt
Tester…

261. 11. SF for Content
Marketing

262. Con tent Marketing

266. So we scraped all the
headlines we could…

271. @ShannonMcGuirk_
HEAD OF PR AND CONTENT
AIRA.NET

272. Scraping news sites to
work out who gives out
the most links:
HTTPS://WWW.SLIDESHARE.NET/SHANNONMCGUIRK/SE
ARCH-LEEDS-MAKING-HEADLINES

273. 12. Scraping
SERPs

274. @patlangridge
HEAD OF SEO
SCREAMING FROG

275. How to scrape SERP
features
HTTPS://WWW.SCREAMINGFROG.CO.UK/HOW-TO-SCRAPE-
GOOGLE-SEARCH-FEATURES-USING-XPATH/

276. Pat wants to know
what questions
people also ask…

279. Google of your choice!
https://www.google.co.uk
Search query parameter
/search?q=
Keywords with + symbols
hmrc+complaints
=CONCATENATE in Excel

280. Google query string Find and replace
spaces with + symbol
=CONCATENATE(A2,B2)
Google Doc template: https://bit.ly/2lAN5sE

282. Configuration > Spider > Rendering > JavaScript
Configuration > robots.txt > Settings > Ignore robots.txt
Configuration > User-Agent > Present User Agents > Chrome
Configuration > Speed > Max Threads = 1 > Max URI/s = 0.5

283. Configuration > Custom > Extraction

284. Configuration > Custom > Extraction

285. Configuration > Custom > Extraction
Snippet page title: (//div[@class='ellip'])[1]/text()
Snippet text paragraph: (//span[@class="e24Kjd"])[1]
Snippet URL: (//div[@class="xpdopen"]//a/@href)[2]

290. Thanks!
Questions?
@OliverBrett
@LordofTheSERPs
@ScreamingFrog
support@screamingfrog.co.uk