With JavaScript SEO, we're crawling, rendering & indexing. We want to work together with Devs and SEOs to get the best outcome for the client.

2. With JavaScript SEO
We're crawling, rendering & indexing
Peter Mead @petermeadit
Disclaimer: bullet points kill robots…
Warning: Advanced SEO ahead, this will get dry.

3. About Peter Mead
● Began developing websites and playing with
Search engines in 1997
● Major focus on Technical SEO
● Advocate for the SEO Community
● Called the Godfather by some
PeterMead.com
StudioHawk.com.au
SEOCollective.org
Peter Mead @petermeadit

4. SEO Hero - Picture this scenario
Working on your/clients SEO for some time now
Getting some good SEO wins:
- rankings, traffic, conversions, woohoo!
You're the SEO hero!
Wait what? Time to build a new website…
In JavaScript!
Let’s be heroes again?

5. What is JavaScript SEO?
https://moz.com/blog/JavaScript-seo-guide

6. JavaScript frameworks
Some popular frameworks include:
● Angular by Google
● React by Facebook
● Vue by Evan You
There’s lots of ways developers can make
JavaScript websites, and loads of frameworks...

7. React is a popular framework

8. It’s about crawling, rendering, and indexing your site
That’s just how we... crawl!

9. How Googlebot processes JavaScript
Googlebot processes JavaScript in three main phases:
1. Crawling
2. Rendering
3. Indexing
https://developers.google.com/search/docs/guides/JavaScript-seo-basics
Lets go…
Down the
rabbit hole.

10. Crawling, discovering resources and pages
Since mobile-first indexing, it will likely be Googlebot smartphone crawler
1. Crawler sends GET requests to server
2. Server gives response headers, & file contents
3. These are saved for processing
Requests usually come from Mountain View, CA, USA
Google will crawl and store ALL resources to build the page, HTML pages,
Javascript files, CSS, APIs.
This begins to build the Crawl queue, with its own specific crawl budget emerging.

11. Processing, preparing HTML & Resources for rendering
1. Finds Resources and Links to pages and files needed to build the page.
2. These are added to the crawl queue and cached.
3. Prioritising and scheduling is is based on the crawl queue.
Google looks for <a> tags with a href attribute.
Looks for Resources (CSS, JS, Img, etc.) and Links (to Pages) to later build the
page and provide relevance and context.
Javascript links to Resources and Pages may not get crawled or processed.
Most restrictive directives are chosen from either HTML or Javascript version.
Eg. noindex overrides index, and noindex in HTML skips rendering.

12. Rendering, seeing Pages the way users see them
Web Rendering Service (WRS) Processed pages are placed in Render queue
- Renders changes made to the Document Object Model (DOM) by Javascript
- Renders Using “evergreen” headless Chrome
Rendering the web is mega resource intensive, so Google needs shortcuts.
Rendering all resources caches during Processing stage
May cause unexpected results if the state of any shared resources has changed.
E.g timer for a popover shared in cache.
There’s no fixed 5 second timeout, rather, uses inactivity or blocked processing.

13. Indexing and crawling content in two waves

14. Rendering is expensive
Rendering can
take longer
Than 5 seconds
But needs to
be rendered
before it can
be indexed
Pages may not
get indexed
https://www.youtube.com/watch?v=LXF8bM4g-J4

15. Recommended render timeout
Google hasn't published anything official about render timeout
5 seconds is 'about right.'
This is roughly what Google may be seeing
https://sitebulb.com/resources/guides/how-to-crawl-javascript-websites/

16. So what does Google want to see rendered?
All our non-interaction on-page content
- Title tags,
- Meta descriptions,
- Alt attributes,
- Meta robots tags,
- Internal links & Navigation
All normal SEO elements apply.
https://ahrefs.com/blog/JavaScript-seo/

17. BAD JavaScript SEO
Can cause page elements not to load or render:
● Rendered core content
● Hyperlinks and Navigation
● Lazy-loaded images
● Page load times
● Document Layouts
Doing BAD work…

18. Common Issues preventing Rendering
Your page is queued for rendering,
It may not render as expected.
A range of factors
can be preventing Google from
rendering your page properly,
such as:
Render, render…

19. Robots directives blocking resources
Robots directives blocking important .js resources, preventing Google from
crawling and therefore rendering the content.

20. Timeouts causing resources to become unreachable
Timeouts causing network resources to take a long time to become unreachable,
so Google gives up and does not render.

21. Errors within the execution of the code
Errors within the execution of the code resultant from state or network connectivity
issues, such that JavaScript is unable to return HTTP status code errors.

22. Lazy loading images
Lazy loading images and text or other content, which is integral for the content,
however, is not yet available in the cached render queue, due to slow JavaScript
execution.

23. Hyperlinks generated by JavaScript
Hyperlinks generated by JavaScript are not crawlable, because Google does not
click links, so use standard hyperlinks links for them to be rendered.

24. Google won't click links, it crawls hyperlinks
Crawlers will not click on links or buttons or interact with the page the way users
do. They crawl and process the <a> tags.
Wont crawl this:
Or this:
Will crawl this:

25. Common Rendering Methods
Some of the most common JavaScript rendering methods include the following:
Pre-rendering
Client-side rendering
Server-side rendering
Dynamic rendering
Hybrid rendering
Isomorphic JavaScript

26. Google recommended rendering
Developers code changes may be costly.
What to recommend:
- Server side rendering
- Prerendering, or
- Dynamic rendering
Instead of client side rendering
...where possible
https://developers.google.com/search/docs/guides
/JavaScript-seo-basics

27. Dynamic rendering
Some common dynamic renderers are Puppeteer, Rendertron, and prerender.io.
Switches between client side and prerender for certain user agents.
Essentially showing a specific rendered version to Googlebot.
Content is crawled and indexed without having to execute JavaScript first.
https://developers.google.com/search/docs/guides/dynamic-rendering

28. Wait, isn’t dynamic rendering just cloaking?
Cloaking is against Google’s policies,
Can result in penalties if it’s implemented on your website.
Dynamic rendering methods are permitted and recommended by Google
However, make sure your rerendered version page does not show different
content to what is being served to users

29. Trisomorphic rendering, and the rest... woohoo
Many other ways to render, including streaming server rendering, if you prefer.
https://developers.google.com/web/updates/2019/02/rendering-on-the-web
https://www.deepcrawl.com/knowledge/ebooks/JavaScript-seo-guide/different-rendering-methods/

30. Consider the costs of DOM vs. vDOM
● Document Object Model (DOM) and Virtual DOM (vDOM)
● Vue and React use a Virtual Document Object Model (vDOM)
● Angular uses a real DOM
● Individual elements can be with a vDOM
● Don't need to update the entire DOM tree
● PageSpeed & Core Web Vitals are integral for Web development and SEO
alike.
● vDOM is considered faster than real DOM
● Pages built with Vue and React are often faster than Angular

31. View the DOM in Chrome Dev Tools

32. Identify JavaScript rendering
1. Just turn off JavaScript in your browser…
2. Do a Google search for specific on-page text you expect to be indexed
Devs and SEOs can use any number of the following tools:

33. Mobile friendly test
Basically shows you the pages the way we think Google sees it.
https://search.google.com/test/mobile-friendly

34. Inspect URL Search Console
Shows a screenshot and rendered HTML

35. Crawling JavaScript sites with Screaming frog
Configure rendering to ‘JavaScript’
Enable rendered screenshots
5 seconds for a render timeout
Select Googlebot Smartphone
https://www.screamingfrog.co.uk/crawl-JavaScript-seo/

36. Ensure external resources get crawled and rendered

37. Crawl & View Rendered Pages
You will get a list of Rendered Pages for easy analysis

38. Compare Raw & Rendered HTML
Allows us to check which HTML is being rendered
Important if content is missing
Helps identify missing links etc. within the DOM

39. Prerender test with screaming frog
Configure Screaming Frog User-Agent
Use a Googlebot crawler so the rerender triggers the crawler to look at the
prerendered version of the page.
(turn off JavaScript rendering options)

40. View Rendered Source
https://chrome.google.com/webstore/detail/view-rendered-source/ejgngohbdedoabanmclafpkoogegdpob?hl=en

41. JS Library Detector
https://chrome.google.com/webstore/detail/library-detector/cgaocdmhkmfnkdkbnckgmpopcbpaaejo?hl=en

42. Use BuiltWith to help
discover various frameworks
In use on your page
https://builtwith.com/

43. Lighthouse to identify JavaScript libraries
https://developers.google.com/web/tools/lighthouse

44. Google’s Steps for Devs fixing JavaScript SEO issues
1. Test with the Mobile-Friendly Test or the URL Inspection Tool
2. Make sure to prevent soft 404 errors
3. Expect Googlebot to decline user permission requests
4. Don't use fragment URLs to load different content
5. Don't rely on data persistence to serve content.
6. Use content fingerprinting to avoid caching issues with Googlebot.
7. Ensure that your application uses feature detection for all critical APIs that it needs and provide a fallback behavior
or polyfill where applicable.
8. Make sure your content works with HTTP connections.
9. Make sure your web components render as expected. Use the Mobile-Friendly Test or the URL Inspection Tool to
check if the rendered HTML has all content you expect.
10. After you fix the items in this checklist, test your page with the Mobile-Friendly Test or the URL inspection tool in
Search Console again.
If you fixed the issue, a green check mark appears and no errors display. If you still see errors, post in the JavaScript Sites in
Search Working Group.
https://developers.google.com/search/docs/guides/fix-search-JavaScript

45. 1. Mobile-Friendly Test or the URL Inspection Tool
To test how Google crawls and renders a URL, use the Mobile-Friendly Test or the
URL Inspection Tool in Search Console.
You can see loaded resources, JavaScript console output and exceptions,
rendered DOM, and more information.

46. 2. Make sure to prevent soft 404 errors
Make sure to prevent soft 404 errors.
In a single-page application (SPA), this can be very difficult.
To prevent error pages being indexed, use one or both of the following strategies:
● Redirect to a URL where the server responds with a 404 status code
● Add or change the robots meta tag to noindex

47. 3. Googlebot to decline user permission requests
Features that require user permission don't make sense for Googlebot, or for all
users.
For example, if you make the Camera API required, Googlebot can't provide a
camera to you.
Instead, provide a way for users to access your content without being forced to
allow camera access.

48. 4. Don't use fragment URLs to load different content
An SPA may use fragment URLs (for example https://example.com/#/products) for
loading different views.
The AJAX-crawling scheme has been deprecated since 2015, so you can't rely on
fragment URLs to work with Googlebot.
Recommend using the History API to load different content based on the URL in
an SPA.

49. 5. Don't rely on data persistence to serve content
WRS loads each URL
However, WRS does not retain state across page loads:
● Local Storage and Session Storage data are cleared across page loads.
● HTTP Cookies are cleared across page loads.

50. 6. Use content fingerprinting to avoid caching issues
Googlebot caches aggressively in order to reduce network requests and resource
usage.
WRS may ignore caching headers. This may lead WRS to use outdated
JavaScript or CSS resources.
Use content fingerprinting avoid this problem, it makes a fingerprint of the content
part of the filename, like main.2bb85.js.
The fingerprint is dependant on the content of the file, so updates generate a
different filename every time.
Check out the web.dev guide on long-lived caching strategies to learn more.

51. 7. Use feature detection for all critical APIs
Use feature detection (to run browser dependant code) for all critical APIs, and
provide a fallback behavior or polyfill (to provide modern functionality on older browsers that do not
support it) where applicable.
Some web features may not be available with all user agents and some may
intentionally disable certain features.
For example, WebGL rendering of photo effects in the browser, feature detection
shows Googlebot doesn't support WebGL.
Fix this by removing the photo effect or use server-side rendering to prerender
photo effects, making content accessible to everyone, including Googlebot.

52. 8. Make sure your content works over HTTP connections
Googlebot uses HTTP requests to retrieve content from your server.
Other types of connections, such as WebSockets or WebRTC are not supported
Provide an HTTP fallback to retrieve content
Also use robust error handling and feature detection to run different code of
browser does not support feature

53. 9. Make sure web components render as expected
Use Mobile-Friendly Test or URL Inspection Tool to make sure the rendered HTML
has all content as expected.
WRS flattens light DOM and shadow DOM (to bundle CSS with markup)
If the web components you use aren't using <slot> (placeholders inside to be filled with
markup) consult the documentation, or use another web component instead
More information on best practices for web components.

54. 10. Test your JavaScript fixes
After you fix the issues, test the page with:
● Mobile-Friendly Test
● URL inspection tool

55. Lean to speak Dev and SEO languages
fluent in over six million forms of communication…
… but that was getting rather dry

56. Your Dev + SEO Collaboration Journey
Devs and SEOs can agree JavaScript can cause SEO-related issues for crawling
and indexing your website’s content.
A collaborative JavaScript SEO approach can be about finding ways for Devs and
SEOs to get on the same page.
By working together we can make better-informed decisions about the rendering
approach that’s are best
Taking time to understand issues from both sides can help us get SEO success

57. JavaScript is not the devil
Jamie Alberico asks…
Is JavaScript the devil?
Nik Ranger says No!
Ulrika Viberg says no!
Kristina Azarenko says no!
Martin Splitt says hell no!
Although Devs + SEOs often have
Issues, when done well, JavaScript
SEO rocks!
https://www.youtube.com/watch?v=bjnrchJdqqw

58. Resources
Tools - https://www.searchenginejournal.com/JavaScript-tips-tools-testing-rendering/
More tools - https://www.searchenginejournal.com/JavaScript-seo-debugging-tools/
Screaming frog - https://www.screamingfrog.co.uk/crawl-JavaScript-seo/
Beginners guide - https://www.contentkingapp.com/academy/JavaScript-seo/
In depth crawling, rendering, indexing - https://ahrefs.com/blog/JavaScript-seo/
Official google documentation -
https://developers.google.com/search/docs/guides/JavaScript-seo-basics

59. Be the hero again! Devs and SEOs working together!
What?
@petermeadit
Questions?