We know that Google wants to align their results with the websites that make searchers happy. We expect to see more and more ranking factors that align with this - whether based on actual usage data, machine-learned equivalents, or simply from evaluating algorithm changes based on their impacts on usage. But on our journey to that point, it's important to realize that changes made to improve conversion rate or UX could harm search performance, and vice versa. How do we navigate this complexity? That's what Will is going to be covering.

1. When “what’s good for users”
isn’t good for Google
Will Critchlow -- @willcritchlow
SearchLove San Diego 2019

2. UX is a ranking
factor

3. UX is a ranking
factor

4. UX is a ranking
factorHow users see
your site

5. UX is a ranking
factor

6. UX is a ranking
factor

7. UX is a ranking
factorAffects organic
search
performance

8. Not everyone agrees
Don’t worry. I have seen the reddit threads and twitter arguments

9. Later, I’ll get onto why I don’t care

10. Round up
UX ranking
factors

11. Make it
actionable

12. Round up
UX ranking
factors
Make it
actionable

13. Controversial Accepted

14. CTR for algorithm
evaluation
This has been
confirmed
repeatedly.

15. Paul Haahr
Worth following @haahr
His online bio dramatically
understates his influence:

17. https://youtu.be/iJPu4vHETXw

19. “I’ve talked explicitly about using
clickthrough rate for experiments”

20. https://youtu.be/eGSGZMI4Z_I?t=979

21. Controversial Accepted

22. CTR for
personalization
Slightly more
controversial as
we don’t know
exactly what they
mean.

23. “We use it [CTR] for personalization”

24. https://youtu.be/eGSGZMI4Z_I?t=991

25. Controversial Accepted

26. Human UX
ratings
Qualitative ratings
of pages that rank:
used to evaluate
algorithms

29. “... make this page a frustratingly
poor user experience”
-- reason for page quality rating
Search quality rater guidelines [PDF]

30. We also got to see the “leaked” UX playbooks come out of Google recently across a
range of industries - getting deep into implementation details. [H/T @gfiorelli1]

32. Controversial Accepted

33. ML models
trained on ratings
E.g. Panda -
trained to like the
things humans
like

34. Back in 2011, I was suggesting we run our own
Panda-like quality surveys (WBF here, instructions here)

35. Sidenote: totally worth running
these surveys for executive buy-in
Since Panda went real-time, quality issues don’t necessarily cause
obvious drops correlated with algorithm history dates

36. Client site 1 Client site 2
Would you trust information from this website? 72% 64%

37. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%

38. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%

39. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%

40. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%

41. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%

42. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%

43. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%
Does this website contain insightful analysis? 72% 62% 81%

44. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%
Does this website contain insightful analysis? 72% 62% 81%
Would you consider bookmarking pages on this site? 44% 38% 56%

45. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%
Does this website contain insightful analysis? 72% 62% 81%
Would you consider bookmarking pages on this site? 44% 38% 56%
Are there excessive adverts on this website? 2% 2% 8%

46. Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%
Does this website contain insightful analysis? 72% 62% 81%
Would you consider bookmarking pages on this site? 44% 38% 56%
Are there excessive adverts on this website? 2% 2% 8%
Could pages from this site appear in print? 54% 54% 59%

47. We also asked for free-text feedback and
found some surprising priorities from
non-SEOs

48. “The reviews seem fake”
Trust is a huge deal for real-world users

49. “There's not enough information
about the company and why I
should use their products”
On a micro-site that doesn’t have an “about” page

50. “In this day and age every page that
has anything at all to do with
business should be https”
Security is a big deal in B2B - even without on-site purchases

51. “The pictures were of low quality
and blurry”
We know this matters to users. It’s at the easier end of ML detection

52. Controversial Accepted

53. Site speed
measured
directly
Real User Metrics
including site
speed from
Chrome

54. https://moz.com/blog/google-chrome-usage-data-measure-site-speed

55. https://moz.com/blog/google-chrome-usage-data-measure-site-speed
First announced at SearchLove San Diego last year

56. “The Chrome User Experience Report is powered by
real user measurement of key user experience
metrics across the public web, aggregated from
users who have opted-in to syncing their browsing
history, have not set up a Sync passphrase, and have
usage statistic reporting enabled.”
Chrome User Experience Report documentation and announcement

57. Soft denial by Gary Illyes in his AMA
Q: Is CrUX data being used to assess page
speed as a ranking factor?
A: “I don't remember where the data is
coming from, but not our publicly
available tools”

58. Controversial Accepted

59. CTR as direct
ranking factor
This has been
explicitly denied
by Google reps.

60. https://www.slideshare.net/randfish/mad-science-experiments-in-seo-social-media

61. MozCon 2014

65. We’ve seen similar things in-person
with experiments at conferences
[Note: others have tried to reproduce results more recently including this failure to do so by Bartosz]

66. More:
https://sparktoro.com/blog/queries-clicks-influence-googles-results/

67. But: not everyone agrees

68. https://www.reddit.com/r/TechSEO/comments/ao3fmk/i_am_gary_illyes_googles_chief_of_sunshine_an

69. “In the most general, broadest way - without
attempting to skirt, skip, duck or dive it, does
Google use interactions of searchers to alter
what position certain results may hold? Those
actions could include, but aren't limited to; CTR,
return and click another listing, adjusting the
query, repeat search and repeat visit etc.”
-- /user/Darth_Autocrat

70. “we primarily use these things in
evaluations.”

71. “Can you please confirm/deny whether
[RankBrain] uses UX signals of any kind?”
-- /user/Darth_Autocrat

72. “Dwell time, CTR, whatever
Fishkin's new theory is, those are
generally made up crap”

73. “Dwell time, CTR, whatever
Fishkin's new theory is, those are
generally made up crap”

74. A little devil’s advocacy: are there any charitable interpretations?

75. Technically correct
Not exactly clickthrough rate
[Something more like “difference between expected and observed CTR”?]

76. Technically correct
Not exactly a “ranking factor”
[They call it something more like a “post-evaluation modifier” or something?]

77. Dodging the question
“primarily use”
[and maybe limiting the “crap” comments to RankBrain?]

78. Dodging the question
“primarily use”
[and maybe limiting the “crap” comments to RankBrain?]

79. Dodging the question
“primarily use”
[and maybe limiting the “crap” comments to RankBrain?]

80. 1. They aren’t marketers

81. 1. They aren’t marketers
2. Their interests aren’t
aligned with yours

82. 1. They aren’t marketers
2. Their interests aren’t
aligned with yours
3. They “don’t
understand what it’s
doing exactly”(*)
(*) Haahr on RankBrain

83. I gave a
presentation here
in SD last year:
“From the horse’s
mouth”
Slides & Video

84. But grrrr:
a) their job is
supposed to be
to help
webmasters

85. and
b) calling it
“made up crap” is
unprofessional
and discourages
the vulnerable

86. 1. They aren’t marketers
2. Their interests aren’t
aligned with yours
3. They “don’t
understand what it’s
doing exactly”(*)
(*) Haahr on RankBrain
Remember:

87. Some SERP
interaction as
some kind of
ranking factor
My theory.

88. We sell custom cigar humidors. Our
custom cigar humidors are handmade. If
you’re thinking of buying a custom cigar
humidor, please contact our custom
cigar humidor specialists at
custom.cigar.humidors@example.com
Keyword stuffing: Doesn’t mean keyword density is the correct metric

89. Controversial Accepted

90. Controversial Accepted

91. Controversial Accepted
Future

92. Controversial Accepted
Future
More intent
measurement?

93. Controversial Accepted
Future
More intent
measurement?
[“Needs met” is
already the
primary metric]

94. Controversial Accepted
Future
More intent
measurement?
Measure task
completion?

95. Controversial Accepted
Future
More intent
measurement?
Measure task
completion?
Consider
conversion rate?

96. So, given all of this, ALL we need to
do is optimize for UX then?

97. Um. No.
Evidence later

98. Google makes mistakes

99. Haahr talked about evaluating
algorithms on “next page click rate”

100. Adding white
space gamed
that metric

101. It’s also really hard, and people are
confusing

102. “There are so many experiments
that we’ve done that have very
misleading live metrics that you
really need to dig into”

103. “We had a long-running experiment
that swapped results two and four”

104. “My guess is more people would
scroll down and click on position
three”

105. “...actually more people click on
[position] one”

106. https://youtu.be/eGSGZMI4Z_I?t=1024

107. It is easy to think you have shown
something, but find it full of
confounding details when you dig
into the individual results
We have seen the same thing with our testing

108. We just need to know that UX is aligned with Google’s goals
But we don’t need the futurist
perspectives or the tinfoil hat conspiracy
theories for my purposes here

109. We just need to know that UX is aligned with Google’s goals
1. Panda (ML-trained on UX signals)
2. Quant and qual measurement of algo
changes using UX metrics

110. We should be building SEO
hypotheses from UX fundamentals

111. If we’re talking UX, we’re talking CRO:
Conversion Rate Optimization

112. Most organisations treat them entirely separately
SEO CRO
Search Engine
Optimization
Conversion Rate
Optimization

113. But in truth there is a lot of overlap
SEO CRO

114. While some SEO changes don’t impact CRO
SEO
Meta info -
invisible to UX CRO

115. And some CRO changes don’t impact SEO
SEO
Non-indexed
pagesCRO

116. An awful lot of changes affect both

117. “How do we know the tests won’t
hurt conversion rate?”

118. Never: “how do we know our CRO
tests aren’t hurting rankings?”

121. Anyway. You ought to worry about BOTH

122. When CRO messes up SEO

123. When CRO messes up SEO
-25%!

124. I wrote a post at moz.com about how big an impact this can have

125. You think this is what happens as you roll out winning CRO tests

126. But you don’t realise that some of them are bad for search

127. What if there was a way to GET the best
of both?

128. You can do better if you only roll out the true winners

129. SEO split testing is template basedOur solution: full funnel testing

130. Conversion rate
optimization focuses on
converting more of your
existing traffic (bottom of
the funnel)
Conversion Rate
Optimization

131. Conversion rate
optimization focuses on
converting more of your
existing traffic (bottom of
the funnel)
Search engine
optimization focuses on
getting additional traffic to
your website (top of
funnel)
Conversion Rate
Optimization
Search Engine
Optimization

132. Conversion rate
optimization focuses on
converting more of your
existing traffic (bottom of
the funnel)
Search engine
optimization focuses on
getting additional traffic to
your website (top of
funnel)
Full funnel testing
combines these
approaches to do both at
the same time resulting in
higher ROI from testing
Conversion Rate
Optimization
Search Engine
Optimization
Full Funnel Testing

133. How does it work?

134. SEO split testing is template basedHere’s how it works - on an example website
Home
Animals
Cats Dogs Unicorns Badgers

135. CatsCats
We want to test two templates for our animals pages

136. Cats Dogs Unicorns Badgers
CRO test - different users see different templates on the same pages
1

137. Cats Dogs Unicorns Badgers
CRO test - different users see different templates on the same pages
1
2
Cats BadgersDogs Unicorns

138. Cats Badgers
SEO test - different users see the same templates on DIFFERENT pages
1
2
Cats Badgers
Cats Badgers

139. Cats Badgers Unicorns Dogs
SEO test - different users see the same templates on DIFFERENT pages
Cats Badgers Unicorns Dogs
Cats Badgers Unicorns Dogs
1
2

140. The site looks like this during the test
Home
Animals
Cats Dogs
Countries
Scotland England Ireland WalesUnicorns Badgers

141. Animals
A B A B A B A B
Animals
A B A B
SEO testingCRO testing vs.

142. Google’s view Animals
Cats Dogs Unicorns Badgers
AB BA

143. User view Animals
Cats Dogs Unicorns Badgers
AB BA
?????

144. Animals
Cats Dogs Unicorns Badgers
AB BA
Landing page
established
What do unicorns eat

145. SEO test
complete
Animals
Cats Dogs Unicorns Badgers
AB BA
What do unicorns eat

146. Cookied
AA AA
What do unicorns eat
Animals
Cats Dogs Unicorns Badgers

147. Second page
view
Animals
Cats Dogs Unicorns Badgers
AA AA
What do unicorns eat

148. Third page view Animals
Cats Dogs Unicorns Badgers
AA AA
What do unicorns eat

149. Sidenote:
This just makes better SEO tests
anyway BTW
By avoiding an inconsistent user experience as they browse the site

151. In summary

152. UX is a ranking
factor?

153. Not everyone agrees
Controversial Accepted

154. We just need to know that UX is aligned with Google’s goals
But we don’t need the futurist
perspectives or the tinfoil hat conspiracy
theories for my purposes here

155. We just need to know that UX is aligned with Google’s goals
1. Panda (ML-trained on UX signals)
2. Quant and qual measurement of algo
changes using UX metrics

156. To know that we should be building
SEO hypotheses from UX
fundamentals

157. BUT: then we affect users and search

158. SEO split testing is template basedWhich is why I believe the future of SEO is testing

159. Thank you
Will Critchlow, CEO Distilled
@willcritchlow
With help, ideas, slides and more from @craigbradford,
@dom_woodman and @tomanthonyseo

160. Questions: @willcritchlow

161. Image credits
● Hornet
● Beach
● Paul Haahr
● Google
● Smile
● Pug
● Chrome
● Tinfoil hat
● Broken plate
● Confused monkey
● Unicyclist
● Horse
● Phone
● Devil cat
● Coloured smoke
● San Diego