SlideShare a Scribd company logo
1 of 15
Download to read offline
11.02.2011




     The Influence of Commercial Intent of Search Results
     on Their Perceived Relevance

     Dirk Lewandowski
     Hamburg University of Applied Sciences, Germany




Introduction

•    Ongoing discussion about results quality
      –  Did Google’s results really get worse?
      –  Do search engine results (and especially, Google’s) get more and more
         commercial?

•    Discussion on biased search results / “search neutrality”
      –  Does Google prefer its own services in its rankings?

•    Objective
      –  To find out whether commercial results have a negative effect on relevance as
         perceived by the users.




                                                                                                 1
11.02.2011




Search engine market share (Germany)




Previous work

•    Retrieval effectiveness tests
      –  Lots of work done
      –  Main objective is to compare results quality of commercial Web search engines
      –  Tests are highly standardized


•    Commercial intent
      –  Some studies on commercial intent of the searchers
      –  No studies on the commercial intent of the documents




                                                                                                 2
11.02.2011




Research questions

•    RQ1: How effective are web search engines in retrieving relevant results?

•    RQ2: What ratio of the top 10 search engine results are pages with a
     commercial intent?

•    RQ3: Are results with a commercial intent comparably relevant to non-
     commercial results?

•    RQ4: Do banner or textual ads have a negative effect on the perceived
     relevance?




Test design

•    Retrieval effectiveness test
      –  To collect relevance judgments


•    Results classification
      –  Classify each result according to its commercial intent
           •  Commercial intent of the page/website itself
           •  Commercial intent as expressed through advertisements on the page




                                                                                          3
11.02.2011




Retrieval effectiveness test design

•    50 queries
      –  “Real life queries” from students asked for their last query entered into a Web
         search engine
      –  Mainly simple queries without an explicit commercial intent
•    3 search engines
      –  Google, Microsoft, Yahoo (German user interfaces)
•    10 results per query
•    Binary relevance judgments
•    50 undergraduate students as jurors




Retrieval effectiveness test: Data collection

•    Results from all search engines collected

•    Results
      –  were made anonymous
      –  were randomized

•    Each juror evaluated the complete results set for one query




                                                                                                   4
11.02.2011




Results classification (1)

•     Commercial
       1.  A product is sold on the page
       2.  Webpage of a company

 •    Non-commercial
       3.  Page of a public authority, government office, administration, etc.
       4.  Page of a non-governmental organization, association, club, etc.
       5.  Private page
       6.  Page of an institution of higher education

 •    Other




Results classification (2)

•     Number of banner ads




 •    Number of text-based ads (“sponsored links”)




                                                                                         5
11.02.2011




Limitations

•     Results cannot be generalised
       –  Commercial intent of results does not have a negative influence on their
          perceived relevance when asking undergraduate students.

•     Relatively small dataset
       –  500 URLs per search engine (total of 1,500 URLs)




     Results




                                                                                             6
11.02.2011




Results

1.  Results from the classification task
2.  Results from the retrieval effectiveness test
3.  Combined results: Influence of commercial intent on the perceived
    relevance




Results

1.  Results from the classification task




                                                                                7
11.02.2011




Results from the classification task (1)

Results with a commercial intent
  account for
      –  82% (Google)
      –  75% (Microsoft)
      –  78% (Yahoo)




Results from the classification task (2)

•    Lowest ratio of results with a commercial intent on the first position.




                                                                                       8
11.02.2011




Results from the classification task (3)

•    Google has a significantly higher ratio of results with sponsored links.
      –  Google: 23.7% of all results
      –  Microsoft: 12.3%
      –  Yahoo: 17.7%

•    No significant difference between the ratios of results with banner ads.
      –  Google: 21.3% of all results
      –  Microsoft: 23.0%
      –  Yahoo: 22.0%




Results

•    Results from the retrieval effectiveness test




                                                                                        9
11.02.2011




Macro precision
Number of queries answered best by an individual search engine




Mean Average Precision
All search engines, all results




                                                                        10
11.02.2011




Results

•    Combined results: Influence of commercial intent on the perceived
     relevance




Relevance of results with a commercial intent

•    Relevance of results with commercial intent is comparable to non-
     commercial results
      –  Google: 59.4% of commercial results relevant (vs. 58.7%)
      –  Microsoft: 34.6% vs. 36.5%
      –  Yahoo: 51.1% vs. 51.7%




                                                                                11
11.02.2011




Relevance of results with advertising

•     Relevance of results with text-based ads is comparable to results without
      text-based ads
       –  Google: 59.8% vs. 59.3%
       –  Microsoft: 40.0% vs. 33.9%
       –  Yahoo: 52.7% vs. 52.0%

•     Relevance of results with banner ads is comparable to results without
      banner ads
       –  Google: 58.5% vs. 59.8%
       –  Microsoft: 33.3 vs.35.1%
       –  Yahoo: 55.8% vs. 49.1%




     Discussion




                                                                                         12
11.02.2011




Discussion

•    Most striking result is that Google has significantly more results with text-
     based ads.

•    Search Engine Optimization (SEO) may be a reason
      –  Many pages are highly optimised for Google.

•    Wouldn’t it be clever to prefer pages carrying ads from the search engine’s
     own ads system?
      –  However, Google says it doesn’t do this.




Google‘s advertising model




                                                                                            13
11.02.2011




 Conclusion & further research




Conclusion

     1.  In terms of retrieval effectiveness, Google outperforms the other two
         search engines (RQ1).

     2.  There is only a slight variation in the ratio of commercial results from
         search engine to search engine (RQ2).

     3.  The results classified as “commercial” are comparably relevant to the
         results classified as “non-commercial” (RQ3).

     4.  Textual or banner ads do not have a negative effect on the perceived
         relevance of the corresponding result (RQ4).




                                                                                           14
11.02.2011




Further research

     1.  Study should be repeated using other user groups
         –  Relevance judgments came from a comparably homogeneous user group.
         –  Data already collected for other user groups (another student group;
            librarians; crowd-sourcing).

     2.  Large ratio of results with text-based ads in Google deserves further
         investigation.
         –  Currently, we conduct a large-scale study where we automatically detect
            text-based ads on search results, comparing Google and Bing.
              •  30,000 queries, 2 search engines, 30 results per query  1.8 Million URLs.




                                                                                                     15

More Related Content

Similar to The Influence of Commercial Intent of Search Results on Their Perceived Relevance

Overview of Search Marketing - SEO & SEM
Overview of Search Marketing - SEO & SEMOverview of Search Marketing - SEO & SEM
Overview of Search Marketing - SEO & SEMSanger & Eby
 
Internet Summit - Preconference Training (2012)
Internet Summit - Preconference Training (2012)Internet Summit - Preconference Training (2012)
Internet Summit - Preconference Training (2012)Dana Todd
 
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Joni Salminen
 
Google strategy evaluation , i click hatem kameli
Google strategy evaluation , i click hatem kameliGoogle strategy evaluation , i click hatem kameli
Google strategy evaluation , i click hatem kameliHatem Kameli
 
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...Wouter Schikhof
 
Paid and Organic Search Co-Optimization
Paid and Organic Search Co-OptimizationPaid and Organic Search Co-Optimization
Paid and Organic Search Co-OptimizationBill Hunt
 
Presentation for Exl Pharma Conference about Future of Search
Presentation for Exl Pharma Conference about Future of SearchPresentation for Exl Pharma Conference about Future of Search
Presentation for Exl Pharma Conference about Future of SearchWO Strategies
 
Digital Prerequisites for Social Media Strategies
Digital Prerequisites for Social Media StrategiesDigital Prerequisites for Social Media Strategies
Digital Prerequisites for Social Media StrategiesRandi Priluck
 
Lead Scoring Conversions Webinar
Lead Scoring Conversions WebinarLead Scoring Conversions Webinar
Lead Scoring Conversions WebinarAct-On Software
 
Sept 20 2012 ona show me the numbers
Sept 20 2012 ona  show me the numbersSept 20 2012 ona  show me the numbers
Sept 20 2012 ona show me the numbersHack the Hood
 
Generative AI and SEO
Generative AI and SEOGenerative AI and SEO
Generative AI and SEOJason Packer
 
Search Trends 2011
Search Trends 2011Search Trends 2011
Search Trends 2011Brian Jones
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
Webová analytika – proč? kdo? kdy? jak? (Insight Model)
Webová analytika – proč? kdo? kdy? jak? (Insight Model)Webová analytika – proč? kdo? kdy? jak? (Insight Model)
Webová analytika – proč? kdo? kdy? jak? (Insight Model)Akce Dobrého webu
 
Seo 4 Fundamentals
Seo 4 FundamentalsSeo 4 Fundamentals
Seo 4 Fundamentalsguesta416c6e
 
Organizational and strategic analysis of GOOGLE
Organizational and strategic analysis of GOOGLEOrganizational and strategic analysis of GOOGLE
Organizational and strategic analysis of GOOGLEDivya Lakshme
 
Keeping Content Searchable In Google's World
Keeping Content Searchable In Google's WorldKeeping Content Searchable In Google's World
Keeping Content Searchable In Google's WorldMartino Flynn
 
Excel with Enterprise SEO
Excel with Enterprise SEOExcel with Enterprise SEO
Excel with Enterprise SEOKirill Kronrod
 

Similar to The Influence of Commercial Intent of Search Results on Their Perceived Relevance (20)

Overview of Search Marketing - SEO & SEM
Overview of Search Marketing - SEO & SEMOverview of Search Marketing - SEO & SEM
Overview of Search Marketing - SEO & SEM
 
Internet Summit - Preconference Training (2012)
Internet Summit - Preconference Training (2012)Internet Summit - Preconference Training (2012)
Internet Summit - Preconference Training (2012)
 
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)Search Engine Marketing (Digital Marketing '15 @ Oulu University)
Search Engine Marketing (Digital Marketing '15 @ Oulu University)
 
Google strategy evaluation , i click hatem kameli
Google strategy evaluation , i click hatem kameliGoogle strategy evaluation , i click hatem kameli
Google strategy evaluation , i click hatem kameli
 
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...
2012-Search University 4 - Knewledge-Gerald Claessens & Wouter Schikhof- ...
 
Paid and Organic Search Co-Optimization
Paid and Organic Search Co-OptimizationPaid and Organic Search Co-Optimization
Paid and Organic Search Co-Optimization
 
Presentation for Exl Pharma Conference about Future of Search
Presentation for Exl Pharma Conference about Future of SearchPresentation for Exl Pharma Conference about Future of Search
Presentation for Exl Pharma Conference about Future of Search
 
Digital Prerequisites for Social Media Strategies
Digital Prerequisites for Social Media StrategiesDigital Prerequisites for Social Media Strategies
Digital Prerequisites for Social Media Strategies
 
Lead Scoring Conversions Webinar
Lead Scoring Conversions WebinarLead Scoring Conversions Webinar
Lead Scoring Conversions Webinar
 
Sept 20 2012 ona show me the numbers
Sept 20 2012 ona  show me the numbersSept 20 2012 ona  show me the numbers
Sept 20 2012 ona show me the numbers
 
Generative AI and SEO
Generative AI and SEOGenerative AI and SEO
Generative AI and SEO
 
Search Trends 2011
Search Trends 2011Search Trends 2011
Search Trends 2011
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
Introduction to Global SEO
Introduction to Global SEOIntroduction to Global SEO
Introduction to Global SEO
 
Webová analytika – proč? kdo? kdy? jak? (Insight Model)
Webová analytika – proč? kdo? kdy? jak? (Insight Model)Webová analytika – proč? kdo? kdy? jak? (Insight Model)
Webová analytika – proč? kdo? kdy? jak? (Insight Model)
 
Seo 4 Fundamentals
Seo 4 FundamentalsSeo 4 Fundamentals
Seo 4 Fundamentals
 
Organizational and strategic analysis of GOOGLE
Organizational and strategic analysis of GOOGLEOrganizational and strategic analysis of GOOGLE
Organizational and strategic analysis of GOOGLE
 
Keeping Content Searchable In Google's World
Keeping Content Searchable In Google's WorldKeeping Content Searchable In Google's World
Keeping Content Searchable In Google's World
 
Excel with Enterprise SEO
Excel with Enterprise SEOExcel with Enterprise SEO
Excel with Enterprise SEO
 
[Study] The Definitive Organic Click Through Rate Study
[Study] The Definitive Organic Click Through Rate Study[Study] The Definitive Organic Click Through Rate Study
[Study] The Definitive Organic Click Through Rate Study
 

More from Dirk Lewandowski

The Need for and fundamentals of an Open Web Index
The Need for and fundamentals of an Open Web IndexThe Need for and fundamentals of an Open Web Index
The Need for and fundamentals of an Open Web IndexDirk Lewandowski
 
In a World of Biased Search Engines
In a World of Biased Search EnginesIn a World of Biased Search Engines
In a World of Biased Search EnginesDirk Lewandowski
 
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...Dirk Lewandowski
 
Künstliche Intelligenz bei Suchmaschinen
Künstliche Intelligenz bei SuchmaschinenKünstliche Intelligenz bei Suchmaschinen
Künstliche Intelligenz bei SuchmaschinenDirk Lewandowski
 
Analysing search engine data on socially relevant topics
Analysing search engine data on socially relevant topicsAnalysing search engine data on socially relevant topics
Analysing search engine data on socially relevant topicsDirk Lewandowski
 
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändert
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändertGoogle Assistant, Alexa & Co.: Wie sich die Welt der Suche verändert
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändertDirk Lewandowski
 
Suchverhalten und die Grenzen von Suchdiensten
Suchverhalten und die Grenzen von SuchdienstenSuchverhalten und die Grenzen von Suchdiensten
Suchverhalten und die Grenzen von SuchdienstenDirk Lewandowski
 
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?Dirk Lewandowski
 
Are Ads on Google search engine results pages labeled clearly enough?
Are Ads on Google search engine results pages labeled clearly enough?Are Ads on Google search engine results pages labeled clearly enough?
Are Ads on Google search engine results pages labeled clearly enough?Dirk Lewandowski
 
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?Dirk Lewandowski
 
Wie Suchmaschinen die Inhalte des Web interpretieren
Wie Suchmaschinen die Inhalte des Web interpretierenWie Suchmaschinen die Inhalte des Web interpretieren
Wie Suchmaschinen die Inhalte des Web interpretierenDirk Lewandowski
 
Perspektiven eines Open Web Index
Perspektiven eines Open Web IndexPerspektiven eines Open Web Index
Perspektiven eines Open Web IndexDirk Lewandowski
 
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?Wie entwickeln sich Suchmaschinen heute, was kommt morgen?
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?Dirk Lewandowski
 
Neue Trends: Google, SEO und Co.?
Neue Trends: Google, SEO und Co.?Neue Trends: Google, SEO und Co.?
Neue Trends: Google, SEO und Co.?Dirk Lewandowski
 
Internet-Suchmaschinen: Aktueller Stand und Entwicklungsperspektiven
Internet-Suchmaschinen: Aktueller Stand und EntwicklungsperspektivenInternet-Suchmaschinen: Aktueller Stand und Entwicklungsperspektiven
Internet-Suchmaschinen: Aktueller Stand und EntwicklungsperspektivenDirk Lewandowski
 
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...Dirk Lewandowski
 
Verwendung von Skalenbewertungen in der Evaluierung von Suchmaschinen
Verwendung von Skalenbewertungen in der Evaluierung von SuchmaschinenVerwendung von Skalenbewertungen in der Evaluierung von Suchmaschinen
Verwendung von Skalenbewertungen in der Evaluierung von SuchmaschinenDirk Lewandowski
 
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)Dirk Lewandowski
 
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)Dirk Lewandowski
 

More from Dirk Lewandowski (20)

The Need for and fundamentals of an Open Web Index
The Need for and fundamentals of an Open Web IndexThe Need for and fundamentals of an Open Web Index
The Need for and fundamentals of an Open Web Index
 
In a World of Biased Search Engines
In a World of Biased Search EnginesIn a World of Biased Search Engines
In a World of Biased Search Engines
 
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...
EIN ANDERER BLICK AUF GOOGLE: Wie interpretieren Nutzer/innen die Suchergebni...
 
Künstliche Intelligenz bei Suchmaschinen
Künstliche Intelligenz bei SuchmaschinenKünstliche Intelligenz bei Suchmaschinen
Künstliche Intelligenz bei Suchmaschinen
 
Analysing search engine data on socially relevant topics
Analysing search engine data on socially relevant topicsAnalysing search engine data on socially relevant topics
Analysing search engine data on socially relevant topics
 
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändert
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändertGoogle Assistant, Alexa & Co.: Wie sich die Welt der Suche verändert
Google Assistant, Alexa & Co.: Wie sich die Welt der Suche verändert
 
Suchverhalten und die Grenzen von Suchdiensten
Suchverhalten und die Grenzen von SuchdienstenSuchverhalten und die Grenzen von Suchdiensten
Suchverhalten und die Grenzen von Suchdiensten
 
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?
Können Nutzer echte Suchergebnisse von Werbung in Suchmaschinen unterscheiden?
 
Are Ads on Google search engine results pages labeled clearly enough?
Are Ads on Google search engine results pages labeled clearly enough?Are Ads on Google search engine results pages labeled clearly enough?
Are Ads on Google search engine results pages labeled clearly enough?
 
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?
Search Engine Bias - sollen wir Googles Suchergebnissen vertrauen?
 
Wie Suchmaschinen die Inhalte des Web interpretieren
Wie Suchmaschinen die Inhalte des Web interpretierenWie Suchmaschinen die Inhalte des Web interpretieren
Wie Suchmaschinen die Inhalte des Web interpretieren
 
Perspektiven eines Open Web Index
Perspektiven eines Open Web IndexPerspektiven eines Open Web Index
Perspektiven eines Open Web Index
 
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?Wie entwickeln sich Suchmaschinen heute, was kommt morgen?
Wie entwickeln sich Suchmaschinen heute, was kommt morgen?
 
Suchmaschinen verstehen
Suchmaschinen verstehenSuchmaschinen verstehen
Suchmaschinen verstehen
 
Neue Trends: Google, SEO und Co.?
Neue Trends: Google, SEO und Co.?Neue Trends: Google, SEO und Co.?
Neue Trends: Google, SEO und Co.?
 
Internet-Suchmaschinen: Aktueller Stand und Entwicklungsperspektiven
Internet-Suchmaschinen: Aktueller Stand und EntwicklungsperspektivenInternet-Suchmaschinen: Aktueller Stand und Entwicklungsperspektiven
Internet-Suchmaschinen: Aktueller Stand und Entwicklungsperspektiven
 
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...
Ordinary Search Engine Users Assessing Difficulty, Effort and Outcome for Sim...
 
Verwendung von Skalenbewertungen in der Evaluierung von Suchmaschinen
Verwendung von Skalenbewertungen in der Evaluierung von SuchmaschinenVerwendung von Skalenbewertungen in der Evaluierung von Suchmaschinen
Verwendung von Skalenbewertungen in der Evaluierung von Suchmaschinen
 
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (3)
 
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)
Neue Entwicklungen bei Suchmaschinen und deren Relevanz für Bibliotheken (2)
 

Recently uploaded

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 

Recently uploaded (20)

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 

The Influence of Commercial Intent of Search Results on Their Perceived Relevance

  • 1. 11.02.2011 The Influence of Commercial Intent of Search Results on Their Perceived Relevance Dirk Lewandowski Hamburg University of Applied Sciences, Germany Introduction •  Ongoing discussion about results quality –  Did Google’s results really get worse? –  Do search engine results (and especially, Google’s) get more and more commercial? •  Discussion on biased search results / “search neutrality” –  Does Google prefer its own services in its rankings? •  Objective –  To find out whether commercial results have a negative effect on relevance as perceived by the users. 1
  • 2. 11.02.2011 Search engine market share (Germany) Previous work •  Retrieval effectiveness tests –  Lots of work done –  Main objective is to compare results quality of commercial Web search engines –  Tests are highly standardized •  Commercial intent –  Some studies on commercial intent of the searchers –  No studies on the commercial intent of the documents 2
  • 3. 11.02.2011 Research questions •  RQ1: How effective are web search engines in retrieving relevant results? •  RQ2: What ratio of the top 10 search engine results are pages with a commercial intent? •  RQ3: Are results with a commercial intent comparably relevant to non- commercial results? •  RQ4: Do banner or textual ads have a negative effect on the perceived relevance? Test design •  Retrieval effectiveness test –  To collect relevance judgments •  Results classification –  Classify each result according to its commercial intent •  Commercial intent of the page/website itself •  Commercial intent as expressed through advertisements on the page 3
  • 4. 11.02.2011 Retrieval effectiveness test design •  50 queries –  “Real life queries” from students asked for their last query entered into a Web search engine –  Mainly simple queries without an explicit commercial intent •  3 search engines –  Google, Microsoft, Yahoo (German user interfaces) •  10 results per query •  Binary relevance judgments •  50 undergraduate students as jurors Retrieval effectiveness test: Data collection •  Results from all search engines collected •  Results –  were made anonymous –  were randomized •  Each juror evaluated the complete results set for one query 4
  • 5. 11.02.2011 Results classification (1) •  Commercial 1.  A product is sold on the page 2.  Webpage of a company •  Non-commercial 3.  Page of a public authority, government office, administration, etc. 4.  Page of a non-governmental organization, association, club, etc. 5.  Private page 6.  Page of an institution of higher education •  Other Results classification (2) •  Number of banner ads •  Number of text-based ads (“sponsored links”) 5
  • 6. 11.02.2011 Limitations •  Results cannot be generalised –  Commercial intent of results does not have a negative influence on their perceived relevance when asking undergraduate students. •  Relatively small dataset –  500 URLs per search engine (total of 1,500 URLs) Results 6
  • 7. 11.02.2011 Results 1.  Results from the classification task 2.  Results from the retrieval effectiveness test 3.  Combined results: Influence of commercial intent on the perceived relevance Results 1.  Results from the classification task 7
  • 8. 11.02.2011 Results from the classification task (1) Results with a commercial intent account for –  82% (Google) –  75% (Microsoft) –  78% (Yahoo) Results from the classification task (2) •  Lowest ratio of results with a commercial intent on the first position. 8
  • 9. 11.02.2011 Results from the classification task (3) •  Google has a significantly higher ratio of results with sponsored links. –  Google: 23.7% of all results –  Microsoft: 12.3% –  Yahoo: 17.7% •  No significant difference between the ratios of results with banner ads. –  Google: 21.3% of all results –  Microsoft: 23.0% –  Yahoo: 22.0% Results •  Results from the retrieval effectiveness test 9
  • 10. 11.02.2011 Macro precision Number of queries answered best by an individual search engine Mean Average Precision All search engines, all results 10
  • 11. 11.02.2011 Results •  Combined results: Influence of commercial intent on the perceived relevance Relevance of results with a commercial intent •  Relevance of results with commercial intent is comparable to non- commercial results –  Google: 59.4% of commercial results relevant (vs. 58.7%) –  Microsoft: 34.6% vs. 36.5% –  Yahoo: 51.1% vs. 51.7% 11
  • 12. 11.02.2011 Relevance of results with advertising •  Relevance of results with text-based ads is comparable to results without text-based ads –  Google: 59.8% vs. 59.3% –  Microsoft: 40.0% vs. 33.9% –  Yahoo: 52.7% vs. 52.0% •  Relevance of results with banner ads is comparable to results without banner ads –  Google: 58.5% vs. 59.8% –  Microsoft: 33.3 vs.35.1% –  Yahoo: 55.8% vs. 49.1% Discussion 12
  • 13. 11.02.2011 Discussion •  Most striking result is that Google has significantly more results with text- based ads. •  Search Engine Optimization (SEO) may be a reason –  Many pages are highly optimised for Google. •  Wouldn’t it be clever to prefer pages carrying ads from the search engine’s own ads system? –  However, Google says it doesn’t do this. Google‘s advertising model 13
  • 14. 11.02.2011 Conclusion & further research Conclusion 1.  In terms of retrieval effectiveness, Google outperforms the other two search engines (RQ1). 2.  There is only a slight variation in the ratio of commercial results from search engine to search engine (RQ2). 3.  The results classified as “commercial” are comparably relevant to the results classified as “non-commercial” (RQ3). 4.  Textual or banner ads do not have a negative effect on the perceived relevance of the corresponding result (RQ4). 14
  • 15. 11.02.2011 Further research 1.  Study should be repeated using other user groups –  Relevance judgments came from a comparably homogeneous user group. –  Data already collected for other user groups (another student group; librarians; crowd-sourcing). 2.  Large ratio of results with text-based ads in Google deserves further investigation. –  Currently, we conduct a large-scale study where we automatically detect text-based ads on search results, comparing Google and Bing. •  30,000 queries, 2 search engines, 30 results per query  1.8 Million URLs. 15