Evaluating Heterogeneous Information Access and Search Evaluation Challenges
This document discusses challenges in evaluating heterogeneous information access systems and proposes areas for future research. It notes that traditional IR evaluation focuses on system-oriented metrics using test collections, while heterogeneous search involves more complex user behaviors like non-linear browsing. Key challenges include accounting for diverse search tasks, coherence, diversity, personalization and different result presentation strategies. The document advocates better understanding user behavior through models and applying this understanding to improve evaluation metrics for heterogeneous search systems. It identifies areas for future work such as modeling task complexity, coherence and click patterns to develop more powerful evaluation frameworks.
Recomendados
Recomendados
Más contenido relacionado
Similar a Evaluating Heterogeneous Information Access and Search Evaluation Challenges
Similar a Evaluating Heterogeneous Information Access and Search Evaluation Challenges (20)
Más de Mounia Lalmas-Roelleke
Más de Mounia Lalmas-Roelleke (20)
Último
Último (20)
Evaluating Heterogeneous Information Access and Search Evaluation Challenges
- 1. Evalua&ng Heterogeneous Informa&on Access (Posi&on Paper) Ke Zhou1, Tetsuya Sakai2, Mounia Lalmas3, Zhicheng Dou2 and Joemon M. Jose1 1University of Glasgow 2MicrosoN Research Asia 3Yahoo! Labs Barcelona SIGIR 2013 MUBE workshop
- 2. IR Evalua&on • System-‐oriented Evalua&on (test collec&on + metrics) • User-‐oriented Evalua&on (interac&ve user study) • Current endeavor to incorporate user into system-‐oriented metrics – Time-‐Biased Gain (Smucker, Clarke.) – U-‐measure (Sakai, Dou) – etc.
- 3. Increasing Heterogeneous Nature on Search
- 4. Increasing Heterogeneous Nature on Search ……
- 5. Posi&on • Compared with tradi&onal homogeneous search, evalua&on in the context of heterogeneous informa&on is more challenging and requires taking into account more complex user behaviors and interac4ons.
- 6. Challenges • Non-‐linear Traversal Browsing • Diverse Search Tasks • Coherence • Diversity • Personaliza&on • etc.
- 7. Various Presenta&on Strategies Non-‐linear BlendedBlended …… Tabbed
- 8. User Browsing Pa^ern “E” Browsing Pa?ern on Aggregated Search Page “F” Browsing Pa?ern on Organic Search Page h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233
- 9. Non-‐linear Traversal Browsing h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233
- 10. Search Tasks: Ver&cal Orienta&on CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.)
- 11. Search Tasks: Complexity SIGIR’12 (Arguello et al.)
- 13. Coherence Car Animal Animal Car Animal Car Car Car Animal Car Animal Car vs. CIKM’12 (Arguello et al.)
- 16. Avenues of Research • Be^er understanding of users – Click models: WSDM’12 (Chen et al.), SIGIR’13 (Wang et al.) – Ver&cal orienta&on: CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.) – Task complexity: SIGIR’12 (Arguello et al.) – Task coherence: CIKM’12 (Arguello et al.) – Diversity: SIGIR’12 (Zhou et al.) – Personaliza&on – Non-‐linear presenta&on strategies
- 17. Avenues of Research • Be^er incorpora&on of learned user behavior into evalua&on metrics – follow SIGIR’13 (Chuklin et al) and convert obtained aggregated search click models into system-‐oriented evalua&on metrics. – model addi&onal features into powerful evalua&on framework (e.g. TBG, U-‐measure, AS-‐ metric).
- 18. Thank you! Ques&ons? Ke Zhou, zhouke@dcs.gla.ac.uk TREC 2013 FedWeb track: h^ps://sites.google.com/site/trecfedweb/