Recomendados
Recomendados
Más contenido relacionado
Similar a PARADE passage representation aggregation for document reranking
Similar a PARADE passage representation aggregation for document reranking (20)
Más de Sunghoon Joo
Más de Sunghoon Joo (20)
Último
Último (20)
PARADE passage representation aggregation for document reranking
- 1. PR-298 주성훈, Samsung SDS 2021. 1. 17. https://www.flaticon.com/kr/authors/freepik https://arxiv.org/pdf/2008.09093.pdf PARADE: Passage Representation Aggregation for Document Reranking Canjia Li1,3∗, Andrew Yates2, Sean MacAvaney4, Ben He1,3, Yingfei Sun1 1 University of Chinese Academy of Sciences, Beijing, China 2 Max Planck Institute for Informatics, Saarbr¨ucken, Germany 3 Institute of Software, Chinese Academy of Sciences, Beijing, China 4 IR Lab, Georgetown University, Washington, DC, USA licanjia17@mails.ucas.ac.cn, ayates@mpi-inf.mpg.de sean@ir.cs.georgetown.edu, {benhe, yfsun}@ucas.ac.cn
- 3. 1. Research Background Information Retrieval (IR) 3/22 Inverted Index Retrieval task https://giyatto.tistory.com/2 https://devopedia.org/information-retrieval 1) Retrieval stage 2) re-ranking stage • 1차 검색 결과를 재정렬함으로서 더 연관성 높은 문서가 상위에 노출되도록 한다.
- 4. 1. Research Background BERT 기반의 re-ranking 4/22 Ying, Chengxuan, and Chen Huo. "An Adaptive Early Stopping Strategy for Query-based Passage Re-ranking.“ (2020) • BERT (http://arxiv.org/abs/1810.04805) 를 많은 NLP task 에 적용해 성공을 거둠 -> IR re-ranking task에 동기 부여 • Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT • Transformer의 self-attention의 높은 계산비용으로 인한 입력 문장 길이 제한이 있음
- 5. BERT P1 Document P2 Pn … rel1 rel2 reln … Q [SEP] Document relevance score 결정하는 방법 • 상위 3개 sentence/passage의 relevance score를 평균 내 문서의 relevance score를 결정 • Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling • Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. • Sentence/passage의 relevance score를 aggregation 하는 방법 1. Sentence/passage의 순서를 고려하는 방법 : non-relevant passag가 들어오면 rel(q,D)를 깎음 • Hui Fang, Tao Tao, and Chengxiang Zhai. 2011. Diagnostic evaluation of information retrieval models. ACMTrans. Inf. Syst., 29(2). 2. passage-level cumulative gain • Zhijing Wu, Jiaxin Mao, Yiqun Liu, Jingtao Zhan, Yukun Zheng, Min Zhang, and Shaoping Ma. 2020. Leveraging passage-level cumulative gain for document ranking. In WWW. ACM. 𝑟𝑒𝑙(𝑞, 𝐷) • Longformer (4096 token) • Beltagy et al., 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. 1. Research Background 5/22
- 6. Objective & Approach : a hybrid retrieval approach • We propose PARADE, an end-to-end document reranking model. • PARADE predicts a document’s relevance by learning passage-level relevance representations that are aggregated in a way that preserves document-level context. 1. Research Background 6/22 BERT P1 Document P2 Pn … Aggregation Q [SEP] 𝑟𝑒𝑙(𝑞, 𝐷)
- 7. 2. Methods
- 8. Representing a Document as Passages 𝑝𝑖 𝑐𝑙𝑠 = BERT(𝑞, 𝑃𝑖) 2. Methods 8/22 BERT BERT 𝑝2 𝑐𝑙𝑠 𝑝 𝑛 𝑐𝑙𝑠 P1: 150 words P2: 150 words 150 words 100 words stride P3: 150 words P1-start P1-endP2-start P2-endP3-start P3-end … 𝐷 = {𝑃1, … , 𝑃𝑛} … BERT 𝑝1 𝑐𝑙𝑠 𝑝3 𝑐𝑙𝑠 100 words stride
- 9. Aggregating Passage Relevance Representations 2. Methods 9/22 BERT BERT 𝑝2 𝑐𝑙𝑠 𝑝 𝑛 𝑐𝑙𝑠𝐷 𝑐𝑙𝑠 = {𝑝1 𝑐𝑙𝑠 , … , 𝑝 𝑛 𝑐𝑙𝑠} … BERT 𝑝1 𝑐𝑙𝑠 𝑝3 𝑐𝑙𝑠 𝑝𝑖 𝑐𝑙𝑠 = BERT(𝑞, 𝑃𝑖) 𝐷 = {𝑃1, … , 𝑃𝑛} 𝐷 𝑐𝑙𝑠 → 𝑑 𝑐𝑙𝑠 ∈ 𝑅 𝑑 Aggregating layer Single-layer feed-forward network 𝑟𝑒𝑙(𝑞, 𝐷) 𝑑 𝑐𝑙𝑠 𝑑 𝑐𝑙𝑠 [𝑗] = max(𝑝1 𝑐𝑙𝑠 𝑗 , … , 𝑝 𝑛 𝑐𝑙𝑠 𝑗 ) 𝑤1, … , 𝑤 𝑛 = softmax(𝑊𝑝1 𝑐𝑙𝑠 , … , 𝑊𝑝 𝑛 𝑐𝑙𝑠) 𝑑 𝑐𝑙𝑠 = 𝑖=1 𝑛 𝑤𝑖 𝑝1 𝑐𝑙𝑠 1) PARADEMax 2) PARADEAttn
- 10. Aggregating Passage Relevance Representations 3) PARADETransformer 2. Methods 10/22 𝑝2 𝑐𝑙𝑠 𝑝1 𝑐𝑙𝑠 𝑝 𝑛 𝑐𝑙𝑠 Single-layer feed-forward network 𝑟𝑒𝑙(𝑞, 𝐷) http://jalammar.github.io/illustrated-transformer/ 변형해 새로 그림 𝑒𝑚𝑏 𝑐𝑙𝑠 … +positional embedding MultiHead Attention Add & LayerNorm 2-layer Feed Forward Add & LayerNorm 𝑑 𝑐𝑙𝑠 𝐷 𝑐𝑙𝑠 = {𝑝1 𝑐𝑙𝑠 , … , 𝑝 𝑛 𝑐𝑙𝑠} 𝑝𝑖 𝑐𝑙𝑠 = BERT(𝑞, 𝑃𝑖) 𝐷 = {𝑃1, … , 𝑃𝑛} 𝐷 𝑐𝑙𝑠 → 𝑑 𝑐𝑙𝑠 ∈ 𝑅 𝑑
- 11. Robust04 is a newswire collection used by the TREC 2004 Robust track. GOV2 is a Web collection crawled from government Websites. Dataset • Robust04, GOV2 2. Methods 11/22 301 0 FBIS3-10082 1 301 0 FBIS3-10169 0 301 0 FBIS3-10243 1 301 0 FBIS3-10319 0 301 0 FBIS3-10397 1 301 0 FBIS3-10491 1 301 0 FBIS3-10555 0 301 0 FBIS3-10622 1 301 0 FBIS3-10634 0 301 0 FBIS3-10635 0 301 0 FBIS3-10721 1 301 0 FBIS3-10805 1 … qrels.robust2004.txt
- 12. P3 Q [SEP] Pn Training • BERT 초기화 (MSMARCO passage ranking) 2. Methods 12/22 301 0 FBIS3-10082 1 301 0 FBIS3-10169 0 301 0 FBIS3-10243 1 301 0 FBIS3-10319 0 301 0 FBIS3-10397 1 301 0 FBIS3-10491 1 301 0 FBIS3-10555 0 301 0 FBIS3-10622 1 301 0 FBIS3-10634 0 301 0 FBIS3-10635 0 301 0 FBIS3-10721 1 301 0 FBIS3-10805 1 … BM25 result BERT BERT 𝑝2 𝑐𝑙𝑠 𝑝 𝑛 𝑐𝑙𝑠 … 𝑝1 𝑐𝑙𝑠 𝑝3 𝑐𝑙𝑠 Aggregating layer Single-layer feed-forward network 𝑟𝑒𝑙(𝑞, 𝐷) 𝑑 𝑐𝑙𝑠 BERT 𝑟𝑒𝑙(𝑞, 𝐷) Q [SEP] D GT 𝐿 𝐶𝐸 • PARADE 학습 (robust2004, GOV2) Q [SEP] P1 Q [SEP] P2 BERT 3 epochs, single Google TPU v3-8 에서 2.5 시간. Document set 1000 Initial ranking (BM25)
- 14. Research Questions • 속도와 정확도 tradeoff 관련하여, RQ1: How can BERT’s efficiency be improved while maintaining its effectiveness? • 입력 문서의 길이와 관련하여, RQ2: How does the number of document passages preserved influence effectiveness? • 다양한 initial ranking method의 대해 PARADE가 성능 향상이 있는지 RQ3: Is it beneficial to rerank documents from a more effective initial ranking method? In particular, is reranking BM25+RM3 better than reranking BM25? 3. Experimental Results 14/22
- 15. *Birch : top 3 passage score와 initial rankin에서 document score를 weighted sum 함. BERT-Large 사용, initial ranking score고려 (https://www.aclweb.org/anthology/D19-3004/) 3. Experimental Results 15/22 PARADE의 reranking 성능 Document set 100 100 Initial ranking (BM25) Re-ranking *BERT-MaxP : 4 passage의 max score로 doc score결정 (https://arxiv.org/pdf/1905.09217.pdf)
- 16. PARADE의 계산 효율성에 대한 실험 3. Experimental Results 16/22
- 17. zt, zs : logits from teacher and student modelLCE : student model에 대한 cross-entropy loss 3. Experimental Results 17/22
- 18. 3. Experimental Results 18/22 입력 passage의 수에 대한 성능 비교 입력 문장이 길수록 성능이 좋음 : 문서의 내용을 잘 보존한다. Passage representation 을 위해 Attention을 사용해야 한다.
- 19. 효율성과 성능을 모두 고려하면, 많은 passage를 사용하도록 학습하고 inference시에는 적은 passage를 사용하는 것이 유리하다 • Trade-off efficiency for effectiveness 입력 passage의 수에 대한 성능 비교 3. Experimental Results 19/22 계산량 증가
- 20. PARADE가 re-ranking 방법으로서 효과적인가? 3. Experimental Results 20/22 D 1000 1000 Initial ranking (BM25, BM25+RM3) Re-ranking (PARADE) DQE 와 DBM25 가 PARADE에 의해 높은 순위로 올라갔다. DQE: BM25 + RM3 에서 검색했지만 BM25 에서 검색하지 않은 관련 문서 DBM25: BM25에서 검색했지만 BM25 + RM3에서 검색하지 않은 관련 문서
- 21. 4. Conclusion
- 22. Thank you. • Language model기반의 PARADE라는 end-to-end document reranking model 을 새로 선보였다. TREC Robust04 dataset에서 SOTA TREC-COVID challenge에서 second round 1위 • PARADE에 대한 knowledge distillation을 통해 parameter수를 줄이면서 성능은 높이는 것이 가능했다. 4. Conclusions 22/22 • BM25, RM3 등 기존의 initial ranking method의 검색 결과 품질을 re-ranking으로 높이는 것이 가능하다는 것을 보였다. • Transformer를 이용한 passage representation aggregation이 문서의 relevance score를 구하는 좋은 방법임을 보였다.