일상대화란 정확히 무엇인가? 왜 필요로 하는가? 핑퐁의 접근 방법은 무엇이 특별한가? FAQ 기반의 챗봇의 한계점 Reply-Centric 기반 답변 Retrieval Model Reply-Centric 기반 Reaction Classification Future Works: 100억건의 데이터를 충분히 활용할 수 있는 모델들은 무엇이 있는가? 핑퐁 봇 & 핑퐁 빌더 데모

1. 사람들과 자연스러운 대화를 나누는
인공지능 ‘핑퐁’ 만들기
스캐터랩(ScatterLab)
Machine Learning Engineer
김준성
스캐터랩(ScatterLab)
CEO
김종윤

2. 1. 🤖
2. &
3. Query Centric Model :
4. Reaction Model: !
5. Reply Retrieval Model: !
6. Query Retrieval Model: QA ?
7. Pingpong Bot & Builder :
8. Future Works: ?

3. 핑퐁과
일상대화 기술은 무엇인가?
스캐터랩(ScatterLab)
CEO
김종윤

5. #0. ?
= “ ?”
= “ ”

6. 20 .
10% .
. .
= “ ?”
= “ ”
#0. ?

7. ( : “ ?”) ( : “ ”)
,
#0. ?

8. Button UI NLI
NLIGUI
#1. ?
Natural Language Interface =

9. ( , 2017 5 3 )
SKT ‘ ’
45%
1) Nass, Clifford, and Li Gong. "Speech interfaces from an evolutionary perspective." Communications of the ACM 43.9 (2000): 36-43.
1.
• AI 1)
AI .
• AI
, “ ”
.
.
• ,
AI (human-likeness)
.
#1. ?
(Jiang J, et al. (2015))
Microsoft Cortana
30%

10. 2.
• AI
,
(CPS, Conversation-turns per session)
1).
• AI
,
.
#1. ?
1) Chen, Chun-Yen, et al. "Gunrock: Building A Human-Like Social Bot By Leveraging Large Scale Real User Data."

11. 3.
• AI ,
AI .
• AI
.
#1. ?

12. ,
• . AI
. .
•
.
#1. ?

13. Vision
Make machines social

14. Vision
Make machines social
“ ”

15. Mission Statement

16. Open-Domain Conversational AI:
Technical Section
스캐터랩(ScatterLab)
Machine Learning Engineer
김준성

17. Source: Optional source placeholder. Delete if not used.
(Junseong Kim)
fb.com/codertimo
ScatterLab Machine Learning Engineer
Machine Learning Team ( )
Naver Clova AI Research Intern
Atlas Guide NLP/Machine Learning Engineer
General Domain Dialog System/Chatbot (Chitchat)
General Sentence Embedding (BERT, ELMo)
And Diverse Natural Language Processing Tasks…
Neural Machine Translation
codertimo/BERT-pytorch 2.4k+ stars
Research Domain
Experience
2017, 2014 Intel ISEF Computer Science Finalist
2016, 2014 Intel KSEF Computer Science Grand Award
Working Experience

18. Dataset
100 + Utterance(2.2TB)
& :
Utterance Variance
Context Aware Utterance
Massive Dialog System
…
NLP/Dialog System 😍

19. Goal
Engineer Main Goal
100 !
?

20. Query Centric Model
!
Query
?!?!

21. Query Centric Model
Query Matching
Sentence Representation Cosine-Similarity Script Retrieval
Response = argmax(cos(Queryembed, Scriptembedi
))

22. Query Centric Model
?
Query Space
!
Query Matching
Sentence Representation Cosine-Similarity Script Retrieval
Response = argmax(cos(Queryembed, Scriptembedi
))

23. Query Centric Model
? 👋
😭
😘
??😰
Query Space
Query Matching
Sentence Representation Cosine-Similarity Script Retrieval
Response = argmax(cos(Queryembed, Scriptembedi
))

24. Query Centric Model
?
!
👋
😭
😘
??😰
Query Space
Query Matching
Sentence Representation Cosine-Similarity Script Retrieval
Response = argmax(cos(Queryembed, Scriptembedi
))

25. Query Centric Model
?
!
👋
😭
😘
??😰
Query Space
Query Matching
Sentence Representation Cosine-Similarity Script Retrieval
Response = argmax(cos(Queryembed, Scriptembedi
))

26. Query Centric Model
https://youtu.be/rhHQMA80TcE
Query Retrieval Model: Pingpong-POC

27. Query Centric Model
Query Retrieval Model: Pingpong-POC
• Rule Based Filtering
• , , Rule Based Filtering
• Human Filtering
• -> Query-Reply Pair

28. Query Centric Model
Query Retrieval Model: Pingpong-POC
Q-> !
Reply select:
Q’:

29. Query Centric Model
Query Retrieval Model: Pingpong-POC
•
• → ( )
• Blackbox → (B2B) control → customization
• Query Search Variance →
Ex ) -> ?
Ex ) ! -> ><>< , ->
Ex ) -> XX XX
Ex ) : ! -> !

30. Query Centric Model
Search Space Q' N
Query Space Query Variance
Query -> Information -> (1:1 )

31. Reply Centric Model
Query Space
Reply Space ? 🤔
?
Query ?

32. Reply Centric Model
Reply Space
Reaction
?
?
…
Answer to Query
,….
Context Aware Reply
?
KB & Background Aware Reply
30

33. Reply Centric Model
Reply Space
Coverage ! (
Context+Query -> Reply Distribution Deep Learning
+ NLU, NLG

34. Reaction
Coverage,
Query Coverage
: { | }
🧐 🤩 😡 🤗 😘
: Reaction

35. Reply Edit Distance Clustering
1200 Reaction Class
Reaction
!!!!
…??
..
.
..
?
..
?!?!
?? ??
Coverage,

36. Reaction
!!!!
…??
..
.
..
?
..
?!?!
?? ??
( Class)
?!?!
( Class)
Coverage,
Reaction Dataset Labeling
Reply Edit Distance Clustering
1200 Reaction Class

37. Reaction
Coverage,
1000 + Reaction Training Pair 👏
Query Variance

38. Reaction
Coverage,
Classification Model Retrieval Model
Queryinput
Query Encoder
Ensemble
XGBoost Reranking
Reaction Class Select Among 1K class
!!
…

39. Reaction
Coverage,
Response ResponseReaction Class
Response

40. Reaction
https://youtu.be/Cabc5JOPqEk
https://goo.gl/yANLGz

41. Pros:
1. . .
2. ,
Reaction
Coverage,
Cons:
1.
2. ,

42. Reply Retrieval Model
,
(Q)
,
Main

43. Reply Retrieval Model
Next Utterance Prediction Model
https://arxiv.org/abs/1705.02364
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Query Reply
Query Reply Continuous Utterance Next Utterance Classification
Response = argmax(Model(Equery, Ereplyi
))

44. Reply Retrieval Model
Next Utterance Prediction Model
Query-Reply 50 Pair (single )
Query, Reply Sentence Encoder concat FNN Binary Classification
Pair Next Utterance Prediction (BERT Next Sentence Prediction)
N:1 Negative Sampling Random ( N Negative )

45. Reply Retrieval Model
Next Utterance Prediction Model
Reply Set

46. Reply Retrieval Model
Next Utterance Prediction Model: Unbalanced
Reaction 😰
Reranking Logic ( )
?
Q
0.79
? 0.92
Insight: ..🤖
…

47. Reply Retrieval Model
Next Utterance Prediction Model: Unbalanced RMM Prediction Result
🤩
!

48. Reply Retrieval Model
Next Utterance Prediction Model
Pros:
1. Reaction ( { , } → ! )
2. (?)
3. Q
Cons
1. (R) Space .
1. Reply set
2. Reply Q , Q .

49. Q Reply retrieval model Cons .
,
Reply Retrieval Model .
Query Retrieval Model
Motivation
1:1

50. Query Retrieval Model
Using Next-Utterance Task Pretrained Encoding
LM+Next Sentence Prediction context
Next Utterance Sub-Task Encoder Embedding ?
Query Reply

51. ???!🤩
Query Retrieval Model
Dataset

52. 😰
Query Retrieval Model
Dataset

53. Query Retrieval Model
Dataset
“ ”
50000

54. Query Retrieval Model
Using Next-Utterance Task Pretrained Encoding
Query Query
Transfer Learning
Dataset Fine-Tuning

55. https://docs.google.com/spreadsheets/d/1LWsrffq93u5U55-O7Yij8Lhl15Dlbr8mlucabTUB9jo/edit#gid=1973099187
Query Retrieval Model
Result: SOTA of Chatbot Builder
!! SOTA!!
RAW &Test Dataset

56. Query Retrieval Model
Result: SOTA of Chatbot Builder

57. Query Retrieval Model
Pros and Cons
Cons:
1. Q variation
Pros:
1.
2.
3. , , Dialogflow , SOTA

58. https://landing.pingpong.us/
Pingping Builder
Chatbot Builder For Open-Domain Chatbot

59. Pingping Bot
Prebuilt Bot & App
^^https://m.me/ai.pingpong

60. Pingping Bot
Prebuilt Bot & App

61. Pingping Bot
Prebuilt Bot & App
COMMING SOOOOON😘
Pingpong Family

62. Future Works
NER, Tokenizer, Typo-Correction, Semantic Parsing
Question and Answering : external KB, user KB
Persona Reply Style Transfer
Reply Generation
Context Aware Reply
End-to-End Dialog System
Hyper-Personalization
SOTA Language Modeling
😭

63. 😭
Context Aware BERT for Open-Domain Dialog System

64. 감사합니다