Pbsmt presenation waleed_oransa_29_april2010

Statistical Machine Translation Waleed Oransa, M.Sc. College of Computing and Information Technology Arab Academy for Science and Technology Cairo, Egypt [email_address]

Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object]

Why is Machine Translation so Hard? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

MT Approaches ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Interlingua Semantic Syntactic Structure Word s Semantic Syntactic Structure Word s Direct Syntactic Transfer Semantic Transfer Source Language Text Target Language Text Conceptual Generation Semantic Generation Syntactic Generation Morphological Generation Conceptual Analysis Semantic Analysis Parsing Morphological Analysis Better Quality & More difficulty

Why Statistical Machine Translation (SMT)? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Example of a parallel corpus ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Statistical Machine Translation (SMT) ,[object Object],[object Object],[object Object],[object Object],[object Object]

How to build SMT System ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SMT Training Phase English Sentences PBSMT System Training Language Model (Arabic) Input: Training Corpus Arabic/English Bi-Text Output: Language Model and Translation Model Arabic Sentences Translation Model (English/Arabic) Language Modeling Training (Tool: SRILM toolkit) Translation Modeling Training (Tool: Giza++ & Moses toolkit) What is the Language Model?

Language Model (LM) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

LM Role in SMT مشرف يجتمعوا مشرف تجتمع مشرف يجتمع مشرف يتقابل مشرف يقابل مشرف يجتمعا مشرف تجتمعا مشرف يجتمعن مشرف يجتمعون مشرف يتقابلان Language Model مشرف xxxx مع كبار المسؤولين المدنيين والعسكريين Language Model يجتمع يجتمعوا تجتمع يجتمع يتقابل يقابل يجتمعا تجتمعا يجتمعن يجتمعون يتقابلان

SMT Training Phase English Sentences PBSMT System Training Language Model (Arabic) Input: Training Corpus Arabic/English Bi-Text Output: Language Model and Translation Model Arabic Sentences Translation Model (English/Arabic) Language Modeling Training (Tool: SRILM toolkit) Translation Modeling Training (Tool: Giza++ & Moses toolkit) What is the Translation Model?

Translation Model (TM) ,[object Object],[object Object],[object Object]

Translation Model (TM) ,[object Object],[object Object],[object Object],[object Object]

Translation Model (TM) ,[object Object],[object Object]

SMT Translation Phase SMT System (Decoder) Source Text Target Text Language Model (Arabic) Translation Model (English/Arabic) Musharraf Meets with Senior Civilian ,[object Object],[object Object],[object Object],[object Object],[object Object],Initial N-best hypotheses p=0.13 p=0.21 p=0.23 p=0.12 p=0.18 ,[object Object],[object Object],[object Object],[object Object],[object Object],p=0.53 p=0.42 p=0.37 p=0.22 p=0.48 Final N-best hypotheses ,[object Object]

How TM & LM work together? Musharraf Meets with Senior Civilian and Military Officials مشرف ***** مع كبار المسؤولين المدنيين والعسكريين يجتمع Language Model Translation Model يجتمعوا تجتمع يجتمع يتقابل يقابل يجتمعا تجتمعا يجتمعن يجتمعون يتقابلان

Agenda ,[object Object],[object Object],[object Object],[object Object]

PBSMT Approach ,[object Object],[object Object],[object Object],[object Object],ولد The prophet Mohamed was born في سنة 570 ميلادية in 570 A.D الرسول محمد

PBSMT Training Phase English Sentences PBSMT System Training Language Model (Arabic) Input: Training Corpus Arabic/English Bi-Text Output: Language Model and Translation Model PBSMT Normal Training Phase Arabic Sentences Translation Model Phrase Table (English/Arabic) Language Modeling Training (Tool: SRILM toolkit) Translation Modeling Training (Tool: Giza++ & Moses toolkit)

Phrase based alignment (The prophet, الرسول ) (The prophet Mohamed, الرسول محمد ) (great man, رجل عظيم ) (Mohamed is a great man, محمد رجل عظيم ) (The prophet Mohamed is a great man, الرسول محمد رجل عظيم ) etc. Extract all phrase: English to Arabic word alignment Arabic to English word alignment Intersection of both alignments

PBSMT drawbacks ,[object Object],[object Object],[object Object]

MT Evaluation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Human Evaluation 5 Adequacy (1-5) 4 Fluency (1-5) مشرف يتقابل مع كبار المسؤولين المدنيين والعسكريين بخصوص ملف الحد من الانتشار النووي MT Musharraf Meets with Senior Civilian and Military Officials to Examine Nuclear Anti-Proliferation Dossier Source

Automatic Evaluation ,[object Object],[object Object],[object Object],[object Object],Higher BLEU Score مشرف يجتمع بكبار المسؤولين المدنيين والعسكريين لبحث ملف الحد من الانتشار النووي Ref3 مشرف يجتمع مع كبار المسؤولين المدنيين والعسكريين لدرس ملف الحد من الانتشار النووي Ref2 مشرف يجتمع بكبار المسؤولين المدنيين والعسكريين لدرس ملف الحد من الانتشار النووي Ref1 مشرف يجتمع بالقادة ال مدنيين و الع سكريين رفيعي المستوى ل دراسة ملف وقف الانتشار النووي MT2 مشرف يتقابل مع كبار المسؤولين المدنيين والعسكريين بخصوص ملف الحد من الانتشار النووي MT1 Musharraf Meets with Senior Civilian and Military Officials to Examine Nuclear Anti-Proliferation Dossier Source

Online MT Services Review البنتان قالتا أنّنا جيّدين SK 8 قال الاثنان بنات " نحن جيّد " SY 7 ان فتاتين " نحن جيدة " MS 6 الفتاتين وقال " نحن جاهزون " GO 5 قالت الفتاتان " نحن جيدات " The two girls said "we are good" B خمسة عشر بنتًا Sakhr Trjem (SK) 4 خمسة عشر بنات Systran translator (SY) 3 خمسة عشر الفتيات MS-Bing Translator (MS) 2 خمسة عشر فتيات Google Translate (GO) 1 خمس عشرة فتاة Fifteen girls A Arabic Translation Sentence/Translation Service

Pbsmt presenation waleed_oransa_29_april2010

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Pbsmt presenation waleed_oransa_29_april2010

Similar a Pbsmt presenation waleed_oransa_29_april2010 (20)

Último

Último (20)

Pbsmt presenation waleed_oransa_29_april2010