As presented at MT Summit XV

1. MACHINE TRANSLATION
QUALITY ESTIMATION
A Linguist’s Approach

2. WHAT IS MT QUALITY ESTIMATION?
Automatically providing a quality indicator for machine
translation output without depending on human reference
translations.
Our objective:
Estimate quality and post-editing effort for eBay listing titles
and descriptions
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 2

3. ONE big CHALLENGE
min W Ʃ T
t=1 ||(W(t)X(t) − Y (t) )||2 2 + λs||S||1 + λb||B||1,∞ subject to: W = S + B
or
“State-of-the-art QE explores different supervised linear or non-linear learning methods
for regression or classification such as Support Vector Machines (SVM), different types
of Decision Trees, Neural Networks, Elastic-Net, Gaussian Processes, Naive Bayes,
among others”
(Machine Translation Quality Estimation Across Domains, de Souza et al, 2014)
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 3

4. A LINGUIST’S APPROACH
Using linguistic features from 3 dimensions:
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 4
COMPLEXITY ADEQUACY FLUENCY

5. FEATURES
Complexity:
• Length
• Polysemy
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 5
Adequacy:
• QA
 Terminology
 Patterns
 Blacklist
 Numbers
• Automated
Post-Editing
• (POS)
• (NER)
Fluency:
• Misspellings
• Grammar errors

6. IMPLEMENTATION
Checkmate+LanguageTool
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 6
Reusable Profile
Detailed Report
Score

7. TESTING
• One Language (es-LA)
• Short samples (~300 words)
• Bigger samples (~1000 words)
• Post-Edited files (~50,000 words)
• pt-BR, ru-RU, zh-CN
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 7

8. RESULTS

9. MEASURING RESULTS
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 9

10. SAMPLES - SCORE AND TIME ALIGN
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 10

11. FILES - SCORE AND ED ALIGN
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 11
Average ED (es-LA, descriptions) = 72

12. MT QE OVER TIME
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 12

13. SAMPLES - OTHER LANGUAGES
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 13

14. CHALLENGES
• False positives
• Matching score and post-editing effort
• Same weight for all features
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 14

15. WHAT’S NEXT
• Tracking scores over time
• Adding scores to our post-editing tool
• Adding new languages
• Researching new features
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 15

16. HOW CAN YOU USE THIS?
• Tailor the model to your needs
• Estimate quality at the file/segment level
• Target post-editing, discard bad content
• Estimate post-editing effort/time
• Compare MT systems
• Monitor MT system progress
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 16

17. Q&A
THANK YOU! jrowda@ebay.com
MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 17