This document proposes a method called "Reversed analysis" to measure machine translation (MT) post-editing effort. The method involves applying a traditional translation memory (TM) leverage analysis technique to the target content of a raw MT output compared to a virtual TM of final post-edited content. This allows the method to evaluate each MT segment and calculate a cost reduction for translation jobs based on the matching percentage to the post-edited TM. The document describes applying this method using the ETMA (SDL TMS 2011) tool.

1. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
MT Reversed Analysis
François Richard, Feb2013

2. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
Objective
Define and apply a cost-effective and repeatable method/process
to measure MT Post-Editing effort within ETMA (SDL TMS 2011)
Objective
Reversed analysis will help in:
• Gaining understanding of main factors that impact raw MT quality.
• Quantifying raw MT quality issues raised by Translation suppliers.
• Deciding on new MT engines strategy (domain specific engines/ New training,...).
• Monitoring quality evolution.
• Identifying source content adequacy.
• Adjusting pricing model.
3 steps:
• Describe and document the method
• Share and explain the method to Translation Suppliers
• Apply the method to start collecting data about MT quality so that it can be further
analysed

3. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
What is PEMT?
PEMT is linguistic work/effort required to bring (modify/correct) raw MT output to a final
linguistic quality equivalent to the quality obtained with a classical translation process.
Many factors can influence the raw MT quality and as a result the required PEMT effort:
Volume used for the assessment
Content-type (Technical, legal, Marketing,…)
Language pair
Quality and consistency of the source
Human factor including experience and motivation, ...

4. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
TM
leverage:
Source
Translation
Low
fuzzy TM
match
High fuzzy
TM match
$=60% of FR
$=30% of FR
$=?? % of FR
$=?? % of FR
Lower
Quality MT
Higher
Quality MT
MT:
}PEMT

5. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Description of the "Reversed analysis" method
• In classical TM leverage analysis (based on TM fuzzy matching algorithm), “similarities”
between source content that you need to translate and source content you have stored
in your TMs are evaluated; resulting in a matching leverage analysis; combined with a
cost model to calculate TM translation savings.
• The principle is to apply this well-understood TM leverage analysis (*) to the target
content: The raw MT (target) content is analyzed against a virtual TM fed with the final
translated content (what has been post-edited); resulting in a matching leverage
analysis; combined to an appropriate cost model (*) to calculate MT translation
savings.

6. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Key properties
• It is a posteriori process (PEMT must be completed)
• It does not incur any extra cost
• It allows to evaluate each raw MT segment by assigning it a matching percentage.
• It allows to calculate a cost reduction for any translation job/task.
• “Target” world (instead of “Source“ world)
• Post-editing :
• Over-editing translation could be a risk/temptation
• Definition of quality level and corresponding post-editing effort (e.g. light PE)
• Provide guidelines for linguists performing the MT post-editing

7. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Detailed ETMA steps
• Select an ETMA LW MT job that is already completed
• Categorize it (Lang pair/Volume/Content-type/Format/Quality of the source)
• Isolate source segments that went through LW MT (“New word” category)
• Process these segments through LW MT engine only (*) - Download corresponding “raw
MT” bilingual file - Reverse it.
• Process these segments through ETMA updated TM (*) - Download corresponding
“post-edited” bilingual file.
• Create a reverse “post-edited” TM using reverse “post-edited” bilingual file
• Generate the leverage analysis of the reverse “raw MT” bilingual file against the reverse
“post-edited” TM

8. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Illustration

9. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Illustration

10. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Illustration

11. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Calibrating cost model
• The accurate (and unique?) method for evaluating translation costs reduction from
PEMT is Productivity Gain evaluation.
• Productivity Gain evaluations are usually not easy to reproduce, take time and money (a
few thousands $/lang).
• So, the idea is to use the results of a single Productivity Gain evaluation to calibrate the
reverse analysis cost model. And then repeat as many time as required the reverse
analysis and associate its results to the calibrated cot model to quantify translation costs
reduction.

12. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12