1. Build Your Own
Statistical Machine
Translation Engines
Ruben de la Fuente

2. About Me
• 4-year degree in translation
• Worked as translator for 10+ years
• Working full time in MT for the past
year

3. Agenda
• Quick comparison with RbMT
• Fundamentals of SMT
• Requirements and preparation
• Using DoMY

4. Disclaimer
• I’m not saying SMT is better
• I’m not saying SMT is right for you

5. Statistical Machine Translation
Computer learns to translate through
statistical analysis of alignment in
bilingual corpora

6. Rule-based Machine Translation
User Dictionaries + Grammar and
translation rules

7. SMT: Pros and Cons
Pros Cons
Quick to build Unpredictable
Cheap Quick
Fluent improvements not
easy

8. Features of an SMT system
• Translation Model: table containing
source and target phrases, together
with a probability score (accuracy)
• Language Model: list of sequences of
n-words in target language together
with a probability score (fluency)

9. Language and Translation Models
• LM (fluency) • TM (accuracy)

10. Tokenization and recasing
Breaking up text in Lowercase all words
meaningul units (tokens)
File > file
file? > file ?
file. > file .
File! > file !

11. Requirements: Computing
•4 GB RAM PC needed
•Ubuntu 10.04 64-bit OS
•Virtual Machine OK

12. Requirements: size
MS Translator Hub recommends at least
10k segments
I have gotten good results with 100-200k
segments
Roughly over 1 million words corpus

13. Publicly Available Corpora
• Opus (ECB, EMA, OpenOffice)
• Acquis Communautaire
• Europarl
• Hansard
• Multilingual websites: Bitextor

14. Bitextor is Cunning
www.mywebsite.com/en/overview.html
www.mywebsite.com/es/overview.html
<title>My source text</title>
<title>My target text</title>

15. Requirements: relevance
Data needs to be in-domain

16. Requirements: quality
Garbage in, garbage out
Diagnose your TMs with automated QA
checks (e.g. glossary adherence, length)

17. CheckMate: General

18. CheckMate: Length

19. CheckMate: Terminology

20. Remove Repetitions

21. Remove Markup
Markup brings noise to the learning
process
Click <strong>Send</strong>
Haga clic en <strong>Enviar</strong>

22. Do-Moses-Yourself (DoMY)
Moses: state-of-the-art extensively used
open source SMT toolkit
DoMY: extension of Moses making
installation and configuration easier

23. Online SMT Portals
Cons
letsmt.eu
NDA-compliance
smartmate.co Availability
Speed

24. DoMY (Basics)
Graphs: import-tmx, clean-LM/TM, build
LM/TM, train, translate.
Ini files: configuration (language pairs,
paths for input and output).
Folder structure: always include
superdomain, domain and subdomain

25. Folder structure
corpus graphs

26. Run from terminal
Edit ini Command line

27. Running from GUI

28. Graphs
Graph Function Input Output
Import-tmx Extract data from Raw Corpora/sa
tmx files
Clean-tm Clean data Corpora/sa Corpora/re
ady
Build-lm Prepares training Corpora/re builds
set for LM ady
Build-tm Prepares training Corpora/re builds
set for TM ady
Train Trains MT engine Builds engines
Translate Translates input Translation Translation
files and produces s/in s/out
tmx output

29. Tips for settings
LM: 7-gram
TM: 9-gram
Aligner: Berkeley for distant languages

30. Troubleshoot
Error message in terminal
Log file in graph folder
DoMT QA

31. Is Your Engine Good?
A set is excluded from training to be used
for evaluation (598 segments)
From 0.5 BLEU points, engine is likely to
perform well

32. Keep Improving
Retrain the engine periodically as more
translation corpus become available
Gather feedback on what needs to be
improved

33. Statistical PE
• Keep a corpus of raw vs. PE
• Treat them as separate language pairs
• Run them thru DoMY
• Create raw vs. PE engine
• 2 engines: source > target, raw > PE

34. Questions?
Speak now…
Or reach me at:
www.facebook.com/xlation
www.wordbonds.es
@rubendelafuente
http://www.linkedin.com/in/rubendelafuente