In this slide, we will discuss Rule-based, Stochastic & Hybrid Taggers.

1. Word Level Analysis
POS (Continued)
L3 NLP (Elective)
1
TJS

2. Words and Word Classes
 Words are classified into categories called
part of speech (word classes or lexical
categories)
2

3. Part of Speech
NN noun student, chair, proof, mechanism
VB verb study, increase, produce
ADJ adj large, high, tall, few,
JJ adverb carefully slowly, uniformly
3
JJ adverb carefully slowly, uniformly
IN preposition in, on, to, of
PRP pronoun I, me, they
DET determiner the, a, an, this, those
 open vs. closed word classes

4. Part of Speech tagging
 process of assigning a part of speech like noun,
verb, pronoun, preposition, adverb, adjective,
etc. to each word in a sentence
4
POS tagger
Words
+
tag set
POS
tag

5. Speech/NN sounds/NNS were/VBD sampled/VBN
by/IN a/DT microphone/NN.
Another tagging possible for the sentence is:
5
Speech/NN sounds/VBZ were/VBD sampled/VBN
by/IN a/DT microphone/NN.

6. Part of speech tagging
methods
 Rule-based (linguistic)
 Stochastic (Data-driven) and
 TBL (Transformation Based Learning)
6
 TBL (Transformation Based Learning)

7. Rule-based (linguistic)
Steps:
1. Dictionary lookup  potential tags
2. Hand-coded Rules
The show must go on.
7
The show must go on.
Step 1  NN, VB
Step 2  discard incorrect tag
Rule: IF preceding word is determiner THEN
eliminate VB tag.

8.  Morphological information
IF word ends in –ing and preceding word is a verb
THEN label it a verb (VB).
8
 Capitalization information

9. +
 Speed
 Deterministic
9
 Deterministic
-
 requires manual work
 usable for only one language

10. Stochastic Tagger
 The standard stochastic tagger algorithm is the
Hidden Markov Model (HMM) tagger.
 A Markov model applies the simplifying
assumption that the probability of a chain of
assumption that the probability of a chain of
symbols can be approximated in terms of its
parts or n-grams.
 The simplest n-gram model is the unigram
model, which assigns the most likely tag (part-of-
speech) to each token.
10

11.  The unigram model requires tagged data to
gather most likely statistics. The context used by
the unigram tagger is the text of the word itself.
For example, it will assign the tag JJ for each
occurrence of fast if fast is used as an adjective
more frequently than it is used as a noun, verb,
or adverb.
or adverb.
 She had a fast
 Muslim fast during Ramadan
 Those who are injured need medical help fast.
 We would expect more accurate predictions if we
took more context into account when making a
tagging decision.
11

12.  A bi-gram tagger uses the current word and the
tag of the previous word in the tagging process.
As the tag sequence “DT NN” is more likely than
the tag sequence “DT JJ”, a bi-gram model will
assign a correct tag to the word fast in sentence
assign a correct tag to the word fast in sentence
(1).
 Similarly, it is more likely that an adverb (rather
than a noun or an adjective) follows a verb.
Hence, in sentence (3), the tag assigned to fast
will be RB (Adverb)
12

13. N- gram Model
 An n-gram model considers the current word
and the tag of the previous n-1 words in
assigning a tag to a word.
Fig. Context used by Tri-gram Model
13

14. HMM Tagger
 Given a sequence of words (sentence), the
objective is to find the most probable tag sequence
for the sentence.
 Let W be the sequence of words:
 Let W be the sequence of words:
W = w1, w2, … , wn
 The task is to find the tag sequence
T = t1, t2, … , tn
which maximizes P(T|W), i.e.,
T’ = argmaxT P(T|W)
14

15.  Applying Bayes Rule, P(T/W) can be
estimated using the expression:
P(T|W) = P(W|T) * P(T) /P(W)
 As the probability of the word sequence,
P(W), remains the same for each tag
sequence, we can drop it. The expression
for the most likely tag sequence becomes:
T’ = argmaxT P(W|T) * P(T)
15

16.  Using the Markov assumption, the probability of
a tag sequence can be estimated as the product
of the probability of its constituent n-grams, i.e.,
P(T) = P(t1) * P(t2|t1) * P(t3|t1t2) … * P(tn|t1 …
 P(T) = P(t1) * P(t2|t1) * P(t3|t1t2) … * P(tn|t1 …
tn-1)
 P(W/T) is the probability of seeing a word
sequence, given a tag sequence
 For ex, what is the probability of seeing ‘The egg
is rotten’ given ‘DT NNP VB JJ’.
16

17.  We make following two assumptions :
1. The words are independent of each other, and
2. The probability of a word is dependent only on
2. The probability of a word is dependent only on
its tag.
Using these assumptions P(W/T) can be expr :
P(W/T) = P(w1/t1) * P(w2/t2) .... P(wi/ti) *
...P(Wn/tn)
i,.e.,
17

18. 18

19. 19

20.  Some of the possible tag sequence:
DT NNP NNP NNP
DT NNP MD VB or DT NNP MD NNP (Output  Most likely)
20

21. 21

22. Brill Tagger: Initial state
 Initial State:
most likely tag
 Transformation:
22
 Transformation:
The text is then passed through an ordered
list of transformations.
Each transformation is a pair of a a rewrite
rule and a contextual condition .

23. Learning Rules
Rules are learned in the following manner
1. each rule, i.e. each possible transformation, is
applied to each matching word-tag-pair.
2. the number of tagging errors is measured
against the correct sequences of the training
corpus ("Truth" ).
23
corpus ("Truth" ).
3. the transformation which yields the greatest
error reduction is chosen.
4. Learning stops when no rules / transformations
can be found that, if applied, reduces errors
beyond some given threshold.

24. • Set of possible ‘transforms’ is infinite, e.g.,
“transform NN to VB if the previous word
was MicrosoftWindoze & word braindead
occurs between 17 and 158 words before
24
occurs between 17 and 158 words before
that”
• To limit: start with small set of abstracted
transforms, or templates

25. Templates used: Change a
to b when…
25

26. Rules learned by TBL tagger
26

27. Lexicalized transformations
Brill complements the rule schemes by so-called
lexicalized rules which refer to particular words in
the condition part of the transformation:
Change a to b if
27
 Change a to b if
1. the preceding (following, current) word is C
2. the preceding (following, current) word is C and
the preceding (following) word is tagged d.
etc.

28. unknown words
 In handling unknown words, a POS-tagger can
adopt the following strategies
 assign all possible tags to the unknown word
 assign the most probable tag to the unknown
word
28
 same distribution as ‘Things seen once’
estimator of ‘things never seen’
 use word features i.e. see how words are
spelled (prefixes, suffixes, word length,
capitalization) to guess a (set of) word
class(es). -- Most powerful

29. Most powerful unknown word
detectors
 32 derivational endings ( -ion,etc.);
 capitalization; hyphenation
 More generally: should use morphological
29
 More generally: should use morphological
analysis! (and some kind of machine learning
approach)