[2019] Class-based N-gram Models of Natural Language
•
1 recomendación•378 vistas
This document discusses class-based n-gram models of natural language. It outlines that n-gram models predict the probability of a word based on the previous n-1 words, and that increasing n improves accuracy but decreases reliability due to more parameters needing estimation. It also discusses using word classes in n-gram models by grouping words with similar contexts together, and finding "sticky pairs" of words that occur near each other more than expected based on their individual frequencies.
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a [2019] Class-based N-gram Models of Natural Language
Similar a [2019] Class-based N-gram Models of Natural Language (15)
Más de Jinho Choi
Más de Jinho Choi (20)
Último
Último (20)
[2019] Class-based N-gram Models of Natural Language
- 1. Class-Based n-gram Models of Natural Language Victoria Lawlor 1/23/2019 Brown, P. F., Desouza, P. V., Mercer, R. L., Pietra, V. J. D., & Lai, J. C. (1992).
- 2. Problem “In a number of natural language processing tasks, we face the problem of recovering a string of English words after it has been garbled by passage through a noisy channel.”
- 3. Outline 1. Language Models 2. Word Classes 3. Sticky Pairs and Semantic Classes
- 4. Language Models Evaluating language models: perplexity • Perplexity is a measurement of how well a probability distribution or probability model predicts a sample • A low perplexity indicates the probability distribution is good at predicting the sample
- 5. Language Models N-gram models—predict the probability of a word based on the words before Predict wn given the previous n – 1 words: P(wn|w1w2...wn – 1)
- 6. Language Models Estimate N-gram probabilities: Number of times the given n – 1 words precedes wn. Sum of all the n- grams that start with the given n – 1 words.
- 7. Language Models Increasing n • Increases the accuracy of the model • Decreases reliability • Increases the number of parameters to estimate Solution: Interpolation
- 9. Word Classes N-gram class model:
- 10. Word Classes N-gram class model:
- 11. Word Classes Monday Tuesday Wednesday Thursday Friday Let’s meet on______ Are you free to hang out ______ Tomorrow is ______ I’ll be there ______ You have until ______ Assigning words to classes:
- 12. Word Classes Assigning words to classes—mutual information:
- 13. Word Classes Assigning words to classes (greedy): • Put each word into its own class • Select desired number of classes, C • Merge the pair of classes for which the loss in average mutual information is least, until the number of classes = C • Iterate through each word and move it to a different class if it will lead to higher mutual information
- 14. Word Classes
- 15. Sticky Pairs Mutual information of w1 and w2: If w2 follows w1 more often than we expect, then the mutual information is positive, and the pair is “sticky”.
- 16. Sticky Pairs Word Pair Mutual Information Humpy Dumpty 22.5 Klux Klan 22.2 Avant garde 22.1 Taj Mahal 21.8 Mumbo jumbo 21.4 Helter skelter 21.4
- 17. Semantic Stickiness We can seek pairs of words that simply occur near one another more than we would expect.
- 18. Semantic Stickiness We can seek pairs of words that simply occur near one another more than we would expect. Prnear(w1w2) > Pr(w1) Pr(w2) (much)
- 19. Semantic Stickiness large size smaller small larger attorney counsel trial court judge morning noon evening night nights midnight bed school classroom teaching grade math letter addressed enclosed letters correspondence performance performed perform performs performing
- 20. Thanks!