Language models & power laws introductory presentation for 7th meetup of Complexity Explorers Krakow group, cekrk.com Covers some comments on statistical properties of language models, brings Markov model and n-grams. Links book Information Theory Meets Power Laws: Stochastic Processes and Language Models (Wiley), by Łukasz Dębowski with thecurrent understanding of language models and differences between statistical regularities of unigrams/random permutations and natural text.

1. Complexity Explorers Kraków #7
cekrk.com Marcin Stępień @marcinstepien Kraków 2023-02-22
Language models
and power laws
“Industrial City” 1920, by Leon Chwistek
The Queen of Laputa, Gulliver's Travels

2. Complexity
Explorers
Kraków

3. Inspired by Santa Fe Institute
Algorithmic
Information
Theory
Systems
Theory
Computational
Complexity
www.santafe.edu

4. discoveries
models
simulations
experiments
physical/numerical/thought
opinions
statements
influences
>

5. ❏ There is hardly any book that contains a large number of
occurrences of both word lemma and word love.
Yaglom and Yaglom [1983]
* Information Theory Meets Power Laws: Stochastic Processes and Language Models (Wiley), Łukasz Dębowski, 2021
[Ł. Dębowski]*

6. Language model
Evaluate
probabilities on
sequence of
(sub)words or
characters
Read
Text
Generate
Text

7. ... this invention had employed all his
thoughts from his youth; that he had emptied
the whole vocabulary into his frame, and
made the strictest computation of the general
proportion there is in books between the
numbers of particles, nouns, and verbs, and
other parts of speech.
[Jonathan Swift, 1726]
Gulliver’s Travels, Part III
A not so new idea
Les moniteurs de la
conversation á Laputa, 1875
The Engine

8. ❏ Language models based on Markov Chain
❏ Andrey Markov [1906]
❏ “memory-less” - next step in the process is only dependent on the
immediate previous step, or a fixed number of previous step
❏ N-gram
❏ Mathematical Theory of Communications, Claude Shannon [1948]
❏ text generation, names generation, word suggestion, speech
recognition, OCR, protein sequencing
Before Generative Pre-trained Transformer GPT
* GPT is a kind of Markov Process (continuous-time version of Markov Chain)

9. src: www.adamwalanus.pl/2016/chaitin/160519-1804-19.jpg
The Zipf Mystery to 1:30 The Zipf Mystery from 6:52 to 12:30

10. ❏ There is hardly any book that contains a large number of
occurrences of both word lemma and word love.
Yaglom and Yaglom [1983]
[Ł. Dębowski]

11. ❏ Rejecting simple determinism from the process of natural text
generation means that we have to consider some model of
randomness.
[Ł. Dębowski]

12. ❏ Why only a certain well-defined number of texts or sequences
of symbols of a given length gets reproduced in the process of
human culture or in the stream of consciousness?
[Ł. Dębowski]

13. ❏ …Thus we can formally demonstrate that natural language is
NOT a hidden Markov process
[Ł. Dębowski]

14. ❏ …natural texts, i.e., texts written in natural language, exhibit certain
statistical regularities. Some of these regularities are similar to
statistical regularities of unigram texts, i.e., random permutations
of characters, whereas some are strikingly different.
[Ł. Dębowski]

15. ❏ For possible applications in engineering and artificial
intelligence, it is also vital to understand general constructions
of stochastic processes that satisfy these power laws as well as
to comprehend reasons why these laws are satisfied.
[Ł. Dębowski]

16. ❏ similar words are close to each
other in the embedding space
❏ unsupervised - Word2Vec [2013] *
Word Embeddings
0.1 0.137 0.01 … 0.07 0.42
Word
❏ rare words, not in training data;
morphologically rich languages -
FastText [2016]
❏ handles homonyms, typos - ELMo
[2019]
* the morning paper introduction:
blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/
King - Man + Woman = Queen

17. WE use case: semantic change of words over time
source: https://github.com/williamleif/histwords

18. Łukasz Dębowski
Language Models and Power Laws

19. Thank you