Unicode is ubiquitous — nearly all modern computers use it to store and transmit text — and yet most developer know very little or get panicked when they see the message "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92". Panic no more! Eddie will give the introduction that you never got about Unicode, and give best practices and the knowledge you need to handle text from all around the world!

1. Introduction to Unicode
(that you never got)
Eddie Antonio Santos
eddieantonio.ca

3. â€
œ
Dia duitâ€
•
, said RoÃsÃn

5. Eddie Antonio Santos
PhD Candidate
Assistant lecturer,
software localisation
Software developer
Indigenous languages tech

7. Beliefs about Unicode

8. Beliefs about Unicode
Unicode is a character
encoding
False

9. Beliefs about Unicode
My app is only in English, so I
don't need to worry about
Unicode
False

10. Beliefs about Unicode
“Isn't Unicode that emoji
company?”
Yes…

11. Beliefs about Unicode
Unicode is frustrating
It can be…
but I'm here to help!

12. What even
is

13. • An international
standard for the
transmission and
storage of text on
computers
• A consortium
• a database of
characters and their
properties

14. database of characters

15. A € ৪ ‽
Letters Symbols Digits Punctuation

16. A 🤔 ৪ ‽
Letters Emoji Digits Punctuation

17. A € ৪ ‽
Letters Symbols Digits Punctuation

18. A € ৪ ‽
U+0041 U+20AC U+09EA U+203D
code points
assigns numbers to characters

19. A € ৪ ‽
U+0041 U+20AC U+09EA U+203D
code points
a string is a sequence of code points

20. = U+0041
€ = U+20AC
৪ = U+09EA
‽ = U+203D
A code point is like a
database primary key
into Unicode
Character Database.
Every code point has
properties.
A

21. Name
A
U+0041
€
U+20AC
৪
U+09EA
‽
U+203D
LATIN CAPITAL LETTER A
BENGALI DIGIT FOUR
EURO SIGN
INTERROBANG

22. General category
A
U+0041
LATIN CAPITAL LETTER A
€
U+20AC
EURO SIGN
৪
U+09EA
BENGALI DIGIT FOUR
‽
U+203D
INTERROBANG
Letter, uppercase
Digit, decimal
Symbol, currency
Punctuation, other

23. General category
A
U+0041
LATIN CAPITAL LETTER A
a
U+0061
LATIN SMALL LETTER A
‫ا‬
U+0627
ARABIC LETTER ALEF
あ
U+3042
HIRAGANA LETTER A
Letter, uppercase
Letter, other
Letter, lowercase
Letter, other

24. Bidirectionality Class
A
U+0041
LATIN CAPITAL LETTER A
€
U+20AC
EURO SIGN
‫ا‬
U+0627
ARABIC LETTER ALEF
😳
U+1F633
FLUSHED FACE

25. Bidirectionality Class
A
U+0041
LATIN CAPITAL LETTER A
€
U+20AC
EURO SIGN
‫ا‬
‫لف‬
U+0627
ARABIC LETTER ALEF
😳
U+1F633
FLUSHED FACE

26. Bidirectionality Class
A
U+0041
LATIN CAPITAL LETTER A
€
U+20AC
EURO SIGN
‫ا‬
U+0627
ARABIC LETTER ALEF
😳
U+1F633
FLUSHED FACE
Strong left-to-right
Strong right-to-left
Weak
Neutral

27. How this
works in

29. (demo)

30. • Unicode assigns code
points to characters
• code points != bytes

31. Bytes
Code points
Encoding

32. character encodings

33. A (very) brief history of character
encodings
before computers
• Morse code (~1837)
• Baudot code (1870)
• BCDIC (1928)
computers invented
• EBCDIC (1963)
• ASCII (1967)
• ISO 8859 (1987)
• Latin-1, Latin-2, Windows
CP1252
Cze¶æ, jestem
Stanis³aw

34. UCS-2
Universal character set
(1991)
🥴🙄😬

35. UTF-8
👑😻💅🏼✨
Unicode Transmission Format, 8 bit
(~1993)

36. UTF-8
• backwards
compatible with
ASCII
• supports all
languages

37. society post utf-8

38. â€
œ
Dia duitâ€
•
,
said RoÃsÃn
Cze¶æ, jestem
Stanis³aw
LÃ&
AMP;SUP3;PEZ
UnicodeDecodeError: 'utf-8'
codec can't decode byte 0x89 in
position 1: invalid start byte

39. mojibak
e
gãrblëd têxt, due to mixed character encodings

40. mojibak
e
garbled text, due to mixed character encodings

42. UnicodeDecodeError: 'utf-8' codec
can't decode byte 0x89 in position
1: invalid start byte

44. (demo)

45. recommendations

46. Always explicitly specify encoding

47. Always use UTF-8

48. Put a 💩 in your test suite
• most emoji use part of Unicode that is poorly supported
• insert a test that encodes a string with an emoji and
decodes it

49. Use your head
• chardet and ftfy are great tools, but they can sometimes be
wrong!

50. (demo)

51. ASK ME
QUESTIONS!