Regexp class is in every Rubyist's toolbox. But do you know the theory behind it, and what goes on under the hood?

1. Secrets of Regexp
Hiro Asari
Red Hat, Inc.

2. Let's Talk About
Regular Expressions

3. Let's Talk About
Regular Expressions
• There is no regular expression

4. Let's Talk About
Regular Expressions
• A good approximation as a name

5. Let's Talk About
Regexp

6. Some people, when confronted
with a problem, think, "I know,
I'll use regular expressions."
Now they have two problems.
Jaime Zawinski
12 Aug, 1997
http://regex.info/blog/2006-09-15/247
http://www.codinghorror.com/blog/2008/06/regular-expressions-now-you-have-two-
problems.html
The point is not so much the evils of regular expressions, but the evils of overuse of it.

7. Formal Language
Theory
• The Language L
• Over Alphabet Σ

8. Formal Language
Theory
• Alphabet Σ={a, b, c, d, e, …, z, λ} (example)

9. Formal Language
Theory
• Alphabet Σ={a, b, c, d, e, …, z, λ} (example)
• Words over Σ: "a", "b", "ab", "aequafdhfad"

10. Formal Language
Theory
• Alphabet Σ={a, b, c, d, e, …, z, λ} (example)
• Words over Σ: "a", "b", "ab", "aequafdhfad"
• Σ*: The set of all words over Σ

11. Formal Language
over Σ
• A subset L of Σ* (with various properties)
• L can be finite, and enumerate well-formed
words, but often infinite

12. Example
• Language L over Σ = {a,b}
• 'a' is a word
• a word may be obtained by appending 'ab'
to an existing word
• only words thus formed are legal

13. Well-formed words
a
aab
aabab

14. Ill-formed words
b
aaaab
abb

15. Succinctly…
• a(ab)*

16. Expression
• Textual representation of the formal
language against which an input is tested
whether it is a well-formed word in that
language

17. Regular Languages
• ∅ (empty language) is regular

18. Regular Languages
• ∅ (empty language) is regular
• For each a ∈ Σ (a belongs to Σ), the
singleton language {a} is a regular language.

19. Regular Languages
• ∅ (empty language) is regular
• For each a ∈ Σ (a belongs to Σ), the
singleton language {a} is a regular language.
• If A and B are regular languages, then A ∪ B
(union), A•B (concatenation), and A*
(Kleene star) are regular languages

20. Regular Languages
• ∅ (empty language) is regular
• For each a ∈ Σ (a belongs to Σ), the
singleton language {a} is a regular language.
• If A and B are regular languages, then A ∪ B
(union), A•B (concatenation), and A*
(Kleene star) are regular languages
• No other languages over Σ are regular.

21. Regular Expressions
• Expressions of regular languages

22. Regular Expressions
ot
• Expressions of regular languages
N

23. Regular? Expressions
• It turns out that some expressions are
more powerful and expresses non-regular
languages
• Language of 'squares': (.*)1
• a, aa, aaaa, WikiWiki

24. How does Regexp
work?
• Build a finite state automaton representing
a given regular expression
• Feed the String to the regular expression
and see if the match succeeds

25. a
a

26. ab*
a
b

27. .*
.

28. a$
a $

29. a?
a
ε

30. a|b
a
b

31. (ab|c)
a b
c

32. (ab+|c)
b
a b
c

33. Match is attempted at
every character, left to
right

34. /a$/
zyxwvutsrqponmlkjihgfedcba
^
Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward
to the end of the line

35. /a$/
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward
to the end of the line

36. /a$/
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward
to the end of the line

37. /a$/
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward
to the end of the line

38. /a$/
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
zyxwvutsrqponmlkjihgfedcba
^
⋮
zyxwvutsrqponmlkjihgfedcba
^
Regexp does not think, 'a$' can match only at the end of the line, so we should fast forward
to the end of the line

39. ^s*(.*)s*$
abc d a dfadg
^
abc d a dfadg
^
abc d a dfadg
^
abc d a dfadg
^
# matches 'abc d a dfadg '

40. a?a?a?…a?aaa…a
def pathological(n=5)
Regexp.new('a?' * n + 'a' * n)
end
1.upto(40) do |n|
print n, ": "
print Time.now, "n" if 'a'*n =~ pathological(n)
end

41. a?a?a?aaa
aaa
^

42. Regexp tips

43. Use /x
UP_TO_256 = /b(?:25[0-5] # 250-255
|2[0-4][0-9] # 200-249
|1[0-9][0-9] # 100-199
|[1-9][0-9] # 2-digit numbers
|[0-9]) # single-digit numbers
b/x
IPV4_ADDRESS = /#{UP_TO_256}(?:.#{UP_TO_256}){3}/

44. A, z for strings
^, $ for lines
• A: the beginning of the string
• z: the end of the string
• ^: after n
• $: before n

45. A, z for strings
^, $ for lines
• A: the beginning of the string
• z: the end of the string
• ^: after n
• $: before n always in Ruby

46. What's the problem?
also note the difference in what /m means

47. What's the problem?
#! /usr/bin/env perl
$a = "abcndef";
if ($a =~ /^d/) {
print "yesn";
}
if ($a =~ /^d/m) {
print "yes nown";
}
# prints 'yes now'
also note the difference in what /m means

48. What's the problem?
#! /usr/bin/env ruby
a = "abcndef";
if (a =~ /^d/)
p "yes"
end
http://guides.rubyonrails.org/security.html#regular-expressions

49. Security Implications
class File < ActiveRecord::Base
validates :name, :format => /^[w.-+]+$/
end
http://guides.rubyonrails.org/security.html#regular-expressions

50. file.txt%0A<script>alert(‘hello’)</script>

51. file.txt%0A<script>alert(‘hello’)</script>

52. file.txtn<script>alert(‘hello’)</script>

53. file.txtn<script>alert(‘hello’)</script>
/^[w.-+]+$/

54. file.txtn<script>alert(‘hello’)</script>
/^[w.-+]+$/
Match succeeds
ActiveRecord validation succeeds

55. file.txtn<script>alert(‘hello’)</script>
/A[w.-+]+z/

56. file.txtn<script>alert(‘hello’)</script>
/A[w.-+]+z/
Match fails
ActiveRecord validation fails

57. Prefer Character Class
to Alterations
require 'benchmark'
# simple benchmark for alternations and character class
n = 5_000
str = 'cafebabedeadbeef'*5_000
Benchmark.bmbm do |x|
x.report('alternation') do
str =~ /^(a|b|c|d|e|f)+$/
end
x.report('character class') do
str =~ /^[a-f]+$/
end
end

58. Benchmarks
Ruby 1.8.7
user system total real
alternation 0.030000 0.010000 0.040000 ( 0.036702)
character class 0.000000 0.000000 0.000000 ( 0.004704)
Ruby 2.0.0
user system total real
alternation 0.020000 0.010000 0.030000 ( 0.023139)
character class 0.000000 0.000000 0.000000 ( 0.009641)
JRuby 1.7.4.dev
user system total real
alternation 0.030000 0.000000 0.030000 ( 0.021000)
character class 0.010000 0.000000 0.010000 ( 0.007000)

59. Beware of Character
Classes
# case-insensitively match any non-word character…
# one is unlike the others
'r' =~ /(?i:[W])/
's' =~ /(?i:[W])/ matches, even if 's' is a word character
't' =~ /(?i:[W])/
https://bugs.ruby-lang.org/issues/4044

60. /^1?$|^(11+?)1+$/

61. /^1?$|^(11+?)1+$/
Matches '1' or ''

62. /^1?$|^(11+?)1+$/
Non-greedily match 2 or more 1's

63. /^1?$|^(11+?)1+$/
1 or more additional times

64. /^1?$|^(11+?)1+$/
matches a composite number

65. /^1?$|^(11+?)1+$/
Matches a string of 1's if and only
if there are a non-prime # of 1's

66. Integer#prime?
class Integer
def prime?
"1" * self !~ /^1?$|^(11+?)1+$/
end
end
No performance guarantee
Attributed a Perl hacker Abigail

67. • @hiro_asari
• Github: BanzaiMan