Automacon 2016 keynote

1. infrastructure as code might
be literally impossible
part 2
joe damato
packagecloud.io

2. part 1
bit.ly/impossible-infra

3. hi, i’m joe
i like computers
i once had a blog
called timetobleed.com
@joedamato

4. packagecloud.io
@packagecloudio

5. follow along
blog.packagecloud.io

6. hi

7. disclaimer

12. infrastructure as code
might be impossible
because nothing works.

15. cognitive load

16. too much stuff

17. coping strategies

18. coping w cognitive load
copy & paste configs
stackoverflow

19. BTW
This is actually part of another talk I’m working on called
Programmers should
get paid more & work
less

20. anw

21. the problem is so
pronounced, that in some
cases it’s impossible to do
seemingly simple tasks

22. some examples then some
thoughts.

23. Today’s cool stories
1. SSL
2. APT
3. Linux Networking
4. Linux Threading (maybe)
5. Python packaging (maybe)

24. SSL

25. SSL is
important

26. agreed?

27. Ubuntu &
Debian
don’t agree

28. SSL doesn't
work on Debian
/ Ubuntu

31. anw

33. LOL gnutls,
who cares?

34. apt-get!
git!
curl!
ngIRCd!
…

35. well, actually
you should use
OpenSSL

38. I like
rabbits.

39. * 3. All advertising materials
mentioning features or use of this
* software must display the
following…
!
* 6. Redistributions of any form
whatsoever must retain the following…
OpenSSL says…

40. GPL says
6. ….You may not impose any
further restrictions on the recipients'
exercise of the rights granted herein.

41. These two licenses are
not compatible.

42. in other
words

43. software licenses force
you to use a particular
SSL library with a very
painful bug.

44. greetings

45. (not sayin that
OpenSSL is
bug free)

46. (but, am sayin NSS
and gnutls have
less mindshare)

47. btw

52. (hi)

53. OK but I don’t
care about SSL,
I use GPG.

55. NO.!
plz stop.

56. anw

57. APT

58. file
compression
is important

59. agreed?

60. Ubuntu &
Debian
don’t agree

62. (more about hash
sum mismatch
later)

64. in other
words

65. APT bug when
decompressing XZ files
makes it impossible to
install software reliably

66. this is unfortunate due
to the slow release
cycle of Debian/
Ubuntu updates

67. “SO easy, that type
of work can be done
over the weekend”

68. -o Acquire::CompressionTypes::Order::=gz

69. … OK …
hopefully that repo has
gzip’d metadata or it’s
gonna be a real short
trip

70. anw

71. hash sum
mismatch

72. have you
seen it?

73. do you know
what it
means?

74. do you know
why it
happens?

75. what it means

77. happens all
the time…

80. how could
that happen?

81. one of (at least) 3 ways
1. stale cache between client/server
2. XZ decompression bug
3. apt race condition

82. how to avoid each
1. better HTTP headers… or use SSL…. but
like gnutls ?? lol
2. don’t generate XZ archives
3. ?????? race condition ??????

83. APT race

84. how it happens
1. Download + cache Release file
2. repo owner updates repo
3. Download Packages files
4. Compare checksums from the (stale) Release file
against Packages file
5. hash sum mismatch

85. this means…

86. it is impossible to
1. update your repository without
breaking clients
2. generate consistent mirrors of other
repositories

87. !!!!!!
this is bad!
!!!!!

88. but i’ve done all of these
before and never had a
problem?

89. congrats you got
lucky!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

90. so, wait, joe, are
you saying that
APT metadata is
inherently racy?

91. yes!

92. and ubuntu
agrees

94. OK so APT repos
and the tools you use
to generate them are
fundamentally racy

96. so now
what?

97. Acquire-by-hash

98. Acquire-by-hash
• Mechanism for downloading metadata by it’s
hash sum
• Server should keep “a few” older copies of
metadata around
• Prevents the race condition from happening

99. Acquire-by-hash
• Added in APT 1.2.0
• Ubuntu Xenial and newer
• Debian Stretch and newer
• not supported by reprepro!
• not supported by aptly

100. only one way to get
working, consistent, not
racy APT metadata

101. use
packagecloud.io

102. Linux
Networking

103. Full networking writeup
literally 90 pages
literally everything about linux networking
literally available here:
http://bit.ly/linux-networking

104. summary

106. [random os] has a better/faster/
leaner/whatever networking stack
than linux

108. lots and lots
and lots of copy
paste coping

109. question

110. an answer

111. an other answer

112. yet an other answer

113. and on and on
and on and on…

114. no one even
knows what these
values mean

115. (p. much no one
knows what these
values mean)

116. example

117. netdev_max_backlog

122. similarish
explanations

123. what does it
actually mean
tho?

124. If

125. if
• driver calls netif_receive_skb (likely)
• and RPS is disabled (default)

126. Then

127. it doesn't do
anything.

128. literally nothing.
it’s not even
checked.

130. if
• driver calls netif_rx (unlikely)
• or RPS is enabled (rare for most ppl)

131. then data is queued
to a backlog length
limited by
netdev_max_backlog

132. coping
strategies
abound

133. here’s a coping
strategy i think
is fine

135. curl | sudo bash

136. you aren’t reading all
of the chef/puppet
source so what’s the
difference?

137. (hi, be mad)

138. too damn hard to
understand how a
computer works

139. on that
note…

140. Linux
Threading

141. “threads are slow”

142. “context switches
are expensive/
slow/…”

145. a 7 year old bugfix
for XFree86 broke
threads on Linux

147. Story Time
TLS
segment selectors
XFree86 Modules

148. Story Time
mmap
MAP_32BIT

149. June 29, 2001

150. “This adds a new mmap flag to
force mappings into the low
32bit address space.
Useful e.g. for XFree86′s ELF
loader or linuxthreads’ thread
local
data structures.”

151. Nov 11, 2002

152. “532. Fixed module loader
to map memory in the low
32bit address space on
x86-64 (Egbert Eich).”

153. Story Time
ELF small code model
31bit mapping

154. Jan 4, 2003

155. “Make MAP_32BIT for 64bit
processes only map in the first 31bit,
because it is usually used to map
small model code. This fixes the X
server crashes. Some cleanups in
this area.”

156. So: MAP_32BIT is
actually MAP_31BIT

159. Mar 4, 2003

160. /* For Linux/x86-64 we have one
extra requirement: the stack must
be in the first 4GB.
Otherwise the segment register base
address is not wide enough.
*/
glibc

161. May 9, 2003

162. /* We prefer to have the stack
allocated in the low 4GB
since this allows faster
context switches.
*/
glibc

163. justification for
MAP_32BIT in
glibc changed

164. Aug 13, 2008

165. “Pardo” report
https://lkml.org/lkml/2008/8/12/423

166. “Pardo” report

167. “Pardo” report
pardo filled the 31bit 1GB space
with thread stacks.
!
subsequent allocations were doing
a linear search for a free address
on the kernel side.

168. MAP_STACK
is added.

169. (it does nothing)

171. June 29, 2001: MAP_32BIT added to kernel
Nov 11, 2002: XFree86 updated to use MAP_32BIT
time or w/e
Jan 4, 2003: MAP_32BIT updated for ELF small code
Feb 12, 2003: wrmsr slowness reported
Mar 4, 2003: MAP_32BIT added to glibc
May 9, 2003: MAP_32BIT retry added to glibc
Aug 13, 2008:“Pardo” report
Aug 13, 2008: MAP_STACK
Aug 15, 2008: glibc updated

173. a few
questions

174. how did we
get here?
question

175. legacy code
backward compat
an thought

176. free open source
doesn’t exist
an thought

177. why so much
copy-paste
coping?
question

178. necessary
complexity
an thought

179. lack of time
an thought

180. an aside:

181. but, why is
there no time?

182. i don’t know, but
could it be that

183. efficiency gains are
captured by
management instead
of engineering?

184. or could it be
that…

185. working software
systems aren’t
economically viable for
99% of companies?

186. hence why no one
found that threading
bug for 5 years?

187. working software
given complex
requirements is
expensive

188. how much did
you pay for your
an Linux?

189. ?packagecloud.io
@packagecloudio

190. Python Packaging

191. 3 types of python packages
1. source distributions (sdists)
2. eggs
3. wheels

192. some …interesting…
behavior with [-_.]

193. setup(name='hi_automacon',
…
!
!
setup(name=‘hi-automacon',
…
!
!
setup(name=‘hi.automacon',
…

194. what do you
think happens?

195. “There are only two hard
things in Computer
Science: cache
invalidation and naming
things.”

196. (literally unknown)

197. hi_automacon

198. setup.py: hi_automacon
metadata: hi-automacon
sdist: hi_automacon-1.0.tar.gz
egg: hi_automacon-1.0-py2.7.egg
wheel: hi_automacon-1.0-py2-none-any.whl

199. OK SO:
wheels and eggs leave “_” in the filename
but !
translate it in the metadata to “-“
!
…. but not sdists

200. OK OK OK OK
OK OK OK OK

201. thats fine not
a big deal

202. weekend work
and all that

203. hi-automacon

204. setup.py: hi-automacon
metadata: hi-automacon
sdist: hi-automacon-1.0.tar.gz
egg: hi_automacon-1.0-py2.7.egg
wheel: hi_automacon-1.0-py2-none-any.whl

205. OK SO:
wheels and eggs translate “-“ to “_” in
the filename
but !
leave it in the metadata
!
…. but not sdists

207. package name file name metadata
dash underscore dash
underscore underscore dash

209. wheels and eggs only

210. sdists are WYSIWYG affff

211. hi.automacon

212. weird
• everything has ‘.’ in it
• file names and metadata for all
package types

214. let’s curl against PyPI….

215. django-allauth

216. curl
https://pypi.python.org/simple/
django-allauth/

217. HTTP 200

219. OK OK OK OK
OK OK OK OK

220. curl
https://pypi.python.org/simple/
django_allauth/

221. HTTP 302

222. < Location: /simple/
django-allauth

224. OK OK OK OK
OK OK OK OK

225. curl
https://pypi.python.org/simple/
django.allauth/

226. HTTP 200

228. (hi)

229. and now what
happens if we try
mixing the case?

230. lol maybe
next time.