Behind the scenes of a grown-up web application - YAPC::EU 2014, Sofia, Bulgaria

1. Behind the
scenes of a
grown-up web
application
Kerstin Puschke
YAPC::EU 2014

2. Who am I
Contact
• http://www.kpuschke.eu
• twitter: @titanoboa42
• https://www.xing.com/profile/
Kerstin_Puschke
software engineer
at XING Hamburg
• xing.com
social network for business
professionals
• about 14M users
• debian, perl, javascript, ruby on
rails, mysql, redis, riak, . . .
We’re hiring!
http://corporate.xing.com/
english/company/careers-at-xing/
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 2

3. Outline
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 3
• Technology & Architecture
• programming languages, databases, storage,...
• applications, APIs & asynchronous communication
• Development Process and Tools
• development and test environments
• version control
• continous integration, deployment

4. How it all began
10 years ago - Perl & MySQL
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 4

5. The camel has been growing
Today - Perl & MySQL & more
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 5

6. xing.com
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 6
perl
rails
rails
rails
mobile
...

7. Backend
• Perl
• Ruby on Rails
• Scala
• Hadoop
• Elastic Search
• Insect / Arachnid
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 7

8. Frontend
• HTML
• Javascript
• jquery
• backbone
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 8

9. Storage, Databases & Cache
• MySQL
• Redis
• Memcache
• Swift / Open Stack
• Riak
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 9

10. Perl „core“
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 10
perl
„core“
rails
rails
rails
mobile
...

11. Rails
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 11
perl
„core“
rails
jobs
rails
...
rails
events
mobile
rails
...
rails
~ 20 apps

12. Communication
• REST API
• RabbitMQ AMQP
• plus Beetle
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 12
perl
„core“
rails
jobs
rails
public API
rails
events
mobile
rails
...

13. The camel has been growing...
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 13

14. Dissolving Profile
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 14
perl
„core“
rails
rails
(public API)
rails
mobile
rails profile
frontend

15. Shadow calls
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 15
old app
new app user request
worker amqp message
shadow call

16. Still at work
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 16

17. Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 17
• ~600k lines of code, ~300k lines of perl code

18. Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 18
• ~30M external requests / day

19. Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 19
• ~360M internal REST requests / day, 6k / second

20. Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 20
DC1
DC2
2 x 6
application server
2 x 18
API Server
2 x 4
worker server

21. Taking care
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 21

22. Office Locations
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 22
Hamburg, Munich, Vienna, Barcelona
~520 people from ~28 countries, ~80 engineers

23. Provisioning Development VMs
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 23
• Vagrant & Chef

24. Version Control
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 24
• Github >1800 repositories

25. Continuous Integration & Deployment
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 25
• Jenkins CI ~30 projects, ~250 build jobs
• weekly to continuous deployment

26. Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 26
• logjam

27. Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 27
• graphite

28. Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 28
• cube

29. famous last words
die "WTF?"; # should not be possible
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 29

30. The professional network
www.xing.com
Thank you
for your
attention!

31. Who am I
Contact
• http://www.kpuschke.eu
• twitter: @titanoboa42
• https://www.xing.com/profile/
Kerstin_Puschke
software engineer
at XING Hamburg
• xing.com
social network for business
professionals
• about 14M users
• debian, perl, javascript, ruby on
rails, mysql, redis, riak, . . .
We’re hiring!
http://corporate.xing.com/
english/company/careers-at-xing/
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 31

32. Credits
Map
• The map on slide 21 was created as a screenshot from
http://umap.openstreetmap.fr
• Map background credits
Outdoors (OSM) Tiles © Gravitystorm / map data OpenStreetMap
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 32