1. from disaster to stability
the scaling challenges of my.opera.com
Surge 2010 – Version 3

3. 1999
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

4. 2001
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

5. 2004
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

6. 2007
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

7. 2009
5,500
Servers
kUsers
2,500
1,640
887
257 205 430
1 10 50
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

8. the current beta

9. the situation
2007

10. crashes every day
too many connections!!!
NFS volume of doom
Team?

11. monitoring

16. many improvements since then
➔ Efficient filesystem cache
➔ "Dogpile effect" AKA stampeding AKA ...
➔ Persistent db + memcached connections
➔ Soft counters
➔ Profiling, profiling, …

17. code profiling
[DML] time=1237308152, user=,
url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,
name=XWA::User, variable=active, type=module,
elapsed=0.068473, host=my.opera.com
[DML] time=1237308152, user=, url=/community/,
name=XWA::User, variable=, type=module,
elapsed=0.015935, host=my.opera.com
[DML] ...

18. top time-intensive modules
XWA::User::Sidebar 2024.919s (27.2%, 0.28 s/call)
XWA::User 1778.445s (23.9%, 0.09 s/call)
XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)
XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)
XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)
XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)
XWA::Skins 180.486s ( 2.4%, 0.49 s/call)
XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)
XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)
XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)
XWA::Albums 93.740s ( 1.3%, 2.04 s/call)
XWA::Journals 92.390s ( 1.2%, 2.37 s/call)

19. many improvements since then
➔ YSlow?
➔ The Expires header is your friend!
➔ Hot MyISAM tables converted to InnoDB
➔ MySQL Master/Master setup
➔ Jet Profiler

21. jet profiler

22. 3
scalability

23. 1. avatars

24. Avatars - 2007
75%
/<user-name>/avatar.pl
/<user-name>/avatar.pl?xscale=8192 (!)

25. Avatars wtf!?
my $sql = DBConnect('master');
my %user = $sql->get(
"SELECT a.blob, a.filename,
FROM avatars a, users u
WHERE u.user=? AND u.id=a.user",
$user);
$req->print( $user{'blob'} );

26. Avatars - reloaded
➔ Export to balanced fs (5 formats)
➔ Zero SQL queries
➔ Storage subsystem
➔ static.myopera.com was born

27. resources
(user uploads, binary blobs, ...)
Pools
or single servers
URLs
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.png
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpg
http://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

28. + x
➔ Load ➔ HTTP::DAV
➔ Flexibility ➔ Precomp URLs
➔ Static scales!

29. 2. varnish

31. Varnish
Most popular RSS feeds
My Opera frontpage
Opera Mini approval
Datacenter emergencies

32. Varnish
Most popular RSS feeds
➔ /desktopteam/blog/
➔ Friends, Groups API
➔ No cookies (remove req.http.cookie)

33. Varnish
My Opera frontpage
➔ Danger, Will Robinson!
➔ Mangle cookies
➔ Accept-Language headers

34. Varnish
Opera Mini 5.0 approval
➔ Global coverage
➔ Traffic surge (5x peak, 2x over 24h)

36. IT NEEDS
TO BE OUT
TOMORROW
!!!
THERE
WILL BE A
PRESS
RELEASE !

37. Varnish
Opera Mini 5.0 approval
➔ Global coverage
➔ Traffic surge (5x peak, 2x over 24h)
➔ No problems!

38. Opera Mini “countup” traffic
Submitted Approved
to Apple Store April, 12th
March, 23rd

39. Varnish
Datacenter emergencies

40. Datacenter emergencies
files.myopera.com
DC1
User Files Storage SAN

41. Datacenter emergencies
files.myopera.com DC2
LVS + Varnish servers
DC1
User Files Storage SAN

42. ~ 1Gbit/s! Varnish

43. + x
➔ Load ➔ Chainsaw!
➔ Flexibility ➔ Purging
➔ Instant scaling

44. 3. geodns

45. geodns

46. + x
➔ Prototype 1 week ➔ Accuracy
➔ Geo-scaling ➔ No DC feedback
➔ Redundant ➔ Monitoring

47. Next steps
➔ Search (Solr?)
➔ Batch activity feed
➔ Real connection pooling
➔ … and on ...

48. Remember!
➔ Team spirit is important
➔ Another level of indirection...
➔ Keep it simple
➔ Keep a log

49. the heroes
http://my.opera.com/devblog/about/
http://my.opera.com/devblog/

50. any questions?
?

51. handout download:
http://tinyurl.com/surge2010-cosimo
thanks!