1. Text
2009.4
hongqn@douban.com

2. • 2005 3
•
•
•

3. • 2.8M 1/4
•
• 20M / 500~600/sec
• 23 PC (1U*15/2U*8)
• 12
• 38G memcached

4. • 1U (frodo)
• AMD Athlon 64 1.8GHz
• 1G 160G SATA*2
• Gentoo Linux
• MySQL 5
• Quixote (a Python web framework)
• Lighttpd + SCGI (shire)
• Memcached (!)

5. Internet
Lighttpd
SCGI
App
FS
MySQL Memcache Static Files

6. Gentoo Linux
•
• emerge mysql
• ebuild patch
•
•
• GLSA(Gentoo Linux Security Advisories)

7. MySQL
• The world’s most popular open source database
• / ==> MyISAM
• ==> InnoDB
• Replicate for backup

8. Python
•
• Battery Included
•
•
• CPUG: http://python.cn/

9. Quixote
• REST URL
• Django, TurboGears, Pylons
ZOPE
• http://www.douban.com/subject/1000001
# luz/subject/__init__.py
def _q_lookup(request, name):
subject = get_subject(name)
return lambda req: subject_ui(req, subject)
# luz/subject/subject_ui.ptl
def subject_ui [html] (request, subject):
site_header(request)
“<h1>%s</h1>” % subject.title
site_footer(request)

10. Lighttpd
•
• SCGI
• SCGI: FastCGI
Quixote
• 80 lighttpd
SCGI localhost
Quixote

11. Memcache
• MySQL
• libmemcache python Pyrex
python 3x+
def get_subject(subject_id):
subject = mc.get(‘s:’+subject_id)
if subject is None:
store.farm.execute(“select xxx, xxx from subject where id=%s”,
subject_id)
subject = Subject(*store.farm.fetchone())
mc.set(‘s:’+subject_id, subject)
return subject

12. • 1.2M /
• IO
•

13. • 1U
• pippin meriadoc ( merry)
• , 4G 250G SATA*3
•
• IP DNS
IP -_-b
• frodo
(subversion, trac, etc...)

14. Internet
Lighttpd (#$) DNS
HTTP
SCGI Proxy
App Lighttpd (!")
FS
Static Files Memcache MySQL

15. •
• innodb_buffer_pool_size
•
• IP

16. • 1.5M /
•
• IP

17. • IP(BGP)
• (arwen)
• 74G 1w SATA * 3
•
•

18. Internet
Data
Lighttpd Mining
SCGI write read
App Replicate
MySQL MySQL
Master Slave
Static Files Memcache

19. • 2M /
• IO
•
, etc...
•

20. • 4G 250G SATA*3
• 10000
• mod_rewrite URL
• lighttpd mod_memcache
• IO
• web
•
•

21. Internet
store.farm
App
SCGI MySQL
Master
Lighttpd Memcache Replicate
store.farmr
HTTP Proxy
WebDAV Web Service
Replicate
Lighttpd
(w/ mod_memcache)
MySQL write
Spiders Slave
Lighttpd
WebDAV
Memcache
Static Files
!"#$%
read
Data
Mining
MySQL
Slave

22. • store farmr
• replicate delay
•
•
• cache
•
•
• ...... but it works

23. replicate delay
def get_subject(sid):
sbj = mc.get(‘s:’+sid)
if sbj is None:
sbj = flush_subject(sid, store.farmr)
return sbj
def flush_subject(sid, cursor=None):
cursor = cursor or store.farm
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
def update_subject(subject, props):
store.farm.execute(“update subject ...”)
store.farm.connection.commit()
flush_subject(subject.id, store.farm)

24. • 2.5M /
•
• /
• SATA
•

25. • Scale Up 1U
• 16G 147G SCSI *2 + 500G SATA
• SCSI RAID-0
• MySQL Slave
• memcached
• MyISAM InnoDB
•
• Sphinx

26. Internet
Sphinx
MySQL
Web Service store.farm Master
Replicate
Lighttpd
SCGI
HTTP Proxy App
Memcache
Lighttpd
(w/ mod_memcache) WebDAV
store.farmr
Memcache
Lighttpd
Web Service
WebDAV
Static Files MySQL
Slave
Spiders
Memcache
Memcache

27. • 5.2M /
•
• Web IO
•
•

28. • :)
•
•
•
• 3 1U 4 32G 1T SATA * 3
• otho.douban.com lotho.douban.com
• lighttpd 1.5 with aio support
• LVS
• Scale Up: 4G -> 8G

29. Internet
www.douban.com otho.douban.com
LVS LB
(Master)
Lighttpd
Lighttpd 1.5
(w/ mod_cache)
HTTP Proxy
Lighttpd Keepalived
LVS LB
(backup)
Lighttpd
WebDAV Lighttpd 1.5
(w/ mod_cache)
Static Files

30. write
replicate !"#$%
read Data
Mining
MySQL MySQL
Master Slave
Replicate read
write
!"#$%
Data
Mining
MySQL
Slave

31. • 6.4M / (5M PV)
•
•
• CPU memcache /

32. •
• lighttpd mod_scgi round-robin
• lighttpd 1.5
• mod_proxy
• proxy.balance = fair (load based, passive
balancing)
•
• spread

33. Internet
Lighttpd
SCGI
HTTP Proxy
App Memcache
Lighttpd
HTTP Proxy
spread
HTTP Proxy
Lighttpd
Lighttpd
Log
Aggregator SCGI
spread
Lighttpd
WebDAV App Memcache
Static Files

34. • 11M / 3
•
• Sphinx
•
•
load

35. •
•
•
• Sphinx -> Xapian
• MogileFS

36. • libmemcache -> libmemcached consistent
hash memcache
• libmemcached consistent hash bug
• CPU
• libmemcached failover bug
• nginx lighttpd load balance
• spread
• nginx

37. !"
Master
replicate
replicate
%&'() !"
Slave
#$
Master read
replicate
#$ read
Data
Slave Mining
write
!" *+,-
Slave Master
replicate
*+,-
Slave write
replicate
%&'()Data
Mining
#$
Slave

38. •
• store.farm[r] -> store.get_cursor(table=‘xxx’,
ro=True/False)
def flush_subject(sid, ro=False):
cursor = store.get_cursor(table=‘subject’, ro=ro)
cursor.execute(“select ... from subject”)
subject = Subject(*cursor.fetchone())
mc.set(‘s:’+sid, subject)
return subject
•

39. Internet
upload.douban.com otho.douban.com
signed POST form Lighttpd 1.5
Uploader
(w/ mod_cache)
HTTP redirect
FileStorage
MogileFS Tracker
Gateway
App
MogileFS MogileFS
Node Node
MogileFS
Master

40. Internet
Lighttpd SCGI App
Lighttpd HTTP Proxy
HTTP Proxy
HTTP Proxy
Lighttpd SCGI App
HTTP Proxy
Nginx
Lighttpd
HTTP Proxy
Lighttpd Lighttpd SCGI App
WebDAV
Static Files

41. • 13M /
• MogileFS
• Tracker DB
•

42. • 8
• 32G CPU
• (300G SCSI×2 + 1T SATA) × 3
• (1T SATA × 3) × 5
•6 2
• DoubanFS

43. DoubanFS
• hash
• hash hash
• Merkle Tree
• consistent hash
• WebDAV
• MogileFS 3 50

44. Consistent Hash

45. Merkle Tree

46. Internet
upload.douban.com otho.douban.com
signed POST form Lighttpd 1.5
Uploader
(w/ mod_cache)
HTTP redirect
FileStorage
Gateway
App
DoubanFS DoubanFS
Node Node

47. • 16M /
•
• DoubanFS IO
•

48. • DoubanDB
•
• MySQL
• MySQL Master
• failover
• replicate delay

49. DoubanDB
• Key-Value
• Amazon Dynamo
• set(key, value), get(key), delete(key)
• memcache
• Merkle Tree
• Consistent Hash
• TokyoCabinet
• DoubanDB
• DoubanFS 2.0 DoubanDB

51. !"
Master1
replicate
replicate
%&'() !"
Slave
#$
Master1 read
replicate
#$ read
Data
Slave Mining
write
!" *+,-
Master2 Master
replicate
*+,-
Slave write
replicate
%&'()Data
Mining
#$
Master2

52. • DoubanFS
•
•
• ngnix
• www.douban.com LVS
• RabbitMQ spread

53. •
• profile
• memcache cache
• join
•
•