The Tuenti architecture is designed to handle over 1500 million pageviews per day. Some key stats about Tuenti's scale include over 40,000 million pageviews and 50,000 million requests per month from 13 million active users. The architecture includes over 1200 servers split across frontends, databases, memcache servers, and other services.
To scale effectively, Tuenti focuses on monitoring systems to detect bottlenecks and issues. They also thoroughly test and understand the tools they use like MySQL, memcache, and NoSQL databases. The architecture evolves through iterations as needs change - responsibility may shift from databases to edges or services can be split. Learning from failures and others is also important to scaling.
The Tuenti backend
1. The Web Scale
Tuenti architecture to withstand
1500+ million pageviews / day
Guillermo Pérez - bisho@tuenti.com
Security & Backend Architecture Tech Lead
7. Tuenti Stats
13M users
REALLY ACTIVE
50%+ active weekly
>1h browsing per DAY!
8. Tuenti Stats
- Each month, over:
40,000 M pageviews
50,000 M requests
100 M new photos
2,000+ Tb served photos
- On peaks:
1,600 million pageviews/day
35,000 requests/second
6,000 million served photos/day
13. Monitoring
- Your crystal ball!
Glimpse of the future
Answer questions
- Detect bottlenecks
- Detect what needs to be optimized
The 90/10 Rule
No premature optimization
- Detect bad usages
- Detect browser patterns
- Detect changes, issues
18. Know your tools
- Stop reading blogs
- Read internals documentation
- Test software
- Test hardware
- Experiment
19. Know your tools
- Mysql (innoDB) IS fast
photos table (photo_id, user_id, ...)
PK photo_id, KEY user_id
PK user_id, photo_id, KEY photo_id
Usage: select * from photos where user=X
sorting
covering index
Even No SQL :)
Hardware limits, replication
21. Know your tools
- Memcache
Tons of persistent TCP conns eats your ram
UDP performance issues
Single thread for UDP
Multiport patch
proxies
Stresses the network to the max
Driver issues, configuration
Variable performance with net devices
22. Know your tools
- No SQL
Not magic!
Good for heavy write loads
Good for data processing
Still needs tweaking partitioning, schemas
24. Evolve, iterate
- All architectures scale till certain point
- Then you must rethink everything
Then, and only then!
Remember premature optimization?
Scale != efficient
Future is hard to predict
29. Architecture
- Design for failure:
Support disabling
Nice degradation, fallbacks
Controlled launches
- Test with dark launches
- Think on storage operations
- Be able to migrate live
- Focus on your core, use CDNs
30. Architecture
- Move work to the browser:
Request routing
Templates
Cache
Pefetch
- Move remaining to your FEs:
Data relations
Consistency
Privacy, access check
Live migrations
Knowledge of the storage infraestructure
31. Architecture
- All teams involved
Frontend
Good JS, templating, caching, prefetching
Backend
Data design, parallelization, optimizations
Systems
Iron benchmarks, tunning, networking
33. Scaling a website
- Setup: 1 server
- Bottleneck: cpu
- Solution: Add fronteds
- Changes: Share sessions
34. Scaling a website
- Setup: N fronteds, 1 DB
- Bottleneck: DB Reads
- Solution: Add DB slaves
- Changes: Split reads to slaves or DB proxy
35. Scaling a website
- Setup: N fronteds, 1 DB Master + N Slaves
- Bottleneck: Limited # of slaves, so DB Reads
- Solution: Chain replication / Add cache layer
- Changes: Big ones!
Some caches in certain places is easy
But for dynamic app, Memcache as storage
Makes your DB nor relational
36. Scaling a website
- Setup: N FEs, 1 DB Master + N Slaves, Caches
- Bottleneck: DB Writes
- Solution: Split tables into DB clusters
- Changes: Add some DB abstraction
37. Scaling a website
- Setup: N FEs, N DB clusters, Caches
- Bottleneck: DB Writes on certain table
- Solution: Partition tables
- Changes: DB abstraction and big changes
DB no longer relational, more key based
Partition key limits queries
Denormalization, duplicity
38. Scaling a website
- Setup: N FEs, N partitioned DBs, Caches
- Bottleneck: Disk space, DB cost
- Solution: Archive tables
- Changes: DB abstraction + migration scripts
42. Backend Framework
- Our mission:
Provide easy to use, productive, easy to
debug, testable, fast, extensible,
customizable, deterministic, reusable,
instrumentalized (stats) framework and
tools to ease developers daily work and
manage the infraestructure.
43. Backend Framework
- From Request routing to Storage
- Simple layers, clean responsibilities
- Clean, organized codebase
- Using:
convention over configuration
configuration over coding
- Queuing system for async execution
- Gathering stats from all levels
44. Backend Framework
- Request routing:
Multiple entry points
Fast request parsers route to Agents
Data centric agents
Printers
45. Backend Framework
- Domain Api:
Expose top-level business actions
Clean, semantic Api
No state, no magic, all data in params
Check privacy (the right place!)
46. Backend Framework
- Domain Backend:
Implement public/internal business actions
Clean, semantic Api
No state, no magic, all data in params
Coordinate transactions
No privacy
47. Backend Framework
- Domain Storages (ORM like)
Configure storage access for a table
Fields, validation, partitioning, primary
key, caching techniques, custom queries.
Provide access to storage via standard apis:
CRUD actions
Cached Lists
Cached Queries
+ Custom
Data container
