Scaling is a real issue for many websites. However, many developers try to implement solutions used by the giants of the internet: Google, Facebook, Twitter, WordPress, etc. However, many of these techniques are designed for very unique and high demand situations. Implementing these techniques prematurely to smaller websites can lead to overly-complex solutions that can be difficult to manage, and even hinder progress.
See real life examples of from two popular websites: cevo.com and datingdna.com. Learn what challenges and bottlenecks they faced, and the solutions they created to overcome them. Find out what optimizations and implementations returned the greatest ROI for scaling with the project.
CEVO is an online gaming league that hosts thousands of matches for hundreds of video game events. They have contracted with DirectTV, Dell, and other companies to offer branded gaming events. Because the company is run by a staff spread across the USA and Canada, the website runs 100% of their business. Hundreds of gaming servers and player clients communicate with CEVO’s APIs to get real-time information. Learn how they’ve handled the ever increasing load on their servers.
Dating DNA is an online dating community that integrates with today’s popular social networking sites. The site has exploded since it released its iPhone App, which is currently the #1 iPhone Dating App. Learn how Dating DNA has scaled from having daily user sign-ups increase 1000% in a matter of weeks.
This presentation will be given by Justin Carmony, a lead developer for both projects.
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Real Life Scaling: A Tale of Two Websites
1. Real Life Scaling:
A Tale of Two Websites
By Justin Carmony - Utah Open Source Conference 2009
2. “It was the best of times, it was
the worst of times...”
- A Tale of Two Cities, Charles Dickens
3. About Me
• Website Hobbyist since 1997
• Professional Since 2005
• Worked in Large Open Source & Proprietary Solutions
• Full Time Contractor & Consultant
• Sponsorship Manager for UTOSC
• Blog: http://www.justincarmony.com/blog/
4. This Presentation
• Point You In The Right Direction
‣ Not a Substitute for Research & Homework
• Developers Perspective, Not So Much Sys Admin
• Q & A Session Afterwards
• Available for Questions Remainder of Conference
• Ask via Email: justin@justincarmony.com
14. “Zero Theory”
200 Current Visitors
500 New Sign-Ups
300 Messages Sent
1,000 Users
10,000 Users
100,000 Users
15. “Zero Theory”
200 Current Visitors 2,000 Current Visitors
500 New Sign-Ups 5,000 New Sign-Ups
300 Messages Sent 3,000 Messages Sent
1,000 Users 10,000 Users
10,000 Users 100,000 Users
100,000 Users 1,000,000 Users
16. Tale #1: Cyber Evolution
Website: www.cevo.com
Online Gaming League
Co-Developed w/ Eric Ping
Communication Between Many
“Clients” & “Servers”
Great Growth Over 4 Years
Learning by Fire for Myself
17. Tale #2: Dating DNA
Website: http://www.datingdna.com
Online Free Dating Service
#1 iPhone Dating App
Overnight 1,000% Growth
Continuing to Grow Rapidly
Unique Scaling Challenges
19. #1 Challenge: The Database
• 80% of Performance Issues were
Database Related
20. #1 Challenge: The Database
• 80% of Performance Issues were
Database Related
• Issues Aren’t Noticeable Until
After Growth & High Load
21. #1 Challenge: The Database
• 80% of Performance Issues were
Database Related
• Issues Aren’t Noticeable Until
After Growth & High Load
• Most Issues Were Due to Poor
Queries and/or Poor Indexes
23. Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs
24. Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs
Prevention: Learn About Database Design
25. Database Abuse
• Excessive Logging (instead of using rotated files, etc)
• Excessive Writes (INSERT, UPDATE, REPLACE, DELETE)
• Shear Volume of Repetitive Queries
• Non-Indexed Searches
• Sub-Queries Instead of JOINs
Prevention: Learn About Database Design
This is NOT for DBAs Only
26. Example: Custom Forums
• In Our Defense, We Made This In
About 24 Hours
• Nested Queries = 1000+ Queries
Per Page - Not Joking
• 10 second Load Times
• End Result: Users Hated It
27. Example: Standings Page
• Before:
‣ Nested Queries To Determine
Stats Each Row
‣ Regenerated Every Time Match
Updated
• After:
‣ Generate Every 30 Minutes
‣ Proper JOINs, Only One Query
28. MySQL Locks
• Problem: Popular Tables Using MyISAM
• Solutions:
‣ Switched to InnoDB
‣ Optimized Logic to Reduce Writes
‣ Reduced JOINs in Queries
30. Open APIs
• Can Get Heavily Abused
• Can Grow Unexpectedly
• Must Be Extremely Efficient
• CEVO’s APIs
‣ 100,000s of Player Lookups
‣ Thousands of CMN Players
‣ Match History Calls
34. Identifying What’s Unique
• “What Unique Part of Your Site
Will be Complicated to Scale?”
• Important to Identify Early
• Allows you to Plan for the Future
• Many Solutions for the Common
Stuff to Scale
35. X 2 -X
Dating DNA’s Compatibility Score Growth
X = # of Users
37. # of Score Records
10 Users
200 Users
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users
38. # of Score Records
10 Users 90 Records
200 Users
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users
39. # of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users
25,000 Users
300,000 Users
1,000,000 Users
40. # of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users
300,000 Users
1,000,000 Users
41. # of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users
1,000,000 Users
42. # of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users 89,999,700,000 Records
1,000,000 Users
43. # of Score Records
10 Users 90 Records
200 Users 39,800 Records
5,000 Users 24,995,000 Records
25,000 Users 624,975,000 Records
300,000 Users 89,999,700,000 Records
1,000,000 Users One Trillion Records!
44. How We Solved It
• Introduce Limits on Records per User
• Only Save Decent Scores
• Smart Logic to Predetermine Good Matches
• Shard Table Into Multiple Tables
• We’re Still Managing & Finding More Optimizations
46. Memcached
• Scales Much Easier than Traditional Databases
• Reduced Load Off Databases by 70%
• Implement Quickly -- Its Awesome!
• Gave Presentation On @ UPHPU
‣ Check My Blog for Recording
47. Hardware
• Be careful of “Throwing Hardware” at Problems
• Run on Realistic Hardware
• Developers, You Need To Be Cost Conscious
• Can You Afford to Scale Up? How About Scale Down?
48. Other Challenges
• Apache Configurations
‣ MaxClient Limits Reached
• MySQL Configurations
‣ Max Connections Reached
• Network Communication
‣ Saturated Bandwidth Between
Servers
49. Tools I’ve Used
• MySQL - JetProfiler (No FOSS, Hopefully One Day)
• Server Performance - top, htop, atop, iotop
• PHP - Zend Debugger & Profiler, xdebug
• FirePHP & FireBug
52. Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
53. Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
54. Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications
55. Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications
• Create Metrics to Monitor the Health of the
Components
56. Getting Ready To Scale
• Compartmentalize “Parts” Into “Components”
• Create “Scaling Strategy” For Each “Component”
• Keep It Simple, Complexity == Complications
• Create Metrics to Monitor the Health of the
Components
• “Zero Theory” For Each Component
59. Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Dating DNA Database
Web Servers
Application Cluster
Score Generation Memcache
System Cluster
60. Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Dating DNA Database
Web Servers
Application Cluster
Score Generation Memcache
System Cluster
61. Makes Components of
Dating DNA
Jobs / Cron User Uploaded
System Content
Database
Web Servers Communication Cluster
Score Generation Memcache
System Cluster
68. Scaling Components
Comp C1 Comp A2
Comp B1 Comp E1
Comp A1 Comp D1
Server #1 Server #2 Server #3 Server #4
69. Scaling Components
Comp C1 Comp A2 Comp C2
Comp B1 Comp E1 Comp B2
Comp A1 Comp D1 Comp A3
Server #1 Server #2 Server #3 Server #4
70. Scaling Components
Comp C1 Comp A2 Comp C2
Comp B1 Comp E1 Comp B2 Comp B3
Comp A1 Comp D1 Comp A3 Comp A4
Server #1 Server #2 Server #3 Server #4
71. Scaling Components
• Should Be Able to “Live” With Each Other
• Deployment & Management all Automated
‣ Not Necessarily “Automatic”
• Fault Tolerance
• Testing & Staging Environments
72. Scaling Web Servers
• Challenges Load Balancer
‣ Routing
‣ Sessions
• Ideas Web Server Web Server
‣ Separate Dynamic &
Static
Cache DB
‣ Use Sub Domains
73. Asynchronous Score
Generation
• Pass Message To “Score Server”
‣ Pass User ID Only
• Quickly Generate a Small Set of Scores
• Queue User for Full Process
‣ Spawn Small Generation
• Update “MEMORY” MySQL Table on Status
74. The Open Source Advantage
• Choice & Options w/ Cost
• Solid Applications, Tested & Proven
• Great Communities
• Give Back, Karma++
75. Last Thoughts
• Once again, K.I.S.S
• Proactive > Reactive
• Monitoring, Monitoring, Monitoring!
• The “Cloud”
• Get Advice from the Community