Watch the full video here: https://vimeo.com/166815139 DrupalCon New Orleans - May 10, 2016 Steven Merrill, Phase2 Director of DevOps, with Brian Aznar of MLS Representing the sport’s highest level of professional soccer in the US and Canada, the Major League Soccer’s Digital division maintains a platform comprised of multiple sites and applications responsible for delivering information, news, videos, and stats to eager MLS fans. In today’s digital landscape, a sports fan’s digital experience is paramount to successful promotion and brand loyalty. In this case study session, you will learn how MLS and Phase2 migrated 25 club sites onto a high performance Drupal platform while maintaining their existing legacy sites to deliver the best experience for site visitors. In addition, we will discuss the performance strategy developed to ensure that the millions of MLS soccer fans can seamlessly access their favorite team's news and stats without interruption. In this session you will learn : - How MLS planned and developed a long term digital strategy with Phase2 for sustainable and high performance management of their digital platform - Techniques to implement sophisticated customization options into multiple sites while managing them on the same platform code base - Best practices to execute autoscaling to save money and automatically handle traffic spikes - How to apply technical and governance changes to make on-demand shifts less burdensome to prevent developer burnout - The steps you need to take to implement microservices alongside a Drupal platform

1. SCORE!
SUPERIOR DIGITAL EXPERIENCES WITH MLS
MAY 10, 2016
Brian Aznar
(Director of Engineering, MLS Digital)
Steven Merrill
(Director of DevOps, Phase2)

2. BACKGROUND
MLS & DIGITAL PRODUCTS

3. MAJOR LEAGUE SOCCER
STATE OF THE LEAGUE
‣ 20 current teams with plans to expand to 24 by 2020
‣ Adding state of the art soccer-specific stadiums around the US -
average in stadium audience exceeds NHL and NBA
‣ Expanding our reach in the US and around the world with great new
TV distribution deals
‣ Young and multicultural fan base that is hungry for digital

4. MLS FANBASE
‣ Avids
‣ Regularly watches and
attends MLS matches
‣ Consume soccer related
content
‣ Regularly visits
MLSsoccer.com and other
soccer specific sites
‣ Casuals
‣ Occasionally
watches and
attends MLS
Matches

5. MLS DIGITAL PRODUCT ECOSYSTEM
‣ Web: 20 club websites + 1 league web site using Drupal 7
‣ Mobile Apps: iOS, Android
‣ Connected TV: Roku, Chomecast, Apple TV
‣ Services: Content API, Video & Stats ingest pipelines,
Content delivery, etc …
‣ Infrastructure: AWS

6. 2015 CASE STUDY

7. GOAL
‣ Modernize and stabilize MLS’ web
platform (backend and frontend)
and successfully migrate 21 sites
by the end of 2015 to AWS and D7.

8. THE TEAM

9. PHASE2
‣ Founded in 2001
‣ Offices in DC, NYC, San Francisco, and Portland
‣ Committed to open source technology
‣ Focus on how content is delivered, how
teams collaborate, and how audiences
experience information

10. PHASE2 & MLS
‣ Long-term partnership over 3 years
‣ Helped with D7 platform build
‣ March 2015 - we kicked off their migration to
MP7 Drupal platform and into AWS

11. PHASE2 & MLS TECHNOLOGY TEAM
‣ Drupal Developers (Phase 2)
‣ Fullstack Javascript Developers: React, Backbone, Node.js & Go (MLS)
‣ DevOps (Phase 2)
‣ Process = Scrumban
‣ Maximize development velocity, agility and responsiveness to business needs.
‣ Centralized Development Team
‣ Moving from segmented teams towards a single cross disciplinary team.

12. CHALLENGES & SOLUTIONS

13. CHALLENGES WE NEEDED TO ADDRESS
1. Outdated UX originally built for desktop viewing only
2. Inflexible Authoring - Needed an efficient and scalable
way to give clubs more control to build out their own content
3. Wasted Server Capacity - High variability in traffic
patterns means planning for the worst

14. CHALLENGES WE NEEDED TO ADDRESS
4. High Fixed infrastructure costs
5. Poor Performance
6. Technology Soup - Go, node, CoreOS/Fleet, Docker,PHP/
Drupal (Multiple Versions) and Salt all managed with a lean
team
7. Aggressive Migration - 21 websites and No downtime
a. Avoiding major tent pole events that varied across clubs

15. 1. MODERN UI EXPERIENCE

16. ‣ Building blocks that editors can put together
‣ All features were designed to be responsive
‣ Video highlights are pushed from MLS’s video providers
‣ Import of MLS entities (teams/players/seasons) from MLS’s pipeline
2. EDITORIAL FLEXIBILITY

17. 3. SPIKEY TRAFFIC
‣ Traffic Spikes (avg 4x) centered around games and events
‣ League represented about half the traffic on average
‣ Great candidate for autoscaling

18. 4.a HIGH INFRASTRUCTURE COST
‣ 2 Infrastructures (One foot in the cloud)
‣ Drupal running in Rackspace Servers
‣ Services Running in AWS
‣ Drupal running on fixed set of large servers configured to
handle the average amount of traffic plus physical
devices (firewalls, NetApp storage…)
‣ Paying for unnecessary resources needs during off peak
and off season traffic

19. 4.b MOVING INTO AWS
‣ Rackspace => AWS - everything in one cloud infrastructure
‣ Autoscaled Instances
‣ Additional operational benefits by using:
‣ Amazon RDS, ElastiCache, S3

20. AUTOSCALING
‣ Reduces stress
‣ Automatic healing
‣ Performance benefits
‣ Manual adjustment up and down
‣ Automatic scaling based on metrics
‣ Cost benefits

21. ‣ For autoscaling, you should build cattle, not pets
‣ Previous MLS autoscaling provisioned from scratch
‣ Phase2 built “fat” AMIs with all software + code
‣ Far faster deployment time
‣ An AMI can come up if everything else is down
‣ See Steven’s DevOps track session tomorrow for more info!
AUTOSCALING BACKGROUND

22. A USUAL MONTH

23. THE OVERALL RESULT

24. 5. POOR PERFORMANCE
‣ Unsuccessful attempt at cloud multi region
architecture
‣ Caching anomalies between Drupal in 2
datacenters
‣ Slow cache clearing bottlenecks
‣ Slow server response times > 1000 ms

25. ‣ Right-sized AWS instances to their workloads
‣ Upgraded to PHP 5.5 for a performance boost
‣ Optimal instance size + PHP 5.5 = 40% increased throughput
‣ Added stability fixes
PERFORMANCE TUNING: QUICK WINS

26. ‣ Load Testing & Application Monitoring to find and remove bottlenecks
‣ Disabled DB read/write split for a massive speed increase
‣ Doubled the size of memcache instance
‣ Killed legacy code that cleared all of Drupal’s caches
‣ Results:
‣ Instance sizes reduced by half
‣ Massive decrease in average page render time
PERFORMANCE TUNING

28. 6. TECH COMPLEXITY
‣ Clubs and League Content Sites
‣ Drupal (6/7)
‣ PHP (5.1/5.3)
‣ Jenkins / Salt for Deployment
& Configuration Management
‣ Web Services / Stats & Video Ingest
Pipeline / Misc Web Apps
‣ Node.js & Go
‣ CoreOS/Fleet/Docker & Teamcity for
Deployment & Configuration
Management
‣ Server Configuration Drift

29. 7. MIGRATION
‣ Seattle, Orlando, NYCFC were in place
‣ Large project management lift
‣ Coordination with stakeholders
‣ Training schedule
‣ Working with club schedules /
game schedules
‣ Central Digital Club Services team played a huge role by
managing coordination w/ clubs and drupal editors

30. ‣ Drupal multisite
‣ Saved server resources
‣ Made deploys smaller
‣ Worked for MLS Digital specifically because of
the integrated nature of their Drupal platform
‣ No bespoke development per-club
‣ Centralized feature development
‣ Far lower support burden (6 people)
MULTI-TENANT SETUP

31. PROJECT MILESTONES
‣ April 2015: Migrated first team site
(NY Red Bulls)
‣ July: Turned on autoscaling
‣ April - November: Migrated MP7
‣ October: League site launched!
‣ October: Decision Day
‣ December: Turned off Rackspace hosting
‣ January 2016: Helped hire in-house
DevOps Engineer
‣ March: Successful 2016 Season Kickoff

32. TOOLS

33. TOOLS
‣ New Relic for Application Performance
Management / Instrumentation
‣ Blaze meter for Load testing
‣ Datadog for Infrastructure Metrics

34. NEW RELIC
‣ Table Stakes Tooling for the Modern
Technology Team
‣ Real Time End to End Client & Server
metrics and performance measurement
‣ Introspection of everything from end
page rendering on a user’s browser to
PHP & DB bottlenecks on the server
‣ Tiers:
‣ Free - 1 day retention -
good enough for load tests
‣ Pro - 90 day retention

35. REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO

36. CLUB ROLLUPS
REALTIME FULL STACK METRICS: CLIENT TO SERVER TO REPO

37. CLUB ROLLUPS
MODULE, HOOK & VIEW PERFORMANCE

38. GLOBAL ROLLUPS
MULTI SITE TOP LEVEL METRICS: AGGREGATE

39. CLUB ROLLUPS
MULTI SITE TOP LEVEL METRICS: INDIVIDUAL SITE

40. SUMMARY

41. OUTCOMES
‣ Migrated fully onto AWS
‣ Lowered hosting bill significantly by leveraging AWS features
‣ Helped reduce support load through policy changes
and new MLS hires
‣ Go see Steven’s session for more info
on immutable infra!

42. PROBLEMS ADDRESSED IN 2015
1. Outdated front-end UI experience originally built for
desktop viewing only
2. Club editorial teams needed more control and flexibility to
build content that met their needs but we needed it all to fit
within the same brand ecosystem
3. High variability in traffic patterns resulting wasted capacity

43. PROBLEMS ADDRESSED IN 2015
4. High fixed infrastructure costs
5. Managing multiple technologies (CoreOS/Fleet,Salt, PHP,
Go, node) and versions (PHP, Drupal) with a small team
6. Migrating 21 websites without any down time and
avoiding major tent pole events that varied across clubs

44. THE FUTURE

45. THE FUTURE
‣ Improving performance
‣ Improving editorial usability
‣ Further consolidation & simplification
of tech stack
‣ #docker!
‣ Preparing for omnichannel

46. CHECK US OUT!
labs.mlssoccer.com
mlssoccer.com
phase2technology.com
SOCCER
STADIUMS
NEAR YOU!

47. QUESTIONS

48. SoHow Was It?- Tell Us What YouThink
Evaluate this session - https://events.drupal.org/neworleans2016/schedule
Thanks!