2020 brought us massive growth in usage but big challenges came along with it (as they say, a good problem to have!).
In this talk, I will share the challenges we encountered and how we responded to them, including the decisions we had to make and the processes we developed.
6. #UXDXAPAC Handling Sudden Growth
Early 2020
Restructuring: Splitting the Japanese business to the rest of the global business
WHY?
1. Difference in Academic calendars
2. Ensures Future business sustainability
7. #UXDXAPAC Handling Sudden Growth
If you want to place an image with full screen,
remove the top blue bar and the footer
But then on March 2020…
14. #UXDXAPAC Handling Sudden Growth
Effects on our product
- More demand for features to facilitate online education
- Higher traffic due to more users with higher activity per user
- More bugs discovered due to increased usage
- Higher urgency in addressing and fixing bug reports
15. #UXDXAPAC Handling Sudden Growth
August 2020:
- Start of some private schools
October 2020:
- Start of public schools
Incidents Happened... Incidents
16. #UXDXAPAC Handling Sudden Growth
Incident Investigation Results
Main Symptoms:
- Slow access on some pages, leading to errors
- Late delivery of activities to students
Causes:
- Slow performance on database need further optimizations
- No CDN on some platforms set up CDN
- Autoscaling setup didn’t work improve scaling based on peak times
19. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
WHY?
- By focusing on developing features, we had less attention on scalability.
- We accumulated technical debt and now it’s the time to pay back.
- Developing new features is useless if users cannot use our core features.
Which lead to…
a freeze on developing new features
20. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
I was chosen to lead the efforts and we broke down our efforts into 3 main
focus:
Technical
Communications
Operations
21. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Technical
Allows us to prevent these incidents from happening again
- Clean up old code
- Remove unnecessary requests
- Improve performance via caching, N+1 fixes, improving algorithms, etc
- Setup auto-scaling and scheduled scaling
- Ensure apps have enough resources so they don’t crash
- Improve database for higher capacity
22. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Communications
Keeps everyone on the same page
- Agree on timeline with the stakeholders
- Communicate with users about the efforts being done
- Weekly checkup with Platform leads on the optimization efforts, to also
maintain cross-platform communication
- Compile common issues encountered so that support officers can
respond to users faster
- Create an incident management flow so that everyone is up-to-date on
status of incidents that may happen
23. #UXDXAPAC Handling Sudden Growth
Platform Optimization Efforts
Operations
Allows us to recover from incidents as quickly as possible
- Have developers on stand by (on a rotation basis) from 6:30 am for
possible incidents.
27. #UXDXAPAC Handling Sudden Growth
Platform Optimization Results
- No major incidents, despite increasing usage
- Our scaling system worked perfectly
- Established cross-platform communications
- Raised awareness on how developments in different departments affect
each other
30. #UXDXAPAC Handling Sudden Growth
- Degradations at some points of
the day
- We found that we hit our limit
of concurrent usage.
Battle is won but the war is not yet over... Incident?
31. #UXDXAPAC Handling Sudden Growth
More Future Improvements
- Establish workflows and processes
- Improve our incident response process
- Switch from reactive to proactive
- Changes on an organizational level from the lessons learned
33. #UXDXAPAC Handling Sudden Growth
Takeaways
- Always analyze the data to make sound decisions
- Don’t procrastinate in addressing technical issues. Technical debt will
come back to bite you when you are less prepared.
- We can prepare all we want but we will always encounter situations we
are not ready for. Learn to adapt!
- Be decisive, for the users. Make a call and own it. Taking a lot of time to
cascade discussions and decisions can be costly.
34. #UXDXAPAC Handling Sudden Growth
Visit https://career.quipper.com/
Be a distributor of wisdom and join us!
We’re looking for:
● Software Engineers (Web & Mobile)
● Product Designers (UI & UX)
● SRE
🇮🇩
🇵🇭
