Abstract: Over the past few years, massive open online courses (MOOCs) powered by Open edX have become wildly popular, bringing free or low-cost education to millions of students around the world. Such success, however, presents a slew of challenging problems in terms of providing a scalable, robust, and secure platform. At Appsembler, we offer customers a fully managed and supported Open edX stack, all the way from the frontend web application to the backend services like ElasticSearch, MySQL, and MongoDB. With so many moving parts, we have come to realize the value of a multi-container, microservices-oriented architecture using Kubernetes. In contrast to a single-container deployment of the Open edX stack, a Kubernetes-based approach allows us to scale different services independently; improves robustness since we can simply spin up new copies of containers if they go down; and results in improved security through greater segmentation and isolation. In addition to discussing these benefits, we'll also cover Kubernetes’ integration with Google Cloud services like Cloud SQL and Cloud Logging. Sched Link: http://sched.co/6BbQ

1. Scaling Open edX
with Kubernetes
KubeCon London
2016.03.10
Morgan Robertson

2. Hi, I’m Morgan

3. What we’ll cover
● Introducing Open edX
● The problem: Walled gardens
● The solution: Open edX SaaS
● Open edX on Kubernetes
● Lessons learned

4. Introducing Open edX

5. Technical components
LMS/CMS (Django/Python)
Forum (Sinatra/Ruby)
User DB (MySQL)
Course DB (Mongo)
Tasks (Celery/RabbitMQ)
Caching (Memcache)
Proxy (Nginx)
Search (ElasticSearch)
Mapreduce (Hadoop)

7. The walled garden

8. Open edX Testdrive

9. Individual deployments don’t scale!
● Inefficient use of resources
● Can’t demo advanced features
● Painful upgrade process

10. Open edX SaaS
AMC

11. Kubernetes vs. Docker Swarm & friends
Kubernetes Swarm Compose
Scheduling ✔ ✔
Service discovery ✔ ✅ ✅
Container scaling ✔ ✔
Health checking ✔
Secret management ✔
Rolling updates ✔

12. Kubernetes vs. Docker Swarm & friends
Kubernetes Swarm Compose Consul
Scheduling ✔ ✔
Service discovery ✔ ✅ ✅ ✔
Container scaling ✔ ✔
Health checking ✔ ✔
Secret management ✔
Rolling updates ✔

14. Containerizing Open edX
edxapp
forum

15. Building images
FROM phusion/baseimage:0.9.15
# Clone the configuration repo
RUN git clone https://github.com/appsembler/configuration.git
/tmp/configuration
# Run Ansible
WORKDIR /tmp/configuration/playbooks
RUN ansible-playbook -vvv -i 127.0.0.1, -c local forum.yml
CMD ["/sbin/my_init"]

16. Open edX on Kubernetes

17. Persistence: Mongo and MySQL
● Mongo replica set on VMs
● Google Cloud SQL
● Kubernetes services can abstract non-Pod backends

18. Using Endpoints for non-Pod backends
# mongodb-endpoint.yml
apiVersion: v1
kind: Endpoints
metadata:
name: mongodb
subsets:
- addresses:
- ip: 10.240.0.3
- ip: 10.240.0.4
- ip: 10.240.0.5
ports:
- name: mongodb
port: 27017
# mongodb-service.yml
apiVersion: v1
kind: Service
metadata:
name: mongodb
labels:
name: mongodb
spec:
clusterIP: None
ports:
- name: mongodb
port: 27017

19. Initialization tasks using Jobs
# Create users and DBs in MySQL
apiVersion: extensions/v1beta1
kind: Job
metadata:
name: mysql-init
spec:
template:
metadata:
labels:
name: mysql-init
spec:
containers:
- name: mysql-init
image: gcr.io/appsemblerx/mysql-init
restartPolicy: OnFailure
● One-time tasks
● Batch tasks
● Run until success

20. Secrets
$ base64 secret.json
ewogICAgInNlY3JldCI6
ICJteSBzZWNyZXQiCn0K
Kubernetes
API
Pod 1
secret.
json
Pod 2
secret.
json

21. Google Cloud Monitoring

22. Lessons learned
● Containerizing an existing application can be difficult
● Once containerized, deploying on Kubernetes is
generally straightforward
● Not everything must be containerized to use
Kubernetes

23. Thank you for your time!