Iguazú: A Long-Running Job Scheduler using Docker and Mesos
•
8 likes•21,943 views
These slides describe how Coursera created its next generation of long-running job scheduler by integrating Docker and Mesos.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (10)
Similar to Iguazú: A Long-Running Job Scheduler using Docker and Mesos
Similar to Iguazú: A Long-Running Job Scheduler using Docker and Mesos (20)
Recently uploaded
Recently uploaded (20)
Iguazú: A Long-Running Job Scheduler using Docker and Mesos
- 1. IGUAZÚ A Job Scheduler Using Mesos and Docker Colleen Lee Software Engineer
- 4. ! ! ! ! "COURSERA" IN 2011 ! • Work to be done: gradebook exports, regrading quizzes, sending batch emails, encoding videos, etc.
- 5. CASCADE
- 6. Cascade: Lifecycle of a Job
- 7. HOW DOES CASCADE WORK? Client
- 8. HOW DOES CASCADE Job data Client SQS Database Job information WORK?
- 9. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Database
- 10. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade New job Database Poll
- 11. HOW DOES CASCADE Client WORK? SQS Worker Worker Running... Cascade Status information Database
- 12. HOW DOES CASCADE Client SQS Worker Worker Cascade No new jobs! Database Poll Zzz... WORK?
- 13. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Status? Job status Database
- 14. HOW DOES CASCADE Client WORK? SQS Worker Worker Cascade Job is done! Database
- 15. LACK OF ISOLATION Worker 2 Worker 1 Worker 2 Cascade Worker 1 CPUs Memory Worker 2 Worker 1
- 16. LACK OF ISOLATION Worker 1 Worker Worker 2 Worker 1 Worker 2 Cascade 2 Worker 1 CPUs Memory
- 18. FRAGILE DEPLOYMENT FLOW System code vs. Job code
- 19. FRAGILE DEPLOYMENT FLOW System code vs. Job code Rarely updated Updated all the time
- 20. FRAGILE DEPLOYMENT FLOW System code vs. Job code Rarely updated Updated all the time Unique system no deploy tooling
- 21. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code???
- 22. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code!
- 23. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job...
- 24. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job...
- 25. FRAGILE DEPLOYMENT FLOW Worker Poll for job... Run job... Poll for job... Run job... Update job code! Worker (new code) Poll for job... Run job... Poll for job... Run job... Worker Worker Worker Cascade Worker Worker Worker Cascade Worker Worker Worker Cascade
- 27. POOR DEVELOPMENT STORY Client SQS Worker Worker Cascade Database
- 28. POOR DEVELOPMENT STORY Dev 1 Client SQS Worker Cascade DB
- 29. POOR DEVELOPMENT STORY Dev 1 Client SQS Worker Cascade DB Client Worker Dev 2 Cascade DB
- 30. POOR DEVELOPMENT STORY Job data Poll Dev 1 Client SQS Worker Cascade DB New job Client Worker Dev 2 Cascade DB Where'd my job go? What is this job?
- 31. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story
- 32. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language
- 33. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language Cascade-Scala? Cascade-Python?
- 34. CASCADE HAS ... SOME ISSUES • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language Cascade-Scala? Cascade-Python? Duplicating work: BAD Static partitioning: BAD
- 37. 2014: CASCADE V2?? DATABASE MIGRATIONS!! SCHEDULED JOBS!! PROGRAMMING ASSIGNMENTS!
- 39. IGUAZÚ!
- 40. IGUAZÚ!
- 42. • Resource isolation: cgroups ! ! ! ! !
- 43. • Resource isolation: cgroups • Master(s) has/have "soft state" • Coordination AND robustness ! ! !
- 44. • Resource isolation: cgroups • Master(s) has/have "soft state" • Coordination AND robustness • Implementation: • Scheduler: accepts and manages resources • Executor: process launched on slaves to run tasks
- 46. • Lightweight, but provides abstraction of a VM
- 47. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting!
- 48. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting!
- 49. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting! • Private Docker registry: convenience, versioning
- 50. • Lightweight, but provides abstraction of VM • Dockerfiles: self-documenting! • Private Docker registry: convenience, versioning • Usage: specify the image and specify a command
- 51. Iguazú: Lifecycle of a Job
- 52. HOW DOES IGUAZÚ WORK? Client
- 53. HOW DOES IGUAZÚ WORK? Client SQS Iguazú Job data Production mode Job id Database Job information
- 54. HOW DOES IGUAZÚ WORK? Client In-memory Iguazú Job data Development mode Job id Database Job information
- 55. HOW DOES IGUAZÚ WORK? Client SQS Poll Manager Iguazú Database
- 56. HOW DOES IGUAZÚ WORK? Client SQS Job data Manager Scheduler Iguazú Database
- 57. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Slave
- 58. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor Slave
- 59. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Check for Slave new image
- 60. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Status updates Slave
- 61. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database TASSKla_vFeINISHED Executor registry Status updates Slave
- 62. HOW DOES IGUAZÚ WORK? All done! TASSKla_vFeINISHED Master(s) Client SQS Manager Scheduler Iguazú Database Executor registry Status updates Slave
- 63. HOW DOES IGUAZÚ WORK? All done! TASSKla_vFeINISHED Master(s) Client SQS Manager Scheduler Iguazú Database Executor registry Status updates Slave OK!
- 64. HOW DOES IGUAZÚ WORK? Master(s) Client SQS Manager Scheduler Iguazú Database Slave Executor registry Job is done! Slave Status? Job status
- 65. REMEMBER CASCADE'S PROBLEMS? • Lack of isolation • Fragile deployment flow • Poor development story • Tied exclusively to one language
- 66. REMEMBER CASCADE'S PROBLEMS? Mesos: cgroups! • Isolation! • Fragile deployment flow • Poor development story • Tied exclusively to one language
- 67. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Poor development story • Tied exclusively to one language Private Docker repo
- 68. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Consistent development story • Tied exclusively to one language Iguazú: proper abstractions
- 69. REMEMBER CASCADE'S PROBLEMS? • Isolation! • Easy deployment flow • Consistent development story • Any language No restrictions
- 70. OTHER BENEFITS
- 71. OTHER BENEFITS • Ease of transition ! ! ! ! Mesos: job management Docker: job packaging
- 72. OTHER BENEFITS • Ease of transition • Performance ! ! ! Mesos: long-running! PHP: blaaargh
- 73. OTHER BENEFITS • Ease of transition • Performance • Flexibility ! ! Use Docker, run Scala code, etc.
- 74. OTHER BENEFITS • Ease of transition • Performance • Flexibility • Fine-grained control over scheduling ! Autoscaling!
- 75. OTHER BENEFITS • Ease of transition • Performance • Flexibility • Fine-grained control over scheduling • Designed to work on a heterogeneous pool of resources Security :)
- 76. THANKS! QUESTIONS? ! We are hiring! See http://coursera.org/jobs ! @firejade0 clee@coursera.org