Keynote delivered at the Asia Pacific Software Engineering Conference, Dec 2017

1. Experience in teaching
DevOps
Len Bass

2. Topics
• What is DevOps
• Class offering
• Student knowledge shortcomings
© Len Bass 2015 2

3. Over the wall development
copyright 2015 Len Bass 3
Board has
idea
Developers
implement
Operators place
in production
Time

4. Where Does theTime Go?
• As Software Engineers our view is that there are the following
activities in software development
• Requirements
• Design
• Implementation
• Test
• Code Complete
• Different methodologies will organize these activities in
different ways.
• Agile focuses on getting to Code Complete faster than with
other methods.
4
Developers
implement
copyright 2015 Len Bass

5. What is wrong?
•Code Complete Code in Production
•Between the completion of the code and
the placing of the code into production is a
step called: Deployment
•Deploying completed code can be very
time consuming because of concern about
errors that could occur.
5copyright 2015 Len Bass

6. What is the work flow for code from a
multiteam development effort
• You develop and test your code in isolation
• Your code is integrated with code developed by other
teams to see if an executable can be constructed.
• The built system is tested for correctness
• The built system is tested for performance and other
qualities (staging)
• The built system is placed into production
copyright 2015 Len Bass 6

7. Errors can occur at every stage
• Every error must either be corrected or prevented.
• Preventing errors can be done through some
combination of
• Process
• Architecture
• Tooling
• Coordination among teams.
• Coordination takes time.
• Correcting errors takes time
© Len Bass 2015 7

8. How much time?
• Historically, releases are scheduled for once a quarter or
once a year to give time to coordinate and adequately
test.
• This means there may be months delay before a new
concept or feature is added to a system.
• This delay has become more and more unacceptable.
• Weekly or daily releases are becoming the norm.
copyright 2015 Len Bass 8

9. Goal of DevOps
• The goal of DevOps is to reduce the time to market
without compromising quality by
• Reducing errors during deployment
• Speeding up incident handling after deployment
• Speeding up deployment
copyright 2015 Len Bass 9

10. Example for reducing errors during
deployment
• Use microservice architecture
• Each team makes its own choices of technology for the service it
is developing
• Avoids inconsistencies in versions and technology choices that
are discovered during deployment process
© Len Bass 2015 10

11. Example for speeding up incident
handling
• Developers carry pagers and are the first respondents to
an incident
• When an incident occurs developers do not need to wait
for the incident to be escalated through a ticketing
system
• Gives developers cradle to grave responsibility for a
service
© Len Bass 2015 11

12. Example for speeding up
Deployment (Deployment pipeline)
• Developer creates and tests code on local machine.
• Checks code into a version control system
• Continuous integration server (CI) builds the system and runs a series of
integration tests.
• After passing the tests, the system is promoted to a staging environment
where it undergoes more tests including performance, security, and user
acceptance tests.
• After passing those tests, the system is promoted to provisional production
where it undergoes even more tests.
copyright 2016 Len Bass 12
Pre-commit
tests
X
Build Image
and Perform
Integration
tests
UAT / staging /
performance
tests
Deploy to
production
Commit
...
Pre-commit
tests
Commit
Developers
promote
to normal
production

13. Just to summarize
• Placing a system into production can be error prone and
time consuming
• Supporting a system after it gets into production can be
time consuming and error prone
• DevOps is a set of practices intended to speed up placing
a robust system into production
© Len Bass 2015 13

14. Topics
• What is DevOps
• Class offering
• Student knowledge shortcomings
© Len Bass 2015 14

15. Context for DevOps course
• I teach a DevOps course as a portion of the CMU MSE
professional master’s program
• Students are graduate students at CMU. Some have
industrial experience, some right out of undergraduate
school
• Currently in the fourth offering of the course
© Len Bass 2015

16. Course structure
• Half theory
• Lectures on video to be seen prior to
class
• Textbook and other readings
• In class discussion about lectures
and readings
• Half assignments
• Assignments involve a variety of
open source tools
• Evaluation based on theory and
assignments © Len Bass 2015

17. Discussion examples
• What was your takeaway from the videos and readings?
• Each student is called on
• Generate a design of a relevant tool, e.g. Continuous
Integration server
• Done in groups
• Time in discussion matches time in video
• Balances out of class reading/video with inclass time spent
discussing
© Len Bass 2015

18. Assignment
• Sample assignment:
• Create two virtual machines
• Install SSH so that one machine can securely connect to another
without being prompted for a password
• One page discussion: Discuss the pluses and minuses of having
“known hosts” in SSH
• Each assignment takes roughly half a day
© Len Bass 2015 18

19. Syllabus
• You can retrieve the syllabus from
https://github.com/devopseducator/2016workshop
© Len Bass 2015

20. Major topics
• DevOps overview
• Networking
• Cloud
• Deployment pipeline
• Architecture
• Deployment
• Security
• Monitoring
© Len Bass 2015 20

21. Elaborating on the syllabus - networking
• it is amazing what Masters students in CS at CMU do not
know. A reflection of their undergraduate curriculum.
• Virtualization
• Virtual machines
• Virtual machine images
• Networks
• IP addresses
• DNS servers andTTL (time to live)
• Ports
• Load Balancers
© Len Bass 2015

22. Elaborating on the syllabus –
cloud
• Failures in the cloud
• Distributed coordination – Zookeeper, Consul
• Scalability
© Len Bass 2015

23. Deployment pipeline
• Introduction to a deployment pipeline
• What are the roles and responsibilities of various
teams with respect to the pipeline
• Case study in business continuity
• Environments of the pipeline in more detail with types
of tests
• Development
• Build
• Staging
• Production
• Case study of company that sets up continuous
deployment pipelines
© Len Bass 2015

24. Elaborating on the syllabus –
architecture and advanced
packaging
• Microservice architecture
• Case study in migrating to a microservice architecture
• Containers
© Len Bass 2015

25. Elaborating on the syllabus –
deployment
• Designing for deployment
• Use of feature toggles
• Backward and forward compatability
• Managing schemas
© Len Bass 2015

26. Elaborating on the syllabus –
security
• Basic security
• Credential management
• Securing the network
• Securing the pipeline
• Securing an app
© Len Bass 2015

27. Elaborating on the syllabus –
monitoring
• What to monitor
• How to monitor
• Interpreting monitoring data
© Len Bass 2015

28. Assignments – all open source
except AWS
• VirtualBox - virtualization
• SSH – secure connection betweenVMs
• Vagrant – tool for creating and provisioning a singleVM
• Jenkins – continuous integration server
• Chef – configuration management
• Docker - containers
• AWS EC2 Container Service
• ChefVault – secure configuration management
• Nagios - monitoring
© Len Bass 2015

29. Topics
• What is DevOps
• Class offering
• Student knowledge shortcomings
© Len Bass 2015 29

30. Student knowledge shortcomings – 1
• Networking 101
• URL - > IP
• Ports
• TCP/IP
• Networking 102
• Subnets
• Firewalls
© Len Bass 2015 30

31. Student knowledge shortcomings – 2
• Security 101
• Certificates
• Trust
• Vulnerabilities/patching
© Len Bass 2015 31

32. Student knowledge shortcomings – 3
• Operations
• Deployment/rollback
• Network management
• Incident handling
© Len Bass 2015 32

33. Proposal
• Establish an “infrastructure” minor to the Software
Engineering degree. It could have the following courses:
• Operating systems
• Networking
• Operations/Security
• The cloud
• DevOps
© Len Bass 2015 33

34. Summary
• DevOps is a set of practices to
• Shorten time to deployment
• Reduce errors during deployment process
• Shorten time to respond to incidents
• A course on DevOps is attractive to students
• There are gaps in knowledge from standard
undergraduate CS or SE curricula
© Len Bass 2015 34

35. More information
• Textbook
• https://github.com/devopseducator/
2016workshop
• https://github.com/devopseducator/
2017workshop
© Len Bass 2015 35