Presentation on topic of using virtualized environment in development and testing, especially for web applications. Prepared for Pycon Ukraine 2011

1. Virtual testing &
development*
* batteries included

2. $ git clone git://server.com/project.git
$ run
Ideal world

4. $ git clone git://server.com/project.git
Real world

5. $ git clone git://server.com/project.git
... read help
$ mkdir, cp, vim config.conf, configure, make, make
install
... ask guru
$ /project/hidden/blackmagic --fix_that --fix_this --
undocumented_parameter=MAGIC_CONST_42
... %^ing @#$@#%@!!!!
Real world

6. $ git clone git://server.com/project.git
... read help
$ mkdir, cp, vim config.conf, configure, make, make
install
... ask guru
$ /project/hidden/blackmagic --fix_that --fix_this --
undocumented_parameter=MAGIC_CONST_42
... %^ing @#$@#%@!!!!
$ run
Real world

8. Why?

9. Operating System
User Space
Music
WebServer
DBAppServer
IM
Editor
Browser

10. Problems
• No isolation (oh, wrong binary on
path...?)
• Not repeatable (that README ain’t
gonna run itself!)
• No guarantees (“..but it works on my
computer!”)

11. Solution?

13. Operating System
Music
WebServer DBAppServer
IM Editor
BrowserUser Space
Virtualized OS

14. Benefits
• Isolation (project and version specific
files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)

15. Business benefits
• Reduce costs (less hardware, lower energy
consumption)
• Backup and data protection
• Application availability
• Less time to respond to changing business
needs
• Less time spent on routine tasks
• Better IT department perception

16. Time spend on routine tasks
according to VMware small & medium business survey

17. Why haven’t we been doing
this for a long time?
• Big companies have been
• Amazon EC2
• Microsoft
• Google
• Only recently possible on local machines
• Cheat hardware
• Cheap RAM
• Desktop virtualization API

18. Ok, let’s go virtual.
Now what?

19. Vagrant is a tool for building and
distributing virtualized development
environments (vagrantup.com)

20. VirtualBox is cross-platform virtualization software
that works on all the major platforms
(virtualbox.org)

21. High level overview
• Describe environment with Vagrantfiles
• Command line interface to manage VM lifecycle
• Automate provisioning and configuration
• Share folder with host OS via NFS
• Provide SSH access to guest OS
• Network multiple VMs
• Package and distribute VMs

22. Vagrant boxes
• Box = pre-packaged VM environment (~OS
image)
• A number of ready to use boxes @ http://
vagrantbox.es
• Ubuntu 32 / 64
• CentOS
• Solaris
• RH

23. Vagrantfile
• Vagrantfile is to Vagrant as Makefile is to
Make
• Describes VM in code
• One Vagrantfile per project (can contain
multiple VMs)
• Commit it to version control
• Actually a Ruby code - easy to extent

24. Vagrantfile
Vagrant::Config.run do |config|
config.vm.box = “base_ubuntu"
end

25. Command line
$ vagrant help
vagrant init
vagrant up
vagrant halt
vagrant reload
vagrant suspend
vagrant resume
vagrant destroy
vagrant provision
vagrant package
…

27. • Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)

28. Chef is an open-source systems
integration framework built specifically
for automating the cloud (opscode.com/
chef)

29. Puppet is a system for automating
system administration tasks, works
on most UNIX-like operating
systems (puppetlabs.com)

30. How Chef/Puppet works
• You create recipes
• Recipes describe desired system state
• Then you manage your server park
• Safely
• Repeatedly
• In automated way

32. Provisioning in Vagrant
• Use Chef, Puppet, bash scripts, etc. to
manage VM configuration
• Repeatable!
• Use same tools for production - Chef,
Puppet

33. Vagrant + Chef
Vagrant::Config.run do |config|
config.vm.box = “base_ubuntu_box"
config.vm.provision :chef_solo do |chef|
chef.recipe_url = "http://server/cookbooks.tar.gz"
chef.add_recipe(“aptitude")
chef.add_recipe(“apache2")
chef.add_recipe(“mysql-server")
chef.add_recipe(“python27")
chef.add_recipe(“pip")
end
end

34. Networking
• Assign IP to VM in Vagrantfile
• Forward ports
• Access VM from your browser

35. Networking
Vagrant::Config.run do |config|
config.vm.box = "lucid32"
config.vm.network("10.20.30.49")
config.vm.forward_port("web", 80, 4567)
end

36. • Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)

37. Packaging
• When environment is stable
• Package and distribute prepared VMs
• Minimize setup time

39. • Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)

40. Beyond that
• Multi-VM Vagrantfile
• DB
• AppServer
• WebServer
• Use vagrant as library
• Call API functions to do the tasks from
Ruby code
• Run custom SSH commands

41. Demo
http://youtu.be/O3-MNsowgHc

42. Demo
github.com/dreamiurg/timetest
•Django application
•Apache2 + MySQL on single VM
•Additional Python packages like used (South, annoying)
•DB migrations handled by South
•Deployment process
• Vagrant is used to create VM, install required packages and
configure services
• Fabric (fabfile.org) is used to populate DB, set up apache2 conf
and install Python modules

43. Prerequisites
$ <install python>
$ <install ruby>
$ <install virtualbox>
$ gem install vagrant
$ git clone http://server/project.git
$ cd project

44. Demo: create VM, deploy project
00:00:00 $ vagrant box add base http://server/base.box
... base box for VM is copied to your computer
00:02:00 $ vagrant up
... vagrant creates VM from base box, installs and configures
required packages – apt, apache2, mysql, python, git, …
00:05:00 $ fab deploy
... DB is populated with initial data, apache config updated, apache
restarted
00:07:00 $ curl http://localhost:8080/
... site is up and running.

45. Real world

46. Our experience
• 10+ projects during last 18 months
• From small (single webserver, no DB) to
medium (several webservers + load balancer,
master/slave DB replication)
• New developer start time reduced from 2-3
days to 1 hr on average
• Number of deployment/environment defects
reduced by 75%

48. Links
• http://vagrantup.com
• http://www.vagrantbox.es
• http://fabfile.org
• http://opscode.com
• http://puppetlabs.com
• http://github.com/dreamiurg/timetest
• http://www.vmware.com/files/pdf/VMware-SMB-
Survey.pdf

49. Dmitry Guyvoronsky
http://about.me/dreamiurg
dmitry.guyvoronsky@gmail.com
@dreamiurg