Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian Vladu

@ITCAMPRO #ITCAMP17Community Conference for IT Professionals
Provisioning Windows instances at
scale on Azure, AWS and OpenStack
Adrian Vladu
Senior Cloud Engineer @ Cloudbase Solutions
@ader1990 @cloudbaseit

Many thanks to our sponsors & partners!
GOLD
SILVER
PARTNERS
PLATINUM
POWERED BY

• 5+ years of experience
• Developer at first
• Engineer on paper
• Problem solver at heart
About me

• Cloudbase-Init and the revolving ecosystem
• How to create and test a Windows image
• Live Demo: Spawn a Windows VM using Cloudbase-
Init on Azure
• Ask.me
Takeaways

• Provisioning agent for cloud (automation) ready
instances
• cloud-init like
• Multi-cloud
• Multi-platform
• Full Windows support
• Cloud-init v2
Cloudbase-Init – what is it?

• Mature project, ~ 5 years old now
• De-facto standard provisioning agent on OpenStack
for Windows
• ~ 10 millions reported executions
Cloudbase-Init – stats

• 100% written in Python (v2.7, v3.4)
• Open source (Apache 2 License) project under OpenStack umbrella
– https://github.com/openstack/cloudbase-init
– https://bugs.launchpad.net/cloudbase-init
– https://review.openstack.org/#/c/427611/
– https://cloudbase-init.readthedocs.io/en/latest/
• Supports Windows + *nix forks
– https://github.com/pellaeon/bsd-cloudinit
• MSI installer/zip archive for x86/x64 with Stable/Nightly Builds
– https://cloudbase.it/downloads/CloudbaseInitSetup_Stable_x64.msi
– https://cloudbase.it/downloads/CloudbaseInitSetup_x86.msi
– https://cloudbase.it/downloads/CloudbaseInitSetup_x86.zip
Cloudbase-Init

• Install it via msi – fully automated
• Install it using a zip with the Python env +
Cloudbase-Init package
• Only as a Python process
• As a Python process managed by a Windows service
Cloudbase-Init (Windows) – how do I install/run it?

• Unattend step (Unattend.xml)
– During sysprep specialize step as a Python process
• First run
– At first boot the service is started by SetupComplete.cmd
– At every other boot as a Python process managed by a
Windows service set on “Automated start”
• Windows Nano Server
– At first boot the service is created and started by
SetupComplete.cmd
– The zip is used in this case
Cloudbase-Init (Windows) – when does it run?

• All Windows versions currently supported by
Microsoft
• Windows 2K8/2K8R2 and Vista/7 x86/x64
• Windows 2K12/2K12R2/Hyper-V and 8/8.1 x86/x64
• Windows 2K16/Hyper-V and 10 x86/x64
• Windows PE
• Windows Nano Server
Cloudbase-Init (Windows) – supported versions

• Execute non-metadata dependent plugins
• Load the first available metadata service
• Execute (non)metadata dependent plugins
• Communicate to the metadata service the instance
status
Cloudbase-Init – what does it do?

• Uses WMI
• Uses Win32 APIs directly
• Wraps exe’s
• Uses PyMI
–https://github.com/cloudbase/PyMI
Cloudbase-Init – what actually does it do?

• OpenStack (http or configdrive)
• EC2 (http or configdrive)
• MAAS (http)
• OpenNebula (http)
• CloudStack (http)
• Azure (http+configdrive+kvp)
• Packet (http)
• NoCloud (configdrive)
• DigitalOcean (http)
Cloudbase-Init – supported metadata services

• Create user
• Set user password and post it to the metadata
service (if supported)
• Set SSH public keys
• Execute local scripts
• Execute user data
• Execute vendor data
Cloudbase-Init – features (1)

• Set hostname / FQDN
• Create ephemeral disk notice
• Enable / Disable TRIM
• Set MTU
• Set NTP server

• Activate Windows
• Set page file
• Extend volumes
• Configure automatic updates
• Configure WINRM listener
• Configure WINRM certificate authentication
• Configure SAN policy
• Configure BCD boot status policy
Cloudbase-Init (Windows) – features (3)

• Set unique boot disk id
• Import certificates from the metadata
• Set display idle timeout
• Communicate RDP certificate thumbprint to the
metadata
• Install platform agents

• Configure networking
– Debian-like metadata OpenStack
– OpenStack JSON
– MAAS
– OpenNebula
– Packet
• Functionality
– Rename / disable adapter
– MTU
– Set IPV4 IP/netmask/gateway/DNS
– Set IPV6 IP/netmask/gateway
– Set IPV6 DNS
– Create NET_LBFO (TBM) / vendor dependent / OVS bonds
– Create VLAN adapters

• Communicate the status of the instance to the
metadata
–Via call-home URL
–Via KVP in Hyper-V
• Log to serial port

• Service can run as LocalSystem or as a dedicated
user
• By default, as a dedicated admin user
• Use LocalSystem if you don’t need impersonation
scripts
• Pass the hash attacks avoided by resetting service
user password at every service run
Cloudbase-Init – security concerns

• Like any other OpenStack project
• Feature / bug management
–https://launchpad.net/cloudbase-init
• Using git and git-review
• Enters the review system
–Unit tests need to pass
–Argus CI needs to +1
–Two core reviewers need to +2
–The code is merged into the github repo
Cloudbase-Init – how do I contribute?

ARGUS
Continuous Integration Framework for Cloudbase-Init

• Continuous integration framework for Cloudbase-Init
• 100% Python
• OpenStack Tempest based (open source Apache 2 license)
• https://github.com/cloudbase/cloudbase-init-ci
• http://argus-ci.readthedocs.io/en/latest/
• Backends
– Tempest OpenStack
– Heat OpenStack
– Azure
– Local
Argus – what is it?

• From a functionality perspective, it can be used to
test other Windows applications
• Has WinRM (pywinrm) scp like feature
• Terminology
–Backend
–Recipe
–Test
–Scenario
Argus

• Create the Windows image
–Create a Windows image with all updates
–Install git client
–Add an administrative user
–Enable HTTP WinRM
–Sysprep
• Add image to the backend
Argus workflow – Preparation

• Start an instance on backend
• Install & configure Cloudbase-Init with the given git
head
• Sysprep & reboot
• Check if Cloudbase-Init executed correctly
Argus workflow – Execution

• All currently supported by MSFT
• Can also test production images
Argus – Windows support

• Big thanks to Microsoft
• “Small” size OpenStack Mitaka deployment
• Runs at every patch change
• Huge increase in productivity after all the integration
testing complexity has been passed to the CI
• Argus is built to be a general purpose application
testing framework on multiple backends
Argus – running it

AZURE DEMO TIME
Start a Windows instance with Cloudbase-Init on Azure

ARESTOR
HTTP Metadata Service for Argus

• Create and control http metadata for an instance
• Otherwise we need access to
complex/heterogeneous/expensive environments
• Easier to have only one powerful backend
• https://github.com/cloudbase/arestor (Apache 2)
• Support for all the http supported metadata services
–OpenStack
– CloudStack, MAAS, Packet
Arestor – what is it?

• Argus controls what Arestor exposes at a given
endpoint through a Python client
• Encrypted interaction Argus->Arestor with
key/secret
• Arestor has:
–Management (admin) endpoint
–Client (instance) endpoint (S)
• Arestor implements GET/POST endpoints
Argus integration with Arestor

WINDOWS IMAGING TOOLS
PowerShell tools to create a Windows image

• Tools to create a Windows image (Apache 2 License)
– https://github.com/cloudbase/windows-openstack-imaging-tools
– https://github.com/cloudbase/cloudbase-init-offline-install
• Supported hypervisors/platforms for the generated images:
– KVM
– Baremetal
– Hyper-V
– MAAS
• Supported hypervisors to generate images:
– KVM (legacy version 1.0.0)
– Hyper-V
• Supported Windows image versions: All supported by MSFT
Windows Imaging Tools – what is it?

• Install VirtIO and other drivers
• Install additional features
• Apply updates
• Clean updates (shrinks considerably the image 20GB->15GB)
+ defragment
• Install Cloudbase-Init
• Sysprep
• Wallpaper :D
• Gold image (experimental)
• Config file based generation (experimental)
• Use qemu-img.exe to convert to qcow2/raw
Windows Imaging Tools – what does it do?

• Trial Windows Server 2012R2 images for KVM and
Hyper-V available
–https://cloudbase.it/windows-cloud-images/
• Zero to hero imaging tools script
–https://github.com/cloudbase/windows-openstack-
imaging-tools/blob/master/Examples/create-windows-
online-cloud-image.ps1
Windows Imaging Tools – fast lane

• Windows Imaging Tools bundles Cloudbase-Init
• Cloudbase-Init is tested with Argus
• The images generated by imaging tools are tested
with Argus
• Arestor provides to Argus the flexibility to test
Cloudbase-Init and the images with various
metadata formats and implementations
• Hard tested by Argus and users
The big fit

AZURE LIVE DEMO RESULTS

TIME FOR QUESTIONS

THANK YOU!

Wait, there’s more

Running Windows in the wild

• On Hyper-V – reboot (report) it now
• On KVM
– VirtIO drivers need to be installed
– https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/
– Set cpu_mode=host-passthrough & libvirt_type=kvm in nova.conf
compute node and create another VM on OpenStack.
• On Baremetal
– Make sure you have the required drivers for the storage/network
adapters
• On VMware
– No nested hyper-v? Set guestOS = "winhyperv“ in the .vmx
– Vlans not working? Set netadapter type to e1000e in the .vmx
What have I done wrong? Wild BSOD appears

What have I done wrong? Wild sysprep fail

• When Cloudbase-init is installed with the MSI on non-
Nano server and is set to perform the sysprep: if
Cloudbase-init process exits with non-zero code, a reboot
is performed (max 10 times). You will get an error saying
that “Windows could not start the installation process”.
• Documented at https://technet.microsoft.com/en-
us/library/cc722061(v=ws.10).aspx
• Why? If Cloudbase-init does not run, the instance can be
unconfigured/exposed/vulnerable or cannot be accessed
via RDP/WinRM.
• If you get “Invalid Unattend.xml”

• Press Shift+F10
• A wild cmd will appear
• Start PowerShell (optional) – literally run `start powershell` in
the cmd
• You won’t have WMI – so don’t try Get-WmiObject like
commands
• Use `notepad.exe` or `cat` to view/edit files
• Check C:WindowsPanther* log files
• Check C:WindowsPantherUnattendGC* log files
• Check Cloudbase-Init log files from
– C:Program FilesCloudbase SolutionsCloudbase-InitLogs*

• losetup/kpartx/mount/ntfs-3g on Linux
• qemu-img.exe convert to vhd/x and mount vhd on
Windows
– Replace C:WindowsSystem32sethc.exe with
C:WindowsSystem32cmd.exe, reboot and press Shift 5
times and you will have an admin cmd.exe
– Use regedit.exe and File->Load Hive-
>m:windowssystem32configsystem|software and update
the registry keys at will. File->Unload Hive afterwards.
– Be a hero and open with notepad.exe the registry file and
update the XML (it’s just an XML). You might need to replace
LocalSystem with admin user as the owner of that file
What have I done wrong? Feeling left out

Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian Vladu

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (16)

Similar a Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian Vladu

Similar a Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian Vladu (20)

Más de ITCamp

Más de ITCamp (20)

Último

Último (20)

Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian Vladu