A helicopter ride over the Ansible testing landscape.

1. Anth Courtney, Solution Architect, anth.courtney@cmdsolutions.com.au
Adam Durbin, Director of Strategy and Architecture, adam.durbin@cmdsolutions.com.au
A helicopter ride over the
Ansible testing landscape

2. Overview
Why should we test our Ansible artifacts?
What can we test?
What should we test?
How should we test?

3. But first, a contrived service!

4. Testing Our Playbook
We can immediately start to test our playbook, without
the need to write large amounts of additional code.
Test the syntax of our playbook and roles
Evaluate the repeatability of our playbooks and roles.
Validate configuration files generated by jinja2
templates.

5. Syntax Checking Playbooks and Roles
What –syntax-check does do
 Validates the syntax of a playbook
 Validates the syntax of any included
roles.
 Validates presence of any other
includes. i.e. other playbooks,
vars_files.
 Validates that tasks/modules
specified within a playbook or role
are valid.
What –syntax-check doesn't do
 Doesn’t execute a playbook.
 Doesn’t test the validity of
parameters to tasks.
 Doesn’t test for the existence of
undefined variables.
 Doesn’t test the ability to
sudo/become as defined.
 Doesn’t test the validity of any in-
line jinja2.
$ ansible-playbook –syntax-check <playbook>

6. "Ask yourself,
what would a build
agent do?"
Rob Shand, General DevOps Legend
Typically, we want to:
 Create an environment or
workspace for execution of
our playbook.
 Prepare the environment
or workspace for execution
of the playbook.
 Apply the playbook
against the target
environment.
 Verify the playbook
execution.
 Destroy the environment
or workspace.

7. Testing Idempotency
An operation is idempotent if the result of performing it once is exactly the same as the
result of performing it repeatedly without any intervening actions.
First run of playbook:
PLAY RECAP
*********************************************************************
default : ok=7 changed=10 unreachable=0 failed=0
Nth run of playbook:
PLAY RECAP
*********************************************************************
default : ok=17 changed=0 unreachable=0 failed=0

8. Non-Idempotent Playbooks and Roles
Non-idempotent playbooks and roles can be indicative of:
 A resource being changed multiple times within the same playbook run.
 A conditional that isn’t being evaluated correctly.
 Tasks not being properly configured for idempotence.
- name: Install custom application - non-idempotent approach
command: /tmp/application-1.0.0/bin/install --dest /opt/application
- name: Install custom application - idempotent approach
command: /tmp/application-1.0.0/bin/install --dest /opt/application
args:
creates: /opt/application

9. Use Case: Using Idempotency, Check-Mode
and Diffs for Security

10. Testing Jinja2 Templates
For copy and template tasks, Ansible provides the ability to perform validation
before copying into place.
- name: Create nginx configuration
template:
src: "etc/nginx/nginx.conf.j2"
dest: "/etc/nginx/nginx.conf"
validate: "nginx -t -c %s"
notify: Reload nginx

11. Testing Jinja2 Templates
For copy and template tasks, Ansible provides the ability to perform validation before
copying into place.
- name: Copy nginx modular configuration
template:
src: "etc/nginx/conf.d/modules.conf.j2"
dest: "/etc/nginx/conf.d/modules.conf"
validate: "echo 'http { include '%s'; }' > /tmp/nginx.conf && nginx -t -c /tmp/nginx.conf"
notify: Reload nginx

12. Batteries Included!
Ansible includes a number of modules which are test-orientated:
 assert
 fail
 stat
 wait_for
 uri
 command

13. Built-in Test Module: assert
- name: Check ansible version is suitable for this playbook
assert:
that: ansible_version.full | version_compare('2.1', '>=')
msg: "Ansible version 2.1 or greater is required for this playbook"

14. Built-in Module: uri
- name: Verify that the nginx status page is reachable
uri:
url: "http://{{ inventory_hostname }}/status"
register: result
until: result.status == 200
retries: 60
delay: 1

15. Using 'assert' With Multiple Conditions
- name: Make authenticated request
uri:
url: "http://localhost/api/users/anth"
method: GET
force_basic_auth: yes
user: "admin"
password: "admin123"
register: api_request
- name: Validate that api request response contains required information
assert:
that:
- api_request.json.name is defined
- api_request.json.username == 'anth'
- api_request.json.email | search("@")

16. Built-in Test Module: stat
- name: Determine if ssh private key exists
stat:
path: /home/ec2-user/.ssh/id_rsa
register: ssh_private_key
- name: Validate that ssh private key exists and has correct mode
assert:
that:
- ssh_private_key.stat.exists
- ssh_private_key.stat.mode == "0640"
msg: "ssh private key does not exist or has wrong mode"

17. Built-in Test Module: fail
 Used to ‘bail out’ of the execution of a playbook when a certain condition is met.
 Uses when to provide the condition for termination.
- name: Check ansible version is suitable for this playbook
fail:
msg: "Ansible version 2.1 or greater is required for this playbook"
when: ansible_version.full | version_compare('2.1', '<')

18. Built-in Module: wait_for
 Can be used to wait for a port to become available
 Can be used to wait for a file state to change or for a string/regex to appear
 Can also be used to wait for active connections to be closed before continuing
- name: Verify that cron ran configured @reboot jobs
wait_for:
path: "/var/log/cron"
search_regex: "Running @reboot jobs"

19. Built-in Parameter: failed_when
 Provides the ability to define ‘what constitutes failure?’ for a task.
- name: Run command to test status of application
command: /opt/cmd/bin/test_application.sh
register: command_result
failed_when: "'FAILED'" in command_result.stdout

20. How Do People Test Ansible Playbooks?
 From experience, we’ve typically seen a ‘whole box and dice’ approach.
 Take a ‘test everything’ approach, including unit and functional testing.
 Use tools like ServerSpec, Goss, Ansible, etc.
 Increased cost due to duplication of effort and maintenance of tests.

21. Test the outcome, not the
implementation.
Michael De Haan, Creator of Ansible

22. The Golden Rule of Testing in Ansible
 Ansible’s modules are designed to “achieve state” and “fail fast”.
 It is not necessary to test that services are started, packages are installed,
etc.
 Try to keep it simple.
“Any intelligent fool can make things bigger, more complex, and more violent. It
takes a touch of genius—and a lot of courage to move in the opposite direction.”
- E. F. Schumacher in “Small Is Beautiful”.

23. A Suggested Playbook Testing Pattern
For a playbook, have the ability to apply the playbook (and run the embedded
functional tests) against a virtual environment.
$ make USE_VM=true clean setup <--- Applies playbook to virtual environment
$ make USE_VM=true test <--- Runs functional test against
virtual environment
For a playbook, have embedded functional tests which can be toggled on and off by
feature flags or tags.
$ make clean setup <--- Applies playbook to environment
$ make test <--- Runs functional tests against environment

24. A Suggested Role Testing Pattern
The objectives of testing a role should be similar to that of a playbook:
 Is the YAML syntax of my role correct?
 Is the role built in an idempotent way?
 Does the role run through all tests without failing?
For a role, have the ability to apply the role and run functional tests against a
virtual environment.
$ make clean setup test <--- Applies role to a VM and runs functional tests

25. I Like Unit Tests! I Do! I Like Them, Sam I Am
If you want to unit test your Ansible playbooks or roles, try to keep these
principles in mind:
 The unit tests should only test a single associated task.
 The unit tests should be tightly associated with the related task that they are
testing.
 The unit tests are read-only in nature.
 The use of tags allows selected execution of the unit tests.

26. Is Your Role Worthy Of A Gold Star?
The maintainers of Ansible Galaxy are discussing automated manners to give a gold
star or other indicator to a role that meets certain criteria which are an indicator of
quality.
 Documentation should exist
 Documentation should include a definition of all default variables
 travis.yml should include asserts and be more than just a syntax check
 Check that the role does not call deprecated modules
 Check that the role does not use deprecated syntax
 Check that all variables have a default or are included in vars/main.yml
 Role repo should not include issues that have been open for a long time

27. Playbook and Role Testing Using Tox
Meet Tox: https://tox.readthedocs.io/en/latest/
$ pip install tox
“tox aims to automate and standardize testing in Python.”
 Allows for testing playbooks/roles against different versions of ansible.
 Allows for testing playbooks/roles against different versions of python.
 Works on Windows and Linux.

28. Role Testing Using Molecule
Meet Molecule: http://molecule.readthedocs.io/
$ pip install molecule
“Molecule is designed to aid in the development and testing of Ansible roles..."
 Supports multiple drivers, including Docker, Vagrant and OpenStack.
 Supports multiple providers, including Virtualbox, Libvrt, Parallels, and VMware
Fusion.
 Supports multiple verifiers, including Testinfra, Serverspec, Goss
 Also includes syntax validation and ansible-lint checking.

29. Molecule is CI Friendly!
Molecule provides a number of commands which are ‘continuous integration’ friendly.
$ molecule create <--- Creates instance(s)
$ molecule converge <--- Apply ansible playbook to instance(s)
$ molecule idempotence <--- Validates idempotence of role
$ molecule verify <--- Runs tests against converged instance(s)
$ molecule destroy <--- Destroys instance(s)
By default, all of these commands are executed when running:
$ molecule test.

30. Miscellaneous Tips and Tricks
 Use ‘blocks’ to execute all tests while delaying assertions until the end.
 Use an ‘MVP’ playbook pattern to test ‘is this how you do it?’ type things.
 Use the ping module to check the readiness of hosts.

31. Summary
 Having one test for a playbook or role is better than zero tests.
 Ask “what would a build agent do?” and model your approach accordingly.
 Test the outcome, not the implementation.