Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Automated Testing with CMake, CTest and CDash
1. Richard Thomson
Senior Software Engineer, NVIDIA
@LegalizeAdulthd
http://LegalizeAdulthood.wordpress.com
legalize@xmission.com
2. Outline
Adding test support in CMakeLists.txt
Running Tests with CTest
Viewing Test Trends in CDash
3. Tests in CMake
Tests are commands
Tests pass when:
Process return code is zero
Process output matches success regex
Process output does not match fail regex
6. add_test Arguments
<name> names the test
Multiple tests can run the same command
<command> can be a CMake target
<arg> can use generator expressions
Use absolute paths for working directory
Use COMMAND_EXPAND_LISTS if your
arguments can expand to empty strings
7. Setting Test Properties
Tests are configured through properties
set_property(
TEST <test>
PROPERTY <name> [<value>...] )
set_tests_properties(
<test> [<test> ...]
PROPERTIES <prop> <value>
[<prop> <value>...] )
Use ; separated value lists with set_tests_properties
9. Configuring Test Result
WILL_FAIL
Inverts the success/fail sense
SKIP_RETURN_CODE
If command returns this code, test is skipped
PASS_REGULAR_EXPRESSION
FAIL_REGULAR_EXPRESSION
SKIP_REGULAR_EXPRESSION
Evaluated against test output to determine test
result
10. LABELS property contains a list of label
names
Use with –L <regex> argument to ctest to
select tests to run by regex match
Organizing Tests
11. Test Dependencies
Does one test depend on another test to be
run first?
Use the DEPENDS property to specify the
relationship
Only controls order of test execution;
dependent test is run regardless of success
of the dependency
12. Test Fixtures
Use a fixture to orchestrate set up and tear
down for a test case
A fixture is just a name
Test fixture set up is just a test with
FIXTURES_SETUP property
Test fixture tear down is just a test with
FIXTURES_CLEANUP property
Test using a fixture has
FIXTURES_REQUIRED property
14. How CTest Works
enable_testing establishes the location of a
cmake script in the build directory
add_test writes commands to the cmake
script
ctest processes the script to run tests
Run ctest from the location of the cmake
script (usually top-level build directory)
15. Running CTest
ctest [args...]
-C <config> select config to run
needed for multi-config generators
-R <regex>, -E <regex>
Specify tests to run/exclude based on test name
--timeout <seconds>
--stop-time <time-of-day>
-j <n> for parallel execution
--resource-spec-file <path>
-L <regex> to select tests by label
16. Test Resource Constraints
Tests might need lots of CPU
Tests might need lots of RAM
Tests might need lots of GPU
etc.
Running tests in parallel can cause false
failures due of resource exhaustion
RUN_SERIAL property can force a test to run
alone
17. RESOURCE_LOCK Property
A resource is just a name
Tests requiring the resource have the name
in its RESOURCE_LOCK property
Only one test per named resource will run
at a time
18. RESOURCE_GROUPS
Property
Specify quantity of resources in each group
needed by a test
Configure the resources available in a
JSON file
Check the documentation for details of
specifying and configuring resources
20. GoogleTest Support 3.9
Scans source files to identify tests
Adds one CTest test case per Google Test
test case
Variable containing test names can be used
to further customize test cases
Misses parameterized tests
Misses tests defined through custom
macros
22. GoogleTest Support 3.10
Runs test executable to get test names
Tests names aren't available until CTest is
run
Can't easily further customize tests via
properties at CMake configuration time
Customization can be done by including
additional hand-written files into the
generated CTest script
23. CDash
CDash is a web based dashborad for test
results and trends (FOSS)
CTest can prepare results for submission to
CDash
CDash organizes data from pipeline steps
according to models that are displayed in
tracks in the dashboard
25. CDash Steps
Start
Update
Configure
Build
Test
Coverage
Always displays results in the Coverage track
MemCheck
Always displays results in the Dynamic Analysis
track
Submit
26. CDash Models
Every pipeline is associated with a model
Model defines default steps and error behavior
Nightly
Excludes MemCheck step, continues if Update fails
Continuous
Excludes MemCheck step, stops if Update fails
Experimental
Excludes Update and MemCheck steps
27. Executing Steps and Pipelines
ctest -M <model> -T <step> --track <track>
At least -M or -T must be specified
Needs a CDash configuration in place
ex: ctest -M Nightly --track "Nightly Master"
28. CDash Configuration
Mostly handled by CTest module
include(CTest)
After project()
CTest module defines BUILD_TESTING
Allows you to add custom test-only build code
CTest module calls enable_testing() for you
Requires CTestConfig.cmake at top-level
29. Sample CTestConfig.cmake
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "my.cdash.org")
set(CTEST_DROP_LOCATION
"/submit.php?project=${CTEST_PROJECT_NAME}")
set(CTEST_DROP_SITE_CDASH YES)
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)
30. Simpler CTestConfig.cmake
(3.14+)
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_SUBMIT_URL
"http://my.cdash.org/submit.php?project=${CTEST_PROJECT_NAME}")
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)
31. Custom Pipeline Execution
Write a custom CMake script that calls
ctest_xxx commands
Invoke with ctest -S <script>
Allows arbitrary payloads to be uploaded to
CDash
32. Custom Test Results
ATTACHED_FILES,
ATTACHED_FILES_ON_FAIL properties
specify a list of files to be attached to the
test results in the Upload step
MEASUREMENT property allows a test to
specify a single value that can be tracked
for that test in CDash
CTest snoops test output for XML fragments
to define measurements; see docs for
details.
33. Going Further
Consult the CMake, CTest and CDash
documentation for more details
More details and examples provided in
"Professional CMake: A Practical Guide" by
Craig Scott, http://crascit.com