Slides for the paper in 18th Int. Conf. on System Design Languages (SDL 2017). We present a method and a case study on how model-based regression testing can be achieved in the context of autonomous robots. The method uses information from several domain-specific languages for modeling the robot’s context and configuration. Our approach is implemented in a prototype tool, and its scalability is evaluated on models from the case study.

1. Budapest University of Technology and Economics
Department of Measurement and Information Systems
Fault Tolerant Systems Research Group
Model-based Regression Testing of
Autonomous Robots
David Honfi, Gabor Molnar,
Zoltan Micskei, Istvan Majzik
1
18th International Conference on System Design Languages (SDL 2017)
DOI: 10.1007/978-3-319-68015-6_8

2. CONTEXT AND MOTIVATION
2

3. Context: R3-COP and R5-COP projects
3
Household service robots
Automated forklifts
http://www.elettric80.com/
http://www.care-o-bot.de
Search and rescue robots
http://www.piap.pl

4. Testing approaches
Simulating robot and environment
• Not yet widespread (but changing)
Replaying captured sensor data
• Based on real data, but coverage?
Testing with real robot in “real” environment
• Expert operators, experience-based
• Resource- and time-intensive
4

5. Standard test method: DHS-NIST-ATSM
 Testing robots in physical environment
 Standardized apparatus and procedure
 E.g.: ramp, gap, sand, sign, door opening
5
Source: NIST. Guide for Evaluating, Purchasing, and Training with Response
Robots Using DHS-NIST-ASTM International Standard Test Methods, 2014
Source: ASTM E2801-11, Standard Test Method for Evaluating Emergency
Response Robot Capabilities: Mobility: Confined Area Obstacles: Gaps, ASTM
International, West Conshohocken, PA, 2011
5

6. Previous work: Model-based approach
6
Z. Micskei, Z. Szatmári, J. Oláh, I. Majzik: A Concept for Testing Robustness and Safety of the Context-
Aware Behaviour of Autonomous Systems, TruMAS 2012. DOI
Context
model
Configuration
model
Safety
properties
Test data
generation
Few, but diverse
scenario
Test
execution

7. New challenges: autonomy, modularity…
Frequent
reconfiguration
New tasks
(+ new SW)
Swapping
HW modules
New,
unknown
environment
7

8. PROPOSED APPROACH
8

9. Reconfiguration  Regression testing
 Regression test selection (RTS)
o Rich related work for code
o Categorization: Re-usable, Re-testable, Obsolete, New
 Model-based development
o Domain-specific languages (DSL)
9
How to perform regression test selection on DSLs?

10. Regression test selection metamodel
10
Represent changes in different DSLs in one model
E.g.: (Config) Robot has a motor.
(Context) Test scenario 1 has sand terrain.
(Mapping) Motor is tested by sand in test scenario 1.

11. Architecture of the prototype tool
11
Input: Models describing
the application + tests
Output: classification
of existing test cases
Adapters to various
models and changes
RTS algorithm is
implemented only once

12. Proposed Workflow
Context and
Component
Models
Model
Transfor-
mation
Change in
the Input
Models
Reduced
Set of
Necessary
Tests
12
MDD/DSL expert Test engineer

13. EXPERIENCES AND EVALUATIONS
13

14. Experiences
 Used technologies scale well (EMF, VIATRA…)
(see paper for evaluations)
 Generic approach proved useful
o Several DSLs, several iterations
 Not easy to get abstraction level right
14

15. Model-based regression testing in action
15
Input Output

16. Summary
16