Presentation by Jean-Luc Dormoy (member IEEE EPPC Working Group on ICT) at the event "Incorporating Ethical Considerations in Autonomous & Intelligent Systems (A/IS) – Policy & Industry Requirements in the Algorithmic Age". The event took place on 11 June 2018 and was jointly organized by IEEE Standards Association (IEEE-SA) and the Delft Design for Values Institute (DDFV). For more info see http://designforvalues.tudelft.nl/event/incorporating-ethical-considerations-in-ai-policy-industry-requirements/

1. The Design of Autonomous
Systems: Changing the Practice
while Keeping the Former Benefits
Jean-Luc Dormoy
The Hague, 11 June 2018

2. This presentation owes much to
discussions with Pr Joseph Sifakis

3. Current systems and design practices
 Safety/Security critical (10-9), Mission critical (10-6), Best effort (10-4)
 Today based on two pillars: Verification, and V-model
 Verification of the system vs/ its requirements
• (Model,Requirements) -> Yes, No, Don‘t Know
• Model: Faithful, Generated; Requirements: Consistent, Complete
 Building models requires understanding the relationship between SW
and the underlying HW
 Verification of the implementation vs. its specification
 Verification is also about faults
 Proof of correctness is not enough, it is a social process: checked by a
trusted entity
3

4. V-model, CPS
 System requirements are known
 Development is top-down
 Correctness by checking
 ISO26262 is based on the V-Model
4
The V-model of the Systems Engineering Process, Source: Wikipedia
 Cyber-physical systems: principles and limitations
– Electrical, mechanical, fluidic, thermal...
– From static to PDEs
– Componentization, discretization, execution
 State-of-the-art still limited in an industrial context

5. Autonomous systems: The issues with
current systems and design practices
 Poor trustworthiness of infrastructures and systems
 Impossibiity to guarantee response time
 Integration of mixed criticality systems difficult to achieve
 Frequent software updates
– E.g. in aircraft systems are not changed after design is completed
– Except for the A380, critical software runs on bare metal
 Impossibility to cope with all possible mishaps at design time
5

6. New design?
 KB design
– Theoryful, theoryless knowledge
– Models, and learnt by experience (including ML models)
– Enforced at design, or left at runtime
– Can also be used for implementation
 Avoiding uncertainty and unpredictability
– Simplify HW architectures (no cache, no out-of-order execution)
– Time-deterministic behavior
– This would make adaptive systems more predictable (or more
verifiable at design time)
 Integrating adaptive monitoring and control mechanisms
6

7. The conditions for putting self-driving
cars on the road
 As soon as possible
– Only condition?
 Two other criteria
– Standardisation of safety assurance (What? How?)
– Scalability
 Should be verifiable by independent players
– Not the case today in automotive and medical systems
7

8. Thank You!
8