1
Governance of trustworthy AI
Samos summit 2020
Prof.dr.ir. Marijn Janssen
@HMarijn
Delft University of Technology
Faculty of Technology, Policy & Management

2
Use of AI in our daily lives
• Avi Schiffmann, web person of the year
• Covid-19 case tracker: https://ncov2019.live/
• providing concise, instantly updated information.
• Used by epidemiologists it to predict the disease’s
spread

3
AI and computational algorithms
developments*
M. Janssen & G. Kuk (2016). The challenges and limits of big data algorithm in technocratic
governance. Government Information Quarterly. Vol. 33, No. 4, pp.371-377. DOI
http://dx.doi.org/10.1016/j.giq.2016.08.011
checking of identity
by civil servants
calculation of
social benefits
admissions of
immigrants
Identifying security
threat on airports

4
What can go wrong?
• Data is no correct or
changes result in error
– Google Flu Trends
– Centers for Disease
Control and Prevention
(CDC) prediction
• Algorithms work
differently than expected
– Husky vs. wolf experiment
– Neural network learned to
differentiate between
snow and grass
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014).
The parable of Google Flu: traps in big data
analysis. Science, 343(6176), 1203-1205.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). Why
should i trust you?: Explaining the predictions of any
classifier. In Proceedings of the 22nd ACM SIGKDD
international conference on knowledge discovery and data
mining (pp. 1135-1144). ACM.

5
What is happening?
• More and diverse data
• Diverse algorithms to process the data
• High complexity and (near) real-time
processing
• How do you know it is working properly?
Impact? How is the situation governed?
• Oversight mechanisms are needed

6
What is trustworthy AI?*
• Trustworthy AI should be*
– lawful - respecting all applicable laws and
regulations
– ethical - respecting ethical principles and
values
– robust - both from a technical perspective
while taking into account its social
environment
* https://ec.europa.eu/digital-single-market/en/news/ethics-
guidelines-trustworthy-ai

7
Key requirements on trustworthy AI*
• Human agency and oversight:, proper oversight mechanisms
needed
• Technical Robustness and safety: AI systems need to be resilient
and secure, accurate, reliable and reproducible.
• Privacy and data governance: into account the quality and
integrity of the data, and ensuring legitimised access to data.
• Transparency:. Traceability mechanisms can help achieving this.
Moreover, AI systems and their decisions should be explained .
• Diversity, non-discrimination and fairness: Unfair bias must be
avoided,
• Societal and environmental well-being: AI systems should benefit
all human beings, including future generations. It must hence be
ensured that they are sustainable and environmentally friendly.
• Accountability: Responsibility and accountability for AI systems
and their outcomes. Auditability is needed.
* https://ec.europa.eu/digital-single-market/en/news/ethics-
guidelines-trustworthy-ai

8
Materiality of governance*
* Janssen, Marijn & Kuk, George (2016). Big and Open Linked Data (BOLD) in Research, Policy and
Practice. Journal of Organizational Computing and Electronic Commerce, Vol. 26, no 1-2, pp. 3-13.
DOI 10.1080/10919392.2015.1124005
“Data processing becomes increasingly
autonomous and invisible, they become harder for
the to detect and scrutinize”*

9
What happens without governance?
• Is the information chain secure and governed? How will be dealt
with attacks and outsiders trying to influence?
• How will the system be improved and complexity reduced?
• Risk of over control; adding too much governance is counter
effective and will only add to the administrative burden and create
more complexity
• Complexity increases
and patching the
system
• Who has the overview?
• Who takes actions
when something goes
wrong?
• How will the use of
wrong data/algorithms
be detected?

10
Different speeds of governance*
* Janssen, M., & Van Der Voort, H. (2016). Adaptive governance: Towards a stable, accountable and
responsive government. Government Information Quarterly, Vol. 33, no. 1, pp. 1-5.
What is the speed of AI?

11
Tripartite forms of governance
• Various approaches to governance
• With big data algorithmic systems a risk-
based approach is crucial, in addition to
other approaches
• Risks are hard to assess and determine in
advance, therefore procedures are important
planning and
control
organizational Risk-based
*Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020). Data governance:
Organizing data for trustworthy Artificial Intelligence. Government Information Quarterly, Vol. 7, no 3,
110493, https://doi.org/10.1016/j.giq.2020.101493.

12
Governance arrangements
• Need for conceptualizing from a (information) chain
perspective
• Value of data is created when it results in better decisions
• Joint decision-making and co-regulation
• Monitoring of exceptions, regular samples audits and other
processes

13
Governance model*
*Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020). Data
governance: Organizing data for trustworthy Artificial Intelligence. Government
Information Quarterly, Vol. 7, no 3, 110493, https://doi.org/10.1016/j.giq.2020.101493.

14
Data governance
• Determine data stewardship
• Agreements about data quality
• Regular assessments
• What happens when changes are
introduced/noticed
• Observe external environment

15
Algorithmic governance
• Determine algorithms stewardship
• Dealing with exceptions
• What happens when changes are
introduced/noticed
• From opaque to explainable AI
• For accountability of rules it is important to show
the causality and rules used to ensure fair and
equality.
1. Use AI techniques (M) to derive rules
2. Let people decide on the rule (accountability)
3. Use the causal results and can be explained
4. Use IA techniques (ML) for improving the rules

16
Trusted data frameworks*
• A trusted framework contains several elements to regulate
data sharing or other types of services, which may
include:
• a list of requirements for trusted information sharing;
• a set of standards for realizing trusted information sharing;
• a collection of contracts and agreements for trusted
information sharing;
• an authorization scheme who should have access to
which data under what circumstances;
• a certification mechanism to record adherence of different
parties to the rules;
• an auditing mechanism to verify compliance with
requirements, agreements, contracts; and
• mechanisms to enforce compliance with the rules and
agreements.
*Janssen et al. (2020) Data Governance: Organizing Data for Trustworthy Artificial
Intelligence. Government information quarterly.

17
Recommendations
• Key is understanding what needs to be governed
• Materiality of algorithms: take into account the broader
socio-tech context of humans, data and systems
• Mindfulness use of AI – is the ideal possible?
• In addition to control cycle and hierarchical accountability,
take a risk-based approach
• Take institutional measures, e.g. ensure an independent
oversight body
• Governance should offer various forms of defense against
mistakes and undesired influences
• Provide support for contesting outcomes, sensitivity
analysis, auditing, and having independent experts looking
at different concerns are key
• Appropriate governance is rigid but creates agility

18
m.f.w.h.a.janssen@tudelft.nl
Questions?