1. Industry Standards as vehicle to
address socio-technical challenges
from AI – the case of
Algorithmic Bias Considerations
ANSGAR KOENE,
HORIZON DIGITAL ECONOMY RESEARCH INSTITUTE, UNIVERSITY OF NOTTINGHAM
ANSGAR.KOENE@NOTTINGHAM.AC.UK
4TH FEBRUARY 2019
1

2. Undesired impact
THE NEED FOR ETHICS, LEGAL, SOCIAL, ECONOMIC INTERVENTION
2

3. Algorithms in the news
3

4. Algorithmic Discrimination
4

5. Complex individuals reduced to
simplistic binary stereotypes
5

6. Case study: Recidivism risk prediction
6
Is the algorithm fair
to all groups?
When base rates differ, no non-trivial solution can achieve similar FPR,
FNR, FDR, FOR!
Machine Bias: There’s
software used across the
country to predict future
criminals. Propublica
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias. ProPublica.
https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.

7. Arkansas algorithmic Medicaid
assessment instrument
When introduced in 2016, many people with cerebral palsy had their care dramatically
reduced – they sued the state resulting in a court case.
Investigation in court revealed:
The algorithm relies on 60 answer scores to questions about descriptions, symptoms and
ailments. A small number of variables could matter enormously: a difference between a
three instead of a four on a handful of items meant a cut of dozens of care hours a month.
One variable was foot problems. When an assessor visited a certain person, they wrote that
the person didn’t have any foot problems — because they were an amputee and didn’t
have feet.
Third-party software vendor implementing the system, mistakenly used a version of the
algorithm that didn’t account for diabetes issues.
Cerebral palsy, wasn’t properly coded in the algorithm, causing incorrect calculations for
hundreds of people, mostly lowering their hours.
https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy

8. Fairness is fundamentally a societally defined construct (e.g. equality of
outcomes vs equality of treatment)
◦ Cultural differences between nations/jurisdictions
◦ Cultural changes in time
“Code is Law”: Algorithms, like laws, both operationalize and entrench spatio-
temporal values
Algorithms, like the law, must be:
◦ transparent
◦ adaptable to change (by a balanced process)
The ‘messy’ problem of fair algorithms
8

9. Algorithmic systems are socio-technical
Algorithmic systems do not exist in a vacuum
They are built, deployed and used:
◦ by people,
◦ within organizations,
◦ within a social, political, legal and cultural context.
The outcomes of algorithmic decisions can have significant impacts
on real, and possibly vulnerable, people.

10. Governance frameworks
for algorithmic systems
PUBLIC AND PRIVATE SECTOR RESPONSES TO UNDESIRED IMPACTS OF AI
10

11. 11
EU response (in addition to GDPR)
EU Parliament Science and Technology Options Assessment
(STOA) panel request for study on “Algorithmic Opportunities
and Accountability”

12. Governance options
12
Florian Saurwein, Natascha Just, Michael Latzer, (2015) "Governance of algorithms: options and limitations",
info, Vol. 17 Issue: 6, pp.35-49, doi: 10.1108/info-05-2015-0025

13. Demand side Market solutions
13

14. Supply side Market solutions
14

15. Companies self-organization (aka CSR)
15

16. Branches self-regulations
16

17. Co-regulation
17

18. State Intervention
18

19. Oversight by regulatory organisations
An FDA for algorithms – Andrew Tutt (2016)
An FAA for algorithms – paraphrasing Alan Winfield
19

20. Setting legal requirements
20

21. Ethics Principles &
Industry Standards
21

22. ACM Principles on Algorithmic
Transparency and Accountability
Awareness
Access and Redress
Accountability
Explanation
Data Provenance
Auditability
Validation and Testing
22

23. FAT/ML: Principles for Accountable Algorithms
and a Social Impact Statement for Algorithms
Responsibility: Externally visible avenues of redress for adverse effects, and
designate an internal role responsible for timely remedy of such issues.
Explainability: Ensure algorithmic decisions and data driving those decisions can
be explained to end-users/stakeholders in non-technical terms.
Accuracy: Identify, log, and articulate sources of error and uncertainty so that
expected/worst case implications can inform mitigation procedures.
Auditability: Enable third parties to probe, understand, and review algorithm
behavior through disclosure of information that enables monitoring, checking,
or criticism, including detailed documentation, technically suitable APIs, and
permissive terms of use.
Fairness: Ensure algorithmic decisions do not create discriminatory or unjust
impacts when comparing across different demographics.
https://www.fatml.org/resources/principles-for-accountable-algorithms 23

24. Artificial Intelligence at Google
Our Principles
24https://dzone.com/articles/ethical-ai-lessons-from-google-ai-principles

25. Are ethics principles the unenforcable
‘self-regulation’?
Wagner, B. (2018). Ethics as an Escape from Regulation: From
ethics-washing to ethics-shopping? In M. Hildebrandt (Ed.), Being
Profiling. Cogitas ergo sum. Amsterdam University Press.
25

26. 26

27. 27

28. 28
IEEE P70xx Standards Projects
IEEE P7000: Model Process for Addressing Ethical Concerns During System Design
IEEE P7001: Transparency of Autonomous Systems
IEEE P7002: Data Privacy Process
IEEE P7003: Algorithmic Bias Considerations
IEEE P7004: Child and Student Data Governance
IEEE P7005: Employer Data Governance
IEEE P7006: Personal Data AI Agent Working Group
IEEE P7007: Ontological Standard for Ethically Driven Robotics and Automation Systems
IEEE P7008: Ethically Driven Nudging for Robotic, Intelligent and Autonomous Systems
IEEE P7009: Fail-Safe Design of Autonomous and Semi-Autonomous Systems
IEEE P7010: Wellbeing Metrics Standard for Ethical AI and Autonomous Systems
IEEE P7011: Process of Identifying and Rating the Trustworthiness of News Sources
IEEE P7012: Standard for Machines Readable Personal Privacy Terms

29. Related standards activities
British Standards Institute (BSI) – BS 8611 Ethics design and application of robots
ISO/IEC JTC1 SC42
◦ Artificial Intelligence Concepts and Terminology
◦ Framework for Artificial Intelligence Systems Using Machine Learning
◦ SG 2 on Trustworthiness
◦ transparency, verifiability, explainability, controllability, etc.
◦ robustness, resiliency, reliability, accuracy, safety, security, privacy, etc.
Jan 2018 China published “Artificial Intelligence Standardization White Paper.”
29

30. 30

31. P7003 - Algorithmic Bias Considerations
All non-trivial* decisions are biased
We seek to minimize bias that is:
◦ Unintended
◦ Unjustified
◦ Unacceptable
as defined by the context where the system is used.
*Non-trivial means the decision space has more than one possible outcome and the choice is not
uniformly random.

32. Causes of algorithmic bias
Insufficient understanding of the context of use.
Failure to rigorously map decision criteria.
Failure to have explicit justifications for the chosen criteria.

33. Key question when developing or
deploying an algorithmic system
33
 Who will be affected?
 What are the decision/optimization criteria?
 How are these criteria justified?
 Are these justifications acceptable in the context where the
system is used?

34. 34
P7003 foundational sections
 Taxonomy of Algorithmic Bias
 Legal frameworks related to Bias
 Psychology of Bias
 Cultural aspects
P7003 algorithm development sections
 Algorithmic system design stages
 Person categorization and identifying affected population groups
 Assurance of representativeness of testing/training/validation data
 Evaluation of system outcomes
 Evaluation of algorithmic processing
 Assessment of resilience against external manipulation to Bias
 Documentation of criteria, scope and justifications of choices

35. 35

36. 36

37. Thank you – Questions?
37
https://reentrust.org/
ansgar.koene@nottingham.ac.uk