Striving to increase citizen engagement with politics using technology, we describe a semi automatic technique for measuring the actions of politicians in televised election debates against the audience’s democratic expectations. The method helps viewers to assess the candidates’ performances by exposing, for instance, when speakers purposefully avoid questions, make 'soundbite' remarks or mislead viewers. We evaluate the technique by correlation with audience feedback involving flashcards through which viewers indicate when their expectations are being violated.

1. Detecting Speakers’ Violation of
Democratic Entitlements in Political Debates
Brian Plüss and Anna De Liddo
Knowledge Media Institute
The Open University, UK
{brian.pluss, anna.deliddo}@open.ac.uk
edv-project.netECA Lisbon 2015, 9-12 June

2. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
The Election Debate Visualisation project
• Aim: to empower democratic citizenship by making political media events
more accessible and engaging, and by giving viewers tools to make sense
of complex political argumentation and debate.
• How: Political Communication, Computational Linguistics, Collective Intelligence
and Design brought together to provide enhanced, interactive online debate replays.

3. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
The Election Debate Visualisation project
• Aim: to empower democratic citizenship by making political media events
more accessible and engaging, and by giving viewers tools to make sense
of complex political argumentation and debate.
• How: Political Communication, Computational Linguistics, Collective Intelligence
and Design brought together to provide enhanced, interactive online debate replays.
Like YouTube, with fancy visualisations of complex analytics.

4. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

5. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

6. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

7. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

8. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

9. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

10. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

11. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

12. Context and Motivation
Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June

13. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Dialogue Rule Compliance
and Fair Play
+
Manual Annotation of
Dialogue Act Functions and
Content Features
Rules of Engagement
(76 in 2010; 28 in 2015)

14. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Dialogue Rule Compliance
and Fair Play
• Instances of violations of the rules
Static: performance of forbidden dialogue acts
Dynamic: failure to discharge discourse obligations
• Aggregated normalised scores per speaker
+
Manual Annotation of
Dialogue Act Functions and
Content Features
Rules of Engagement
(76 in 2010; 28 in 2015)

15. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Dialogue Rule Compliance
and Fair Play
• Instances of violations of the rules
Static: performance of forbidden dialogue acts
Dynamic: failure to discharge discourse obligations
• Aggregated normalised scores per speaker
Fair Play Summary
+
Manual Annotation of
Dialogue Act Functions and
Content Features
Rules of Engagement
(76 in 2010; 28 in 2015)

16. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Dialogue Rule Compliance
and Fair Play
+
Manual Annotation of
Dialogue Act Functions and
Content Features
Rules of Engagement
(76 in 2010; 28 in 2015)
• Instances of violations of the rules
Static: performance of forbidden dialogue acts
Dynamic: failure to discharge discourse obligations
• Aggregated normalised scores per speaker

17. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
Qualitative research: citizens’
perceptions of election debates
12 focus groups conducted at Leeds:
• Disengaged Voters
• Committed Party Supporters
• Undecided Voters
• First-time Voters
• Active Users of the Internet
• (Performers)
Male/Female; 8~10 people per group

18. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
Democratic entitlements
1. Be respected as a rational and independent decision-
maker
2. Be able to evaluate political claims and make an
informed decision
3. Feel part of the debate as a democratic cultural event
4. Be able to communicate with and be recognised by the
leaders who want to represent me
5. Be able to make a difference to what happens in the
political world
Qualitative research: citizens’
perceptions of election debates
12 focus groups conducted at Leeds:
• Disengaged Voters
• Committed Party Supporters
• Undecided Voters
• First-time Voters
• Active Users of the Internet
• (Performers)
Male/Female; 8~10 people per group

19. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements

20. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements

21. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements

22. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
• Not with the rules of the broadcasters and political parties,
but with the rules of the citizens.
As before, only…

23. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
• Not with the rules of the broadcasters and political parties,
but with the rules of the citizens.
Examples:
- Answer the question
- Do not mislead
- Do not insult the audience
As before, only…

24. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
• Not with the rules of the broadcasters and political parties,
but with the rules of the citizens.
Examples:
- Answer the question
- Do not mislead
- Do not insult the audience
• Crucially, on the same coding scheme.
As before, only…

25. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
• Static obligations: role-based allowed and disallowed dialogue acts.
Example:
- Speaker must not insult the audience
Rules formalised to capture discourse obligations:

26. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
• Static obligations: role-based allowed and disallowed dialogue acts.
Example:
- Speaker must not insult the audience
• Dynamic obligations (introduction and discharge): dialogue acts to be
performed as a consequence of actions of self or others.
Example:
- When asked a valid question, addressee must provide a valid answer
- Providing the information requested for by an question, discharges the
obligation to provide an valid answer.
Rules formalised to capture discourse obligations:

27. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Democratic Entitlements
and Fair Play
Init-Inform
Follows | Interrupts
Polite | Impolite
On-Topic | Off-Topic
Accurate | Inaccurate
New | Repeated
Init-InfoRequest
Follows | Interrupt
Polite | Impolite
On-Topic | Off-Topic
Neutral | Loaded
Reasonable | Unreasonable
New | Repeated
Resp-Inform
Follows | Interrupts
Polite | Impolite
Relevant | Irrelevant
Accurate | Inaccurate
New | Repeated
Complete | Incomplete
Coding scheme
Resp-Accept
Follows | Interrupts
Polite | Impolite
Resp-Reject
Follows | Interrupts
Polite | Impolite
Init-Cue
Follows | Interrupts
Polite | Impolite
Init-Greeting
Follows | Interrupts
Polite | Impolite
Resp-Greeting
Follow | Interrupts
Polite | Impolite

28. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback
A method for gauging audience responses
to political media events in real-time
• Controlled and nuanced
• Engaging, voluntary and non-intrusive
• Suitable for analytics and visualisations

29. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback
A method for gauging audience responses
to political media events in real-time
• Controlled and nuanced
• Engaging, voluntary and non-intrusive
• Suitable for analytics and visualisations

30. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

31. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

32. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

33. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

34. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

35. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Evaluation through
Instant Audience Feedback

36. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Current and Future Work
• Annotation and visualisation of the April UK election debates

37. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Current and Future Work
• Annotation and visualisation of the April UK election debates

38. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Current and Future Work
• Annotation and visualisation of the April UK election debates
• Evaluation
Reliability: inter-annotator agreement study
Validity: correlation with instant audience feedback

39. Brian Plüss and Anna De Liddo edv-project.netECA Lisbon 2015, 9-12 June
Current and Future Work
• Annotation and visualisation of the April UK election debates
• Evaluation
Reliability: inter-annotator agreement study
Validity: correlation with instant audience feedback
• Explore automation of annotations

40. Thank you!
Brian Plüss and Anna De Liddo
Knowledge Media Institute
The Open University, UK
{brian.pluss, anna.deliddo}@open.ac.uk
edv-project.netECA Lisbon 2015, 9-12 June