- Learn how to "usability test" AI interactions with humans and measure success
- Understand the two distinct ways that humans construct commands to AI systems and how, using physiological measurements, you can measure the human response to the AI system responses
John Whalen explores the concept of cognitive design, describing how humans structure their commands to AI systems (syntax, word usage, prosody) and how to measure human reactions to AI responses using biometrics (facial emotion recognition, heart rate, GSR). Along the way, John shares insights into how to optimally architect the customer experience.
John offers an overview of the results of an evaluation of four major AI systems (Siri, Cortana, Alexa, and Google Assistant), tested by the young and old, those new to AI systems and those that use these tools every day, native and non-native speakers, and techies and non-techies. Each were asked to interact with the systems to request facts, complex information, jokes, commands, and calendar information while the evaluators recorded their commands, the AI response, and the human’s physiological response to the AI response (facial emotion, heart rate, and GSR).
There were several intriguing findings:
- There were two distinct ways humans constructed commands for the AI systems.
- The testers’ favorite AI systems were not always the ones that performed the best in terms of giving correct answers.
- There was a distinct physiological signature associated with a positive experience.
John explains how these findings can help you determine how you should measure the success of your AI system or chatbot and suggests new ways to predict market success that go beyond AI answer accuracy.