For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/09/incorporating-continuous-user-feedback-to-achieve-product-longevity-in-chaotic-environments-a-presentation-from-observa/ Erik Chelstad, CTO and Co-founder of Observa, presents the “Incorporating Continuous User Feedback to Achieve Product Longevity in Chaotic Environments” tutorial at the May 2022 Embedded Vision Summit. In many computer vision applications, a key challenge is maintaining accuracy when the real world is changing. In this presentation, Chelstad explores techniques for designing hardware and software to stave off obsolescence by allowing users to interact with the output of the computer vision system as quickly and cheaply as possible, so that future errors are avoided. He also discusses ways to motivate users to provide useful feedback, and illustrates concepts using a deployed commercial application.

1. Incorporating Continuous
User Feedback to Achieve
Product Longevity in
Chaotic Environments
Erik Chelstad
CTO & Co-Founder
Observa

2. What is a chaotic environment?
• Fast paced
• Rapidly changing
• Uncontrollable external factors
© 2022 Observa, Inc. 2

3. What is a product and who is a user?
• Products
• Hardware
• Software
• Users
• Single human
• Teams or companies
• Autonomous entities
© 2022 Observa, Inc. 3

4. History of Feedback
(not a rock band)

5. Traditional user feedback
• Purposes
• Bug reporting
• Business development
• Mollify users
• Future strategy
• Measure user satisfaction
© 2022 Observa, Inc. 5
• Methods
• Surveys (polling)
• Social media
• Customer support
• Live chat
• Customer success

6. Traditional user feedback
• Problems
• Slow
• Too late
• Focus on negative issues
• Not enough data
• Qualitative
• Reactive – bug fixes
© 2022 Observa, Inc. 6

7. New reasons for getting user feedback
• Market fit
• Proactive releases make users
feel good
• Personalization
• Users no longer expect
products and systems to be
static
• Stay ahead of competition and
disruptors
© 2022 Observa, Inc. 7
• Environment fit
• Natural experiments
• Controlled experiments
• New scenarios and usages
• Changing environment
• New things appearing
• Old things disappearing

8. No excuses for not getting user feedback
• Data harvesting
• Storage is cheap
• Sensors are cheap
• Processing is cheap
• Users are NOT cheap
• Domain specific data is
invaluable
© 2022 Observa, Inc. 8
• Data collection
• Constant feedback is possible
• Products NEED some
connectivity for updates, etc.

9. Designing for Feedback
(not a section on guitar amps)

10. Design – When?
• 😀 Immediately!
• Build into presentation
• 🙂 Soon
• Within a usage session
• 😐 Before too long
• Before a user forgets
everything
© 2022 Observa, Inc. 10

11. Design – How?
• With interaction
• Ease of use
• Not distracting
• Optional
• Not good:
© 2022 Observa, Inc. 11
• Simple & quick:
• Logging off or shutting
down

12. Design – How?
• Passive data collection
• Did they stay on the results
screen for a while or bounce?
• Did the user upload the
results they got back to HQ?
• Are they redoing the same
task?
© 2022 Observa, Inc. 12
• Did they save the results to
their favorites?
• Did they mash the button or
yell at their device?

13. Design – How?
• Non-human
• Did the navigation system
make a sudden correction or
cause human interaction?
© 2022 Observa, Inc. 13
• Did a plotted route result in a
stuck robot?

14. Design – Why?
• Why would a person do it?
• How rewarded?
• Ideas like recognition - your
input was used in the latest
release and performance fixes
for efficiency
• Frustration
• Happiness
© 2022 Observa, Inc. 14
• Why would a machine or
corporation do it?
• Because their humans told
them to
• Maximize ROI on fixed costs
• Extend useful life
• No new training

15. Bringing it Back In

16. When to integrate
• The other side of the collection coin is that
you must actually do something with the
feedback!
• Make it part of the data pipeline
• Automate as much cleaning and validation as
possible
• Make it part of the update workflow
• Review with humans as needed
© 2022 Observa, Inc. 16

17. How to integrate
• Use locally
• Example - Auto adjust overlap
tolerance on panoramic
camera based on user
accepting/rejecting their own
photos
© 2022 Observa, Inc. 17
• At the edge
• Upload back to servers in
batches or at scheduled
intervals
• Always connected
• Can be real-time

18. Case Studies

19. Examples from Observa (my company)
• Viewing a report allows for
corrective input at any time
• No need to “correct” the
system, just report mistakes
• On any chart of a report
• By any viewer of the report
• Remove from reporting
immediately
© 2022 Observa, Inc. 19

20. Examples from Observa (my company)
• Low confidence results “passively” come back for human review in the pipeline
© 2022 Observa, Inc. 20
Incoming
Image
IR System
Low
Confidence
Results
Label Tool
High Priority
Review
Training
Data
Reporting
High
Confidence
Results
Low Priority
Review
Training
Data

21. Examples from Observa (my company)
• Observa app when field
service users are in front
of the shelf
• Passive if they agree or
need to reanalyze
© 2022 Observa, Inc. 21

22. In Conclusion

23. In Conclusion
• User feedback keeps your product relevant
• User feedback can be collected cheaply and
unobtrusively if you design for it
• Your user feedback is unique and
unobtainable by competitors
• Users are motivated to help you:
• Intrinsically
• Financially
• Emotionally
© 2022 Observa, Inc. 23

24. Resources
Natural Experiments
https://www.nobelprize.org/uploads/2021/10/advanced-
economicsciencesprize2021.pdf
The Virtuous Cycle of AI Products
https://www.eriktrautman.com/posts/the-virtuous-cycle-of-ai-products
Evolution of intelligent data pipelines
https://www.technologyreview.com/2021/12/06/1040716/evolution-of-
intelligent-data-pipelines/
© 2022 Observa, Inc. 24