Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
1
User Sentiment to Determine App Quality
Heidi Young
VP Engineering, Applause
User Sentiment to Determine App Quality
Heidi Young
VP of Engineering
2
Getting to Know Each Other
3
Who am I?
• Roles/Titles: Engineer, Data Scientist, Software Junkie
• Worked at/for:
– Applause (App Quality)
– Bing (Micr...
Applause: Who we are and what we do
5©2015 Apigee. All Rights Reserved.
Level Setting
6
Building an App is Easy
7
Step 1: Write code Step 2: Ship to App
Store
©2015 Apigee. All Rights Reserved.
Building a Good App is Hard
• Device Fragmentation
• Multiple App Stores
• Native vs Hybrid vs Web
• Server/Client Tradeof...
Building a High Quality App is Harder
• No crashes
• High star ratings
• Lots of downloads
• Happy customers
9©2015 Apigee...
Different Needs at Different Times
10©2015 Apigee. All Rights Reserved.
Ideation
Design
Development
TestingBeta
Production...
Ideation Design Development Testing Beta Production Maintenance
We think about things like this
11©2015 Apigee. All Rights...
User Sentiment
12
User Reviews and Feedback are Important
1. They are very public
2. They end up making headlines
3. They inform user downlo...
Store Star Rating vs Review Star Rating
• In a lot of cases, what you can extract from review text is very different than ...
A Systematic Approach to Measuring App Quality
15©2015 Apigee. All Rights Reserved.
Extracting something out of the noise
16©2015 Apigee. All Rights Reserved.
• Step 1: Classify Every Review
• Step 2: Use r...
Using Signals to Help with Classification
17©2015 Apigee. All Rights Reserved.
Applause Analytics Scoring and Sentiment
Analysis
18©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Acr...
Applause Analytics Scoring and Sentiment
Analysis
19©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Acr...
Applause Analytics Scoring and Sentiment
Analysis
20©2015 Apigee. All Rights Reserved.
• Histogram of Attribute Scores Acr...
Opens Up Competitive Benchmarking
21©2015 Apigee. All Rights Reserved.
Applause Analytics Clustering and Ranking
22©2015 Apigee. All Rights Reserved.
The How:
• Modified Agglomerative Clusterin...
So… what does this help us do?
23©2015 Apigee. All Rights Reserved.
Smarter, More Context to Alert You to the Right Issues
24©2015 Apigee. All Rights Reserved.
Step 1: Use ratings and review...
Food for thought: Pre- vs Post- Release Issues
25©2015 Apigee. All Rights Reserved.
Top Pre Release Bug Issues
– Creating/...
Food for thought: Pre- vs Post- Release Issues
26©2015 Apigee. All Rights Reserved.
Top Pre Release Bug Issues
– Creating/...
Credit Where Credit is Due
• Everyone at Applause!
• Especially the people doing all of the actual work behind Applause An...
Próxima SlideShare
Cargando en…5
×

User Sentiment to Determine App Quality

477 visualizaciones

Publicado el

User Sentiment to Determine App Quality
Heidi Young, VP Engineering, Applause discusses user sentiment as an approach to determine app quality at I Love APIs 2015

Publicado en: Software
  • Sé el primero en comentar

User Sentiment to Determine App Quality

  1. 1. 1 User Sentiment to Determine App Quality Heidi Young VP Engineering, Applause
  2. 2. User Sentiment to Determine App Quality Heidi Young VP of Engineering 2
  3. 3. Getting to Know Each Other 3
  4. 4. Who am I? • Roles/Titles: Engineer, Data Scientist, Software Junkie • Worked at/for: – Applause (App Quality) – Bing (Microsoft, Search) – Rotella Capital Management (Financial Industry) – SAIC (Defense Contracting) – Other smaller startups • Passionate about: – Search – Information Retrieval – Optimizing and Quantifying Large Data Sets • Member of the Apigee Technical Advisory Board 4©2015 Apigee. All Rights Reserved.
  5. 5. Applause: Who we are and what we do 5©2015 Apigee. All Rights Reserved.
  6. 6. Level Setting 6
  7. 7. Building an App is Easy 7 Step 1: Write code Step 2: Ship to App Store ©2015 Apigee. All Rights Reserved.
  8. 8. Building a Good App is Hard • Device Fragmentation • Multiple App Stores • Native vs Hybrid vs Web • Server/Client Tradeoffs • International, globalization, localization, etc. • Security 8©2015 Apigee. All Rights Reserved. ? ? ?
  9. 9. Building a High Quality App is Harder • No crashes • High star ratings • Lots of downloads • Happy customers 9©2015 Apigee. All Rights Reserved.
  10. 10. Different Needs at Different Times 10©2015 Apigee. All Rights Reserved. Ideation Design Development TestingBeta Production Maintenance
  11. 11. Ideation Design Development Testing Beta Production Maintenance We think about things like this 11©2015 Apigee. All Rights Reserved. • Use humans to touch your apps in the wild, Who better than an actual user? • Use software to collect data and measure quality of your your apps, Too much data, too little time • Use data to analyze where to focus, Actionable Metrics are the only way to get things done Crowd Testers SD K User Sentiment
  12. 12. User Sentiment 12
  13. 13. User Reviews and Feedback are Important 1. They are very public 2. They end up making headlines 3. They inform user download decisions 13©2015 Apigee. All Rights Reserved.
  14. 14. Store Star Rating vs Review Star Rating • In a lot of cases, what you can extract from review text is very different than the store star rating • Don’t let rating inflation fool you! 14©2015 Apigee. All Rights Reserved. Store Store Star Rating App Reviews (only) Rating Delta Google Play (US) 3.97 3.83 -0.14 Apple iTunes 3.76 3.81 +0.05 App Store Star Rating App Reviews (only) Rating Delta HBO Go (Android) 3.89 2.64 -1.25 Netflix (iOS) 3.43 2.67 -0.76 Bank of America 4.16 2.94 -1.22
  15. 15. A Systematic Approach to Measuring App Quality 15©2015 Apigee. All Rights Reserved.
  16. 16. Extracting something out of the noise 16©2015 Apigee. All Rights Reserved. • Step 1: Classify Every Review • Step 2: Use ratings to assign a score/importance to put a measurement stake in the ground Max Score Mean for Category Your app score
  17. 17. Using Signals to Help with Classification 17©2015 Apigee. All Rights Reserved.
  18. 18. Applause Analytics Scoring and Sentiment Analysis 18©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores
  19. 19. Applause Analytics Scoring and Sentiment Analysis 19©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores People talk about these things a LOT
  20. 20. Applause Analytics Scoring and Sentiment Analysis 20©2015 Apigee. All Rights Reserved. • Histogram of Attribute Scores Across App Stores • Mean Attribute Scores Across App Stores People talk about these things a LOT People talk more positively about these things People talk more negatively about these things
  21. 21. Opens Up Competitive Benchmarking 21©2015 Apigee. All Rights Reserved.
  22. 22. Applause Analytics Clustering and Ranking 22©2015 Apigee. All Rights Reserved. The How: • Modified Agglomerative Clustering Algo (Patent Pending) • Represent reviews per app as term review vectors and calculate cosine similarities • Cluster based on size of review corpus, vocabulary for app, etc. • Rank based on magnitude, date, avg sentiment, and rarity The What: • Grouped, related app reviews • Most actionable reviews Example: Can’t turn off notifications
  23. 23. So… what does this help us do? 23©2015 Apigee. All Rights Reserved.
  24. 24. Smarter, More Context to Alert You to the Right Issues 24©2015 Apigee. All Rights Reserved. Step 1: Use ratings and review count trends for detection Step 2: Use clusters for smart recommendation of focus areas Security issues with UK accounts Bidding isn’t sticking Items are no longer copyable Search is broken on HTC devices Auto-detected, pushed to your inbox, or app notification Trend over N timeunits, then pop alert if >90th% over N-1
  25. 25. Food for thought: Pre- vs Post- Release Issues 25©2015 Apigee. All Rights Reserved. Top Pre Release Bug Issues – Creating/Modifying Account information – Navigation flow breaks – Order forms – Video playback (typically on mobile due to rotated screens) – Search experiences – Broken pictures/photos Top Post Release User Issues - Latest update is regression in functionality - Bad graphics - Music or video playback is poor - Slow to load, freezes on load - Login broken - Upsell is annoying
  26. 26. Food for thought: Pre- vs Post- Release Issues 26©2015 Apigee. All Rights Reserved. Top Pre Release Bug Issues – Creating/Modifying Account information – Navigation flow breaks – Order forms – Video playback (typically on mobile due to rotated screens) – Search experiences – Broken pictures/photos Top Post Release User Issues - Latest update is regression in functionality - Bad graphics - Music or video playback is poor - Slow to load, freezes on load - Login broken - Upsell is annoying What’s common before and after release? So that perhaps we can tell you…
  27. 27. Credit Where Credit is Due • Everyone at Applause! • Especially the people doing all of the actual work behind Applause Analytics: – Jason Stredwick – Matthew Johnson – Scott Ferguson – Yashas Mavinakere – Lin Zhong – Rob Reid – Josh Wilcox – Christina Green – Peter Bell 27©2015 Apigee. All Rights Reserved.

×