Raj Singh, CEO, Tempo AI A blueprint for collecting & analyzing actionable mobile metrics

1. Daily Active Users Don’t Matter
A Blueprint for Collecting & Analyzing Actionable Mobile Metrics
Raj Singh
CEO, Tempo AI, Inc.
510-282-4229 (M)
raj@tempo.ai
@mobileraj

2. About Me – Mobile First Dude
• 20+ mobile titles since ‘99
• 7+ apps hit #1 in the App Store/Catalog (Vanity Metric!)
• MobiTV, Kodak Mobile, Dell Mobile, Skyfire, Recipes, Tempo
• Mobile blogger for O’reilly; guest blogger at numerous

3. 2 Kinds of Metrics
Vanity or Actionable?
Examples
# of webpages indexed by Google
% of users uploading a video on YouTube
DAU/MAU (Daily Actives / Monthly Actives)
Registration vs skip registration CTR (click-thru rate) over time

4. Growth Hacking
The process is EXACTLY the same on mobile or any screen for that matter!
• Test
• Measure
• Repeat

5. But Mobile Has Caveats
• 3-4 week App Review cycles suck!
• No HTML; native code is slow!
• Disconnected data across screens; stats from web to App Store to app are
hard to connect
Prioritize your A/B tests!

6. Start With the Framework
AARRR (Dave McClure’s Pirate Framework)
• Acquisition
• Activation
• Retention
• Referral
• Revenue
Other frameworks like Acquisition, Activation, Engagement, Virality are the
same.

7. Identify What Type of App You Are
• Free / Advertising
• 1-time Paid
• In-App (Trending)
• Subscription (Trending)

8. Why App Type Matters?
Retention
Time
Retention
Time
Retention
Time
Retention
Time
Productivity
Smile Curve?
Network
Effect?
Common…Paid App?

9. Case Study: Tempo Smart Calendar
App Type: Subscription
I care about longer-term retention
App Category: Productivity
I care about unique acquisition strategies since we are not social
Goal for Launch: MVP
MVP = Minimum Viable Product
Viable = Until 50% of our Alpha group were DAUs

10. Tempo’s Timeline
Brainstorming
12/10 8/11
Founded
3/12
First Proto
2/13
Public Launch
Seeking Core Product Value
7/12
Optimize Activation &
Engagement
6/13
1. Digest Data
2. Peaks & Valleys
3. Acquisition
4. Activation
Engagement
/ Retention
AARRR is a great framework but you
don’t necessarily need to attack it in
that order as demonstrated with our
timeline.

11. Seeking Core Product Value
• Created an Alpha Group: ~150 users
‒ Initially 50 users; moved to iOS Enterprise acct (~150 was what we could manage)
‒ Friends, advisors, investors (too biased)
‒ Added friends of friends, non-tech types, high school buddies, former colleagues
• Test, learn & iterate w/ Alpha Group
‒ Monthly builds for 9 months (until Launch)
‒ Interviewed each member of the Alpha group regularly
‒ Except for identifying peaks and valleys, data wasn’t useful (sample too small)
‒ Interviews were a MUST; lots of inductive learning
• Searching for Hooks and Must-Haves
‒ Hook = 3 Days (of active use); initial delight and activation
‒ Must-Have = 3 Weeks (of active use)

12. Seeking Core Product Value (Cont)
• Conducted surveys to identify Must-Haves
‒ Started with fill-in-the-blank Qs (eg What do use Tempo for)
‒ Ended with “Choose from the list” and rank 1-5
‒ Must-Haves = When >50% of people in our survey said they “Can’t live without this feature”
Survey #1
1. Why do you use Tempo? [Fill in the Blank]
I like how it looks; The auto-dialing feature is great; 100+ responses
2. Why don’t you use Tempo? *Fill in the Blank+
It doesn’t have the date on the icon; I can’t set 2 alerts; 100+ responses
Survey #2
• Distill results to just 5-8 things
1. What features you can’t live without (Rate 1-5)

13. Seeking Core Product Value (Cont)
• Perform Craigslist UX testing
‒ 50/50 split between target demographic and random
‒ Testing focused on Hooks & Education
‒ Always completed a test with a simple Q&A and a word game (ie What words amongst the
following adjectives best describe Tempo for you)
Accessible Desirable Gets in the way Patronizing Stressful
Appealing Easy to use Hard to use Personal Time-consuming
Attractive Efficient High quality Predictable Time-saving
Busy Empowering Inconsistent Relevant Too technical
Collaborative Exciting Intimidating Reliable Trustworthy
Complex Familiar Inviting Rigid Uncontrollable
Comprehensive Fast Motivating Simplistic Unconventional
Confusing Flexible Not valuable Slow Unpredictable
Connected Fresh Organized Smart Usable
Consistent Frustrating Overbearing Stimulating Useful
Customizable Fun Overwhelming Straight Forward Valuable

14. We Identified Core Product Value?
Tempo Alpha Group Users loves Auto-Conference Calls and I’m Late!
But… Alpha data can be misleading
• Small # of Users = Small Learnings
• Big # of Users = Big Learnings
• Goal for Launch was to get Big Users (10K) to learn big things
• Why 10K?
‒ That’s a big # in the productivity category!
‒ Productivity apps are generally not social
‒ We thought we could achieve it!

15. • Actionable Stats
‒ Funnel drop-off rate by screen?
‒ Time spent on each screen (speed is important)
‒ Download to conversion rate
• Alpha group wasn’t good data b/c they all activated! (Don’t let friends test apps)
So how did we test activation pre-launch?
Performed an out-of-market test in Canada
• Used a different application name: Teempo Smart Calendar
• Spent $500 (total) to acquire ~2-3K users with mobile ads; we optimized the ad
buying as well
Why Canada?
• Because it was convenient
Results
• Our Alpha group connected all their data but the Canadians did not? Privacy?
• Retention wasn’t very good
Still Pre-Launch: Moved Focus to Activation

16. Canada Taught Us 2 Things
1. We need to improve the “don’t connect anything” education in Tempo
2. Users acquired via cheap mobile ads was not the right segment for us;
mostly gamers
We improved education in Tempo but we didn’t have enough time to run
another Canadian test (running out of money)
• Focused on blind Craigslist testing and heuristics usability testing
• Heuristics Usability = Asking experts for feedback

17. Still Pre-Launch: Moved Focus to ASO (App Store Optimization)
• Identified what we could A/B test
‒ App Store description
‒ App Store images
‒ App Title
‒ App Keywords
• Created 10s of permutations and tested
• Couldn’t measure acquisition to retention (unfortunately)

18. Buying Ads – We Tested A Lot!
We even created a fake HTML App Store landing
page to test different descriptions
Results
• Not enough statistical variance between words:
“Smart,” “AI,” “Intelligent,” “Magic” etc (not
conclusive)
• Users preferred icons that looked like calendars
• App screenshots with descriptions outperformed
screenshots without descriptions
Living Cal
Calendar Assistant
Useful Calendar
Smart Cal
Smart Calendar
Smart Calendar #3
Cal Assistant
Intelligent Calendar 0.50
0.50
Actionable Cal
0.60
Smart Calendar #4
Magic Calendar
0.51
0.52
Smart Calendar #1
0.61
0.53
Living Calendar
0.55
0.63
Smart Calendar #2
Learning Cal
0.50
0.40
0.43
0.45
0.46
0.59

19. We’re Ready – Gearing Up For Launch
My Pre-Launch Blueprint
• Identify Core Product Value (Hooks & Must-Haves: UI, Conference Calls &
I’m Late)
• Optimize Activation / On-Boarding (Reduced the # of screens and clicks)
• Identify Peaks and Valleys from Alpha data (Improved in-app education for
the more “private user”)
• Optimize Acquisition / Discovery (ASO: A/B test keywords, app
title, icon, description)
(See slide 10 for the timeline)

20. Launch Day
• Exceeded our goal of 10K users
• What do we focus on now???
‒ Look for surprises: Users were 24X more data than our Canada Alpha!
‒ New Peaks & Valleys
‒ Activation, On-Boarding
• It takes 3 months before you have enough data to really understand
engagement and retention

21. Post-Launch: Studied On-Boarding
• 30-40% bounce rate on registration :/
• Possible A/B Tests:
‒ Facebook or LinkedIn registration
‒ Skip
• Users confused between Tempo registration and
connecting their calendar
More tests planned for future (but not the priority
at present)
100%
70%
62%
62%
60%
59%
Downloads*
Account Creation
Attempted
Account Creation
Successful

22. Post-Launch: Studied Initial Bounce Rate
• 34% of users left after the 1st day
• Data did not answer Day 1 bounce rates, so:
‒ Identified 100 users from support who bounced after 1-3 days
‒ Sent a personal email and spoke to a handful on the phone
‒ Considered a survey but it doesn’t work with dis-engaged users. Surveys yield responses
from your fans!
Results From Discussions
• Email connectivity issues
• Sync issues
• Privacy
• Confusion / Education
We can fix these things!

23. Post-Launch: Seeking Peaks & Valleys
Examples
• The Settings button was the most popular button? (Peak)
• FB connect errors as a % of FB actions were really high (Peak)
• Adding Event was the most popular action (Peak)
• Few customized their Agenda view image (Valley)
• Few clicked on Foursquare or Yelp (Valley)

24. Post-Launch: Seeking Peaks & Valleys (Cont)
Valleys
• Discovery/education or uninteresting feature?
• Create simple A/B tests to answer
Example A/B Test
Hypothesis: Users didn’t customize Agenda View photos because they didn’t
know they could
Test: Sent an email with tips after Tempo user registration
Result: Was no longer a valley. Test succeeded

25. Post-Launch: Seeking Peaks & Valleys (Cont)
Peaks
• Unexpected or a feature
• Create simple A/B tests to answer
Example A/B Test
Hypothesis: Users clicked Settings because
they didn’t know how to change views
Test: Move the menu from the top-middle to
the top-left
Result: Settings was no longer a peak. Test
succeeded

26. Post Launch: Seeking Peaks & Valleys (Cont)
Not all A/B tests are as cut and dry
A/B Test
Hypothesis: “Post Happy B-Day” is a popular action
Test: Let people “Post Happy B-Day” from the
Calendar View
Result: No change. Test was not successful?
Prioritize your A/B tests! Test, Measure, Repeat

27. 3 Months After Launch: Understanding Retention
• Defined an internal goal of 20% after 3 months
‒ Got comps from other apps in the category (eg Evernote etc)
‒ It’s a great # and it’s good to have goals
• Tempo: 10-20% Retention
‒ Current priority!
And so we started with the questions to answer:
• How many days until we see the biggest drop-off in retention?
• Why after using Tempo for X weeks, do you leave?
• Are certain features in the app more or less used by more or less engaged
users?
• Who is our user?

28. 3 Months After Launch: Understanding Retention (Cont)
The process is exactly the same!
• Identify users from support and other channels and contact them!
• Create a survey but it doesn’t substitute for speaking!
• Looked for peaks & valleys but on a per active user and per session basis
over time (actionable stats)
• Created A/B tests and executed
But one difference:
• Look for correlations

29. 3 Months After Launch: Looking For Correlations
Correlation = Ratios
Ratios ~= Actionable Data
Tempo Examples
Users with Exchange are more engaged
Users who use conference calling are more engaged
Users with Facebook events only are less engaged
Create hypothesis and measure the data over time

30. My Post-Launch Blueprint
• Look for surprises (Fixed settings button click rate)
• Study activation workflow in aggregate (Fixed email registration confusion)
• Identify, interview & talk to users (Improved education)
• Find peaks and valleys (A/B tested Facebook birthday post)
• Look for correlations (Lots of Excel!)
• A/B test and prioritize your tests

31. What’s Next? (Conversation For Next Year?)
• Achieve our 20% 3 month retention goal
• Then we focus on acquisition (Virality = Frequency X Reach X Yield)
• Then we focus on monetization
No matter where you are in AARRR, the process is the same:
Test, Measure, Repeat
Talk to your users
Prioritize your A/B tests