#PAWNYCDAY1

1. JENNIFER PRENDKI | HEAD OF DATA SCIENCE | ATLASSIAN
Converting Churn
into Opportunity
Predicting Customer Churn from Product Usage
DANIEL CHUNG | DATA SCIENTIST | ATLASSIAN

2. Talking point
Supporting info for talking point.
Talking point
Manage audience focus by reducing to 30%
opacity. Use no more than two lines.
Talking point
Stay consistent with using the light or dark
throughout the related big statement.
About
Atlassian

3. Daniel specializes in
behavioral models, growth
metrics, and experimentation.
He brings years of experience
in analyzing data to bring
business insight and
influence strategic decisions.
Jennifer is Head of Data
Science at Atlassian.
She leads the Search &
Smarts Engineering team, in
charge of loading the suite of
Atlassian products with smart
features.
JENNIFER PRENDKI, PHD DANIEL CHUNG

4. Agenda
Context & Problem
Model
Measuring Success
Retrospective

5. No Sales Army
Rely on optimization of customer funnel and
solving customer pain points
Dedicated CSS Team
Customer Support & Success - lower customer
friction and champion their success
Cloud Customers
Focusing on monthly subscription customers of
our flagship cloud products
Context

6. No Sales Army
Rely on optimization of customer funnel and
solving customer pain points
Dedicated CSS Team
Customer Support & Success - lower customer
friction and champion their success
Cloud Customers
Focusing on monthly subscription customers of
our flagship cloud products
Context

7. No Sales Army
Rely on optimization of customer funnel and
solving customer pain points
Dedicated CSS Team
Customer Support & Success - lower customer
friction and champion their success
Cloud Customers
Focusing on monthly subscription customers of
our flagship cloud products
Context

8. • New: Active and did not exist 28 days ago
• Returning: Active in the current period and was also active in the previous period
• Dormant: Not Active in the entire current period but was active in the previous period
• Resurrected: Active in the current period and was not active in the entire previous period
MAU, WAU, and DAU?
What does this look like?
Legend: New, Returning, Dormant, Resurrected
Credits to Amit Tiroshi, Product Analyst, Bitbucket
Metrics tracking users who were “active” in the past month / week / day

9. A Slice of the Customer Funnel
Wk 2 WAU Wk 10 MAUWk 6 MAUActivation
…
• Product growth view of the customer funnel
• Excludes acquisition and churn - but the aim is predict their tendency to churn /
upgrade with early signals
License purchased
after evaluation
1st Wk Activity

10. Timing is Key
Too late to intervene when
too far into the customer
lifecycle
Problem Statement
Cannot Target All
To effectively address
customer needs and learn
with limited resources
MAU vs License
MAU increase not necessarily
correlated with license
retention

11. Agenda
Context & Problem
Model
Measuring Success
Retrospective

12. Model
Exploration
Factors
Methodology
Performance
Output
• Churned in 1 year
• Upgraded + not churned in 1 year
Feature Selection
• Profile (static data) vs. activity (behavioral data)
• Company-level, license-level, user-level features
Sample Features

13. Model
Exploration
Factors
Methodology
Performance
Key Important Factors
Output:
Upgraded & Not Churned

14. Model
Exploration
Factors
Methodology
Performance
Key Important Factors Cont’d

15. Model
Exploration
Factors
Methodology
Performance
Tree-based Models
Practical reasons:
• Interpretability
• Works well with raw data
• Ranks features by importance
Analytical reasons:
• Ensemble methods are robust to overfitting
• Parameter optimization is intuitive
• Categorical and numerical data
Classification Techniques Considered
• Random Forest
• Gradient Boosting Machines
• Logistic Regression

16. Model
Exploration
Factors
Methodology
Performance
Measure Model Performance
• k-fold cross-validation
• At least 1 year of post-purchase data to measure
upgrade / churn
Methodology Response Churned in 1 Year
Upgraded and Not
Churned in 1 Year
Random Forest
AUC: 0.655
Variance: 3.50 x 10^-4
AUC: 0.771
Variance: 2.47 x 10^-5
GBM
AUC: 0.764
Variance: 4.12 x 10^-5
AUC: 0.882
Variance: 2.52 x 10^-5
Evaluation Metric
• Area under the ROC (AUC score)

17. JIRA SOFTWARE CONFLUENCE
Methodology Response Churned in 1 Year
Upgraded and Not
Churned in 1 Year
Best Methodology
AUC: 0.764
Variance: 4.12 x 10^-5
AUC: 0.882
Variance: 2.52 x 10^-5
Methodology Response Churned in 1 Year
Upgraded and Not
Churned in 1 Year
Best Methodology
AUC: 0.764
Variance: 5.96 x 10^-5
AUC: 0.878
Variance: 6.22 x 10^-5

18. Agenda
Context & Problem
Model
Measuring Success
Retrospective

19. 1st Wk Activity
Product Cohort
% of Active Licenses with Activity
Wk 2 WAU Wk 6 MAU Wk 10 MAU
JSW
Target 100.0% 100.0% 99.4%
Holdout 81.4% 86.7% 77.9%
Conf
Target 100.0% 99.2% 99.2%
Holdout 76.0% 83.3% 76.1%
Wk 2 WAU Wk 10 MAUWk 6 MAU
LICENSE SURVIVAL
Activation
License purchased
after evaluation

20. 100+% Growth in MRR
from upgrades, despite license churn throughout the year.

21. Agenda
Context & Problem
Model
Measuring Success
Retrospective

22. FURTHER MODEL IMPROVEMENTS
1
2
3
Bundles and Expansions
• Customers with multiple products vs. single product behave differently
• Further segment by other license types, user activity on those licenses, etc.
Event-level Factors
• Currently user activity is our most granular data, aggregated at the license level
• User actions that improve likelihood of license retention / upgrades
Types of Users
• Admins vs. end user vs. admins who also behave like an end user

23. Learnings and Next Steps
User Activity
User activity, even as a early
signal, is a strong indicator
of license upgrade / churn
Reusable &
Scalable
Support numerous use
cases and data and
features at all levels
Measure
Work with the CSS team to
consistently measure
success and make model
adjustments

24. JENNIFER PRENDKI | HEAD OF DATA SCIENCE | ATLASSIAN
DANIEL CHUNG | DATA SCIENTIST | ATLASSIAN
Thank you

26. Appendix

27. DICTIONARY: FEATURES 1 - 10
Feature Data Type Description
is_personal_domain bool Flag indicating if domain of customer is a personal vs enterprise
license_tier string User tier of the license - 10 User, 25 User, 500 User, etc.
billing_period string Billing period frequency - Monthly, Annual
level string License type - Starter, Full (no Evals since post-activation)
customer string True if customer is labelled as a customer (depends on license type)
start_month int Start month of the license
sale_type string Type of sale - N2N, N2E, etc.
sale_action string Sales action - New, Renewal
subregion string Subregion of the customer
region string Region of the customer

28. DICTIONARY: FEATURES 11 - 20
Feature Data Type Description
efficiency_score double
Efficiency ratio of the country tied to the Global Innovation Index
See: http://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2016.pdf
innovation_score double
Global innovation index of the customer’s country
See: https://www.globalinnovationindex.org/
gdp_percapita double GDP per capita for the customer’s country
industry string Industry that the customer belongs to
company_size int Company size of the customer (in # of employees)
other_prod_count int # of other Atlassian products the customer uses
avg_active_users_in_1wk double Average # of weekly active users across week 1
avg_hv_active_users_in_1wk double Average # of high value (engaged) weekly active users across week 1
avg_new_users_in_1wk double Average # of new weekly active users across week 1
avg_returning_users_in_1wk double Average # of returning weekly active users across week 1

29. DICTIONARY: FEATURES 21 - 30
Feature Data Type Description
avg_resurrected_users_in_1wk double Average # of resurrected weekly active users across week 1
avg_dormant_users_in_1wk double Average # of weekly dormant users across week 1
avg_active_users_in_1mth double Average # of weekly active users across week 4
avg_hv_active_users_in_1mth double Average # of highly engaged weekly active users across month 1
avg_new_users_in_1mth double Average # of new weekly active users across month 1
avg_returning_users_in_1mth double Average # of returning weekly active users across month 1
avg_resurrected_users_in_1mth double Average # of resurrected weekly active users across month 1
avg_dormant_users_in_1mth double Average # of weekly dormant users across month 1

30. DICTIONARY: RESPONSES
Feature Data Type
Type
Description
upgraded_in_1yr bool
Flag indicating whether the license upgraded within 1 year post
purchase
upgraded_in_6mth bool
Flag indicating whether the license upgraded within 6 months post
purchase
churned_in_1yr bool
Flag indicating whether the license churned within 1 year post
purchase
upgraded_not_churned_in_1yr bool
Flag indicating whether the license upgraded and not churned within 1
year post purchase

31. FEATURE CORRELATION

32. EXTRA DATA FOR CONFLUENCE
Product # of Active Users / Instance # of Interactions / User
JSW
Avg: 2.84
Median: 1
Avg: 73.04
Median: 9
Conf
Avg: 2.00
Median: 1
Avg: 58.85
Median: 3

33. EXTRA DATA & PLOTS FOR CONFLUENCE 2
Credits to Orpheus Mall, Data Scientist, Product Analytics