In this session, we take a specific business problem—predicting Telco customer churn—and explore the practical aspects of building and evaluating an Amazon Machine Learning model. We explore considerations ranging from assigning a dollar value to applying the model using the relative cost of false positive and false negative errors. We discuss all aspects of putting Amazon ML to practical use, including how to build multiple models to choose from, put models into production, and update them. We also discuss using Amazon Redshift and Amazon S3 with Amazon ML.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Denis V. Batalov, PhD
AWS Solutions Architect, EMEA
November 30, 2016
Predicting Customer Churn
with Amazon Machine Learning
@dbatalov
MAC307

2. Customer churn

3. Machine learning
Science
• Computer Science
• Statistics
• Neuroscience
• Operations Research
Artificial Intelligence
• Rule extraction from data
• Inspired by human learning
• Adaptive algorithms
Engineering
• Training: Data  Models
• Prediction: Models  Forecast
• Decision: Forecast  Actions

4. ML: Robotics

5. ML: Robotics

6. ML: Image recognition

7. Supervised learning

8. Supervised learning
Input Outcome

9. Supervised learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome

10. Supervised Learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome
Supervised
Learning
known historical data
Amazon ML

11. Supervised learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome
Supervised
Learning
Unseen Input Same Outcome
known historical data
Amazon ML

12. Amazon Machine Learning service

13. Amazon Machine Learning service

14. Amazon Machine Learning service

15. Amazon Machine Learning service

16. Telco churn dataset
• US telco customers, their cell phone plans and usage
• 21 attributes, 3333 rows:
• Customer: State, Area_Code, Phone
• Plan: Intl_Plan, VMail_Plan
• Behavior: VMail_Messages, Day_Mins, Day_Calls,
Day_Charge, Eve_Mins, Eve_Calls, Eve_Charge,
Night_Mins, Night_Calls, Night_Charge, Intl_Mins,
Intl_Calls, Intl_Charge
• Other: Account_Length, CustServ_Calls, Churn

17. Telco churn dataset
• US telco customers, their cell phone plans and usage
• 21 attributes, 3333 rows:
• Customer: State, Area_Code, Phone
• Plan: Intl_Plan, VMail_Plan
• Behavior: VMail_Messages, Day_Mins, Day_Calls,
Day_Charge, Eve_Mins, Eve_Calls, Eve_Charge,
Night_Mins, Night_Calls, Night_Charge, Intl_Mins,
Intl_Calls, Intl_Charge
• Other: Account_Length, CustServ_Calls, Churn

18. Telco churn dataset
KS, 128, 415, 382-4657, 0, 1, 25, 265.100000, 110, 45.070000, 197.400000, 99,
16.780000, 244.700000, 91, 11.010000, 10.000000, 3, 2.700000, 1, 0
OH, 107, 415, 371-7191, 0, 1, 26, 161.600000, 123, 27.470000, 195.500000, 103,
16.620000, 254.400000, 103, 11.450000, 13.700000, 3, 3.700000, 1, 0
NJ, 137, 415, 358-1921, 0, 0, 0, 243.400000, 114, 41.380000, 121.200000, 110,
10.300000, 162.600000, 104, 7.320000, 12.200000, 5, 3.290000, 0, 0
OH, 84, 408, 375-9999, 1, 0, 0, 299.400000, 71, 50.900000, 61.900000, 88, 5.260000,
196.900000, 89, 8.860000, 6.600000, 7, 1.780000, 2, 0
OK, 75, 415, 330-6626, 1, 0, 0, 166.700000, 113, 28.340000, 148.300000, 122, 12.610000,
186.900000, 121, 8.410000, 10.100000, 3, 2.730000, 3, 0
AL, 118, 510, 391-8027, 1, 0, 0, 223.400000, 98, 37.980000, 220.600000, 101, 18.750000,
203.900000, 118, 9.180000, 6.300000, 6, 1.700000, 0, 0

19. Console: Creating datasource for Amazon ML

20. Console: Creating datasource for Amazon ML

21. Console: Building the Amazon ML model

22. Recipe
{ "groups": {
"NUMERIC_VARS_NORM":
"group('Intl_Charge','Night_Calls','Day_Calls','Eve_Calls','Eve_Mins','Int
l_Mins','VMail_Message','Intl_Calls','Day_Mins','Night_Mins','Day_Charge',
'Night_Charge','Eve_Charge','Account_Length')” },
"assignments": {},
"outputs": [
"ALL_BINARY",
"State",
"Area_Code",
"normalize(NUMERIC_VARS_NORM)",
"CustServ_Calls"
]
}

23. Recipe: normalize() function
Account_Length Normalized Value
128 0.808771865
107 -0.047574816
137 1.175777586
84 -0.985478323
75 -1.352484044
118 0.400987732

24. Building the Amazon ML model

25. Cost of errors
• Cost of customer churn and acquisition (false negative):
• Foregone cash flow
• Advertising costs
• POS and sign-up admin costs
• Customer retention cost (false + true positive)
• Discounts
• Phone upgrades
• Etc.

26. Financial outcome of applying a model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46

27. Financial outcome of applying a model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46
False Negative True + False Pos Retention Cost Cost with ML
4.80% 12.10% + 14.30% $100.00 $50.40

28. Financial outcome of applying a model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46
False Negative True + False Pos Retention Cost Cost with ML
4.80% 12.10% + 14.30% $100.00 $50.40
• Threshold 0.3  0.17
• $22.06 of savings per customer
• With 100,000 customers over $2MM in savings with ML

29. What’s next?
• https://aws.amazon.com/getting-started/projects/build-
machine-learning-model/
• https://aws.amazon.com/machine-learning/developer-
resources/
• https://github.com/dbatalov/cost_based_ml

30. Thank you!
Denis V. Batalov, PhD
AWS Solutions Architect, EMEA
@dbatalov

31. Remember to complete
your evaluations!