Web Optimization is a Reinforcement Learning problem. Q-Learning is introduced as a way to integrate AB Testing, Attribution, and Predictive Targeting.
6. What’s In it For You
www.Conductrics.com
@conductrics
7. • Reinforcement Learning (RL):
• AB Testing
What’s In it For You
www.Conductrics.com
@conductrics
8. • Reinforcement Learning (RL):
• AB Testing
• Attribution
What’s In it For You
www.Conductrics.com
@conductrics
9. • Reinforcement Learning (RL):
• AB Testing
• Attribution
• Predictive Targeting
What’s In it For You
www.Conductrics.com
@conductrics
10. • Reinforcement Learning (RL):
• AB Testing
• Attribution
• Predictive Targeting
• An RL Solution from AI
What’s In it For You
www.Conductrics.com
@conductrics
11. • Reinforcement Learning (RL):
• AB Testing
• Attribution
• Predictive Targeting
• An RL Solution from AI
• Tell EveryoneYou Know AI!!!
What’s In it For You
www.Conductrics.com
@conductrics
14. Reinforcement Learning Problem:
Learn to make a Sequence
of Decisions by Trial &
Error in order to Achieve
some Goal(s)
www.Conductrics.com
@conductrics
20. Single Location Decisions/AB Test
Home
Page
Hero
Decision
Point
Decision
Simple
Image
Fancy
Version
Fancy
Version
RL Agent
www.Conductrics.com
@conductrics
27. How to Solve:
A
B
Page A Convert
Don’t
Convert
Location Decision Objective/Payoff
Single Location Decisions/AB Test
28. How to Solve:
1. AB/MV Testing
A
B
Page A Convert
Don’t
Convert
Location Decision Objective/Payoff
Single Location Decisions/AB Test
www.Conductrics.com
@conductrics
29. How to Solve:
1. AB/MV Testing
2. Multi-Arm Bandit
A
B
Page A Convert
Don’t
Convert
Location Decision Objective/Payoff
Single Location Decisions/AB Test
30. Only need Conversion Data
Option Value
A 5%
B 6%
Single Location Decisions/AB Test
www.Conductrics.com
@conductrics
31. 31
Part 2: Attribution as Sequential Decisions
AB
Testing
Attribution
Targeting
www.Conductrics.com
@conductrics
32. Single Location Compound: MVT
Home
Page
Banner
Hero
Decision
Point
Decision #1
Decision #2
Banner A
Banner B
Banner C
Simple
Image
Fancy
Version
Banner C
Fancy
Version
RL Agent
Search
Results
Special
Offers
CheckoutSignup
Home
Page
www.Conductrics.com
@conductrics
54. Forward View: Q Learning
Analytics Interpretation of Q-Learning
1)Treat Landing on the Next Page like a
regular conversion!
www.Conductrics.com
@conductrics
55. Forward View: Q Learning
Analytics Interpretation of Q-Learning
1)Treat Landing on the Next Page like a
regular conversion!
2)Use the estimates at the next step as the
conversion value!
www.Conductrics.com
@conductrics
56. Page 1 A B
1) Take an action
Forward View: Q Learning
www.Conductrics.com
@conductrics
57. Page 1 A
1) Take an action – Pick A
Forward View: Q Learning
www.Conductrics.com
@conductrics
58. Page 1 A
2) Measure what user does after
Forward View: Q Learning
www.Conductrics.com
@conductrics
59. 2) Do they Convert?
$10
Page 1 A
Forward View: Q Learning
www.Conductrics.com
@conductrics
65. 3) Yes! Now in Dynamic part of Path
Goal
Page 1
Page 2
A
Forward View: Q Learning
www.Conductrics.com
@conductrics
66. Page 2 C D
4) Check Current Estimated Values ‘C’ & ‘D’
Forward View: Q Learning
www.Conductrics.com
@conductrics
67. 4) Check Current Estimated Values ‘C’ & ‘D’
Of course initially C=$0; D=$0
Page 2 C D
$0 $0
Forward View: Q Learning
www.Conductrics.com
@conductrics
68. 4) Check Current Estimated Values ‘C’ & ‘D’
But lets just assume a mean of C=$1; D=$5
Page 2 C D
$1 $5
Forward View: Q Learning
www.Conductrics.com
@conductrics
75. Attribution in just two simple steps:
Forward View: Q Learning
www.Conductrics.com
@conductrics
76. Forward View: Q Learning
Attribution in just two simple steps:
1)Treat Landing on Next Page like a regular
conversion!
www.Conductrics.com
@conductrics
77. Attribution in just two simple steps:
1)Treat Landing on Next Page like a regular
conversion!
2)Use Predictions of future values at the
next step as the conversion value!
Forward View: Q Learning
www.Conductrics.com
@conductrics
78. Attribution in just two simple steps:
1)Treat Landing on Next Page exactly like a
conversion!
2)Use estimates at the next step as the conversion
value!
3)This is guaranteed to converge to optimum result!!!
Forward View: Q Learning
www.Conductrics.com
@conductrics
90. 90
Case Study: Web & Call Center
Optimize Marketing Site and Call Center
IVR
1. WebSite
• Initial Offer
• Online Chat option
2. Call Center
• Choice of IVR prompts
Result: Call Center Conversion by 26%
Increased average call value by $9.28www.Conductrics.com
@conductrics
91. Sales drive
Optimization
User Features
•Time of Day
•Geo-Data
•Browser Data
Get Decision
Website IVR
Options
Response
IVR Prompts: A;B;C;….
Get Decision
Call Agent
IVR ‘Rewards’
Web Decision
Case Study: Web and Call Center
www.Conductrics.com
@conductrics
92. Application
Layer
Targeted Multi-Touch Optimization
Touch-Point 1
User
2) Option
Response
3) Touch-Point Transition
1) Option
Request
Local Options
Optimizer
F1
F2
Fn
S
Local Model 1
Touch-Point 2
User
7) Option
Response
4) Option
Request
Local Options
Optimizer
F1
F2
Fn
S
Local Model 25) Calculates Attribution Credit
and sends to Model1
8)Conversion
6) Update Local Model1 using
credit as a conversion
9) Update
Local Model2 using
conversion value
• Attribution Credit enables Local Optimizers to Solve
Global Multi-Touch Optimization
www.Conductrics.com
@conductrics
93. 1) Attribution can be solved by
hacking ‘AB Testing’ (Q-Learning)
2) Extended Attribution to include
decisions/experiments
3) Looked into the eye of AI and Lived
What did we Do/Learn?
www.Conductrics.com
@conductrics