1.
#abtesting@csaintamant #abtesting@csaintamant

2.
#abtesting@csaintamant
we can do better

3.
#abtesting@csaintamant
brainstorm. test.brainstorm. test. measure.

4.
#abtesting@csaintamant
data driven
product development

5.
#abtesting@csaintamant
continuous
improvement

6.
vs ba

7.
a b

8.
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7

9.
#abtesting@csaintamant
it gets
messy

10.
#abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

11.
#abtesting@csaintamant
(testable)
hypothesis

12.
#abtesting@csaintamant
individual profiles in a multi-
user household will create a
more personalized experience
and result in higher retention

13.
#abtesting@csaintamant
test
variables

14.
#abtesting@csaintamant
test cells

15.
Variables Gate
Gate w/
Remember
Simple Setup Detailed Setup
Cell 1 (control)
Cell 2 ✓
Cell 3 ✓ ✓
Cell 4 ✓ ✓
Cell 5 ✓ ✓

16.
#abtesting@csaintamant
be mindful

17.
#abtesting@csaintamant
iterate later

18.
#abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

19.
#abtesting@csaintamant
test identifiers

20.
#abtesting@csaintamant
test # 1234
test cells 1 - 5

21.
var tests = user.getTests();
if (tests.in(1234, 2) || tests.in(1234, 3) ||
tests.in(1234, 4) || tests.in(1234, 5)) {
// profiles enabled?
if (tests.in(1234, 3) || tests.in(1234, 4) ||
tests.in(1234, 5)) {
// gate?
if (tests.in(1234, 4)) {
// gate with remember?
}
}
}
else {
// control behavior
}

22.
#abtesting@csaintamant
externalize config

23.
{
meta: {
id: 1234,
name: 'profiles',
},
cells: {
'1': { control: true, profilesEnabled: false },
'2': { profilesEnabled: true, simpleSetup: true },
'3': { profilesEnabled: true, simpleSetup: true, gate: true },
'4': { profilesEnabled: true, simpleSetup: true, gate: true,
gateRemember: true },
'5': { profilesEnabled: true, detailedSetup: true, gate: true }
}
}

24.
var profilesTest = user.getTests().get('profiles');
if (profilesTest.attr('profilesEnabled')) {
// profiles enabled?
if (profilesTest.attr('gate')) {
// gate?
if (profilesTest.attr('gateRemember')) {
// gate with remember?
}
}
}
else {
// control behavior
}

25.
#abtesting@csaintamant
feature flags?
yep.

26.
profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check test flag
var profilesTest = context.getTests().get('profiles');
return profilesTest.attr('profilesEnabled');
});
rules

27.
profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check app config
return context.getAppConfig().profilesEnabled;
});
rules

28.
var tests = user.getTests();
if (tests.in(1234, 3) ||
tests.in(1234, 4) ||
tests.in(1234, 5)) {
// gate?
}
before after
if (rules.profilesGate) {
// gate!
}

29.
#abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

30.
#abtesting@csaintamant
UI assembly / delivery
HTML + (CSS + JS)

31.
#abtesting@csaintamant
templating

32.
profiles-setup.template
<div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
<input id="profile1" /><label for-"profile1">That's you!</label>
<input id="profile2" />
<input id="profile3" />
<input id="profile4" />
<input id="profile5" />
<button>Continue</button>
</div>
<div id="profiles-promo">
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
</div>

33.
<div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
<input id="profile1" /><label for-"profile1">That's you!</label>
<input id="profile2" />
<input id="profile3" />
<input id="profile4" />
<input id="profile5" />
<button>Continue</button>
</div>
<div id="profiles-promo">
{#control}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
{/control}
{#promo1}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li>
</ul>
{/promo1}
{#promo2}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li>
</ul>
{/promo2}
{#promo3}
<img src="images/profiles-setup.png" />
<h2>Each person added will get suggestions based on what they like to watch</h2>
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
</ul>
{/promo3}
{#promo4}
<img src="images/profiles-setup.png" />
<h2>Have up to 5 profiles at no extra cost</h2>
<ul>
<li>Each person will get suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
{/promo4}
</div>

34.
profiles-setup.tmpl
if ifif if if
Control Promo 1 Promo 2 Promo 3 Promo 4

35.
profiles-setup.tmpl
promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl
if ifif if if

36.
<div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
[...]
</div>
<div id="profiles-promo">
{> promo /}
</div>

37.
promo.json
profiles-setup.tmpl
?
promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl

38.
promo.json
{
"rules": [],
"templateName": "promo"
},
{
"rules": ["profilesPromo(1)"],
"templateName": "promo1"
},
{
"rules": ["profilesPromo(2)"],
"templateName": "promo2"
},
{
"rules": ["profilesPromo(3)"],
"templateName": "promo3"
},
{
"rules": ["profilesPromo(4)"],
"templateName": "promo4"
}

39.
profilesPromo =
new Rule('profilesPromo', function(context, params) {
// check test membership
var test = context.getTests().get('profilesSetup');
return test && test.cell(params.id-1);
});
rules

40.
#abtesting@csaintamant
how does it work?

41.
profiles-setup.tmpl template engine partial

42.
resolver promo.json
(mappings)
rules

43.
rules
promo.tmpl
promo1.tmpl
promo2.tmpl

44.
profiles-setup.tmpl template engine resolver

45.
#abtesting@csaintamant
template loading
simplified

46.
#abtesting@csaintamant
client-side
packaging

47.
app.js
import backbone from 'backbone';
import search from 'search';
import profiles from 'profiles';
export ...

48.
#abtesting@csaintamant
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

49.
#abtesting@csaintamant
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

50.
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

51.
#abtesting@csaintamant
100s of files..?
Millions of combinations..?

52.
#abtesting@csaintamant
conditional
dependencies

53.
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

54.
rules again!
profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check test flag
var profilesTest = context.getTests().get('profiles');
return profilesTest.attr('profilesEnabled');
});

55.
/*
* @includewhen rule.profilesEnabled
*/
profiles.js

56.
/*
* @includewhen rule.searchEnabled
*/
search.js

57.
#abtesting@csaintamant
build time:
static analysis

58.
#abtesting@csaintamant
minify / compress
build time:

59.
profiles.js
backbone
search.js
app.js
registry

60.
"app.js": {
"deps": [
"backbone",
"search.js",
"profiles.js",
],
"depsFull": [
"react",
"searchDep2.js",
"searchDep1.js",
"search.js",
"profilesDep2.js",
"profilesDep1.js",
"profiles.js"
]
}

61.
"profiles.js": {
"rule": "profilesEnabled",
"deps": [
"backbone",
"profilesDep2.js",
"profilesDep1.js",
],
"depsFull": [
"backbone",
"profilesSubDep3.js",
"profilesSubDep2.js"
"profilesSubDep1.js"
"profilesDep2.js",
"profilesDep1.js"
]
}
rule!

62.
javascript
registry
build

63.
#abtesting@csaintamant
request time:
evaluate rules

64.
#abtesting@csaintamant
assemble package
request time:

65.
registry
rulespackager
request

66.
#abtesting@csaintamant
packaging
simplified

67.
#abtesting@csaintamant
everything is
a module

68.
#abtesting@csaintamant
define dependencies

69.
#abtesting@csaintamant
only load
what you need
…per user

70.
#abtesting@csaintamant
isolate complexity

71.
#abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

72.
#abtesting@csaintamant
multivariate testing
is powerful…

73.
#abtesting@csaintamant
…but complex

74.
#abtesting@csaintamant
plan for complexity

75.
“Simplicity is the
ultimate sophistication”
- Leonardo da Vinci

76.
go experiment!

77.
questions?
Chris Saint-Amant
@csaintamant
#abtesting