Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

[HTML5DevConf2014] Scaling AB Testing on Netflix.com with Node.js

2.276 visualizaciones

Publicado el

This is the extended (full) version of the talk given at HTML5DevConf 2014. A condensed version of this talk was previously given at NodeConfEU 2014.

At Netflix we run hundreds of A/B tests every year. Maintaining multivariate experiences quickly adds strain to any UI engineering team. Join us to explore the patterns we’ve built in Node.js to tame this beast - ultimately enabling quick feature development and rapid test iteration on our service used by over 50 million people around the world.

Video from NodeConfEU:
https://www.youtube.com/watch?v=gtjzjiTI96c

Publicado en: Software
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

[HTML5DevConf2014] Scaling AB Testing on Netflix.com with Node.js

  1. 1. Scaling A/B testing on Netflix.com with _________ Alex Liu @stinkydofu
  2. 2. data driven product development
  3. 3. Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G
  4. 4. 2,097,152 unique experiences across seven tests
  5. 5. hundreds of new A/B tests per year
  6. 6. 43351892955034 94860861172181 85493567650… 72061153709996
  7. 7. 2105 566 685 templates CSS JS
  8. 8. 2.5M unique packages every week
  9. 9. <html/> <link/> <script/>
  10. 10. problem: conditional dependencies
  11. 11. ▶ Templating ▶ Packaging ▶ Bonus Round
  12. 12. Templating
  13. 13. payment.dust <div id="payments"> <input id="first-name"><input id="last-name"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="card-number"><input id="security-code"> <select name="month"></select><select name="year"></select> <checkbox id="agree-to-terms"/> <button>Start Your Trial</button> </div>
  14. 14. <input id=“first-name"><input id=“last-name"> {@inTest id="10" cell=“2a"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class=“payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“3"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“4"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“5"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} <checkbox id="agree-to-terms"></checkbox> <button>Start Your Trial</button> </div>
  15. 15. payment.dust if if if if if Control Cell 2 Cell 3 Cell 4 Cell 5
  16. 16. <div class="payment-types"> <div id="CC"> {> payment_type_cc /} </div> <div id="DD"> {> payment_type_dd /} </div> </div>
  17. 17. payment.dust if if if if if Control Cell 2 Cell 3 Cell 4 Cell 5 payment_type_cc.dust payment_type_dd.dust
  18. 18. payment.dust Control.dust if if if if if Cell2.dust Cell3.dust Cell4.dust Cell5.dust payment_type_cc.dust payment_type_dd.dust
  19. 19. <div id="payments"> <input id="first-name"> <input id="last-name"> {> payment_method /} <input type="checkbox" id="terms"> <button>Start Your Trial</button> </div>
  20. 20. payment.dust ? payment.json Control.dust Cell2.dust Cell3.dust Cell4.dust Cell5.dust payment_type_cc.dust payment_type_dd.dust
  21. 21. payment.json { "rules": [], "templateName": "control" }, { "rules": ["PaymentTest(2)"], "templateName": "payment_cell2" }, { "rules": ["PaymentTest(3)"], "templateName": "payment_cell3" }, { "rules": ["PaymentTest(4)"], "templateName": "payment_cell4" }, { "rules": ["PaymentTest(5)"], "templateName": "payment_cell5" }
  22. 22. require('nf-rule-infrastructure')
  23. 23. anatomy of a rule var Rule = require('nf-rule-infrastructure'), PaymentTest; PaymentTest = new Rule('PaymentTest', function(context, params, cb) { var test = context.abtests.get(10); cb(test && test.cell(params.id)); }); module.exports = PaymentTest;
  24. 24. require('nf-template-resolver')
  25. 25. payment.dust dustjs partial
  26. 26. resolver payment.json (mappings) rule
  27. 27. rules control.dust cell2.dust cell3.dust
  28. 28. payment.dust dustjs resolver
  29. 29. <input id=“first-name"><input id=“last-name"> {@inTest id="10" cell=“2a"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class=“payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“3"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“4"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} {@inTest id="10" cell=“5"} <div class="payment-types"> <div id="CC"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id=“card-number"><input id=“security-code”> </div> <div id="DD"> <span class="payment-logos"> <img src="logo1.png"><img src="logo2.png"> <img src="logo3.png"><img src="logo4.png"> </span> <input id="CPF"><input id="bank-name"> <input id="branch-number"><input id="account-number"> <input id="digit"> </div> </div> {/@inTest} <checkbox id="agree-to-terms"></checkbox> <button>Start Your Trial</button> </div>
  30. 30. <div id="payments"> <input id="first-name"> <input id="last-name"> {> payment_method /} <input type="checkbox" id="terms"> <button>Start Your Trial</button> </div>
  31. 31. Wins ▶ combine rules ▶ improve template legibility ▶ increase template reuse
  32. 32. ▶ Templating ▶ Packaging ▶ Bonus Round
  33. 33. Packaging
  34. 34. everything is a module
  35. 35. oldSearch app.js newSearch dep1 dep2 dep3 dep4 dep5 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep
  36. 36. oldSearch app.js newSearch dep1 dep2 dep3 dep4 dep5 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep
  37. 37. app.js import jquery from 'jquery'; import oldSearch from 'oldSearch'; import newSearch from 'newSearch'; export ...
  38. 38. oldSearch app.js newSearch dep1 dep2 dep3 dep4 dep5 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep
  39. 39. 685 files…? 2.5M packages…?
  40. 40. oldSearch app.js newSearch dep1 dep2 dep3 dep4 dep5 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep
  41. 41. problem: conditional dependencies
  42. 42. build request
  43. 43. require('nf-include-when')
  44. 44. oldSearch.js /* * @includewhen rule.notInNewSearch */
  45. 45. newSearch.js /* * @includewhen rule.inNewSearch */
  46. 46. anatomy of a rule var Rule = require('nf-rule-infrastructure'), inNewSearch; inNewSearch = new Rule('inNewSearch', function(context, cb) { var test = context.abtests.get(1534); cb(test && test.cell(1)); }); module.exports = inNewSearch;
  47. 47. require('nf-asset-registry')
  48. 48. app.js import jquery from 'jquery'; import oldSearch from 'oldSearch'; import newSearch from 'newSearch'; export ...
  49. 49. app.js jquery oldSearch.js newSearch.js registry
  50. 50. "app.js": { "deps": [ "jquery", "oldSearch.js", "newSearch.js", ], "depsFull": [ "jquery", "oldSearchDep2.js", "oldSearchDep1.js", "oldSearch.js", "newSearchDep2.js", "newSearchDep1.js", "newSearch.js" ] }
  51. 51. "newSearch.js": { "rule": "inNewSearch", "deps": [ "jquery", "newSearchDep2.js", "newSearchDep1.js", ], "depsFull": [ "jquery", "newSearchSubDep3.js", "newSearchSubDep2.js" "newSearchSubDep1.js" "newSearchDep2.js", "newSearchDep1.js" ] } nf-include-when
  52. 52. require('nf-packager')
  53. 53. var packager = require('nf-packager'), includeWhen = require('nf-include-when'), registries = require('nf-asset-registry'); function getScriptUrl() return packager.getPackageDefinition('app.js', registries, includeWhen); }
  54. 54. "app.js": { "deps": [ "jquery", "oldSearch.js", "newSearch.js", ], "depsFull": [ "jquery", "oldSearchDep2.js", "oldSearchDep1.js", "oldSearch.js", "newSearchDep2.js", "newSearchDep1.js", "newSearch.js" ], "fileSize": "4.41 kB", "fileSizeFull": "120.52 kB" } Step 1: Get the full dependency tree for the requested package from the registry.
  55. 55. [ "jquery", /* no rule */ "oldSearchDep2.js", /* no rule */ "oldSearchDep1.js", /* no rule */ "oldSearch.js", /* rules.notInNewSearch */ "newSearchDep2.js", /* no rule */ "newSearchDep1.js”, /* no rule */ "newSearch.js" /* rules.inNewSearch */ ] Step 2: Determine which files have rules.
  56. 56. [ "jquery", /* no rule */ "oldSearchDep2.js", /* no rule */ "oldSearchDep1.js", /* no rule */ "oldSearch.js", /* rules.notInNewSearch */ "newSearchDep2.js", /* no rule */ "newSearchDep1.js”, /* no rule */ "newSearch.js" /* rules.inNewSearch */ ] Step 3: Run the rules. Filter out all deps that resolved false. ✓
  57. 57. [ "jquery", /* no rule */ "oldSearchDep2.js", /* no rule */ "oldSearchDep1.js", /* no rule */ "oldSearch.js", /* rules.notInNewSearch */ "newSearchDep2.js", /* no rule */ "newSearchDep1.js”, /* no rule */ "newSearch.js" /* rules.inNewSearch */ ] Step 4: Filter out all extraneous sub deps. ✓
  58. 58. Step 5: Concatenate the files. [ "jquery", /* no rule */ "newSearchDep2.js", /* no rule */ "newSearchDep1.js”, /* no rule */ "newSearch.js" /* rules.inNewSearch */ ]
  59. 59. build javascript registry
  60. 60. request registry packager rules
  61. 61. Wins ▶ leverage build time tools ▶ leverage the server ▶ divide and conquer with modules
  62. 62. ▶ Templating ▶ Packaging ▶ Bonus Round
  63. 63. Bonus Round
  64. 64. be creative with the registry
  65. 65. "account/bb/models/ratingHistoryModel.js": { "rule": null, "deps": [...], "depsFull": [...], "depsCount": { "underscore": 2, "backbone": 1, "jquery": 2, "common/requirejs-plugins.js": 4, "requirejs-text": 4, "utils/contextData.js": 1, "common/nfNamespace.js": 1 }, "hash": "dd23b163", "fileSize": "1.21 kB", "fileSizeFull": "173.04 kB" } dependency counting dependency pruning file sizes
  66. 66. @import modules @import (reference) "/common/_nf_defs.less"; @import (reference) "/member/memberCore.less"; @import (reference) "/components/menu.less"; @import (reference) "/components/breadcrumbs.less";
  67. 67. "account/containerResponsive.css": { "rule": null, "deps": [...], "depsFull": [...], "depsCount": [...], "hash": "65a431f3", "fileSize": "709 B", "fileSizeFull": "709 B", "css": { "selectors": 8, "declarationBlocks": 6, "declarations": 17, "mediaQueries": 3 } } css analysis
  68. 68. the best part
  69. 69. "cache": { "account/pin.js": "define('account/pin.js', ['member/memberC…", "account/bb/models/changePlanModel.js": "define('account/b…", "account/bb/models/ratingHistoryModel.js": "define('account…", "account/bb/models/viewingActivityModel.js": "define('account…", "account/bb/views/changePlanView.js": "define('account/bb/vi…", "account/bb/views/changePlanView.js": "define('account/bb/vi…", "account/bb/views/emailSubView.js": "define('account/bb/views…", "account/bb/views/viewingActivityView.js": "define('account…", "common/UITracking.js": "define('common/UITracking.js, ['me…", "common/UITrackingOverlay.js": "define('common/UITrackingOve…", … … …
  70. 70. templates templates mappings javascript css mappings javascript css
  71. 71. templates mappings javascript css UI Bundle
  72. 72. deploy UI bundles anytime
  73. 73. never touch the file system
  74. 74. < 5ms package response times
  75. 75. Wins ▶ static analysis FTW ▶ independent UI deployments ▶ requests never touch the fs ▶ fast package response times
  76. 76. Our Learnings
  77. 77. learn by doing
  78. 78. fail fast move faster
  79. 79. “I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison
  80. 80. simplify
  81. 81. thank you Alex Liu @stinkydofu
  82. 82. questions? Alex Liu @stinkydofu

×