Everyone in the performance community is trying to better understand RUM and how to use it. In this presentation I'll introduce, compare, and contrast the two primary methods of performance monitoring. This will be done by using Synthetic and Real User Measurement side by side for 3 months and aggregating and presenting the data that the two methods yield.

1. Jon Fox
@jfox85
Performance Monitoring:
Synthetic
vs
RUM
Velocity 2013
Friday, June 28, 13

2. Performance Monitoring
Friday, June 28, 13

3. Performance Monitoring
Two primary methods:
- Synthetic (Active)
- Real User Measurement - RUM (Passive)
Friday, June 28, 13

4. Synthetic Overview
- Browser in the cloud
- Usually has extra instrumentation
Friday, June 28, 13

5. Synthetic Overview
- Often in controlled environments
- Reliable bandwidth
- Fixed specs
- Dedicated hardware
Friday, June 28, 13

6. Synthetic Overview
- Generally in several geographic locations
- Offer various browsers / rendering engines
Friday, June 28, 13

7. Synthetic - How It Works
Provider Your Website
Cloud Agent
Friday, June 28, 13

8. Synthetic - How It Works
Provider Your Website
Tell provider to run a test
Friday, June 28, 13

9. Synthetic - How It Works
Provider Your Website
Provider sends the test to an agent
Friday, June 28, 13

10. Synthetic - How It Works
Provider Your Website
The agent requests your website
Friday, June 28, 13

11. Synthetic - How It Works
Provider Your Website
The agent receives the page and renders the response
while measuring load times
Friday, June 28, 13

12. Synthetic - How It Works
Provider Your Website
The results are sent back to the provider to be viewed
and possibly aggregated with other results
Friday, June 28, 13

13. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

14. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

15. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

16. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

17. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

18. Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13

19. Real User Measurement (RUM)
On the other hand we have RUM
Friday, June 28, 13

20. RUM Overview
- Add JavaScript to the page
- Gather timing data from the browser
<script type='text/javascript'>var TBRUM=TBRUM||{};TBRUM.q=TBRUM.q||
[];TBRUM.q.push(['mark','firstbyte',(new Date).getTime()]);</script>
Friday, June 28, 13

21. RUM Overview
- Every page and every visitor
- Actual visitor timings
Friday, June 28, 13

22. RUM Overview
- Covers every device and browser that visits your site
Friday, June 28, 13

23. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
Friday, June 28, 13

24. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The site visitor decides to come to your site
Friday, June 28, 13

25. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
They request your page and view it as normal
Friday, June 28, 13

26. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The page includes the RUM Javascript, which collects
timing data from the browser
SiteVisitor
Friday, June 28, 13

27. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The timing data is then sent to the collection service
via a beacon image request
SiteVisitor
Friday, June 28, 13

28. RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The data is then aggregated and processed to
generate pretty charts and graphs
SiteVisitor
Friday, June 28, 13

29. RUM - Navigation Timing Spec
Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
Friday, June 28, 13

30. RUM - Navigation Timing Availability
Source: http://caniuse.com/nav-timing
Friday, June 28, 13

31. RUM - Other Timing Methods
- GoogleToolbar
Source: http://www.google.com/toolbar/ie/index.html
Friday, June 28, 13

32. RUM - Other Timing Methods
- Page unload cookie
Friday, June 28, 13

33. RUM - Other Timing Methods
- Start of document timer
Friday, June 28, 13

34. RUM - Other Data
- Estimated Bandwidth
- IP Address
- URL
- Referer
- Cookies
- Local Storage support
- Number of images, JavaScript, CSS, etc
- User actions on the page
- Memory Usage (very limited availability)
- Anything else you can get from JavaScript!
Friday, June 28, 13

35. RUM - Resource Timing Spec
Source: http://www.w3.org/TR/resource-timing/
Friday, June 28, 13

36. State of Resource Timing Spec
- IE10, Chrome, Chrome Mobile
-Various degrees of support / compatibility
- Same origin policy by default
- Override withTiming-Allow-Origin HTTP response header
Friday, June 28, 13

37. RUM - What kind of data do we get?
Friday, June 28, 13

38. RUM - What kind of data do we get?
Friday, June 28, 13

39. RUM - What kind of data do we get?
Friday, June 28, 13

40. RUM - What kind of data do we get?
Friday, June 28, 13

41. Comparison
So how do these methods compare?
Friday, June 28, 13

42. Methodology
Ran both on velocityconf.com
Torbit used for RUM
Webpagetest for Synthetic
Approximately 3 months (3/18/13 - 6/14/13)
Friday, June 28, 13

43. Methodology
Synthetic hit the following URLs once an hour:
http://velocityconf.com/velocity2013/
http://velocityconf.com/velocity2013/public/schedule/topic/935
http://velocityconf.com/velocity2013/public/content/exhibitors
http://velocityconf.com/velocity2013/public/schedule/grid
http://velocityconf.com/velocity2013/public/content/about
http://velocityconf.com/velocity2013/public/schedule/speakers
Friday, June 28, 13

44. Methodology
From the following synthetic agents:
Dulles
- Chrome
- IE8
- IE9
- IE10
- Firefox
- Safari
Chicago
- Chrome
San Jose
- IE9
Los Angeles
- Chrome
Dublin
- IE8
Sao Paulo
- IE8
Toronto
- IE8
Amsterdam
- IE9
- Chrome
Frankfurt
- IE9
- Firefox
Moscow
- Chrome
- IE8
Stockholm
- Safari
Delhi
- IE8
China
- IE 8
Tokyo
- IE 8
Sydney
- Chrome
Friday, June 28, 13

45. Methodology
Real User Measurement:
RUM was run on every page
(as far as I know)
Friday, June 28, 13

46. Results - Data Overview
Synthetic Load Times
RUM Load Times
Friday, June 28, 13

47. Results - Data Overview
Synthetic Load Times
RUM Load Times
Friday, June 28, 13

48. Results - Browser Coverage
Synthetic RUM
- Chrome
- IE 10
- IE 9
- IE 8
- Firefox
- Safari
- Chrome
- IE 10
- IE 9
- IE 8
- IE 7
- Firefox
- Safari
- Opera
- Android
- iPhone
- And more...
Friday, June 28, 13

49. Results - Geographic Coverage
Synthetic:
Friday, June 28, 13

50. Results - Geographic Coverage
RUM:
Friday, June 28, 13

51. Results - Data Insights
RUM:
Synthetic:
0"
5"
10"
15"
20"
25"
30"
35"
40"
45"
50"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24"
Load%Time%(sec)%
Time%(hrs)%
RUM"Avg"
90th"
Syn"Avg"
Friday, June 28, 13

52. Results - Data Insights
0"
1"
2"
3"
4"
5"
6"
7"
8"
9"
10"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29"
Load%Times%(sec)%
Time%(hrs)%
RUM"Avg"
RUM"90th"
Syn"Avg"
0"
200"
400"
600"
800"
1000"
1200"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29"
Pageviews%
Pageviews"
Friday, June 28, 13

53. Results - Data Insights
Friday, June 28, 13

54. Results - Data Insights
Friday, June 28, 13

55. When To Use?
When should you use one over the other?
Friday, June 28, 13

56. When to use Synthetic
- Pre-production
- Low / no traffic pages
- Going deep on a problem
- Critical pages and paths
- Monitoring competitors
Friday, June 28, 13

57. When to use RUM
- Ultimate source of truth
- Finding out you have a problem
- Monitoring trends over time
-Vendor or departmental SLAs
Friday, June 28, 13

58. Build or Buy? - Synthetic
Should you build it yourself or buy a solution?
Friday, June 28, 13

59. Build - Synthetic
Open source
Self-host options
Friday, June 28, 13

60. Reasons to Buy - Synthetic
Need lots of agents
Friday, June 28, 13

61. Reasons to Buy - Synthetic
Need lots of locations
Friday, June 28, 13

62. Reasons to Buy - Synthetic
Simpler aggregation and reporting
Friday, June 28, 13

63. Build or Buy? - RUM
What about building your own RUM?
Friday, June 28, 13

64. Build - RUM
Collection is pretty easy...
Episodes - Steve Souders
http://stevesouders.com/episodes/
Boomerang - PhilipTellis
https://github.com/lognormal/boomerang
Friday, June 28, 13

65. Reasons to Buy - RUM
Storage and processing are hard...
Friday, June 28, 13

66. Reasons to Buy - RUM
Dealing with outliers & bad data
Friday, June 28, 13

67. Conclusions
So which method should you use?
Friday, June 28, 13

68. Conclusions
A mix of both is best
Friday, June 28, 13

69. Why do we care?
And in case you need a reminder...
Friday, June 28, 13

70. Performance Matters
Aggregated data from 1,000+ Torbit Sites
Friday, June 28, 13

71. Performance Matters
Aggregated data from 1,000+ Torbit Sites
Friday, June 28, 13

72. Conclusions
Free options are available:
Synthetic RUM
Friday, June 28, 13

73. Conclusions
Start collecting data now!
Friday, June 28, 13

74. Questions?
Jon Fox
jon@torbit.com
@jfox85
Friday, June 28, 13