1. Not Waving But Drowning Understanding Data Andrew Hingston switchsolutions.com.au

2. Not Waving But Drowning by Stevie Smith Nobody heard him, the dead man, But still he lay moaning: I was much further out than you thought And not waving but drowning. Poor chap, he always loved larking And now he's dead It must have been too cold for him his heart gave way, They said. Oh, no nono, it was too cold always (Still the dead one lay moaning) I was much too far out all my life And not waving but drowning. Name and what you do Hobby others don’t know Waving or drowning in data?

3. 3 Why understanddata ?

4. 4 Sometimes we are like this

5. 5 … and other times like this!

6. Memorability Anchoring and adjustment Status quo Self-serving Negative comparisons Framing BIAS 6

7. Sources of POWER Legitimate power Referent power Expert power Reward power Coercive power French and Raven (1959)“The bases of social power”See Wikipedia:“Power (philosophy)” 7

8. Persuasion Reciprocity Consistency Social proof Authority Liking Scarcity Robert Cialdini (2001)“Influence: Science and practice”See Wikipedia “Robert Cialdini” 8

9. Busting jargon 9

10. Steps … Specify problem Propose answers Identify the right tools Obtain your data Visualise it Crunch numbers Interpret, persuade, apply 10

11. Today 1. Visualising data 2. Measuring middle 3. Measuring spread 4. Histograms 5. Box plots 6. Bell-shaped curves 7. Exercises using R 11 Course 1. Understanding data 2. Monitoring processes 3. Exploring relationships

12. 12 1 Visualising

13. Why visualise your data? For you Fast understanding Build solid ‘foundation’ Flags problems 13 For others Easier to follow Memorable Less info overload More convincing

14. 14 GOOD CHARTS BAD CHARTS

15. Lots of charts … 15

16. Bar charts 16 Most revenueis still from ads

17. Column charts 17 And that storygoes back a while

18. Line charts 18 Go Android! Source: StatCounterGlobalStats

19. Scatterplot 19 When NASDAQ  So does Google!

20. GUI Shoot Out 20 iOS Win Phone 7 Symbian Android BlackBerryOS Score out of 10 Form Functionality

21. Bubble chart 21

22. 22

23. Pie chart 23

24. Stacked column chart 24 Check outour profits! Tax General Admin Sales & Marketing R&D Traffic acquisition& other

25. Radar charts 25

26. Compound charts (eg. Stock Chart) 26 1. Visualising data with charts

27. PRESENTATIONS! 5 second rule Simplicity Colour Font size Handouts Jargon Pictures 27

28. 28 2 Middle

29. Mode = 1Median = 3 Mean = 4 Trimmed Mean = 3 29

30. Weighted average 30 = 0.25 x 0.04 + 0.50 x -0.08 + 0.25 x 0.04 = 2% Not the mean which is ZERO!

31. 31 3 Spread

32. 32

33. 33 4 Histograms

34. 34 LOOK Shape Peak Multi-peak Lumpy Long tail

35. FreedmanDiaconisEquation Suggests good bin width Very robust Use common sense though! 35 IQR = interquartile range, n = data points

36. Double - Peaked Bell - Shaped Comb Plateau Skewed Truncated Edge - Peaked Isolated - Peaked What is the story? 36

37. HISTOGRAMS Visualise spread Easy to interpret Indicates skewness Indicates multiple peaks 37 BAD GOOD Data points? Width of bins? Sample size?

38. 38 5 Box plots

39. 39 50 45 40 35 30 25 20 Outliers Upper fence Highest datainside fence 1.5 × IQR Quartile 3 BoxIQR Middle 50% Mean 99.3%if normallydistributed Median Quartile 1 1.5 × IQR Lowest datainside fence Lower fence

40. BOX PLOTS Mean and median Spread Symmetry Outliers 40 BAD GOOD Not intuitive Need stats package Bad for presentations

41. Interpreting shape 41 Right-Skewed Left-Skewed Symmetric Q Median Q Q Median Q Q Median Q 1 3 1 3 1 3 * * * Mean

42. Side-by-side boxplots If boxes don’t overlap then difference between groups is ‘statistically significant’ BUT THIS IS A PRETTY ROUGH TEST! 42 Boxes DON’T overlap

43. Why?Visualise Middle Spread Histograms Boxplots Recap 43

44. 44 6 Bell-shapedcurves

45. WARNING! 45 

46. 46 NORMALdistribution 68.2% chance 95.4% chance 99.7% chance stddev stddev Bell shaped mean=median=mode symmetricalGoes from - to + Area under curve = 1

47. Mean return 10% Std deviation 10% Likelihood of -50% 47 ?

48. Different meansbut both still Normal 48 mean = 1 mean = +1

49. Different standard deviationsbut both still Normal 49 Stddev = 1 Stddev = 2

50. Calculating probability 50 stddev =1 mean = 0 X = How many stddev from the mean? This is called Z Put Z into spreadsheet = NORMSDIST ( 1 ) which gives 84% P.S.  is mean is stddev

51. … is the same as … 51 stddev =1 mean = 2 X = How many stddev from the mean? This is called Z Put Z into spreadsheet = NORMSDIST ( 1 ) which gives 84% P.S.  is mean is stddev

52. … or take a shortcut! 52 stddev =1 mean = 2 X = In OpenOfficeCalc = NORMSDIST ( 1 ) which gives 84% = NORMDIST ( 3 ; 2 ; 1; 1 ) also gives 84% P.S.  is mean is stddev   X

53. NEVERCALCULATE THE PROBABILITYOF ONE POINT 53 YOU CAN ONLY CALCULATE PROBABILITY OF A REGION

54. SPEED challenge 54

55. WARNING! 55 

56. Double - Peaked Bell - Shaped Comb Plateau Skewed Truncated Edge - Peaked Isolated - Peaked But data might not be Normal 56

57. Central Limit Theorem The shape of the data doesn’t matter … if you take a large enough sample ( > 30 ) the means of the sample will follow a Normal distribution shape around the mean of the underlying data 57 DEMO

58. Use modified Z-score formula for probabilitythat mean of a sampletakes on certain values 58  is mean is stddevn is sample size Use Z-score formulaif your data follows aNormal distribution

59. 59 7 Exercises

60. PROJECT 60

61. Exercises in R Test Data Exercise 4 Employee Expenses Exercise 5 Toll Booth Exercise 6 Cycle World Exercise 9 Chan’s Linen Exercise 10 Stock returns For the brave! Exercise 11 Social Insight Test Exercise 13 Quality Exercise 16 Car production 61

62. THANKS Feedback please! 62