Invited talk at SETQA-2009 workshop http://compbio.uchsc.edu/SETQA-NLP2009/index.shtml

1. The road from good software engineering

2. to good science

3. ...is a two way street...

4. Good

6. Quality

7. Science

8. Develop theories or models that let us make predictions about the world

9. Our world is language...

10. Do our work so that others can reproduce our results

11. Science

12. Reproduce Results

13. Good Software Engineering

14. ...are those methods that result in software that anyone can use, anytime, anywhere...

15. ...to reproduce our results...

16. Experimental Results in NLP/CL Papers

17. Results are output from test cases

18. Empirical / experimental results that you publish are the test cases for your ideas

19. ...and your software...

20. Can't discount role of softwawe

21. ...although many try...

22. “It's really the ideas that count...”

23. “Well, the algorithm is described in the paper...”

24. “It's really just a prototype...”

25. “Well, I got a new computer and I don't think the software made it to the new one...”

26. “Ummm....my student left and I don't quite know how he did all this...”

27. I did this experiment on X

28. Here are the results...

29. Accept them

30. No, the software isn't available

31. Neither is the data

32. I simply assume you have 8 months available to reinvent my method

33. And that you can do that from an incomplete description

34. Cheers!

35. That's many things

36. It's not science

37. Empiricism is Not a Matter of Faith Computational Linguistics September 2008

38. Software and NLP

41. Good Software

42. It should work

43. Anytime

44. Anywhere

45. For Anyone

46. ...and it should certainly work for you 6 months in the future

47. ...or 5 years from now...

48. ...and it should work for someone else now, and 5 years from now..

49. ...even after you've moved on and aren't answering email, even after the project is over

50. If your software can do that, it's pretty well engineered

51. Will your software work in 40 years?

52. You should hope so

53. Make choices that make that at least possible

54. Think of your software as a time capsule

55. Think of it as your chance for immortality

57. How many hours have you spent away from loved ones, friends, adventure, nature, romance, and life...to create software?

58. At least make it last...

59. Let someone 100 years from now unpack your code and data, and be able to read it, understand it, run it, and modify it

60. Maybe it's your grandchildren, wondering why they never saw you?

61. Let yourself be able to do the same thing in 10 years

62. Will the Linux Kernel be available and running in X years?

63. There's a good chance

64. Company won't go out of business

65. ANSI C will be around a long time

66. Virtualization will keep architectures alive even when hardware is gone

67. Make choices that give your code (and your legacy) a chance too

68. Don't rely on the newest priceiest weirdest goofball proprietary bleeding edge hardware and software

71. Anyone

72. ...with $200?

73. ...with $20,000

74. ...with a PhD in Computer Science?

75. ...and a staff of 10?

76. ...with 4 weeks available to debug?

77. ...and another 6 months to reimplement?

78. Good

79. Software won't stop children from dying of war and disease...

80. So be a little humble

81. Appreciate your good fortune

82. And push yourself a little harder

83. Don't act all whiny and put-upon when people ask you for code or data

84. Especially since a lot of that work is funded by people working very hard making less in a month than we make a in a week

85. ...or that we spend going to NAACL

86. Appreciate our good fortune

87. Good Science

88. Produces theories that make reliable predictions about the world

89. Reproducible Experimental Results

90. Anytime

91. Anywhere

92. For Anyone

93. Gravity

94. A Good Theory

95. Works now

96. Will work in 10 years

97. Works here

98. Works on the moon

99. Works for me

100. Works for you

101. Falling Objects

102. Gravity is a force, not an artifact

103. Telescope

104. Works anytime, anywhere, for anyone

105. The old ones still work

107. We share the big ones...

109. If we have access to the same resources, we can reproduce each other's results

110. If we have access to the same resources, we can reproduce each other's results

111. We need to work a lot harder (and engineer systems a lot better) to make that happen

112. Not convinced?

113. Conduct the following experiment

114. Randomly select 1 of your papers

115. Reproduce your results

116. If you can't...

117. Do you think anyone else can?

118. What if nobody could have reproduced Galileo's falling objects experimental results? Would we simply believe him?

119. They barely believed him at the time

120. If you can, try it for someone else's papers

121. Lessons from Science

122. We don't get it right the first time

123. If I have seen further it is only by standing on the shoulders of giants

125. (who were mostly wrong)

126. Aristotle (384 – 322 BC)

127. There are 4 elements

128. The heavens are different

129. Different rules apply

130. Before the telescope, the heavens really were different

131. Other planets were balls of fire, like the stars, like the sun

133. Ptolemy (90 – 168)

136. Crazy?

137. Very reliably predicts the movement of heavenly bodies

138. Instrumentalist

139. A theory that reliably explains and predicts the existing data

140. Realistic

141. A theory that describes things as they “really” are

144. A great instrumentalist A great data preserver and collector

145. Copernicus (1473 - 1543)

147. Wasn't much of an observer

149. Found Ptolmey's model overly complicated

150. Wanted a simpler explanation

151. Wanted a simpler explanation

152. Came up with another model that was consistent with Ptolmey's data

154. Great!

155. (Well, better)

156. Uniform Motion

157. Perfect circles

160. Brahe (1546 - 1601)

162. A great observational astronomer, the last naked eye astronomer

164. Galileo (1564 - 1642)

167. 1609 Telescope

168. 1610 Observed 4 moons of Jupiter

169. Back to Tycho

170. Made remarkably accurate observations for 20 years

171. Knew about Copernicus

172. Arrived at his own theory

174. A hybrid model

175. Fits and predicts the observed data

176. Data Sharing

178. Kepler (1571 - 1630)

180. Why are there 6 planets?

181. Why are they so positioned?

182. Geometry and Perfect Solids

184. In 1601 Tycho bequeathed his data...

185. Kepler's Laws of Planetary Motion

186. Varying velocity

187. Elliptical Orbits

188. ...around the Sun

189. It was left to Newton to work out what held the planets in place and made them move...

190. History of Science?

191. We are wrong many many times before we are right

192. Progress happens when people leave their data and instruments behind

193. Ptolemy (90 - 168) Copernicus (1473 - 1543) Tycho (1546 – 1601) Galileo (1564 - 1642) Kepler (1571 - 1630) Newton (1642 - 1727)

194. Good science and good software assume you don't get it right at first

195. Leave something behind for your successors to build on

196. Here lies our fried Ted We're sad that's he dead http://www.d.umn.edu/~tpederse