Natalia Juristo
University of Oulu
&
Universidad Politécnica de Madrid
FiDiProFinland Distinguished Professor Program
Project start, January 2013
Experimental Software Engineering
Industrial La...
MISSION
!"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)%
+/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15%
%
6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$...
!"#$%#&'$()*$'+$,-.$*+/0&#12$
3&$'+$1+&#$*+/0&#12$3-$'"#$
&#0435#$+2$+'"#0&
6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$
:#-#0)'#$"...
!"#$%&'$()*+&*+),$+-(./+01/++
)-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+
8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+
;$,&(9+),$+;$,12...
!"#$"%%&$"#'($)*$+,$"%)'
!"#$"%&'($)(*+(),-.%/$0&"s and market-speak to
1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$
Fac...
What Use Are Experiments for Industry?
!"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+
)$56(-1-4&$*+
7  !"#$%&'$()*+#%-2&8$...
What an Experiment Is Not
!"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$
1*%"$-%"#&$,*'*/2&$%#&',$
$
3+#4,"-%$#()#&*#+5#$
6 ...
A SE Experiment
Task i
A
B
M1
M1
Task i
M2
M2
Software process
Software process
Task i
Task i
Task i
Task i
What is Needed to
Conduct an Experiment?
!  "#$%&%'()%*+#
,  -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+#
!  6&%*#780#2...
Train people in
exchange
Researchers provide free training
Companies provide participants for the experiment
Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Reminder of
Unit Testing
concepts
3 Exercises to
warm up
CO...
Quality
@ F-Secure TDD Experiments
•  The mean for ITL is 54% of
correctly implemented user
stories
•  The mean for TDD is 76%
•  Subjects applying
TDD get a...
Productivity
@ F-Secure TDD Experiments
•  The mean for ITL is 23%
•  The mean for TDD is
greater at 48%
•  Subjects applying TDD
show a 24%
improvement in
produc...
Other Data Analysis
Dissecting TDD: Post-hoc analysis
!  "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%,
/)8'*#+,*37&96%$,:8)2*(...
Schedule: Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Testing tool
Concepts
2 Tool Exercises
1 Motivati...
Replications of Our TDD Experiment
!"#$%&'$()*+,-$.&/(,01,#%$2#3.),,
4  5*6*,7,58(&),.$9(/,
:  ;2<$=>%$,?@,.&)$.A,2B,CD,#*...
Quality: TL vs TDD
F-Secure Elektrobit Univ. Politecnica Valencia
TL < TDD (42%)
Java
31 Students
No need more subjects
TL...
Productivity: ITL vs TDD
ITL < TDD (41%)
Java
31 Students
No need of more subjects
ITL < TDD (21%)
C++
9 Professionals
No ...
!"#$$%$&'#$%()*+,+-'+.%
/)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<%
=>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(%
6,&(4'@5)2...
We Are Now Recruiting !
!  "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2#&#12#3$*+$&#.#*4*+,$(5&$.(5&1#$
(+$677$
8  "#$9*''$&5+$2:#$#;0...
Natalia Juristo
University of Oulu
&
Universidad Politécnica de Madrid
Email: natalia@fi.upm.es
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Conducting Experiments in Software Industry
Próxima SlideShare
Cargando en…5
×

Conducting Experiments in Software Industry

355 visualizaciones

Publicado el

What use are experiments for industry?
1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions
2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development

Publicado en: Ingeniería
0 comentarios
0 recomendaciones
Estadísticas
Notas
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Sin descargas
Visualizaciones
Visualizaciones totales
355
En SlideShare
0
De insertados
0
Número de insertados
9
Acciones
Compartido
0
Descargas
6
Comentarios
0
Recomendaciones
0
Insertados 0
No insertados

No hay notas en la diapositiva.

Conducting Experiments in Software Industry

  1. 1. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid
  2. 2. FiDiProFinland Distinguished Professor Program Project start, January 2013 Experimental Software Engineering Industrial Laboratory (ESEIL)
  3. 3. MISSION !"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)% +/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15% % 6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$+&&% "7*.)%)(+%)(%*)$&%((%*8%+/0+1#2+$)&%% 9.&(%8*14"1-%)(+%%+,%-./%&0#$1,#-#2.'/1 "00:#+-%)*%3&45$%2'%1'%&%-#64&1"7*.)% &*34"1+%-+;+:*02+$)%
  4. 4. !"#$%#&'$()*$'+$,-.$*+/0&#12$ 3&$'+$1+&#$*+/0&#12$3-$'"#$ &#0435#$+2$+'"#0& 6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$ :#-#0)'#$"3:"$;/)13'*$$ 8-+(1#.:#$ <+-./59-:$#=>#037#-'&$%03-:&$'"#$ &53#-9,5$7#'"+.$'+$?+@()0#$ A-:3-##03-:$
  5. 5. !"#$%&'$()*+&*+),$+-(./+01/++ )-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+ 8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+ ;$,&(9+),$+;$,12&-%+0$+-;*$%2$+ <9$(7=/+!"#$"%&'(+#.1/&(:+1+%-.$++ &(+*->01%$+9$2$.-#'$()+1(9+),$++ )*++'),*+(+;$)0$$(+),$'+
  6. 6. !"#$"%%&$"#'($)*$+,$"%)' !"#$"%&'($)(*+(),-.%/$0&"s and market-speak to 1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$ Fact-based management -*$%"./*0,,1'20,$('/"($"#)' )345,('3%,+'64'$(%".71' 5))4&+"0&4&6#$+-.'&,-.%/$7("8.(%%(%$".'$%6*(.96:%$ -2$%-;7"*($'(<(4-)3(.6$6(+:.-4-9&(%$ $ =10>(+,<($-)&.&-.%$+".$4("'$6-$3&%6"8(.$ )(*+(),-.%$-2$*("4&6#$
  7. 7. What Use Are Experiments for Industry? !"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+ )$56(-1-4&$*+ 7  !"#$%&'$()*+#%-2&8$+,(+-9:$532$+5-'#,%&*-(+%,)6$%+)6,(+*09:$532$+ -#&(&-(*+ ;6$+%$*01)*+-/+$"#$%&'$()*+*6-018+,11-.+&(80*)%<+)-+ 7  =,*$+8$5&*&-(*+-(+-9:$532$+4%-0(8*+ >  ?$*01)*+/%-'+,(+$"#$%&'$()+,%$+2,1&8+,)+5-'#,(<+1$2$1+ 7  @(5%$,*$+5-()%-1+-(+*-A.,%$+8$2$1-#'$()+ >  @8$(3/<+%$1$2,()+2,%&,91$*+-/+*-A.,%$+)$56(-1-4&$*+ 1. 2.
  8. 8. What an Experiment Is Not !"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$ 1*%"$-%"#&$,*'*/2&$%#&',$ $ 3+#4,"-%$#()#&*#+5#$ 6  78$-+5#$1#+%$92//--+*+:;$ 6  <2++-%$9#$:#+#&2/*=#0$%-$-%"#&$02>,?$-%"#&$)#-)/#$ @.9A#5BC#$&#)#2%#0$#()#&*#+5#$ 6  7D2./$"2,$2$/-%$-E$#()#&*#+5#$'2+2:*+:$)&-A#5%,;$ 6  <2+$:#+#&2%#$&./#,4-E4%".'9,?$*,$5-+F+#0$%-$D2./?$*,$,.9A#5BC#$ <-//#5%#0$02%2$E&-'$)&-A#5%,$ 6  73.&$0#)2&%'#+%$,%-&#,$02%2$E&-'$2//$)&-A#5%,;$ 6  8+$&#2/$)&-A#5%,?$C2&*29/#,$52++-%$9#$5-+%&-//#0$2,$+#5#,,2&>$%-$.+0#&,%2+0$1"2%$%"#$ 52.,#,$2+0$#G#5%,$2&#$$
  9. 9. A SE Experiment Task i A B M1 M1 Task i M2 M2 Software process Software process Task i Task i Task i Task i
  10. 10. What is Needed to Conduct an Experiment? !  "#$%&%'()%*+# ,  -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+# !  6&%*#780#29#)(++24'%# !  "#$%&%'()%*+#5:$%*#1.%#+3/%#;(:$2<(:+# ,  =(:>:%$#2:#3#*((/0#5+2:?#+3/%#2:9*3+1*5;15*%0#@# !  "#$%&%'()%*+#$(2:?#1.%#+3/%#13+A#5:$%*#+15$B# ,  C#13+A#1B)2;3''B#13A%+#31#'%3+1#1D(#.(5*+#1(#;(/)'%1%# !  E%)%:$2:?#(:#1.%#%F)%*2/%:13'#$%+2?:#D%#;3:#+)'21#)3*<;2)3:1+#(*#%&%*B4($B# $(2:?#21#1D2;%#
  11. 11. Train people in exchange Researchers provide free training Companies provide participants for the experiment
  12. 12. Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Reminder of Unit Testing concepts 3 Exercises to warm up CONTROL TASK (Test-last) DAY 2 Test-driven Development 3 TDD Exercises DAY 3 2 TDD Exercises TREATMENT TASK (TDD)
  13. 13. Quality @ F-Secure TDD Experiments
  14. 14. •  The mean for ITL is 54% of correctly implemented user stories •  The mean for TDD is 76% •  Subjects applying TDD get a 17% improvement in quality compared to subjects applying ITL Quality @ F-Secure TDD Experiments •  The observed difference between ITL and TDD is not significant EFFECT SIZE HYPOTHESIS TESTING
  15. 15. Productivity @ F-Secure TDD Experiments
  16. 16. •  The mean for ITL is 23% •  The mean for TDD is greater at 48% •  Subjects applying TDD show a 24% improvement in productivity compared to subjects applying ITL •  The observed differences between ITL and TDD is significant Productivity @ F-Secure TDD Experiments
  17. 17. Other Data Analysis
  18. 18. Dissecting TDD: Post-hoc analysis !  "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%, /)8'*#+,*37&96%$,:8)2*(;, <  =)#9,/;/2%', <  0%'(,>&'(?,(.%,/9$%, <  @%A)/(9&*#+,)B%&,&)(.%&,(.)#,$%'*+#,>&'(, !  =)#9C$%6%2973%#(,.)',39'(,4%)&*#+,9#,*37&96%$,:8)2*(;,,, <  0*#;,DC3*#8(%,$%6%2973%#(,/;/2%', <  E,8#*(,*#/&%)'%,*#,:8)2*(;,A9&,%6%&;,FGDH,&%$8/59#,*#,/;/2%,'*I%, !  =%%$,(9,4%,/9#>&3%$,-*(.,)#,%J7%&*3%#(K,
  19. 19. Schedule: Our TDD Experiment TRAINING EXERCISES EXPERIMENTAL TASKS DAY 1 Testing tool Concepts 2 Tool Exercises 1 Motivational exercise (ITL) BASELINE TASK (Your Way) DAY 2 Slicing 2 Slicing Exercises CONTROL TASK (ITL) DAY 3 Test-driven Development 3 TDD Exercises TREATMENT TASK (TDD)
  20. 20. Replications of Our TDD Experiment !"#$%&'$()*+,-$.&/(,01,#%$2#3.),, 4  5*6*,7,58(&),.$9(/, :  ;2<$=>%$,?@,.&)$.A,2B,CD,#*%E=&#*()., :  8FG,?<#*&(A,2B,@0,#*%E=&#*()., 4  HII,7,JK$.),.$9(/, :  !+$L)%3M&),2B,N,#*%E=&#*()., !"#$%&'$()*+,-$.&/(,C1,=%3..,7,.+&=&(/,)%*&(&(/, 4  5*6*,7,58(&),.$9(/, :  H3'#*(O,P,?!.)3(&*A, :  8FQ,?<#*&(A, :  !<F!,?!=>*R3%A, !"#$%&'$()*+,-$.&/(,@1,!-C,7,S3>%T*O,=3()%3+,)%$*)'$(),, 4  HII,7,U(O,.$9(/, :  !%&=..3(, 4  FO)V3(,7,WW, :  ;2<$=>%$,
  21. 21. Quality: TL vs TDD F-Secure Elektrobit Univ. Politecnica Valencia TL < TDD (42%) Java 31 Students No need more subjects TL = TDD (32%) C++ 9 Professionals Need more subjects TL = TDD (17%) Java 24 Professionals No need more subjects
  22. 22. Productivity: ITL vs TDD ITL < TDD (41%) Java 31 Students No need of more subjects ITL < TDD (21%) C++ 9 Professionals No need of more subjects ITL < TDD (24%) Java 24 Professionals No need of more subjects F-Secure Elektrobit Univ. Politecnica Valencia
  23. 23. !"#$$%$&'#$%()*+,+-'+.% /)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<% =>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(% 6,&(4'@5)28% A&-B.)0-)C'#-2%()*+,+-'+.%9&,%?4#$)28% !)0-)C'#-2%()*+,+-'+.%9&,%6,&(4'@5)28% % !2#3$+%,+.4$2.%9&,%()*+,+-2%)-9,#.2,4'24,+.% D#5#%E%FGG% DH-)2%E%I2+.2% %
  24. 24. We Are Now Recruiting ! !  "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2#&#12#3$*+$&#.#*4*+,$(5&$.(5&1#$ (+$677$ 8  "#$9*''$&5+$2:#$#;0#&*/#+2$%1$0%&2$(-$2:#$2&%*+*+,$ <&##$=$3%>1$2&%*+*+,$(+$677$ ?(5&$'(.%'$#;0#&*/#+2$&#15'21$ @*3#$50$2($<%.2AB%1#3$C%+%,#/#+2$ natalia.juristo@oulu.fi
  25. 25. Natalia Juristo University of Oulu & Universidad Politécnica de Madrid Email: natalia@fi.upm.es

×