Survey on Requirements Engineering Tools

1. Survey on Requirements Engineering Tools 1 / 26 Survey on Requirements Engineering Tools REFSQ Industry Track 2011 Juan M. Carrillo de Gea Christof Ebert jmcdg1@um.es, Vector, Stuttgart, Germany Joaquín Nicolás, José L. Fernández Alemán, Aurora Vizcaíno Ambrosio Toval Universidad de Castilla-La Universidad de Murcia, Spain Mancha, Spain Essen, Germany, March 29, 2011

2. Survey on Requirements Engineering Tools 2 / 26 Contents 1 Introduction 2 Related work 3 Classiﬁcation framework for RE tools’ capabilities 4 Research methodology Research goals Instrumentation Experimental procedure 5 Results Participants Experimental results Discussion Threats to validity 6 Conclusions and future work Conclusions Future work

3. Survey on Requirements Engineering Tools 3 / 26 Introduction Introduction Beneﬁts from automated support to RE RE tools and support to the RE process RE tools and developer’s expectations What are the RE tools’ desirable features? How are they supported by current RE tools? Description of the state-of-the-art on RE tools Current RE tools identiﬁcation Framework selection and adaptation Survey conduction Updated overview on RE tools’ capabilities and potentials

6. Survey on Requirements Engineering Tools 4 / 26 Related work Related work Surveys on RE tools Surveys on RE [Zowghi and Coulin, 2005] [Liu et al., 2010] [Alenljung and Persson, 2008] [Winkler and von Pilgrim, 2010] [Hall, 2008] [Carlshamre et al., 2001] [Gregoriades and Sutcliffe, 2005] [Johansson et al., 2001] [Maiden et al., 2006] [Benslimane et al., 2007] [Portillo Rodríguez et al., 2010]

7. Survey on Requirements Engineering Tools 5 / 26 Classiﬁcation framework for RE tools’ capabilities Regulations and guidelines ISO/IEC TR 24766:2009 Technical Report (TR) of Type 2 Set of capabilities that the RE tools should support Six major categories Requirements elicitation Requirements analysis Requirements speciﬁcation Requirements V&V Requirements management Other tool capabilities Supplements ISO/IEC 14102:2008 International Standard Evaluation of CASE tools

8. Survey on Requirements Engineering Tools 6 / 26 Classification framework for RE tools’ capabilities RE tools’ capabilities ISO/IEC TR 24766 tool capabilities Category Amount Requirements elicitation 37 Requirements analysis 36 Requirements specification 16 Requirements verification and validation 34 Requirements management 17 Other tool capabilities 17 Total 157

9. Survey on Requirements Engineering Tools 7 / 26 Research methodology Research goals Goal/Question Metric (GQM) framework Goal/Question Metric (GQM) framework [Basili and Rombach, 1988] GQM template [Basili et al., 1999] Goal: To investigate the state-of-the-art on RE tools using a questionnaire aimed at software vendors Question: Do current RE tools address industry challenges? Metric: Capabilities of RE tools (questionnaire scores)

10. Survey on Requirements Engineering Tools 8 / 26 Research methodology Instrumentation Instrumentation I Databases hosting RE tools lists (July, 2010–August, 2010) Database Amount Ian Alexander 67 Alarcos Research Group 7 INCOSE 34 Ludwig Consulting Services 40 Qaguild 7 Volere 71 @WEBO 41 Total sample size after discarding invalid tools 94

11. Survey on Requirements Engineering Tools 9 / 26 Research methodology Instrumentation Instrumentation II DESMET [Kitchenham, 1996] method for evaluating software engineering methods and tools Feature Analysis—Survey 146-items questionnaire 6 ISO TR 24766 categories of features plus modelling plus traceability 157 ISO TR 24766 features → 126 questions 126 technical questions + 20 general, administrative questions Web-based survey using LimeSurvey

14. Survey on Requirements Engineering Tools 10 / 26 Research methodology Experimental procedure Experimental procedure Questionnaire and survey system preparation: September, 2010–November, 2010 Tool representatives ﬁlled in the survey: December 20, 2010–January 15, 2011 Follow-up email and deadline extension to February 7, 2011

15. Survey on Requirements Engineering Tools 11 / 26 Results Participants Participants I 38 participants out of 94 candidates invited (40.42%) Acclaro DFSS Leap SE CASE Spec Aligned Elements MacA&D/WinA&D Cognition Cockpit Avenqo PEP MKS Integrity Cradle Blueprint PACE G-MARC Bright Green Projects Polarion inteGREAT Caliber RM Requirements IRQA Cameo Psoda jUCMNav Requirements+ QFDcapture

16. Survey on Requirements Engineering Tools 12 / 26 Results Participants Participants II QPack RTIME TestTrack RM RaQuest RequisitePro TopTeam Analyst Rational DOORS RMTrak TraceCloud ReqMan Rommana TrackStudio Reqtify Scenario Plus VisibleThread On- Requirements premise/On-demand SpiraTeam Composer

17. Survey on Requirements Engineering Tools 13 / 26 Results Experimental results Administrative information I Year of ﬁrst release Year of last release 8 30 7 25 6 5 20 4 15 3 10 2 1 5 0 0 1975 1977 1979 1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2008 2009 2010 2011 No. of tools No. of tools

18. Survey on Requirements Engineering Tools 14 / 26 Results Experimental results Administrative information II Platform required Type of license Windows/ Windows/ Windows/ Open- Open- Mac OS Linux Mac source and source and OS/Linux free not-free Windows/ Proprietary Mac OS/ and free UNIX/Linux Windows/ UNIX/Linux Windows Web-based Proprietary and not- free

19. Survey on Requirements Engineering Tools 15 / 26 Results Experimental results Administrative information III Cost per individual license Amount of licenses in use 18 14 16 12 14 10 12 10 8 8 6 6 4 4 2 2 0 0 No answer Less than 100 to 500 501 to More than No answer 0 to 100 101 to 1001 to More than 100 1000 1000 1000 10000 10000 Cost No. of licenses

20. Survey on Requirements Engineering Tools 16 / 26 Results Experimental results Correlation between variables 11 variables—one for each category of features plus global score plus cost per individual license plus amount of licenses in use Bivariate correlation tests (Pearson’s Correlation Coefficients) Correlation is significant at the (**) 0.01 level/(*) 0.05 level (1-tailed) Strong direct correlation between each distinct category of features Cost per individual license–analysis (0.336*), modelling (0.404*), traceability (0.329*), specification (0.545**) and global score (0.358) Number of licenses in use–other tool capabilities (0.513**), cost per individual license (0.243) and global score (0.183)

24. Survey on Requirements Engineering Tools 17 / 26 Results Experimental results Technical information I Tools’ scores 20 18 18 17 17 16 16 14 14 1212 12 12 12 11 11 10 10 10 9 8 8 7 6 6 6 6 5 5 5 5 4 4 4 4 4 3 3 3 2 2 2 1 1 1 1 1 0 0 0 0 Very high High Medium Low Very low Elicitation Analysis Specification Modelling V&V Management Traceability Other tool capabilities

25. Survey on Requirements Engineering Tools 18 / 26 Results Experimental results Technical information II Global score (level of accomplishment of the entire ISO/IEC TR 24766)—only calculated for those tools participating in all categories 12 10 8 6 4 2 0 Very high High Medium Low Very low Global

26. Survey on Requirements Engineering Tools 19 / 26 Results Discussion Scenario 1: Requirements elicitation and V&V Storing and managing templates for elicitation (57%), elicitation checklists (60%), prioritization forms (57%) Providing OMG ReqIF/RIF compatibility (26%) Generating exception reports on verification/validation plan cases ↔ verification/validation procedures (57%/60%) Providing standard format for interfacing to verification/validation tools (57%/57%) Implementing some built-in requirements checks (50%) Both: Cockpit, Cradle, QPack, Reqtify Elicitation: MKS Integrity, Polarion Requirements V&V: Aligned Elements, CASE Spec, G-MARC, IRQA, PACE, ReqMan, TraceCloud

29. Survey on Requirements Engineering Tools 20 / 26 Results Discussion Scenario 2: Requirements modelling and specification Providing storage and display of BPMN (44%), goal models (39%), SysML artifacts (31%), DFDs (44%) Checking the document through spell checking, grammar checking, data dictionaries, and acronym tables (60%) Generating the output of the specification in a finished form (65%) Complete loop between RE tool and formatted document (39%) Both: Cockpit, Cradle, PACE Modelling: Reqtify Specification: G-MARC, inteGREAT, MKS Integrity, QPack, DOORS, TraceCloud, VisibleThread

32. Survey on Requirements Engineering Tools 21 / 26 Results Discussion Scenario 3: Requirements traceability Generating reports that compare current and previous versions when a source document is updated (55%) Tracing across the tools’ boundaries (55%) Tracing text to graphics (52%), graphics to graphics (47%), elements within graphics (42%), tables and cells within a table (36%) Generating reports of traceability attributes (60%) Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, Reqtify Close to the best: Avenqo PEP, IRQA, Polarion Requirements, QPack, TopTeam Analyst

33. Survey on Requirements Engineering Tools 21 / 26 Results Discussion Scenario 3: Requirements traceability Generating reports that compare current and previous versions when a source document is updated (55%) Tracing across the tools’ boundaries (55%) Tracing text to graphics (52%), graphics to graphics (47%), elements within graphics (42%), tables and cells within a table (36%) Generating reports of traceability attributes (60%) Best: CASE Spec, Cockpit, Cradle, G-MARC, inteGREAT, Reqtify Close to the best: Avenqo PEP, IRQA, Polarion Requirements, QPack, TopTeam Analyst

34. Survey on Requirements Engineering Tools 22 / 26 Results Discussion Scenario 4: Other features Providing Eclipse support (42%) Providing data federation (28%) Providing an open data model (18%)

35. Survey on Requirements Engineering Tools 23 / 26 Results Threats to validity Threats to validity Internal validity (causal relationships) Mortality percentage (7.32%) Commitment of the RE tools representatives Truthfulness of the answers (Hawthorne effect) Triangulation technique Questionnaire planning and design External validity (generalizations) Participants are representative of the RE tools’ community

36. Survey on Requirements Engineering Tools 23 / 26 Results Threats to validity Threats to validity Internal validity (causal relationships) Mortality percentage (7.32%) Commitment of the RE tools representatives Truthfulness of the answers (Hawthorne effect) Triangulation technique Questionnaire planning and design External validity (generalizations) Participants are representative of the RE tools’ community

37. Survey on Requirements Engineering Tools 24 / 26 Conclusions and future work Conclusions Conclusions The RE process is well-covered by current RE tools Connection between the scores accomplished by the RE tools in each category of features More expensive tools offer better requirements analysis, speciﬁcation, modelling and traceability support No association between the cost per individual license and the global score More extended tools are stronger in ISO/IEC TR 24766 other tool capabilities The amount of licenses in use is not associated with the cost per individual license nor with the global score

41. Survey on Requirements Engineering Tools 25 / 26 Conclusions and future work Future work Future work Support to GSD settings Concrete RE capabilities Elicitation Modelling Traceability ... Dissemination of results

44. Survey on Requirements Engineering Tools 26 / 26 Thank you for your attention Thank you! Any questions?

45. Survey on Requirements Engineering Tools 26 / 26 Thank you for your attention Thank you! Any questions?

46. Survey on Requirements Engineering Tools 27 / 26 Additional information Score calculation Participation of the tool t in the category c: true, NA(t , c) ≥ 0.5 · NQ(c); participant (t , c) = false, otherwise. NA(t , c): number of answers of the tool t in the category c NQ(c): number of questions of the category c Score s of the tool t in the category c: NQ(c) q=1 score(t ,q) ·4 score(t , c) = NQ(c) score(t , q) ∈ {0, 1}: score of the t in the question q tool  Very low, s ∈ [0, 0.5];   Low, s ∈ (0.5, 1.5];   ∀s ∈ [0, 4], discretisation(s) = Medium, s ∈ (1.5, 2.5];  High, s ∈ (2.5, 3.5];    Very high, s ∈ (3.5, 4]. 

49. Survey on Requirements Engineering Tools 28 / 26 Bibliography Bibliography I Alenljung, B. and Persson, A. (2008). DESCRY: a method for evaluating decision-supporting capabilities of requirements engineering tools. In Proc. of the 14th Int. Conf. on Requir. Eng.: Foundation for Software Quality, pages 52–57. Springer-Verlag. Basili, V. and Rombach, H. (1988). The TAME project: towards improvement-oriented software environments. IEEE Trans. Softw. Eng., 14:758–773. Basili, V. R., Shull, F., and Lanubile, F. (1999). Building knowledge through families of experiments. IEEE Trans. Softw. Eng., 25:456–473.

50. Survey on Requirements Engineering Tools 29 / 26 Bibliography Bibliography II Benslimane, Y., Cysneiros, L. M., and Bahli, B. (2007). Assessing critical functional and non-functional requirements for web-based procurement systems: a comprehensive survey. Requir. Eng., 12:191–198. Carlshamre, P., Sandahl, K., Lindvall, M., Regnell, B., and Dag, J. N. (2001). An industrial survey of requirements interdependencies in software product release planning. In Proc. of the Fifth IEEE Int. Symp. on Requir. Eng., pages 84–92, Washington, DC, USA. IEEE Computer Society. Gregoriades, A. and Sutcliffe, A. (2005). Scenario-based assessment of nonfunctional requirements. IEEE Trans. Softw. Eng., 31:392–409.

51. Survey on Requirements Engineering Tools 30 / 26 Bibliography Bibliography III Hall, R. J. (2008). A method and tools for large scale scenarios. Autom. Softw. Eng., 15:113–148. Johansson, E., Wesslén, A., Bratthall, L., and Höst, M. (2001). The importance of quality requirements in software platform development - a survey. In Proc. of the 34th Annual Hawaii Int. Conf. on Sys. Sci., volume 9, Washington, DC, USA. IEEE Computer Society. Kitchenham, B. (1996). DESMET: a method for evaluating software engineering methods and tools. Tech. Report TR96-09, Dept. of Computer Science, University of Keele.

52. Survey on Requirements Engineering Tools 31 / 26 Bibliography Bibliography IV Liu, L., Li, T., and Peng, F. (2010). Why requirements engineering fails: a survey report from China. In Proc. of the 2010 18th IEEE Int. Requir. Eng. Conf., pages 317–322. IEEE Computer Society. Maiden, N., Seyff, N., Grunbacher, P., Otojare, O., and Mitteregger, K. (2006). Making mobile requirements engineering tools usable and useful. In Proc. of the 14th IEEE Int. Requir. Eng. Conf., pages 26–35. Portillo Rodríguez, J., Ebert, C., and Vizcaíno, A. (2010). Technologies and tools for distributed teams. IEEE Softw., 27:10–14. Winkler, S. and von Pilgrim, J. (2010). A survey of traceability in requirements engineering and model-driven development. Softw. and Sys. Modeling, 9:529–565.

53. Survey on Requirements Engineering Tools 32 / 26 Bibliography Bibliography V Zowghi, D. and Coulin, C. (2005). Requirements elicitation: a survey of techniques, approaches, and tools. In Aurum, A. and Wohlin, C., editors, Engineering and Managing Software Requirements, pages 19–46. Springer.

Survey on Requirements Engineering Tools

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (9)

Similar a Survey on Requirements Engineering Tools

Similar a Survey on Requirements Engineering Tools (20)

Último

Último (20)

Survey on Requirements Engineering Tools