In the context of a European ERDF project, researchers from UMONS and FUNDP in Belgium carried out a survey on the use of software quality practices in software producing and maintaining companies in Wallonia. Contact: tom.mens@umons.ac.be

1. A survey on software quality practice -
Pilot study in the Walloon region
Javier Perez, Tom Mens, Service de Génie Logiciel, Université de Mons
Flora Kamseu, Naji Habra, PRECISE lab, FUNDP
Presented at SATToSE seminar
University of Koblenz, August 2012
SATTOSE 2012

2. Context
• Portefeuille TIC
– ERDF project lead by CETIC (2007-2013)
• CEIQS: Center of expertise in engineering and
quality of systems
– aimed at developing a portfolio of innovative techniques
allowing local companies to master the diversity,
complexity, quality and rapid evolution of information
systems
• workpackage QUALGEN
– collaboration between FUNDP and UMONS since 2010
– Supported by Wallonia
SATTOSE 2012 2

3. Objectives
• Explore how quality-related software development
practice is being performed in industry
• Compare this across different regions and
countries
• Relates this to what is being taught in academia
– Is there a gap between teaching and industry needs?
– What good methodologies, practices, tools are not
being used and why?
SATTOSE 2012 3

4. About the survey
• Online survey carried out in Walloon region
– Using LimeSurvey, from 29/5 till 30/6 2012
– Companies involved in software development or
software maintenance
• Addressed topics
– Use of processes during software development and
maintenance
– Use of software quality measurement and improvement
– Use of quality models and quality standards
– Use of testing
– Organisational support of development teams
SATTOSE 2012 4

5. Structure of the questionnaire
• Introductory questions (6)
– Details of respondent and company
• General development questions (5)
– Perspective on dev. practices carried out by company
• Structural software quality (5)
• Software testing and maintenance (5)
• Quality models and quality standards (5)
• Organisational support of development teams (5)
SATTOSE 2012 5

6. Respondents – Number
– Initial mailing sent out to 145 companies
– 71 responses out of 188 contacted respondents
• Response rate 37,8%
– Responses from 47 different companies
• Multiple responses from same companies were
aggregated into a single one
– Incomplete responses were ignored
– 44 fully completed questionnaires kept for analysis
• Corresponds to 62% of received responses
SATTOSE 2012 6

7. Respondents - Origin
Mainly from Wallonia and Brussels (the target of our pilot study)
SATTOSE 2012 7

8. Respondents – Company size
Good balance between company size (number of employees) of respondents
SATTOSE 2012 8

9. Respondents – Training Level
Many at master+engineer level (18+17) and bachelor (20) level
Few or none with PhD (10+26), other degree (10+16) or no degree (3+27)
SATTOSE 2012 9

10. Process – Dev. process support
• Use of a well-defined and well-documented
development process? 45,5%
– No: 19
– Yes: 20 (agile or scrum, Prince 2, RUP, ISO
certification, proprietary)
– Don’t know: 5
• Use of agile practices or methods? 63,6%
– No: 12
– Yes: 28 (17 mention SCRUM, 2 mention Prince2)
– Don’t know: 4
SATTOSE 2012 10

11. Process - change or configuration
management process
Use of change or configuration management process is highly popular
73,8% (31/42)
SATTOSE 2012 11

12. Process - perceived importance factors
for software project success
• Rated from (1) not important to (5) essential
Average : 4 4,2
SATTOSE 2012 12

13. Process - Generation of
artefacts/documents/deliverables
during
SATTOSE 2012 13

14. Process - Creation and modification of
software artefacts
Are arch. descr. (36,4%) and design models (39,5%) being evolved?
SATTOSE 2012 14

15. Tools - Use of integrated
platform for
Version control (97,6%) and bug tracking (92,7%) well established
Platforms for continuous integration (57,1%), configuration (55,6%)
and testing (60,5%) a bit less
SATTOSE 2012 15

16. Tools - Programming languages
used followed by scripting languages.
OO languages most popular (Java, C#, C++),
SQL and Cobol legacy also remains important.
SATTOSE 2012 16

17. Tools - Development
environments used
Dichotomy between Java and .Net visible at IDE level.
Many others but much less frequently used.
SATTOSE 2012 17

18. Tools - support for design models,
documentation and code synchro
• Very basic
– Visio (10), Word (8), Sparx Enterprise Architect
(5), Doxygen (3), StarUML (2), Confluence (2)
and many others
– Little use of UML modeling tools
• Little or no support for model-code
synchronisation
SATTOSE 2012 18

19. Quality - Use of design patterns
Use of design patterns is highly popular 72,7% (32/44)
SATTOSE 2012 19

20. Quality - Use of quality
improvement techniques
Only moderately popular (35,7%<x<43%) except for refactoring
Often or
(58,5%) continuously
36,6% (15/41)
42,9% (18/42)
35,7% (15/42)
58,5% (24/41)
SATTOSE 2012 20

21. Use of quality support continued
Poor support for quality (no quality tools, processes or models)
36,8% (14/38)
32,4% (12/37)
19,4% (7/36)
SATTOSE 2012 21

22. Popularity of quality
improvement techniques
• Most popular
 Version control 97,6% and bug tracking 92,7% platforms
 Change and configuration management: 73,8%
 Design patterns: 72,7%
 Refactoring: 58,5%
• Less popular
 Design improvement (e.g. code smell reduction): 42,9%
 Bad quality detection tools: 36,8%
 Metrics and visualisation tools: 35,7%
 Dynamic analysis tools (profiling etc.): 36,6%
 Quality support or improvement process: 32,4%
• Unpopular
 Use of a quality model: 19,4%
SATTOSE 2012 22

23. Reuse of libraries, components
and platforms
Reuse is highly successful
86% from own company
75% from open source
60% from other companies
(excluding “don’t know” results)
SATTOSE 2012 23

24. Testing - Use of test process
Testing is done by nearly all respondents (97,7%), but
test process used by only 46,5% of the respondents
SATTOSE 2012 24

25. Testing – development time spent
SATTOSE 2012 25

26. Testing - tool support
• Very varied
– Mostly unit testing frameworks (8)
– Others: Mantis (4), HP Quality Center (4),
Selenium (3), Hudson (2), Jenkins (2), Quick
Test Pro (2), and many more
SATTOSE 2012 26

27. Preliminary conclusions
• Wide range of respondents, from very small to big
companies, using many different programming
languages and development environments
• Strong points
– All respondents believe that quality assurance and
testing are very important for project success
– Wide use of testing (97,7%), agile practices (63,6%),
design patterns (72,9%) and refactoring (56,5%)
– High level of reuse of components/libraries/platforms
(60% to 86% depending on source of reuse)
SATTOSE 2012 27

28. Preliminary conclusions
• Weak points
– Mitigated success of processes
• development processes (45,5%), test processes (46,5%),
quality support/improvement process (32,4%), quality
models (19,4%)
• Exception: change management process (73,8%)
SATTOSE 2012 28

29. Preliminary conclusions
• Weak points
– Mitigated success of static and dynamic code analysis
tools for detecting quality issues, visualisation,
computing metrics, profiling, etc… (popularity
between 35% and 43%)
– Does not reflect the high
perceived importance of
quality assurance for
software project success
SATTOSE 2012 29

30. What’s next?
• Analyse results in more detail
– Correlate results to company size and training level
• Are bigger companies more process-driven and smaller
ones more agile? Does training level play a role?
• Report on the results
• Repeat the study in other countries
– Identify regional or national trends
– Compare differences and commonalities
– We need your help here!
studies
SATTOSE 2012 30

31. Collaborative study
• Research 2.0
– Carry out follow-up / more narrow studies
– Carry out this study as a collaborative
community effort?
• Store / share / reuse our data with others
– Which format/platform/… to use?
• Have a working session on this topic during
SATTOSE/SOTESOLA?
SATTOSE 2012 31