mdh.sePublications
Change search
Link to record
Permanent link

Direct link
BETA
Publications (6 of 6) Show all publications
Wiklund, K. (2015). Impediments for Automated Software Test Execution. (Doctoral dissertation). Västerås: Mälardalen University
Open this publication in new window or tab >>Impediments for Automated Software Test Execution
2015 (English)Doctoral thesis, monograph (Other academic)
Abstract [en]

Automated software test execution is a critical part of the modern software development process, where rapid feedback on the product quality is expected. It is of high importance that impediments related to test execution automation are prevented and removed as quickly as possible. An enabling factor for all types of improvement is to understand the nature of what is to be improved. The goal with this thesis is to further the knowledge about common problems encountered by software developers using test execution automation, in order to enable improvement of test execution automation in industrial software development. The research has been performed through industrial case studies and literature reviews. The analysis of the data have primarily been performed using qualitative methods, searching for patterns, themes, and concepts in the data. 

Our main findings include: (a) a comprehensive list of potential impediments reported in the published body of knowledge on test execution automation, (b) an in-depth analysis of how such impediments may affect the performance of a development team, and (c) a proposal for a qualitative model of interactions between the main groups of phenomena that contribute to the formation of impediments in a test execution automation project. In addition to this, we contribute qualitative and quantitative empirical data from our industrial case studies. 

Through our results, we find that test execution automation is a commonly under-estimated activity,  not only in terms of resources but also in terms of the complexity of the work. There is a clear tendency to perform the work ad hoc, down-prioritize the automation in favor of other activities,  and ignore the long-term effects in favor of short-term gains. This is both a technical and a cultural problem that need to be managed by awareness of the problems that may arise, and also have to be solved in the long term through education and information. We conclude by proposing a theoretical model of the socio-technical system that needs to be managed to be successful with test execution automation.

Abstract [sv]

Syftet med denna avhandling är att utöka den vetenskapliga kunskapen om problem som kan uppstå under användning av automatiserad testning i industriell programvaruutveckling.

Utveckling av programvara består principiellt av ett antal processteg:

kravbehandling, detaljerad systemkonstruktion, implementation i form av programmering, och slutligen testning som säkerställer att kvaliteten på programvaran är tillräcklig för dess tilltänkta användare. Testning representerar en stor del av tiden och kostnaden för utveckling av programvaran, och detta gör att det är attraktivt att automatisera testningen.

Automatiserade tester kan bidra med många positiva effekter. Testning som utförs om och om igen, för att säkerställa att gammal funktionalitet inte slutar att fungera när ändringar görs i programvaran, kan med fördel göras automatiserat. Detta frigör kvalificerad personal till kvalificerat arbete. Automatisering kan även minska ledtiden för testningen och därmed möjliggöra snabbare leveranser till kund. Ett annat mål med testautomatisering är att vara säker på att samma tester utförs på ett likartat sätt varje gång produkten testas, så att den håller en jämn och stabil kvalitetsnivå.

Automatiserad testning är dock en mer komplex och kostsam verksamhet än vad man kan tro, och problem som uppstår under dess användning kan ha stora konsekvenser. Detta gäller i ännu större utsträckning i organisationer som använder moderna utvecklingsmetoder där automatisering är grundstenen för en effektiv kvalitetskontroll. För att kunna undvika så många problem som möjligt, är det därför mycket viktigt att förstå vad som händer när man använder testautomatisering i stor skala.

Denna avhandling presenterar resultat från fallstudier i svensk industri, som, kombinerat med en systematisk genomgång av befintlig forskning inom området, har utförts för att söka djupare kunskap och möjligheter till generalisering. Arbetet har varit beskrivande, och förklarande, och bygger på empirisk forskningsmetodik. 

I avhandlingen bidrar vi med (a) information om de problem relaterade till automatiserad testning som vi har identifierat i de empiriska fallstudierna, (b) en diskussion av dessa problem i relation till andra studier i området, (c) en systematisk litteraturstudie som ger en översikt över relevanta publikationer i området, (d) en analys av bevisen som insamlats genom litteraturstudien, samt (e) en modell av det organisatoriska och tekniska system som måste hanteras för att man skall nå framgång med ett testautomatiseringsprojekt.

Resultaten tyder på att rent tekniska problem inte utgör huvuddelen av de problem som upplevs med testautomatisering. Istället handlar det till stora delar om organisationell dynamik: hantering av förändringen det innebär att införa automatisering, planering av automatisering och dess användning, samt vilka finansiella förväntningar man har på automatiseringen.

 

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2015
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 180
Keywords
software engineering, software testing, empirical software engineering, impediments, industrial research, empirical research, test execution, test automation, test execution automation
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Computer Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-27830 (URN)978-91-7485-211-0 (ISBN)
Public defence
2015-06-15, Gamma, Mälardalens högskola, Västerås, 10:00
Opponent
Supervisors
Projects
ITS-EASY Post Graduate School for Embedded Software and Systems
Funder
Knowledge Foundation
Available from: 2015-04-30 Created: 2015-04-16 Last updated: 2018-01-11Bibliographically approved
Wiklund, K., Sundmark, D., Eldh, S. & Lundqvist, K. (2014). Impediments for Automated Testing - An Empirical Analysis of a User Support Discussion Board. In: Seventh IEEE International Conference on Software Testing, Verification and Validation ICST 2014: . Paper presented at Seventh IEEE International Conference on Software Testing, Verification and Validation ICST 2014, March 31-April 4 2014, Cleveland, United States (pp. 113-122). IEEE
Open this publication in new window or tab >>Impediments for Automated Testing - An Empirical Analysis of a User Support Discussion Board
2014 (English)In: Seventh IEEE International Conference on Software Testing, Verification and Validation ICST 2014, IEEE , 2014, p. 113-122Conference paper, Published paper (Refereed)
Abstract [en]

To better understand the challenges encountered by users and developers of automatic software testing, we have performed an empirical investigation of a discussion board used for support of a test automation framework having several hundred users. The messages on the discussion board were stratified into problem reports, help requests, development information, and feature requests. The messages in the problem report and help request strata were then sampled and analyzed using thematic analysis, searching for common patterns. Our analysis indicate that a large part of the impediments discussed on the board are related to issues related to the centralized IT environment, and to erroneous behavior connected to the use of the framework and related components. We also observed a large amount of impediments related to the use of software development tools. Turning to the help requests, we found that the majority of the help requests were about designing test scripts and not about the areas that appear to be most problematic. From our results and previous publications, we see a clear need to simplify the use, installation, and configuration of test systems of this type. The problems attributable to software development tools suggest that testers implementing test automation need more skills in handling those tools, than historically has been assumed. Finally, we propose that further research into the benefits of centralization of tools and IT environments, as well as structured deployment and efficient use of test automation, is performed.

Place, publisher, year, edition, pages
IEEE, 2014
Keywords
software testing, test automation, test tools, empirical software engineering
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-25144 (URN)10.1109/ICST.2014.24 (DOI)000355985000014 ()2-s2.0-84903195625 (Scopus ID)978-0-7695-5185-2 (ISBN)
Conference
Seventh IEEE International Conference on Software Testing, Verification and Validation ICST 2014, March 31-April 4 2014, Cleveland, United States
Available from: 2014-06-09 Created: 2014-06-05 Last updated: 2015-07-16Bibliographically approved
Eldh, S., Andersson, K., Ermedahl, A. & Wiklund, K. (2014). Towards a Test Automation Improvement Model (TAIM). In: Proc. - IEEE Int. Conf. Softw. Test., Verif. Valid. Workshops, ICSTW: . Paper presented at 7th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014, 31 March 2014 through 4 April 2014, Cleveland, OH (pp. 337-342).
Open this publication in new window or tab >>Towards a Test Automation Improvement Model (TAIM)
2014 (English)In: Proc. - IEEE Int. Conf. Softw. Test., Verif. Valid. Workshops, ICSTW, 2014, p. 337-342Conference paper, Published paper (Refereed)
Abstract [en]

In agile software development, industries are becoming more dependent on automated test suites. Thus, the test code quality is an important factor for the overall system quality and maintainability. We propose a Test Automation Improvement Model (TAIM) defining ten key areas and one general area. Each area should be based on measurements, to fill the gap of existing assessments models. The main contribution of this paper is to provide the outline of TAIM and present our intermediate results and some initial metrics to support our model. Our initial target has been the key area targeting implementation and structure of test code. We have used common static measurements to compare the test code and the source code of a unit test automation suite being part of a large complex telecom subsystem. Our intermediate results show that it is possible to outline such an improvement model and our metrics approach seems promising. However, to get a generic useful model to aid test automation evolution and provide for comparable measurements, many problems still remain to be solved. TAIM can as such be viewed as a framework to guide the research on metrics for test automation artifacts.

Series
Proceedings - IEEE 7th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014
Keywords
improvement model, industry, measurement, software, test automation, test code
National Category
Software Engineering
Identifiers
urn:nbn:se:mdh:diva-25697 (URN)10.1109/ICSTW.2014.38 (DOI)2-s2.0-84903647674 (Scopus ID)9780769551944 (ISBN)
Conference
7th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014, 31 March 2014 through 4 April 2014, Cleveland, OH
Available from: 2014-07-21 Created: 2014-07-18 Last updated: 2018-01-11Bibliographically approved
Wiklund, K., Eldh, S., Sundmark, D. & Lundqvist, K. (2013). Can we do useful industrial software engineering research in the shadow of Lean and Agile?. In: : . Paper presented at First Intl. Workshop on Conducting Empirical Studies in Industry (CESI 2013) - An ICSE 2013 Workshop, 20th May 2013, San Francisco, USA (pp. 67-68).
Open this publication in new window or tab >>Can we do useful industrial software engineering research in the shadow of Lean and Agile?
2013 (English)Conference paper, Published paper (Refereed)
Abstract [en]

The software industry is rapidly changing from traditional ways of working to lean and agile development methods using self-organized feature development teams that are performing a much larger part of the development process than before. Face to face communication will replace many of the design artifacts used for work-in-progress, such as defect reports and feature system design specifications. This type of data will cease to exist when the feature is developed or the problem is solved, and will not be readily available to researchers. As a consequence, software engineering research in industry will have to rely primarily on participatory and observational methods.

National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-21342 (URN)10.1109/CESI.2013.6618475 (DOI)2-s2.0-84888865068 (Scopus ID)9781467362863 (ISBN)
Conference
First Intl. Workshop on Conducting Empirical Studies in Industry (CESI 2013) - An ICSE 2013 Workshop, 20th May 2013, San Francisco, USA
Projects
ITS-EASY Post Graduate School for Embedded Software and Systems
Available from: 2013-09-18 Created: 2013-09-11 Last updated: 2016-03-10Bibliographically approved
Wiklund, K., Sundmark, D., Eldh, S. & Lundqvist, K. (2013). Impediments in Agile Software Development: An Empirical Investigation. In: Product-Focused Software Process Improvement, Lecture Notes in Computer Science Volume 7983: 14th International Conference, PROFES 2013, Paphos, Cyprus, June 12-14, 2013. Proceedings (pp. 35-49). Springer
Open this publication in new window or tab >>Impediments in Agile Software Development: An Empirical Investigation
2013 (English)In: Product-Focused Software Process Improvement, Lecture Notes in Computer Science Volume 7983: 14th International Conference, PROFES 2013, Paphos, Cyprus, June 12-14, 2013. Proceedings, Springer , 2013, p. 35-49Chapter in book (Refereed)
Abstract [en]

In this paper, we report on a case study on development impediments encountered in the early phase of a transformation to agile methods in a software development organization. Drawing from literature and anecdotal evidence, it was assumed that the majority of the impediments were related to software testing. To investigate this, we performed a case study seeking qualitative and quantitative evidence from task boards, interviews, and observations. Our analysis indicates that the major challenge in the transformation undertaken by the studied organization was coordination and communication in the large, and that testing was the major challenge only when the unit of analysis was restricted to the teams in the department.

Place, publisher, year, edition, pages
Springer, 2013
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 7983
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mdh:diva-21350 (URN)10.1007/978-3-642-39259-7_6 (DOI)000342989700006 ()2-s2.0-84884929541 (Scopus ID)978-3-642-39258-0 (ISBN)978-3-642-39259-7 (ISBN)
Projects
ITS-EASY Post Graduate School for Embedded Software and Systems
Note

14th International Conference of Product Focused Software Development and Process Improvement (PROFES)

Available from: 2013-09-18 Created: 2013-09-11 Last updated: 2018-07-30Bibliographically approved
Wiklund, K., Eldh, S., Sundmark, D. & Lundqvist, K. (2012). Technical Debt in Test Automation. In: Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012. Paper presented at 5th IEEE International Conference on Software Testing, Verification and Validation, ICST 2012; Montreal, QC;17 April 2012 through21 April 2012 (pp. 887-892). IEEE Computer Society
Open this publication in new window or tab >>Technical Debt in Test Automation
2012 (English)In: Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012, IEEE Computer Society, 2012, p. 887-892Conference paper, Published paper (Refereed)
Abstract [en]

Automated test execution is one of the more popular and available strategies to minimize the cost for software testing, and is also becoming one of the central concepts in modern software development as methods such as test-driven devel- opment gain popularity. Published studies on test automation indicate that the maintenance and development of test automation tools commonly encounter problems due to unforeseen issues. To further investigate this, we performed a case study on a telecommunication subsystem to seek factors that contribute to inefficiencies in use, maintenance, and development of the automated testing performed within the scope of responsibility of a software design team. A qualitative evaluation of the findings indicates that the main areas of improvement in this case are in the fields of interaction design and general software design principles, as applied to test execution system development.

Place, publisher, year, edition, pages
IEEE Computer Society, 2012
Keywords
software testing, test automation
National Category
Computer Engineering Engineering and Technology
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-14504 (URN)10.1109/ICST.2012.192 (DOI)2-s2.0-84862310671 (Scopus ID)978-076954670-4 (ISBN)
Conference
5th IEEE International Conference on Software Testing, Verification and Validation, ICST 2012; Montreal, QC;17 April 2012 through21 April 2012
Projects
ITS-EASYSWELL
Note

(c) 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.

Available from: 2012-05-03 Created: 2012-04-21 Last updated: 2018-01-12Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0001-8228-813X

Search in DiVA

Show all publications