The Comparative Evaluation of Test Prioritization Approaches in an Industrial StudyShow others and affiliations
2023 (English)In: Proc. - IEEE Int. Conf. Softw. Qual., Reliab., Secur. Companion, QRS-C, Institute of Electrical and Electronics Engineers Inc. , 2023, p. 35-44Conference paper, Published paper (Refereed)
Abstract [en]
Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We compared three different prioritisation approaches (i.e., diversity, failure history and time) in terms of their impact on coverage, failure detection rates and reduction on test execution time. Our results reveal that DBT is the best candidate to provide feature coverage, whereas failure rate prioritisation yields better failure coverage. Time-based prioritisation is not a reliable approach to provide cost-effective testing. Moreover, DBT would allow stakeholders to receive quick feedback on many combinations of integrated features to verify their code changes. Our participants report that developers are mainly interested in: (i) receiving quick feedback on a high combination of integrated features to verify their code changes, and (ii) associate their test suites to confidence scores representing the risk of missing failures given that fewer tests are executed.
Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc. , 2023. p. 35-44
Keywords [en]
comparative evaluation, continuous integration, diversity based testing, regression testing, Cost effectiveness, Failure rate, Integration testing, Code changes, Comparative evaluations, Continuous integrations, Evaluation of test, Prioritization, Prioritization techniques, Test effectiveness, Test prioritization, Pipelines
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:mdh:diva-66246DOI: 10.1109/QRS-C60940.2023.00025Scopus ID: 2-s2.0-85186749050ISBN: 9798350359398 (print)OAI: oai:DiVA.org:mdh-66246DiVA, id: diva2:1844219
Conference
Proceedings - 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion, QRS-C 2023
Note
Conference paper; Export Date: 13 March 2024; Cited By: 0; Correspondence Address: A. Ahmad; Ericsson AB, Linköping, Sweden; email: azeem.ahmad@ericsson.com; F.G.D.O. Neto; Gothenburg University, Gothenburg, Sweden; email: francisco.gomes@cse.gu.se; Conference name: 23rd IEEE International Conference on Software Quality, Reliability, and Security Companion, QRS-C 2023; Conference date: 22 October 2023 through 26 October 2023; Conference code: 197421
2024-03-132024-03-132024-03-13Bibliographically approved