mdh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Cost-Benefit Analysis of Using Dependency Knowledge at Integration Testing
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-8724-9049
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0003-1597-6738
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-1512-0844
SICS, Sweden.ORCID iD: 0000-0002-0921-0878
Show others and affiliations
2016 (English)In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016, Vol. 10027, p. 268-284Conference paper, Published paper (Refereed)
Abstract [en]

In software system development, testing can take considerable time and resources, and there are numerous examples in the literature of how to improve the testing process. In particular, methods for selection and prioritization of test cases can play a critical role in efficient use of testing resources. This paper focuses on the problem of selection and ordering of integration-level test cases. Integration testing is performed to evaluate the correctness of several units in composition. Further, for reasons of both effectiveness and safety, many embedded systems are still tested manually. To this end, we propose a process, supported by an online decision support system, for ordering and selection of test cases based on the test result of previously executed test cases. To analyze the economic efficiency of such a system, a customized return on investment (ROI) metric tailored for system integration testing is introduced. Using data collected from the development process of a large-scale safety-critical embedded system, we perform Monte Carlo simulations to evaluate the expected ROI of three variants of the proposed new process. The results show that our proposed decision support system is beneficial in terms of ROI at system integration testing and thus qualifies as an important element in improving the integration testing process.

Place, publisher, year, edition, pages
2016. Vol. 10027, p. 268-284
Keywords [en]
Process improvement, Software testing, Decision support system, Integration testing, Test case selection, Prioritization, Optimization, Return on investment
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:mdh:diva-32887DOI: 10.1007/978-3-319-49094-6_17Scopus ID: 2-s2.0-84998880972OAI: oai:DiVA.org:mdh-32887DiVA, id: diva2:956169
Conference
THE 17TH INTERNATIONAL CONFERENCE ON PRODUCT-FOCUSED SOFTWARE PROCESS IMPROVEMENT PROFES'16, 22-24 Nov 2016, TRONDHEIM, Norway
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)IMPRINT - Innovative Model-Based Product Integration Testing (Vinnova)Available from: 2016-08-29 Created: 2016-08-24 Last updated: 2018-11-01Bibliographically approved
In thesis
1. A Decision Support System for Integration Test Selection
Open this publication in new window or tab >>A Decision Support System for Integration Test Selection
2016 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software testing generally suffers from time and budget limitations. Indiscriminately executing all available test cases leads to sub-optimal exploitation of testing resources. Selecting too few test cases for execution on the other hand might leave a large number of faults undiscovered. Test case selection and prioritization techniques can lead to more efficient usage of testing resources and also early detection of faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Test case prioritization schedules test cases for execution in an order to increase their effectiveness at achieving some performance goals such as: earlier fault detection, optimal allocation of testing resources and reducing overall testing effort. In practice, prioritized selection of test cases requires the evaluation of different test case criteria, and therefore, this problem can be formulated as a multi-criteria decision making problem. As the number of decision criteria grows, application of a systematic decision making solution becomes a necessity. In this thesis, we propose a tool-supported framework using a decision support system, for prioritizing and selecting integration test cases in embedded system development. The framework provides a complete loop for selecting the best candidate test case for execution based on a finite set of criteria. The results of multiple case studies, done on a train control management subsystem from Bombardier Transportation AB in Sweden, demonstrate how our approach helps to select test cases in a systematic way. This can lead to early detection of faults while respecting various criteria. Also, we have evaluated a customized return on investment metric to quantify the economic benefits in optimizing system integration testing using our framework.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2016
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 242
National Category
Computer Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-33118 (URN)978-91-7485-282-0 (ISBN)
Presentation
2016-10-25, Omega, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2016-09-12 Created: 2016-09-09 Last updated: 2016-10-04Bibliographically approved
2. Multi-Criteria Optimization of System Integration Testing
Open this publication in new window or tab >>Multi-Criteria Optimization of System Integration Testing
2018 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Optimizing software testing process has received much attention over the last few decades. Test optimization is typically seen as a multi-criteria decision making problem. One aspect of test optimization involves test selection, prioritization and execution scheduling. Having an efficient test process can result in the satisfaction of many objectives such as cost and time minimization. It can also lead to on-time delivery and a better quality of the final software product. To achieve the goal of test efficiency, a set of criteria, having an impact on the test cases, need to be identified. The analysis of several industrial case studies and also state of the art in this thesis, indicate that the dependency between integration test cases is one such criterion, with a direct impact on the test execution results. Other criteria of interest include requirement coverage and test execution time. In this doctoral thesis, we introduce, apply and evaluate a set of approaches and tools for test execution optimization at industrial integration testing level in embedded software development. Furthermore, ESPRET (Estimation and Prediction of Execution Time) and sOrTES (Stochastic Optimizing of Test Case Scheduling) are our proposed supportive tools for predicting the execution time and the scheduling of manual integration test cases, respectively. All proposed methods and tools in this thesis, have been evaluated at industrial testing projects at Bombardier Transportation (BT) in Sweden. As a result of the scientific contributions made in this doctoral thesis, employing the proposed approaches has led to an improvement in terms of reducing redundant test execution failures of up to 40% with respect to the current test execution approach at BT. Moreover, an increase in the requirements coverage of up to 9.6% is observed at BT. In summary, the application of the proposed approaches in this doctoral thesis has shown to give considerable gains by optimizing test schedules in system integration testing of embedded software development.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2018
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 281
National Category
Embedded Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-41273 (URN)978-91-7485-414-5 (ISBN)
Public defence
2018-12-21, Lambda, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2018-11-02 Created: 2018-11-01 Last updated: 2018-11-20Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records BETA

Bohlin, MarkusSaadatmand, MehrdadLarsson, StigAfzal, WasifSundmark, Daniel

Search in DiVA

By author/editor
Tahvili, SaharBohlin, MarkusSaadatmand, MehrdadLarsson, StigAfzal, WasifSundmark, Daniel
By organisation
Embedded Systems
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 87 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf