mdh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Measuring Combinatorial Coverage of Manual Testing
Mälardalen University, School of Innovation, Design and Engineering.
2018 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Introduction: Software testing is a very important activity which assures the quality of the software under test. It becomes crucial in safety-critical systems, where an unexpected behavior of the software can even cause loss of human life or environmental disasters. However, in such complex systems it becomes infeasible to test all possible software scenarios for possible faults. Experience shows that software faults, which can cause unexpected software behavior, are caused by the interactions of variables of the tests. Combinatorial testing is the technique which focuses on the variable interactions of the tests and aims to reduce the number of tests needed to cover all software scenarios while still preserving a high fault detection rate.

Background: Manual testing is the technique used to assure software quality in Bombardier Transportation AB, a Swedish company whose focus is on rail transport and development of trains. Since this process depends on the skills of the engineers, it can result in a large portion of tests not created and consequently in a large number of scenarios uncovered with tests. Therefore, combinatorial testing technique is used to measure the combinatorial coverage of these tests created from experienced engineers. Many comparisons of manual testing and other testing techniques in terms of test coverage, code coverage or mutation analysis. To the best of our knowledge there are no other studies in literature that have measured the combinatorial coverage of manual tests designed from experienced engineers, for different strength interactions of the variables of the tests nor other available tools that generate the number of missing tests to achieve full combinatorial coverage for specific interactions.

Aim: The goal of this thesis is to answer the two research questions: RQ1. What is the combinatorial coverage achieved by tests manually created by experienced engineers in industry? RQ2. Can the effectiveness of manually created tests be improved in terms of combinatorial coverage using combinatorial testing?

Method: In this thesis we investigate the combinatorial coverage of manually created tests by engineers working in industry and the implications of using combinatorial testing in practice. The Combinatorial Coverage Measurement2(CCM) NIST tool is used to measure the test coverage achieved. The research questions are answered by the following steps: 1) Review the scientific literature for related work, 2) Refine thesis research questions based on the previous step, 3) Propose the case study design and perform the measurements needed for data analysis, 4) and the results are analyzed and discussed in terms of test efficiency (i.e., number of test cases) and effectiveness (i.e., achieved combinatorial coverage).

Results: The 2-way interaction combinatorial coverage achieved by manual tests is 78.6% on average, 57% for 3-way combinatorial coverage, 40.2% for 4-way combinatorial coverage, 20.2% for 5-way combinatorial coverage and 13% for 6-way combinatorial coverage. Full combinatorial coverage can be achieved for 2-way and 3-way interactions by adding eight and 66 missing tests on average, respectively. For 4-way interactions full combinatorial coverage can achieved by adding 658 missing tests. For 5-way and 6-way interactions full combinatorial coverage can be achieved by adding 5163 and 6170 missing tests on average respectively.

Conclusion: The combinatorial coverage decreases as the strength of interactions increases. The effectiveness of the tests can be improved for 2-way and 3-way interactions and fully or partially improved for 4-way interactions, depending on the decision of engineers to add all missing tests or part of them, since the number of missing tests is increasing significantly, thus resulting in a very large number of tests to be added. It is not particularly efficient to improve test effectiveness by augmenting manual tests cases using combinatorial testing for higher strength interactions, since in most of the tests suites we studied one would need to generate additional 10.000 missing tests. This is explained by the exponential growth of the number of variable combination for such interactions.

Place, publisher, year, edition, pages
2018. , p. 115
Keywords [en]
Combinatorial Testing, Measuring Combinatorial Coverage, Manual Testing
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
URN: urn:nbn:se:mdh:diva-41211OAI: oai:DiVA.org:mdh-41211DiVA, id: diva2:1257858
External cooperation
Bombardier Transportation AB
Presentation
2018-09-20, R2-141, Mälardalen University, Västerås, 09:00 (English)
Supervisors
Examiners
Available from: 2018-10-23 Created: 2018-10-23 Last updated: 2018-10-23Bibliographically approved

Open Access in DiVA

DVA 501 - Thesis Report - Miraldi Fifo(3459 kB)48 downloads
File information
File name FULLTEXT01.pdfFile size 3459 kBChecksum SHA-512
efaea78788cacbe7251c6335005e25cb14c46f42ca8b5524d4fd8178fa677d8a4f447d8ef96d195616714328326f1c86188178cb13334712e76c56efe032011c
Type fulltextMimetype application/pdf

By organisation
School of Innovation, Design and Engineering
Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 48 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 463 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf