https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Model-based System Testing of Safety-Critical Embedded Software
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. (Software Testing Laboratory)ORCID iD: 0000-0001-8746-7209
2022 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

System-level testing of safety-critical embedded systems is complex and costly. MBT has shown promising results in terms of fault detection effectiveness and efficiency of test generation and execution. However, the industrial adoption of MBT approaches is slow and limited to specific industries and domains. Moreover, the strengths and weaknesses of MBT in industrial settings need to be thoroughly evaluated to find an optimal testing strategy.

The objective of this thesis is to adapt, compare, and evaluate the effectiveness and efficiency of MBT to help industrial practitioners in the testing of safety-critical embedded software. We have divided this objective into three subgoals. To achieve the first subgoal, we have explored multiple state-of-the- art MBT tools and evaluated the selected tool, GraphWalker (GW), in terms of modeling notations, generation algorithm, stopping conditions, and model completeness. To achieve the second subgoal, we have proposed a Model-Based Test scrIpt GenEration fRamework (TIGER), based on GW, to generate system-level test artifacts (i.e., test cases and test scripts). Based on the proposed framework, we implemented two test script generation tools for combinatorial testing (CT) and MBT. Finally, to achieve the last subgoal, we performed a comparative analysis between test suites developed using MBT, CT, and manual industrial practices. 

The results showed that the MBT-generated test suites using the edge coverage criterion tend to cover each requirement multiple times while achieving the same level of requirement coverage as the manually written test suites. Moreover, MBT provided higher Modified Decision and Condition Coverage (MC/DC) than CT and manual testing. On the other hand, CT came out as the most efficient technique in terms of the time required to generate and execute tests as well as achieving the highest fault detection rate with 3-ways and 4-ways interaction strength. Hence, based on the results, we conclude that manual industrial testing will benefit from MBT and CT for improved coverage and fault detection.

Abstract [sv]

Testning på systemnivå av säkerhetskritiska inbyggda system är komplext och kostsamt. MBT har visat lovande resultat när det gäller feldetekteringseffektivitet och effektivitet i testgenerering och testutförande. Den industriella utvecklingen av MBT-metoder är dock långsam och begränsad till specifika branscher och domäner. Dessutom måste styrkorna och svagheterna hos MBT i industriella miljöer utvärderas noggrant för att hitta en optimal teststrategi.

Syftet med denna avhandling är att anpassa, jämföra och utvärdera effektiviteten hos MBT för att hjälpa industriutövare att testa säkerhetskritisk inbyggd programvara. Vi har delat in detta mål i tre delmål. För att uppnå det första delmålet har vi utforskat flera av de främsta MBT-verktygen och utvärderat det valda verktyget, GraphWalker (GW), i termer av modelleringsnotationer, genereringsalgoritm, stoppförhållanden och modellfullständighet. För att uppnå det andra delmålet har vi föreslagit ett Model-Based Test ScrIpt Generation Framework (TIGER), baserat på GW, för att generera testartefakter på systemnivå (d.v.s. testfall och testskript). Baserat på det föreslagna ramverket implementerade vi två testskriptgenereringsverktyg för kombinatorisk testning (CT) och MBT. Slutligen, för att uppnå det sista delmålet, utförde vi en jämförande analys mellan testsviter utvecklade med MBT, CT och manuell industriell praxis.

Resultaten visade att de MBT-genererade testsviterna som använder kanttäckningskriteriet tenderar att täcka varje krav flera gånger samtidigt som de uppnår samma nivå av kravtäckning som de manuellt skrivna testsviterna. Dessutom gav MBT högre Modified Decision and Condition Coverage (MC/DC) än CT och manuell testning. Å andra sidan kom CT ut som den mest effektiva tekniken när det gäller den tid som krävs för att generera och utföra tester samt att uppnå den högsta feldetekteringshastigheten med 3-vägs och 4-vägs interaktionsstyrka. Baserat på resultaten drar vi därför slutsatsen att manuell industriell testning kommer att dra nytta av MBT och CT för förbättrad täckning och feldetektering.

Place, publisher, year, edition, pages
Västerås: Mälardalens universitet, 2022.
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 328
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:mdh:diva-59792ISBN: 978-91-7485-560-9 (print)OAI: oai:DiVA.org:mdh-59792DiVA, id: diva2:1690023
Presentation
2022-10-03, Alfa, Mälardalens Universitet, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2022-08-25 Created: 2022-08-24 Last updated: 2022-11-08Bibliographically approved
List of papers
1. Model-Based Testing in Practice: An Industrial Case Study using GraphWalker
Open this publication in new window or tab >>Model-Based Testing in Practice: An Industrial Case Study using GraphWalker
Show others...
2021 (English)In: ISEC 2021: 14th Innovations in Software Engineering Conference, 2021, p. 1-11, article id 5Conference paper, Published paper (Refereed)
Abstract [en]

Model-based testing (MBT) is a test design technique that supports the automation of software testing processes and generates test artefacts based on a system model representing behavioural aspects of the system under test (SUT). Previous research has shown some positive aspects of MBT such as low-cost test case generation and fault detection effectiveness. However, it is still a challenge for both practitioners and researchers to evaluate MBT tools and techniques in real, industrial settings. Consequently, the empirical evidence regarding the mainstream use, including the modelling and test case generation using MBT tools, is limited. In this paper, we report the results of a case study on applying GraphWalker, an open-source tool for MBT, on an industrial cyber-physical system (i.e., a Train Control Management System developed by Bombardier Transportation in Sweden), from modelling of real-world requirements and test specifications to test case generation. We evaluate the models of the SUT for completeness and representativeness, compare MBT with manual test cases written by practitioners using multiple attributes as well as share our experiences of selecting and using GraphWalker for industrial application. The results show that a model of the SUT created using both requirements and test specifications provides better understanding of the SUT from testers’ perspective, making it more complete and representative than the model created based only on the requirements specification alone. The generated model-based test cases are longer in terms of the number of test steps, achieve better edge coverage and can cover requirements more frequently in different orders while achieving the same level of requirements coverage as manually created test cases.

National Category
Engineering and Technology Computer Systems
Identifiers
urn:nbn:se:mdh:diva-53940 (URN)10.1145/3452383.3452388 (DOI)2-s2.0-85105444710 (Scopus ID)9781450390460 (ISBN)
Conference
Innovations in Software Engineering Conference 2021 ISEC 2021, 25 Feb 2021, Online
Projects
XIVT - eXcellence in Variant TestingADEPTNESS - Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems
Available from: 2021-04-27 Created: 2021-04-27 Last updated: 2022-08-24Bibliographically approved
2. A Model-Based Test Script Generation Framework for Embedded Software
Open this publication in new window or tab >>A Model-Based Test Script Generation Framework for Embedded Software
Show others...
2021 (English)In: Proceedings: 2021 IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, 2021, p. 192-198, article id 9440149Conference paper, Published paper (Refereed)
Abstract [en]

The abstract test cases generated through modelbased testing (MBT) need to be concretized to make them executable on the software under test (SUT). Multiple researchers proposed different solutions, e.g., by utilizing adapters for concretization of abstract test cases and generation of testscripts. In this paper, we propose our Model-Based Test scrIptGenEration fRamework (TIGER) based on GraphWalker, an open source MBT tool. The framework is capable of generating test scripts for embedded software controlling functions of a cyber physical system such as passenger trains developed at Bombardier Transportation AB. The framework follows somedefined mapping rules for the concretization of abstract test cases.We have evaluated the generated test scripts using an industrial case study in terms of fault detection. We have induced faultsin the model of the SUT based on three mutation operatorsto generate faulty test scripts. The aim of generating faulty test scripts is to produce failed test steps and to guarantee the absence of faults in the SUT. Moreover, we have also generated the test scripts using the correct version of the model and executed it to analyse the behaviour of the generated test scripts in comparisonwith manually-written test scripts. The results show that the test scripts generated by GW using the proposed frameworkare executable, provide 100% requirements coverage and can beused to uncover faults at software-in-the-loop simulation level of sub-system testing.

National Category
Engineering and Technology Computer Systems
Identifiers
urn:nbn:se:mdh:diva-53967 (URN)10.1109/ICSTW52544.2021.00041 (DOI)000680833800028 ()2-s2.0-85108022667 (Scopus ID)9781665444569 (ISBN)
Conference
14th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, 12 April 2021 - 16 April 2021
Projects
ADEPTNESS - Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of SystemsVeriDevOps - Automated Protection and Prevention to Meet Security Requirements in DevOps Environments
Available from: 2021-05-24 Created: 2021-05-24 Last updated: 2022-08-24Bibliographically approved
3. Evaluating System-Level Test Generation for Industrial Software: A Comparison between Manual, Combinatorial and Model-Based Testing
Open this publication in new window or tab >>Evaluating System-Level Test Generation for Industrial Software: A Comparison between Manual, Combinatorial and Model-Based Testing
2022 (English)In: Proceedings - 3rd ACM/IEEE International Conference on Automation of Software Test, AST 2022, Institute of Electrical and Electronics Engineers Inc. , 2022, p. 148-159Conference paper, Published paper (Refereed)
Abstract [en]

Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such systems requires a lot of effort, thus automated testing techniques have found a certain degree of success. However, automated testing has not replaced the need for manual testing, rather a common industrial practice exhibits a balance between automated and manual testing. In this respect, comparing manual testing with automated testing techniques continues to be an interesting topic to investigate. The need for this investigation is most apparent at system-level testing of industrial systems, where there is a lack of results on how different testing techniques perform concerning both structural and system-level metrics such as Modified Condition/Decision Coverage (MC/DC) and requirement coverage. In addition to the coverage, the cost of these techniques will also determine their efficiency and thus practical viability. In this paper, we have developed cost models for efficiency measurement and performed an experimental evaluation of manual testing, model-based testing (MBT), and combinatorial testing (CT) in terms of MC/DC and requirement coverage. The evaluation is done in an industrial context of a safety-critical system that controls several functions on-board the passenger trains. We have reported the dominant conditions of MC/DC affected by each technique while generating MC/DC adequate test suites. Moreover, we investigated differences and overlaps of test cases generated by each of the three techniques. The results showed that all test suites achieved 100% requirement coverage except the test suite generated by the pairwise testing strategy. However, MBT-generated test suites were more MC/DC adequate and provided a higher number of both similar and unique test cases. Moreover, unique test cases generated by MBT had an observable effect on MC/DC, which will complement manual testing to increase MC/DC coverage. The least dominant MC/DC condition fulfilled by the generated test cases by all three techniques is the 'independent effect of a condition on the outcomes of a decision'. Lastly, the evaluation also showed CT as the most efficient testing technique amongst the three in terms of time required for test generation and execution, but with an added cost parameter of manual identification of expected outcomes. CCS CONCEPTS •Computer systems organization →Embedded software; • Software and its engineering →Software verification and validation.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc., 2022
Keywords
Safety Critical Systems, System-level Tests, Test Coverage, Test Generation, Accident prevention, Automation, Model checking, Safety testing, Security systems, Software testing, Automated testing, Manual testing, Model based testing, Modified condition decision coverage, System-level test, Test case, Test generations, Test-coverage, Testing technique, Efficiency
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:mdh:diva-59578 (URN)10.1145/3524481.3527235 (DOI)000850254300016 ()2-s2.0-85133436269 (Scopus ID)9781450392860 (ISBN)
Conference
3rd ACM/IEEE International Conference on Automation of Software Test, AST 2022, 17 May 2022 through 18 May 2022
Available from: 2022-07-13 Created: 2022-07-13 Last updated: 2022-11-18Bibliographically approved
4. An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software
Open this publication in new window or tab >>An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software
(English)Manuscript (preprint) (Other academic)
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:mdh:diva-59791 (URN)
Projects
ADEPTNESS
Available from: 2022-08-24 Created: 2022-08-24 Last updated: 2024-07-30Bibliographically approved

Open Access in DiVA

fulltext(1007 kB)635 downloads
File information
File name FULLTEXT02.pdfFile size 1007 kBChecksum SHA-512
4b1b807b7dc6d937512daa063d64ad5618eb6be52a1eab6e229d52f4bc0ca3b6b0082c512ce7836728aafe70301a617bda54c8c15d40133dca713bca6ac67bec
Type fulltextMimetype application/pdf

Authority records

Zafar, Muhammad Nouman

Search in DiVA

By author/editor
Zafar, Muhammad Nouman
By organisation
Embedded Systems
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 638 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 454 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf