mdh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Using Logic Coverage to Improve Testing Function Block Diagrams
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. (IS (Embedded Systems))ORCID iD: 0000-0003-2416-4205
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. (IS (Embedded Systems))ORCID iD: 0000-0002-5032-2310
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. (IS (Embedded Systems))ORCID iD: 0000-0003-4040-3480
2013 (English)In: Testing Software and Systems: Lecture Notes in Computer Science, Volume 8254, Springer Berlin Heidelberg , 2013, 1-16 p.Chapter in book (Refereed)
Abstract [en]

In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams.

Place, publisher, year, edition, pages
Springer Berlin Heidelberg , 2013. 1-16 p.
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 8254
Keyword [en]
logic coveragemodel-based testingfunction block diagramtimed automataIEC 61131-3test automation
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:mdh:diva-22882DOI: 10.1007/978-3-642-41707-8_1Scopus ID: 2-s2.0-84893417933ISBN: 978-3-642-41706-1 (print)OAI: oai:DiVA.org:mdh-22882DiVA: diva2:665770
Projects
ATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova)
Note

25th IFIP WG 6.1 International Conference, ICTSS 2013, Istanbul, Turkey, November 13-15, 2013, Proceedings

Available from: 2013-11-20 Created: 2013-11-19 Last updated: 2016-10-07Bibliographically approved
In thesis
1. Model Checking-Based Software Testing for Function-Block Diagrams
Open this publication in new window or tab >>Model Checking-Based Software Testing for Function-Block Diagrams
2014 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Software testing becomes more complex, more time-consuming, and more expensive. The risk that software errors remain undetected and cause critical failures increases. Consequently, in safety-critical development, testing software is standardized and it requires an engineer to show that tests fully exercise, or cover, the logic of the software. This method often requires a trained engineer to perform manual test generation, is prone to human error, and is expensive or impractical to use frequently in production. To overcome these issues, software testing needs to be performed earlier in the development process, more frequently, and aided by automated tools.

We devised an automated test generation tool called COMPLETETEST that avoids many of those problems. The method implemented in the tool and described in this thesis, works with software written in Function Block Diagram language, and can provide tests in just a few seconds. In addition, it does not rely on the expertise of a researcherspecialized in automated test generation and model checking. Although COMPLETETEST itself uses a model checker, a complex technique requiring a high level of expertise to generate tests, it provides a straightforward tabular interface to the intended users. In this way, its users do not need to learn the intricacies of using this approach such as how coverage criteria can be formalized and used by a model checker to automatically generate tests. If the technique can be demonstrated to work in production, it could detect and aid in the detection of errors in safety-critical software development, where conventional testing is not always applicable and efficient.

We conducted studies based on industrial use-case scenarios from Bombardier Transportation AB, showing how the approach can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, it was applied on real-world programs. The results indicate that it is efficient in terms of time required to generate tests and scales well for most of the software. There are still issues to resolve before the technique can be applied to more complex software, but we are already working on ways to overcome them. In particular, we need to understand how its usage in practice can vary depending on human and software process factors.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2014
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 182
National Category
Embedded Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-26003 (URN)978-91-7485-166-3 (ISBN)
Presentation
2014-11-07, Zeta, Västerås, 10:15 (English)
Opponent
Available from: 2014-09-25 Created: 2014-09-24 Last updated: 2014-11-03Bibliographically approved
2. Automatic test generation for industrial control software
Open this publication in new window or tab >>Automatic test generation for industrial control software
2016 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.

In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. 

The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.

However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.

In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2016
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 214
Keyword
automatic test generation, software testing, automated test generation
National Category
Embedded Systems
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-33364 (URN)978-91-7485-291-2 (ISBN)
Public defence
2016-11-22, Pi, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2016-10-07 Created: 2016-10-07 Last updated: 2016-11-02Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Enoiu, Eduard PaulSundmark, DanielPettersson, Paul
By organisation
Embedded Systems
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar

Altmetric score

Total: 140 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf