mdh.sePublications
Change search
Link to record
Permanent link

Direct link
BETA
Alternative names
Publications (10 of 93) Show all publications
Enoiu, E. P., Daniel, S., Causevic, A. & Pettersson, P. (2017). A Comparative Study of Manual and Automated Testing for Industrial Control Software. In: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017: . Paper presented at 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017; Tokyo; Japan; 13 March 2017 through 17 March 2017 (pp. 412-417). , Article ID 7927994.
Open this publication in new window or tab >>A Comparative Study of Manual and Automated Testing for Industrial Control Software
2017 (English)In: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017, 2017, p. 412-417, article id 7927994Conference paper, Published paper (Refereed)
Abstract [en]

Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.

National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-34088 (URN)10.1109/ICST.2017.44 (DOI)000403393600037 ()2-s2.0-85020701655 (Scopus ID)9781509060313 (ISBN)
Conference
10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017; Tokyo; Japan; 13 March 2017 through 17 March 2017
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)AGENTS - Automated Generation of Tests for Simulated Software Systems (KKS)
Available from: 2016-12-15 Created: 2016-12-13 Last updated: 2018-10-31Bibliographically approved
Johnsen, A., Lundqvist, K., Hänninen, K. & Pettersson, P. (2017). AQAT: The Architecture Quality Assurance Tool for Critical Embedded Systems. In: Proceedings - International Symposium on Software Reliability Engineering, ISSRE, Volume 2017: . Paper presented at 28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017; Toulouse; France; 23 October 2017 through 26 October 2017 (pp. 260-270). , Article ID 8109092.
Open this publication in new window or tab >>AQAT: The Architecture Quality Assurance Tool for Critical Embedded Systems
2017 (English)In: Proceedings - International Symposium on Software Reliability Engineering, ISSRE, Volume 2017, 2017, p. 260-270, article id 8109092Conference paper, Published paper (Refereed)
Abstract [en]

Architectural engineering of embedded systems comprehensively affects both the development processes and the abilities of the systems. Verification of architectural engineering is consequently essential in the development of safety- and mission-critical embedded system to avoid costly and hazardous faults. In this paper, we present the Architecture Quality Assurance Tool (AQAT), an application program developed to provide a holistic, formal, and automatic verification process for architectural engineering of critical embedded systems. AQAT includes architectural model checking, model-based testing, and selective regression verification features to effectively and efficiently detect design faults, implementation faults, and faults created by maintenance modifications. Furthermore, the tool includes a feature that analyzes architectural dependencies, which in addition to providing essential information for impact analyzes of architectural design changes may be used for hazard analysis, such as the identification of potential error propagations, common cause failures, and single point failures. Overviews of both the graphical user interface and the back-end processes of AQAT are presented with a sensor-to-actuator system example.

National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-37453 (URN)10.1109/ISSRE.2017.32 (DOI)000426939700025 ()2-s2.0-85040780004 (Scopus ID)9781538609415 (ISBN)
Conference
28th IEEE International Symposium on Software Reliability Engineering, ISSRE 2017; Toulouse; France; 23 October 2017 through 26 October 2017
Available from: 2017-12-14 Created: 2017-12-14 Last updated: 2018-03-29Bibliographically approved
Johnsen, A., Lundqvist, K., Pettersson, P., Hänninen, K. & Torelm, M. (2017). Experience Report: Evaluating Fault Detection Effectiveness and Resource Efficiency of the Architecture Quality Assurance Framework and Tool. In: Proceedings - International Symposium on Software Reliability Engineering, ISSRE. Volume 2017: . Paper presented at 28th International Symposium on Software Reliability Engineering (ISSRE), Toulouse, France, 2017 (pp. 271-281). , Article ID 8109093.
Open this publication in new window or tab >>Experience Report: Evaluating Fault Detection Effectiveness and Resource Efficiency of the Architecture Quality Assurance Framework and Tool
Show others...
2017 (English)In: Proceedings - International Symposium on Software Reliability Engineering, ISSRE. Volume 2017, 2017, p. 271-281, article id 8109093Conference paper, Published paper (Refereed)
Abstract [en]

The Architecture Quality Assurance Framework (AQAF) is a theory developed to provide a holistic and formal verification process for architectural engineering of critical embedded systems. AQAF encompasses integrated architectural model checking, model-based testing, and selective regression verification techniques to achieve this goal. The Architecture Quality Assurance Tool (AQAT) implements the theory of AQAF and enables automated application of the framework. In this paper, we present an evaluation of AQAT and the underlying AQAF theory by means of an industrial case study, where resource efficiency and fault detection effectiveness are the targeted properties of evaluation. The method of fault injection is utilized to guarantee coverage of fault types and to generate a data sample size adequate for statistical analysis. We discovered important areas of improvement in this study, which required further development of the framework before satisfactory results could be achieved. The final results present a 100% fault detection rate at the design level, a 98.5% fault detection rate at the implementation level, and an average increased efficiency of 6.4% with the aid of the selective regression verification technique.

National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-37457 (URN)10.1109/ISSRE.2017.31 (DOI)000426939700026 ()2-s2.0-85040780151 (Scopus ID)9781538609415 (ISBN)
Conference
28th International Symposium on Software Reliability Engineering (ISSRE), Toulouse, France, 2017
Available from: 2017-12-14 Created: 2017-12-14 Last updated: 2018-07-25Bibliographically approved
Johnsen, A., Dodig-Crnkovic, G., Lundqvist, K., Hänninen, K. & Pettersson, P. (2017). Risk-based decision-making fallacies: Why present functional safety standards are not enough. In: Proceedings - 2017 IEEE International Conference on Software Architecture Workshops, ICSAW 2017: Side Track Proceedings. Paper presented at 2017 IEEE International Conference on Software Architecture Workshops, ICSAW 2017, 3 April 2017 through 7 April 2017 (pp. 153-160). Institute of Electrical and Electronics Engineers Inc.
Open this publication in new window or tab >>Risk-based decision-making fallacies: Why present functional safety standards are not enough
Show others...
2017 (English)In: Proceedings - 2017 IEEE International Conference on Software Architecture Workshops, ICSAW 2017: Side Track Proceedings, Institute of Electrical and Electronics Engineers Inc. , 2017, p. 153-160Conference paper, Published paper (Refereed)
Abstract [en]

Functional safety of a system is the part of its overall safety that depends on the system operating correctly in response to its inputs. Safety is defined as the absence of unacceptable/unreasonable risk by functional safety standards, which enforce safety requirements in each phase of the development process of safety-critical software and hardware systems. Acceptability of risks is judged within a framework of analysis with contextual and cultural aspects by individuals who may introduce subjectivity and misconceptions in the assessment. While functional safety standards elaborate much on the avoidance of unreasonable risk in the development of safety-critical software and hardware systems, little is addressed on the issue of avoiding unreasonable judgments of risk. Through the studies of common fallacies in risk perception and ethics, we present a moral-psychological analysis of functional safety standards and propose plausible improvements of the involved risk-related decision making processes, with a focus on the notion of an acceptable residual risk. As a functional safety reference model, we use the functional safety standard ISO 26262, which addresses potential hazards caused by malfunctions of software and hardware systems within road vehicles and defines safety measures that are required to achieve an acceptable level of safety. The analysis points out the critical importance of a robust safety culture with developed countermeasures to the common fallacies in risk perception, which are not addressed by contemporary functional safety standards. We argue that functional safety standards should be complemented with the analysis of potential hazards caused by fallacies in risk perception, their countermeasures, and the requirement that residual risks must be explicated, motivated, and accompanied by a plan for their continuous reduction. This approach becomes especially important in contemporary developed autonomous vehicles with increasing computational control by increasingly intelligent software applications.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc., 2017
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-36194 (URN)10.1109/ICSAW.2017.50 (DOI)000413089000029 ()2-s2.0-85025634021 (Scopus ID)9781509047932 (ISBN)
Conference
2017 IEEE International Conference on Software Architecture Workshops, ICSAW 2017, 3 April 2017 through 7 April 2017
Available from: 2017-08-10 Created: 2017-08-10 Last updated: 2017-11-02Bibliographically approved
Lindström, B., Offutt, J., Sundmark, D., Andler, S. F. & Pettersson, P. (2017). Using mutation to design tests for aspect-oriented models. Information and Software Technology, 81, 112-130
Open this publication in new window or tab >>Using mutation to design tests for aspect-oriented models
Show others...
2017 (English)In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 81, p. 112-130Article in journal (Refereed) Published
Abstract [en]

Context: Testing for properties such as robustness or security is complicated because their concerns are often repeated in many locations and muddled with the normal code. Such "cross-cutting concerns" include things like interrupt events, exception handling, and security protocols. Aspect-oriented (AO) modeling allows, developers to model the cross-cutting behavior independently of the normal behavior, thus supporting model-based testing of cross-cutting concerns. However, mutation operators defined for AO programs (source code) are usually not applicable to AO models (AGMs) and operators defined for models do not target the AO features. Objective: We present a method to design abstract tests at the aspect-oriented model level. We define mutation operators for aspect-oriented models and evaluate the generated mutants for an example system. Method: AOMs are mutated with novel operators that specifically target the AO modeling features. Test traces killing these mutant models are then generated. The generated and selected traces are abstract tests that can be transformed to concrete black-box tests and run on the implementation level, to evaluate the behavior of the woven cross-cutting concerns (combined aspect and base models). Results: This paper is a significant extension of our paper at Mutation 2015. We present a complete fault model, additional mutation operators, and a thorough analysis of the mutants generated for an example system. Conclusions: The analysis shows that some mutants are stillborn (syntactically illegal) but none is equivalent (exhibiting the same behavior as the original model). Additionally, our AOM-specific mutation operators can be combined with pre-existing operators to mutate code or models without any overlap.

Keywords
Model-based testing, Aspect-oriented model, Mutation testing
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-34018 (URN)10.1016/j.infsof.2016.04.007 (DOI)000387634200009 ()2-s2.0-84963813590 (Scopus ID)
Available from: 2016-12-02 Created: 2016-12-02 Last updated: 2017-06-05Bibliographically approved
Enoiu, E. P., Causevic, A., Sundmark, D. & Pettersson, P. (2016). A Controlled Experiment in Testing of Safety-Critical Embedded Software. In: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016: . Paper presented at 9th IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, 10 April 2016 through 15 April 2016 (pp. 1-11).
Open this publication in new window or tab >>A Controlled Experiment in Testing of Safety-Critical Embedded Software
2016 (English)In: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, 2016, p. 1-11Conference paper, Published paper (Refereed)
Abstract [en]

In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification-and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.

Keywords
automated test generation, controlled experiment, embedded software, manual testing, safety-critical systems, specification-based testing, Automatic test pattern generation, Automation, Embedded systems, Fault detection, Safety engineering, Safety testing, Security systems, Software engineering, Specifications, Standards, Verification, Automated test generations, Safety critical systems, Specification Based Testing, Software testing
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:mdh:diva-33113 (URN)10.1109/ICST.2016.15 (DOI)000391252900001 ()2-s2.0-84983326093 (Scopus ID)9781509018260 (ISBN)
Conference
9th IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, 10 April 2016 through 15 April 2016
Available from: 2016-09-08 Created: 2016-09-08 Last updated: 2018-10-31Bibliographically approved
Enoiu, E. P., Causevic, A., Ostrand, T., Weyuker, E., Sundmark, D. & Pettersson, P. (2016). Automated Test Generation using Model-Checking: An Industrial Evaluation. International Journal on Software Tools for Technology Transfer (STTT), 18(3), 335-353
Open this publication in new window or tab >>Automated Test Generation using Model-Checking: An Industrial Evaluation
Show others...
2016 (English)In: International Journal on Software Tools for Technology Transfer (STTT), ISSN 1433-2779, E-ISSN 1433-2787, Vol. 18, no 3, p. 335-353Article in journal (Refereed) Published
Abstract [en]

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.

Place, publisher, year, edition, pages
Germany: Springer, 2016
Keywords
test generation, model checking, automated software testing, function block diagram, safety-critical software, industrial study
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-28100 (URN)10.1007/s10009-014-0355-9 (DOI)000374974200007 ()2-s2.0-84911400022 (Scopus ID)1433-2779 (ISBN)
Projects
ATAC - Advanced Test Automation for Complex Software-Intensive System (ITEA2/Vinnova)ITS-EASY Post Graduate School for Embedded Software and Systems
Available from: 2015-06-08 Created: 2015-06-08 Last updated: 2018-10-31Bibliographically approved
Enoiu, E. P., Sundmark, D., Causevic, A., Feldt, R. & Pettersson, P. (2016). Mutation-Based Test Generation for PLC Embedded Software using Model Checking. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): . Paper presented at 28th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2016; Graz; Austria; 17 October 2016 through 19 October 2016; Code 185379 (pp. 155-171). , 9976
Open this publication in new window or tab >>Mutation-Based Test Generation for PLC Embedded Software using Model Checking
Show others...
2016 (English)In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016, Vol. 9976, p. 155-171Conference paper, Published paper (Refereed)
Abstract [en]

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 9976
Keywords
Engineering controlled terms: Accident prevention; Application programs; Automation; Embedded software; Fault detection; Java programming language; Model checking; Programmable logic controllers; Safety engineering; Safety testing; Standards Automated test generations; Decision coverage; Detection ability; Engineering software; Experimental evidence; Mutation analysis; Mutation operators; Uppaal model checkers Engineering main heading: Software testing
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-32886 (URN)10.1007/978-3-319-47443-4_10 (DOI)000389932400010 ()2-s2.0-84992445107 (Scopus ID)
Conference
28th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2016; Graz; Austria; 17 October 2016 through 19 October 2016; Code 185379
Projects
ITS-EASY Post Graduate School for Embedded Software and SystemsTOCSYC - Testing of Critical System Characteristics (KKS)AGENTS - Automated Generation of Tests for Simulated Software Systems (KKS)
Available from: 2016-08-29 Created: 2016-08-24 Last updated: 2018-10-31Bibliographically approved
Seceleanu, C. & Pettersson, P. (2015). Foreword. JOURNAL OF LOGICAL AND ALGEBRAIC METHODS IN PROGRAMMING, 84(1), 1-1
Open this publication in new window or tab >>Foreword
2015 (English)In: JOURNAL OF LOGICAL AND ALGEBRAIC METHODS IN PROGRAMMING, ISSN 2352-2208, Vol. 84, no 1, p. 1-1Article in journal, Editorial material (Other academic) Published
National Category
Computer and Information Sciences
Identifiers
urn:nbn:se:mdh:diva-27521 (URN)10.1016/j.jlamp.2014.11.002 (DOI)000347601600001 ()2-s2.0-84938692531 (Scopus ID)
Available from: 2015-02-13 Created: 2015-02-13 Last updated: 2018-01-11Bibliographically approved
Seceleanu, C. & Pettersson, P. (Eds.). (2015). Journal of Logical and Algebraic Methods in Programming: NWPT 2011 Special Issue. Paper presented at The 23rd Nordic Workshop on Programming Theory (NWPT 2011). Sweden: Elsevier
Open this publication in new window or tab >>Journal of Logical and Algebraic Methods in Programming: NWPT 2011 Special Issue
2015 (English)Conference proceedings (editor) (Other academic)
Place, publisher, year, edition, pages
Sweden: Elsevier, 2015
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-35436 (URN)
Conference
The 23rd Nordic Workshop on Programming Theory (NWPT 2011)
Available from: 2017-06-01 Created: 2017-06-01 Last updated: 2017-06-01Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0003-4040-3480

Search in DiVA

Show all publications