mdh.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Automated System Level Software Testing of Networked Embedded Systems
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. Westermo Research and Development AB, Sweden. (Software Testing Laboratory)
2018 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. However, important challenges for successful software test automation are: lack of time for testing, lack of test environment availability, and an excess of test results information that renders decision-making hard.

In this thesis these challenges are tackled in three ways. First, in order to combat lack of time for testing, a method for automated system level regression test selection was implemented and evaluated using data from several years of nightly testing in a real world industrial setting. It was shown that automated test selection can be integrated into system level nightly testing and solve problems such as nightly testing not finishing on time. Second, in order to improve the hardware coverage of devices in the test environment and how test cases map to hardware, an algorithm based on the subgraph isomorphism problem was implemented and evaluated using industrial data. This implementation was significantly faster than the previous implementation, and the mapping process was done in such a way that hardware coverage increased over iterations. Third, to better understand decision-making in critical steps of the software development process in an industrial setting, two empirical studies were conducted. The results showed how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware, and back to developers via the test results database.

Automated system level software testing of networked embedded systems can be difficult to achieve. This thesis addresses several important challenges and provides results that are of interest both to industrial practitioners and researchers.

Place, publisher, year, edition, pages
Västerås: Mälardalen University , 2018.
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 275
National Category
Software Engineering
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-41109ISBN: 978-91-7485-408-4 (print)OAI: oai:DiVA.org:mdh-41109DiVA, id: diva2:1252751
Presentation
2018-11-13, Room Zeta, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2018-10-02 Created: 2018-10-02 Last updated: 2018-10-09Bibliographically approved
List of papers
1. Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors
Open this publication in new window or tab >>Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors
Show others...
2016 (English)In: 27th International Symposium on Software Reliability Engineering ISSRE'16, 2016Conference paper, Published paper (Refereed)
Abstract [en]

We propose a new method of determining an effective ordering of regression test cases, and describe its implementation as an automated tool called SuiteBuilder developed by Westermo Research and Development AB. The tool generates an efficient order to run the cases in an existing test suite by using expected or observed test duration and combining priorities of multiple factors associated with test cases, including previous fault detection success, interval since last executed, and modifications to the code tested. The method and tool were developed to address problems in the traditional process of regression testing, such as lack of time to run a complete regression suite, failure to detect bugs in time, and tests that are repeatedly omitted. The tool has been integrated into the existing nightly test framework for Westermo software that runs on large-scale data communication systems.  In experimental evaluation of the tool, we found significant improvement in regression testing results. The re-ordered test suites finish within the available time, the majority of fault-detecting test cases are located in the first third of the suite, no important test case is omitted, and the necessity for manual work on the suites is greatly reduced.

Keywords
Automation; Data communication systems; Fault detection; Program debugging; Regression analysis; Software reliability Automated systems; Experience report; Experimental evaluation; Large scale data; Regression suites; Regression testing; Regression tests; Research and development
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-33812 (URN)10.1109/ISSRE.2016.23 (DOI)000391437700002 ()2-s2.0-85013287871 (Scopus ID)
Conference
27th International Symposium on Software Reliability Engineering ISSRE'16, 23-27 Oct 2016, Ottawa, Canada
Available from: 2016-11-21 Created: 2016-11-21 Last updated: 2018-10-02Bibliographically approved
2. Automated test mapping and coverage for network topologies
Open this publication in new window or tab >>Automated test mapping and coverage for network topologies
Show others...
2018 (English)In: ISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, Inc , 2018, p. 73-83Conference paper, Published paper (Refereed)
Abstract [en]

Communication devices such as routers and switches play a critical role in the reliable functioning of embedded system networks. Dozens of such devices may be part of an embedded system network, and they need to be tested in conjunction with various computational elements on actual hardware, in many different configurations that are representative of actual operating networks. An individual physical network topology can be used as the basis for a test system that can execute many test cases, by identifying the part of the physical network topology that corresponds to the configuration required by each individual test case. Given a set of available test systems and a large number of test cases, the problem is to determine for each test case, which of the test systems are suitable for executing the test case, and to provide the mapping that associates the test case elements (the logical network topology) with the appropriate elements of the test system (the physical network topology). We studied a real industrial environment where this problem was originally handled by a simple software procedure that was very slow in many cases, and also failed to provide thorough coverage of each network's elements. In this paper, we represent both the test systems and the test cases as graphs, and develop a new prototype algorithm that a) determines whether or not a test case can be mapped to a subgraph of the test system, b) rapidly finds mappings that do exist, and c) exercises diverse sets of network nodes when multiple mappings exist for the test case. The prototype has been implemented and applied to over 10,000 combinations of test cases and test systems, and reduced the computation time by a factor of more than 80 from the original procedure. In addition, relative to a meaningful measure of network topology coverage, the mappings achieved an increased level of thoroughness in exercising the elements of each test system.

Place, publisher, year, edition, pages
Association for Computing Machinery, Inc, 2018
Keywords
Network topology, Subgraph isomorphism, Test coverage, Testing, Embedded systems, Mapping, Test facilities, Topology, Communication device, Computational elements, Industrial environments, Physical network topologies, Prototype algorithms, Software testing
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-40528 (URN)10.1145/3213846.3213859 (DOI)2-s2.0-85051515196 (Scopus ID)9781450356992 (ISBN)
Conference
27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, 16 July 2018 through 21 July 2018
Available from: 2018-08-23 Created: 2018-08-23 Last updated: 2018-10-02Bibliographically approved
3. Decision Making and Visualizations Based on Test Results
Open this publication in new window or tab >>Decision Making and Visualizations Based on Test Results
2018 (English)In: Empirical Software Engineering and Measurement, 12th International Symposium on ESEM18, 2018, article id 34Conference paper, Published paper (Refereed)
Abstract [en]

Background: Testing is one of the main methods for quality assurance in the development of embedded software, as well as in software engineering in general. Consequently, test results (and how they are reported and visualized) may substantially influence business decisions in software-intensive organizations. Aims: This case study examines the role of test results from automated nightly software testing and the visualizations for decision making they enable at an embedded systems company in Sweden. In particular, we want to identify the use of the visualizations for supporting decisions from three aspects: in daily work, at feature branch merge, and at release time. Method: We conducted an embedded case study with multiple units of analysis by conducting interviews, questionnaires, using archival data and participant observations. Results: Several visualizations and reports built on top of the test results database are utilized in supporting daily work, merging a feature branch to the master and at release time. Some important visualizations are: lists of failing test cases, easy access to log files, and heatmap trend plots. The industrial practitioners perceived the visualizations and reporting as valuable, however they also mentioned several areas of improvement such as better ways of visualizing test coverage in a functional area as well as better navigation between different views. Conclusions: We conclude that visualizations of test results are a vital decision making tool for a variety of roles and tasks in embedded software development, however the visualizations need to be continuously improved to keep their value for its stakeholders.

Keywords
Software Testing, Visualizations, Decision Making
National Category
Engineering and Technology Computer Systems
Identifiers
urn:nbn:se:mdh:diva-40902 (URN)10.1145/3239235.3268921 (DOI)000469776800034 ()2-s2.0-85053207166 (Scopus ID)978-1-4503-5823-1 (ISBN)
Conference
Empirical Software Engineering and Measurement, 12th International Symposium on ESEM18, 11 Oct 2018, Oulu, Finland
Projects
TOCSYC - Testing of Critical System Characteristics (KKS)The Volvo chair of vehicular electronics and software architectureITS ESS-H Industrial Graduate School in Reliable Embedded Sensor SystemsTESTMINE - Mining Test Evolution for Improved Software Regression Test Selection (KKS)
Available from: 2018-09-13 Created: 2018-09-13 Last updated: 2019-06-18Bibliographically approved
4. Information Flow in Software Testing: An Interview Study with Embedded Software Engineering Practitioners
Open this publication in new window or tab >>Information Flow in Software Testing: An Interview Study with Embedded Software Engineering Practitioners
Show others...
2019 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 7, p. 46434-46453Article in journal (Refereed) Published
Abstract [en]

Activities in software testing is a challenge for companies that develop embedded systems where multiple functional teams and technologically difficult tasks are common. This study aims at exploring the information flow in software testing, the perceived challenges and good approaches, for a more effective information flow. We conducted semi-structured interviews with twelve software practitioners working at five organizations in the embedded software industry in Sweden. The interviews were analyzed by means of thematic analysis. The data was classified into six themes that affect the information flow in software testing: testing and troubleshooting, communication, processes, technology, artifacts and organization. We further identified a number of challenges such as poor feedback and understanding exactly what has been tested; and approaches such as fast feedback as well as custom automated test reporting; to achieve an improved information flow. Our results indicate that there are many opportunities to improve this information flow: a first mitigation step is to better understand the challenges and approaches. Future work is needed to realize this in practice, for example to shorten feedback cycles between roles, as well as enhance exploration and visualization of test results

National Category
Software Engineering
Identifiers
urn:nbn:se:mdh:diva-40930 (URN)10.1109/ACCESS.2019.2909093 (DOI)000465621200001 ()2-s2.0-85064750453 (Scopus ID)
Funder
Knowledge Foundation, 20150277
Available from: 2018-09-13 Created: 2018-09-13 Last updated: 2019-05-09Bibliographically approved

Open Access in DiVA

fulltext(1330 kB)94 downloads
File information
File name FULLTEXT02.pdfFile size 1330 kBChecksum SHA-512
8e6bb97b97afcbd58efae95ea19704aca3fa061e6d489403dcfc0206d41658fd6df2b97e79d5a8bf513a5b3507d01ec7731c1133adc535ffb051b293deb4e216
Type fulltextMimetype application/pdf

Authority records BETA

Strandberg, Per Erik

Search in DiVA

By author/editor
Strandberg, Per Erik
By organisation
Embedded Systems
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 94 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 301 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf