https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Information Flow in Software Testing: An Interview Study with Embedded Software Engineering Practitioners
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems. Westermo Reserarch and Development AB. (Software Testing Laboratory)ORCID iD: 0000-0003-1688-6937
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0003-2416-4205
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0003-0611-2655
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-5032-2310
Show others and affiliations
2019 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 7, p. 46434-46453Article in journal (Refereed) Published
Abstract [en]

Activities in software testing is a challenge for companies that develop embedded systems where multiple functional teams and technologically difficult tasks are common. This study aims at exploring the information flow in software testing, the perceived challenges and good approaches, for a more effective information flow. We conducted semi-structured interviews with twelve software practitioners working at five organizations in the embedded software industry in Sweden. The interviews were analyzed by means of thematic analysis. The data was classified into six themes that affect the information flow in software testing: testing and troubleshooting, communication, processes, technology, artifacts and organization. We further identified a number of challenges such as poor feedback and understanding exactly what has been tested; and approaches such as fast feedback as well as custom automated test reporting; to achieve an improved information flow. Our results indicate that there are many opportunities to improve this information flow: a first mitigation step is to better understand the challenges and approaches. Future work is needed to realize this in practice, for example to shorten feedback cycles between roles, as well as enhance exploration and visualization of test results

Place, publisher, year, edition, pages
2019. Vol. 7, p. 46434-46453
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:mdh:diva-40930DOI: 10.1109/ACCESS.2019.2909093ISI: 000465621200001Scopus ID: 2-s2.0-85064750453OAI: oai:DiVA.org:mdh-40930DiVA, id: diva2:1247851
Funder
Knowledge Foundation, 20150277Available from: 2018-09-13 Created: 2018-09-13 Last updated: 2022-11-09Bibliographically approved
In thesis
1. Automated System Level Software Testing of Networked Embedded Systems
Open this publication in new window or tab >>Automated System Level Software Testing of Networked Embedded Systems
2018 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. However, important challenges for successful software test automation are: lack of time for testing, lack of test environment availability, and an excess of test results information that renders decision-making hard.

In this thesis these challenges are tackled in three ways. First, in order to combat lack of time for testing, a method for automated system level regression test selection was implemented and evaluated using data from several years of nightly testing in a real world industrial setting. It was shown that automated test selection can be integrated into system level nightly testing and solve problems such as nightly testing not finishing on time. Second, in order to improve the hardware coverage of devices in the test environment and how test cases map to hardware, an algorithm based on the subgraph isomorphism problem was implemented and evaluated using industrial data. This implementation was significantly faster than the previous implementation, and the mapping process was done in such a way that hardware coverage increased over iterations. Third, to better understand decision-making in critical steps of the software development process in an industrial setting, two empirical studies were conducted. The results showed how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware, and back to developers via the test results database.

Automated system level software testing of networked embedded systems can be difficult to achieve. This thesis addresses several important challenges and provides results that are of interest both to industrial practitioners and researchers.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2018
Series
Mälardalen University Press Licentiate Theses, ISSN 1651-9256 ; 275
National Category
Software Engineering
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-41109 (URN)978-91-7485-408-4 (ISBN)
Presentation
2018-11-13, Room Zeta, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2018-10-02 Created: 2018-10-02 Last updated: 2022-11-09Bibliographically approved
2. Automated System-Level Software Testing of Industrial Networked Embedded Systems
Open this publication in new window or tab >>Automated System-Level Software Testing of Industrial Networked Embedded Systems
2021 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. Out of the many challenges for successful software test automation, this thesis addresses five: (i) understanding how updated software reaches a test environment, how testing is conducted in the test environment, and how test results reach the developers that updated the software in the first place; (ii) selecting which test cases to execute in a test suite given constraints on available time and test systems; (iii) given that the test cases an run on different configurations of connected devices, selecting which hardware to use for each test case to be executed; (iv) analyzing test cases that, when executed over time on evolving software, testware or hardware revisions, appear to randomly fail; and (v) making test results information actionable with test results exploration and visualization.

The challenges are tackled in several ways. First, to better understand the flow of information in the embedded systems software development process, interviews at five different companies were conducted. The results show how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware in the test environment, and back to developers. Second, in order to address the challenges of test selection and hardware selection, automated approaches for testing given resource constraints were implemented and evaluated using industrial data stemming from years of nightly testing. It was shown that these approaches could solve problems such as nightly testing not finishing on time, as well as increasing hardware coverage by varying hardware selection over test iterations. Third, the challenge of intermittently failing tests was addressed with a new metric that can classify test cases as intermittently or consistently failing. Again, by using industry data, factors that lead to intermittent failures were identified, and similarities and differences between root causes for intermittently and consistently failing tests were observed. Finally, in order to better render test results actionable, a tool was implemented for test results exploration and visualization. The implementation was evaluated using a reference group and logging of the tool’s usage. Solution patterns and views of the tool were identified, as well as challenges for implementing such a tool.

Abstract [sv]

Inbyggda system finns överallt och fyller viktiga roller i ledningssystem för industri och transport. Här kan mjukvarufel leda till produktionsbortfall eller till och med dödsfall, så mjukvaran i systemen måste vara tillförlitlig. Mjukvarutestning är en standardmetod för att kvalitetssäkra mjukvaran i inbyggda system, och många processer för mjukvaruutveckling strävar efter automatiserad testning. Av de många utmaningarna för framgångsrik testautomatisering täcker denna avhandling fem: (i) att förstå hur uppdaterad mjukvara når en testmiljö, hur testning utförs i testmiljön och hur testresultat når tillbaka till utvecklarna som uppdaterade mjukvaran; (ii) att välja vilka testfall som ska exekveras i en testsvit givet begränsningar i tillgänglig tid och tillgängliga testsystem; (iii) givet att testfall körs på olika konfigurationer av anslutna enheter, hur väljs hårdvaran som ska användas för varje testfall ut; (iv) att analysera testfall som, när de körs upprepade gånger med mjukvara, testvara eller hårdvara under utveckling, verkar slumpmässigt påvisa fel; och (v) att göra testresultat mer användbara genom göra det möjligt att utforska och visualisera den.

Utmaningarna möts på flera sätt. För det första, för att bättre förstå informationsflödet i utvecklingsprocessen för mjukvaran i inbyggda system så genomfördes intervjuer på fem olika företag. Resultaten visar hur visualiseringar och en testresultatsdatabas stödjer beslutsfattande. Resultaten beskriver också det övergripande informationsflödet i mjukvarutestning: från utvecklare till hårdvara i testmiljön och tillbaka till utvecklarna. För det andra, för att ta itu med utmaningarna vid testselektion och hårdvaruselektion, implementerades och utvärderades automatiserade metoder för testning givet resursbegränsningar, med hjälp av industriella data från flera år av nattliga tester. Dessa tillvägagångssätt kunde lösa problem som att nattliga tester inte avslutas i tid, och kan även öka hårdvarutäckningen genom att variera hårdvaruselektionen över testiterationer. För det tredje angrips utmaningen med intermittenta tester med en ny metrik för att klassificera testfall som intermittent eller konsekvent påvisar fel. Genom att återigen använda industriellt data så identifierades faktorer som leder till intermittenta tester, och likheter och skillnader i rotorsaker för intermittenta och konsekventa tester identifierades. Slutligen, för att göra testresultaten mer användbara, implementerades ett verktyg för utforskning och visualisering av testresultat. Implementationen utvärderades med hjälp av en referensgrupp samt av loggning av verktygets användning. Lösningsmönster och verktygets vyer identifierades, liksom utmaningar för att implementera ett sådant verktyg.

Place, publisher, year, edition, pages
Västerås: Mälardalen University, 2021
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 349
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-56036 (URN)978-91-7485-529-6 (ISBN)
Public defence
2021-11-22, Gamma och online via Teams, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Note

Available from: 2021-10-13 Created: 2021-10-13 Last updated: 2022-11-08Bibliographically approved

Open Access in DiVA

fulltext(11357 kB)312 downloads
File information
File name FULLTEXT02.pdfFile size 11357 kBChecksum SHA-512
738d40effc568fb4e801b44a8d5fd7fbb6e0c2d53eb9854b84b1e20e9ffa57f7a01970f92c76646e2ab573b84e30859eb887bd4f4bfed560f99a2eb85b9795ac
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Strandberg, Per ErikEnoiu, Eduard PaulAfzal, WasifSundmark, Daniel

Search in DiVA

By author/editor
Strandberg, Per ErikEnoiu, Eduard PaulAfzal, WasifSundmark, DanielFeldt, Robert
By organisation
Embedded Systems
In the same journal
IEEE Access
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 322 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 621 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf