https://www.mdu.se/

Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.

Communication devices such as routers and switches play a critical role in the reliable functioning of embedded system networks. Dozens of such devices may be part of an embedded system network, and they need to be tested in conjunction with various computational elements on actual hardware, in many different configurations that are representative of actual operating networks. An individual physical network topology can be used as the basis for a test system that can execute many test cases, by identifying the part of the physical network topology that corresponds to the configuration required by each individual test case. Given a set of available test systems and a large number of test cases, the problem is to determine for each test case, which of the test systems are suitable for executing the test case, and to provide the mapping that associates the test case elements (the logical network topology) with the appropriate elements of the test system (the physical network topology). We studied a real industrial environment where this problem was originally handled by a simple software procedure that was very slow in many cases, and also failed to provide thorough coverage of each network's elements. In this paper, we represent both the test systems and the test cases as graphs, and develop a new prototype algorithm that a) determines whether or not a test case can be mapped to a subgraph of the test system, b) rapidly finds mappings that do exist, and c) exercises diverse sets of network nodes when multiple mappings exist for the test case. The prototype has been implemented and applied to over 10,000 combinations of test cases and test systems, and reduced the computation time by a factor of more than 80 from the original procedure. In addition, relative to a meaningful measure of network topology coverage, the mappings achieved an increased level of thoroughness in exercising the elements of each test system.

Working Memory (WM) processing is central for human cognitive behavior. Using neurofeedback training to enhance the individual WM capacity is a promising technique but requires careful consideration when choosing the feedback signal. Feedback in terms of univariate spectral power (specifically theta and alpha power) has yielded questionable behavioral effects. However, a promising new direction for WM neurofeedback training is by using a measure of WM that is extracted by multivariate pattern classification. This study recorded EEG oscillatory activity from 15 healthy participants while they were engaged in the n-back task, n[1,2]. Univariate measures of the theta, alpha, and theta-over-alpha power ratio and a measure of WM extracted from multivariate pattern classification (of n-back task load conditions) was compared in relation to individual n-back task performance. Results show that classification performance is positively correlated to individual 2-back task performance while theta, alpha and thetaover-alpha power ratio is not. These results suggest that the discriminability of multivariate EEG oscillatory patterns between two WM load conditions reflects individual WM capacity. 

Westermo Research and Development has developed SuiteBuilder, an automated tool to determine an effective ordering of regression test cases. The ordering is based on factors such as fault detection success, the interval since the last execution, and code modifications. SuiteBuilder has enabled Westermo to overcome numerous regression-testing problems, including lack of time to run a complete regression suite, failure to detect bugs in a timely manner, and repeatedly omitted tests. In the tool's first two years of use, reordered test suites finished in the available time, most fault-detecting test cases were located in the first third of suites, no important test case was omitted, and the necessity for manual work on the suites decreased greatly. 

The authors have held both academic and industrial research positions, and have designed and carried out many empirical studies of large software systems that were built and maintained in industrial environments. Their experiences show that the most crucial component of a successful study is the participation of at least one industrial collaborator who is committed to the study’s goals and is able to provide advice and assistance throughout the course of the study. This paper describes studies carried out in three different industrial environments, discusses obstacles that arise, and how the authors have been able to overcome some of those obstacles. 

This paper discusses third-level educational programmes that are intended to prepare their graduates for a career building systems in which software plays a major role. Such programmes are modelled on traditional Engineering programmes but have been tailored to applications that depend heavily on software. Rather than describe knowledge that should be taught, we describe capabilities that students should acquire in these programmes. The paper begins with some historical observations about the software development field. 

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.

We describe the application and evaluation of fault prediction algorithms to a project developed by a Swedish company that transitioned from waterfall to agile development methods. The project used two different version control systems and a separate bug tracking system during its lifetime. The algorithms were originally designed for use on systems implemented with a traditional waterfall process at American companies that maintained their project records in an integrated database system that combined bug recording and version control. We compare the performance of the original prediction model on the American systems to the results obtained in the Swedish environment in both its pre-agile and agile stages. We also consider the impact of additional variables in the model.

This chapter is a survey of close to ten years of software fault prediction research performed by our group. We describe our initial motivation, the variables used to make predictions, provide a description of our standard model based on Negative Binomial Regression, and summarize the results of using this model to make predictions for nine large industrial software systems. The systems range in size from hundreds of thousands to millions of lines of code. All have been in the field for multiple years and many releases, and continue to be maintained and enhanced, usually at 3 month intervals. Effectiveness of the fault predictions is assessed using two different metrics. We compare the effectiveness of the standard model to augmented models that include variables related to developer counts, to inter-file calling structure, and to information about specific developers who modified the code. We also evaluate alternate prediction models based on different training algorithms, including Recursive Partitioning, Bayesian Additive Regression Trees, and Random Forests.

Previous research has provided evidence that a combination of static code metrics and software history metrics can be used to predict with surprising success which files in the next release of a large system will havethe largest numbers of defects. In contrast, very little research exists to indicate whether information about individual developers can profitably be used to improve predictions. We investigate whether files in a large system that are modified by an individual developer consistently contain either more or fewer faults than the average of all files in the system. The goal of the investigation is to determine whether information about which particular developer modified a file is able to improve defect predictions. We also extend earlier research evaluating use of counts of the number of developers who modified a file as predictors of the file's future faultiness. We analyze change reports filed for three large systems, each containing 18 releases, with a combined total of nearly 4 million LOC and over 11,000 files. A buggy file ratio is defined for programmers, measuring the proportion of faulty files in Release R out of all files modified by the programmer in Release R-1. We assess the consistency of the buggy file ratio across releases for individual programmers both visually and within the context of a fault prediction model. Buggy file ratios for individual programmers often varied widely across all the releases that they participated in. A prediction model that takes account of the history of faulty files that were changed by individual developers shows improvement over the standard negative binomial model of less than 0.13% according to one measure, and no improvement at all according to another measure. In contrast, augmenting a standard model with counts of cumulative developers changing files in prior releases produced up to a 2% improvement in the percentage of faults detected in the top 20% of predicted faulty files. The cumulative number of developers interacting with a file can be a useful variable for defect prediction. However, the study indicates that adding information to a model about which particular developermodified a file is not likely to improve defect predictions.