https://www.mdu.se/

In software system development, testing can take considerable time and resources, and there are numerous examples in the literature of how to improve the testing process. In particular, methods for selection and prioritization of test cases can play a critical role in efficient use of testing resources. This paper focuses on the problem of selection and ordering of integration-level test cases. Integration testing is performed to evaluate the correctness of several units in composition. Further, for reasons of both effectiveness and safety, many embedded systems are still tested manually. To this end, we propose a process, supported by an online decision support system, for ordering and selection of test cases based on the test result of previously executed test cases. To analyze the economic efficiency of such a system, a customized return on investment (ROI) metric tailored for system integration testing is introduced. Using data collected from the development process of a large-scale safety-critical embedded system, we perform Monte Carlo simulations to evaluate the expected ROI of three variants of the proposed new process. The results show that our proposed decision support system is beneficial in terms of ROI at system integration testing and thus qualifies as an important element in improving the integration testing process.

Prioritization, selection and minimization of test cases are well-known problems in software testing. Test case prioritization deals with the problem of ordering an existing set of test cases, typically with respect to the estimated likelihood of detecting faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Most existing approaches for test case prioritization and selection suffer from one or several drawbacks. For example, they to a large extent utilize static analysis of code for that purpose, making them unfit for higher levels of testing such as integration testing. Moreover, they do not exploit the possibility of dynamically changing the prioritization or selection of test cases based on the execution results of prior test cases. Such dynamic analysis allows for discarding test cases that do not need to be executed and are thus redundant. This paper proposes a generic method for prioritization and selection of test cases in integration testing that addresses the above issues. We also present the results of an industrial case study where initial evidence suggests the potential usefulness of our approach in testing a safety-critical train control management subsystem.

In industrial software testing, development projects typically set up and maintain test suites containing large numbers of test cases. Executing a large number of test cases can be expensive in terms of effort and wall-clock time. Moreover, indiscriminate execution of all available test cases typically lead to sub-optimal use of testing resources. On the other hand, selecting too few test cases for execution might leave a large number of faults undiscovered. Limiting factors such as allocated budget and time constraints for testing further emphasizes the importance of test case prioritization in order to identify test cases that enable earlier detection of faults while respecting such constraints. In this paper, we propose a multi-criteria decision making approach for prioritizing test cases in order to detect faults earlier. This is achieved by applying the TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) decision making technique combined with fuzzy principles. Our solution is based on important criteria such as fault detection probability, execution time, complexity, and other test case properties. By applying the approach on a train control management subsystem from Bombardier Transportation in Sweden, we demonstrate how it helps, in a systematic way, to identify test cases that can lead to early detection of faults while respecting various criteria.

A significant part of industrial systems have requirements on long life times. Such requirements on the complete system impose requirements on its corresponding embedded systems to be operational for an equally long time. As a consequence it is of paramount importance to be able to replace obsolete components of the embedded systems during the life time of the system, and to be able to update part of the design due to new requirements. In this paper we present a method to manage component replacement in such systems, and we present an industrial case study highlighting the work needed to replace an FPGA chip with another, including all corresponding legacy FPGA design challenges that comes with such a replacement. We have found one larger problem inherent in the ability to use the included components in a way that is not possible with the new circuits replacing the old ones. This problem significantly increased the work needed when performing the conversion and migration from the old design to the new, since parts of the design had to be redesigned from a functional perspective.

Engineering system architectures for complex systems involves the tasks of analyzing architectural drivers, identifying architectural concerns, identifying valid architecture candidates, and evaluation of alternatives. One problem to overcome when architecting a system is the identification of valid of architectural candidates. We have developed a step-wise method for performing system architecture analysis and tested it on a sub-system in a project developing a drive system for heavy automotive applications. In this paper we present the complete method of nine steps for engineering an architecture and we elaborate in detail on the procedure to identify architectural candidates based on previously identified architectural drivers. We present a diagram depicting the proposed information model, its concepts and their relationships. In addition, the expectations on such a method as expressed by practitioners have been elicited, and we elaborate on the validity by examining how well the method indicate fulfillment. Our conclusion is that the proposed method does not fail to deliver on any of the needs and this gives an indication of usefulness. When identifying architectural candidates it is important to use proper criteria in the process. Our conclusion is that the practitioners should focus on candidates that affect the system at hand (within system boundaries), and on the candidates that address the architecturally significant system use. This is reflected in our method where we prescribe evaluation of the design candidates by validating that they solve only the right problem and by ensuring that they address the system at hand.

Context: Organizational performance measurements in software product development have received a lot of attention in the literature. Still, there is a general discontent regarding the way performance is evaluated in practice, with few studies really focusing on why this is the case. In this paper research focusing on the context of developing software-intensive products in large established multi-national organizations is reported on. Objective: The purpose of this research is to investigate performance measurement practices related to software product development activities. More specifically, focus is on exploring how managers engaged in software product development activities perceive and evaluate performance in large organizations from a managerial perspective. Method: The research approach pursued in this research consist of exploratory multiple case studies. Data is collected mainly through 54 interviews in five case studies in large international organizations developing software-intensive products in Sweden. Focused group interviews with senior managers from eight companies have also been used in the data collection. Results: The results of this research indicate that managers within software product development in general are dissatisfied with their current way of evaluating performance. Performance measurements and the perception of performance are today focused on cost, time, and quality, i.e. what is easily measurable and not necessarily what is important. The dimensions of value creation and learning are missing. Moreover, measurements tend to be result oriented, rather than process oriented, making it difficult to integrate these measurements in the management practices. Conclusion: Managers that are dissatisfied with their performance measurement system and want to improve the current situation should not start by focusing on the current measurements directly; instead they should focus on how the organization perceives performance and how important performance criteria are being developed. By developing relevant performance criteria the first step in developing an effective performance measurement system is made. Moreover, it is concluded that manager's perception of performance is affected by the currently used measurements, hence limiting the scope of the performance criteria. Thus, a change in the way managers perceive performance is necessary before there can be any changes in the way performance is evaluated. 

Large infrastructure systems with a life time of more than 30 years, such as telecommunication or power transmission systems, are difficult to maintain since they suffer from the end-of-life plague of software, hardware and knowledge. Large companies have traditionally tackled this problem successfully, but maybe not with complete efficiency in all cases. We find system evolution to be an increasingly interesting problem as infrastructure becomes more complicated. Our increasingly complex and advanced society demands more of the infrastructure making system evolution an interesting alternative to system replacement. From the point of view of the ISO/IEC 15288 development process we have identified life cycle issues connected to long life time scenarios and the different life cycle stages. In this paper we contribute with a modification of the utilisation and support stage in ISO/IEC 15288 into an evolution stage where a system is not only retired and replaced but rather evolved into the next generation. Using this approach changes the view of system development for this specific type of systems towards a way of incremental development, where new functions can be added at the same time as old legacy parts are replaced with functionally equivalent modules based on new hardware. We have based our solution on the experience from investigations of life cycle issues for two large infrastructure systems.

A very important task in systems architecting is to understand the needs of the system and identify which ones have architectural ramifications, i.e., architectural drivers. The understanding of architectural drivers enables the later engineering tasks including evaluation of architectural alternatives. Systems engineering guidelines provide models and advice for what information entities to consider, but only limited proposals of how to proceed. In this paper, we device and present a method to perform analysis of architectural drivers and we apply it to an industrial case of developing a hybrid electric drive system for heavy automotive applications. We present data on what practitioners expect from such a method, we present the method and rationale, and preliminary results from applying the method to the case. We note that the process and information model are fairly general and could be considered useful for any developer of a complex system. We believe the proposed method closes some of the gap between the general models described in the system engineering guidelines and an industrially applicable method.

Handling of obsolete software and/or hardware components together with management of function updates in a complex embedded system with an expected life time of more than 30 years can be a very difficult to almost impossible task. This types of challenges can be found in a large number of companies in, for example, the power transmission industry, power plants, aviation etc. In this paper we present the basic steps in a proposed method for handling evolvability in such embedded systems with long expected life cycles. The key elements of the proposed method are the definition of function dependencies, release planning, and test requirements.

Management of obsolete software- and hardware-components along with the addition of new components and functionalities in existing embedded systems can be a very difficult or almost impossible task. If there at the same time is a requirement put on the system that it should be in operation for more than 30 years, then the evolution of the embedded system architecture over such a long duration of time will be an even more challenging problem to solve. A large number of different industries, for example the process and power transmission industries, are faced with this type of challenges on a daily basis. The contribution of our research presented in this paper is a set of questions and answers on how it is possible to replace and update an old control system, with examples inherent in process and power transmission applications. We also look at different methods that can be used during the development of new systems such that they will provide a natural support for evolvability, targeting future control system applications.