https://www.mdu.se/

In this paper, we look at identifying temporal differences between different versions of Complex Real-Time Embedded Systems (CRTES) by using timing traces representing response times and executiontimes of tasks. In particular, we are interested in being able to reason about whether a particular change to CRTES will impact on their temporal performance, which is difficult to answer due to the complicatedtiming behavior such CRTES have. To be specific, we first propose a sampling mechanism to eliminate dependencies existing in tasks' response time and execution time data in the traces taken from CRTES, which makes any statistical inference in probability theory and statistics realistic. Next, we use a mature statistical method, i.e., the non-parametric two-sample Kolmogorov-Smirnov test, to assess the possible temporal differences between different versions of CRTES by using timing traces. Moreover, we introduce a method of reducing the number of samples used in the analysis, while keeping the accuracy ofanalysis results. This is not trivial, as collecting a large amount of samples in terms of executing real systems is often costly. Our evaluation using simulation models describing an industrial robotic controlsystem with complicated tasks' timing behavior, indicates that the proposed method can successfully identify temporal differences between different versions of CRTES, if there is any. Furthermore, our proposed method outperforms the other statistical methods, e.g., bootstrap and permutation tests, that are often widely used in contexts, in terms of bearing on the accuracy of results when other methods have failed.

The ability to measure performance is both fundamental to and critical in successful product development. Previous research has focused on adding new measurements, not on evaluating those currently used. The PMEX is a tool to evaluate the performance-measurement system based on success factors in the development of software-intensive products. Three case studies have been conducted to test the PMEX and the results indicate that the PMEX enables managers to determine explicitly what is and what is not measured. Time, cost and quality seem to be the focus of the performance measurements while technology and planning activities are absent.

This chapter presents a framework for a conceptual evaluation of the performance of industrial product innovation activities. The framework promotes a holistic view of performance by considering three categories of activities: Planning, Implementation, and Sales and Delivery. Successful performance evaluation comes from acknowledging the fact that there are different objectives for each of the three activity categories. Moreover, performance may be expressed as a function of the performance of the Planning, the Implementation, and the Sales and Delivery activities. In this chapter the results of research involving seven large companies in Sweden, with the objective of improving the understanding of what is required to be successful when developing complex industrial products, are presented. Key factors for success as well as some general conclusions are discussed.

This paper presents RapidRT, a novel statistical approach to Worst-Case Response-Time (WCRT) analysis targeting complex embedded real-time systems. The proposed algorithm combines Extreme Value Theory (EVT) and other statistical methods in order to produce a probabilistic WCRT estimate. This estimate is calculated using response time data from either Monte Carlo simulations of a detailed model of the system, or from response-time measurements of the real system. The method could be considered as a pragmatic approach intended for complex industrial systems with real-time requirements. The target systems contain tasks with many intricate dependencies in theirtemporal behavior, which violates the assumptions of traditional analytical methods for response time analysis and thereby makes them overly pessimistic. An evaluation ispresented using two simulation models, inspired by an industrial robotic control system, and five other methods as reference.

Simulation-based analysis methods make few restrictions on the system design and scale to very large and complex systems, therefore they are widely used in timing analysis of complex industrial embedded systems. This paper presents a statistical approach to validation of temporal simulation models extracted from complex embedded systems, by introducing existing mature statistical methods to the context. The proposed approach firstly collects sampling distributions of response time and execution time data of tasks in both the modeled system and the model, based on simple random samples (SRS). The second step of the approach is to compare the sampling distributions, regarding interesting timing properties, by using the non-parametric two-sample Kolmogorov-Smirnov test. The evaluation using a fictive system model inspired by a real robotic control system with a set of change scenarios, shows a promising result. The proposed algorithm can identify temporal differences between the target system and its extracted model, i.e., the algorithm can assess whether the extracted model is a sufficiently accurate approximation of the target system.

The purpose of this paper is to from a managerial perspective investigate performance evaluation practices in a product development context. The focus is on exploring how managers perceive and evaluate performance in a software-intensive product development context. The research approach pursued in this research consist of a literature review combined with focused group interviews and exploratory multiple case studies. Moreover, a number of seminars have been held to discuss the findings both in academia and in practice. The result of this research indicates that product development managers are dissatisfied with their current way of evaluating performance. Performance measurements and the perception of performance are focused on cost, time and quality, i.e. what is easily measurable and not necessarily what is important. The dimensions of value creations and learning are missing. It is argued that manager perception of performance is affected by how it is measured, hence limiting the scope of the performance criteria. Thus, a change in the way managers perceive performance is necessary before there can be any change in the way performance is evaluated. Managers within product development that are dissatisfied with their performance measurement system should focus on how the organization perceive performance. By developing relevant performance criteria the first step in developing an effective performance evaluation system is made.

In today's competitive environment the need to deploy product-development investments more effectively is greater than ever. To assist managers, two conceptual tools have been developed to support the evaluation of performance in product development. The Performance Measurement Evaluation Matrix (PMEX) presented here helps managers evaluate performance-measurement systems they currently use, in order to identify areas requiring improvement. Results from using the PMEX indicate that it is common to associate performance measurements with the efficiency aspects of time, cost, and quality, without monitoring the value created. Performance is largely perceived by managers in terms of time, cost, and quality of the activities in the later phases of the development process. We argue that an effective performance-measurement system is based on performance criteria and then derive measurements based on these. It is argued that there should be a change in the perception of performance before performance-evaluation systems can be improved. The Product Development Organizational Model (PDOPM) assists in developing the perception of performance by relating uncertainty, efficiency and effectiveness at three generic activity levels within the product-development function. The use of our tools provides an improved perception of performance and its measurement, thus enabling improvements to the evaluation of performance.