https://www.mdu.se/

Testing of safety-critical embedded systems is an important and costly endeavor. To date researchers and practitioners have been mainly focusing on the design and application of diverse testing strategies, but leaving the test stopping criteria as an ad hoc decision and an open research issue. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of tasks and hence should be stopped. This algorithm was shown to be successful but its trial and error tuning of parameters was an issue. In this paper, we use the Design of Experiment (DOE) approach to optimise the algorithm's performance and to improve its scalability. During our experimental evaluations the optimised algorithm showed improved performance by achieving relatively the same results with 42% less testing cost as compared to our previous work. The algorithm also has better scalability and opens up a new path towards achieving cost effective non-functional testing of real-time embedded systems.

Testing of safety-critical systems is an important and costly endeavor. To date work has been mainly focusing on the design and application of diverse testing strategies. However, they have left the important decision of “when to stop testing” as an open research issue. In our previous work, we proposed a convergence algorithm that informs the tester when it is concluded that testing for longer will not reveal sufficiently important new findings, hence, should be stopped. The stoptest decision proposed by the algorithm was in the context of testing the worst-case timing characteristics of a system and was evaluated based on the As Low As Reasonably Practicable (ALARP) principle. The ALARP principle is an underpinning concept in many safety standards which is a cost-benefit argument. ALARP implies that a tolerable risk should be reduced to a point at which further risk-reduction is grossly disproportionate compared to the benefit attained. An ALARP stop-test decision means that the cost associated with further testing, after the algorithm stops, does not justify the benefit, i.e., any further increased in the observed worst-case timing.

In order to make a stop-test decision, the convergence algorithm used the Kullback-Leibler DIVergence (KL DIV) statistical test and was shown to be successful while being applied on system’s tasks having similar characteristics. However, there were some experiments in which the stop-test decision did not comply to the ALARP principle, i.e., it stopped sooner than expected by the ALARP criteria. Therefore, in this paper, we investigate whether the performance of the algorithm could be improved in such experiments focusing on the KL DIV test. More specifically, we firstly determine which features of KL DIV could adversely affect the algorithm performance. Secondly, we investigate whether another statistical test, i.e., the Earth Mover’s Distance (EMD), could potentially cover weaknesses of KL DIV. Finally, we experimentally evaluate our hypothesis of whether EMD does improve the algorithm where KL DIV has shown to not perform as expected.

Testing of safety-critical embedded systems is an important and costly endeavor. To date work has been mainly focusing on the design and application of diverse testing strategies. However, they have left an open research issue of when to stop testing a system. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of system tasks, hence, should be stopped. This algorithm was shown to be successful while being applied across task sets having similar characteristics. For the convergence algorithm to become robust, it is important that it holds even if the task set characteristics here called nuisance factors, vary. Generally speaking, there might be either the main factors under analysis, called design factors, or nuisance factors that influence the performance of a process or system. Nuisance factors are not typically of interest in the context of the analysis. However, they vary from system to system and may have large effects on the performance, hence, being very important to be accounted for. Consequently, the current paper looks into a set of nuisance factors that affect our proposed convergence algorithm performance. More specifically, it is interested in situations when the convergence algorithm performance significantly degrades influencing its reliability. The work systematically analyzes each nuisance factor effect using a well-known statistical method, further, derives the most influential factors.

ALARP is an important concept in many safety standards. It helps in making a decision about how tolerable a risk is. A tolerable risk should be reduced to a point that is As Low As Reasonably Practicable (ALARP) which implies further risk-reduction is grossly inappropriate compared to the benefit attained. To date work has considered the process, safety arguments, and influencing factors of how to make an ALARP decision but not shown how to make a quantified judgement for it. In this paper a method for making an ALARP judgement decision is proposed in the context of testing the worst-case timing properties of systems. The method is based around a convergence algorithm that informs the tester when it is believed that testing for longer will not reveal sufficiently important new findings, i.e. any significant increase in observed worst-case timing needs a disproportionate amount of testing time.