https://www.mdu.se/

Changes to safety-critical systems are inevitable and can impact the safety confidence about a system as their effects can refute articulated claims about safety or challenge the supporting evidence on which this confidence relies. In order to maintain the safety confidence under changes, system developers need to re-analyse and re-verify the system to generate new valid items of evidence. Identifying the effects of a particular change is a crucial step in any change management process as it enables system developers to estimate the required maintenance effort and reduce the cost by avoiding wider analyses and verification than strictly necessary. This paper presents a sensitivity analysis-based technique which aims at measuring the ability of a system to contain a change (i.e., robustness) without the need to make a major re-design. The proposed technique exploits the safety margins in the budgeted failure probabilities of events in a probabilistic fault-tree analysis to compensate for unaccounted deficits or changes due to maintenance. The technique utilises safety contracts to provide prescriptive data for what is needed to be revisited and verified to maintain system safety when changes happen. We demonstrate the technique on an aircraft wheel braking system.

Changes to safety critical systems are inevitable and can impact the safety confidence about a system as their effects can refute articulated claims about safety or challenge the supporting evidence on which this confidence relies. In order to maintain the safety confidence due to changes, system developers need to re-analyse and re-verify the system to generate new valid items of evidence. Moreover, identifying the effects of a particular change is a crucial step in any change management process as it enables system developers to estimate the required maintenance effort and reduce the cost by avoiding wider analyses and verification than strictly necessary. This paper presents a sensitivity analysis-based technique which aims at measuring the ability of a system to contain a change (i.e., robustness) without the need to make a major re-design. The technique exploits the safety margins in the assigned failure probabilities to the events of a probabilistic fault-tree analysis to compensate some potential deficits in the overall failure probability budget due to changes. The technique also utilises safety contracts to provide prescriptive data for what is needed to be revisited and verified to maintain system safety when changes happen. We demonstrate the technique on a realistic safety critical system.

Testing of safety-critical systems is an important and costly endeavor. To date work has been mainly focusing on the design and application of diverse testing strategies. However, they have left the important decision of “when to stop testing” as an open research issue. In our previous work, we proposed a convergence algorithm that informs the tester when it is concluded that testing for longer will not reveal sufficiently important new findings, hence, should be stopped. The stoptest decision proposed by the algorithm was in the context of testing the worst-case timing characteristics of a system and was evaluated based on the As Low As Reasonably Practicable (ALARP) principle. The ALARP principle is an underpinning concept in many safety standards which is a cost-benefit argument. ALARP implies that a tolerable risk should be reduced to a point at which further risk-reduction is grossly disproportionate compared to the benefit attained. An ALARP stop-test decision means that the cost associated with further testing, after the algorithm stops, does not justify the benefit, i.e., any further increased in the observed worst-case timing.

In order to make a stop-test decision, the convergence algorithm used the Kullback-Leibler DIVergence (KL DIV) statistical test and was shown to be successful while being applied on system’s tasks having similar characteristics. However, there were some experiments in which the stop-test decision did not comply to the ALARP principle, i.e., it stopped sooner than expected by the ALARP criteria. Therefore, in this paper, we investigate whether the performance of the algorithm could be improved in such experiments focusing on the KL DIV test. More specifically, we firstly determine which features of KL DIV could adversely affect the algorithm performance. Secondly, we investigate whether another statistical test, i.e., the Earth Mover’s Distance (EMD), could potentially cover weaknesses of KL DIV. Finally, we experimentally evaluate our hypothesis of whether EMD does improve the algorithm where KL DIV has shown to not perform as expected.

The development of safety cases has become common practice in many safety critical system domains. Safety cases are costly since they need a significant amount of time and efforts to be produced. More- over, safety critical systems are expected to operate for a long period of time and constantly subject to changes during both development and operational phases. Hence, safety cases are built as living documents that should always be maintained to justify the safety status of the associated system and evolve as these system evolve. However, safety cases document highly interdependent elements (e.g., safety goals, evidence, assumptions, etc.) and even seemingly minor changes may have a major impact on them, and thus dramatically increase their cost. In this paper, we identify and discuss some challenges in the maintenance of safety cases. We also present two techniques that utilise safety contracts to facilitate the maintenance of safety cases, we discuss the roles of these techniques in coping with some of the identified maintenance challenges, and we finally discuss potential limitations and suggest some solutions.

Safety cases need significant amount of time and effort to produce. The required amount of time and effort can be dramatically increased due to system changes as safety cases should be maintained before they can be submitted for certification or re-certification. Anticipating potential changes is useful since it reveals traceable consequences that will eventually reduce the maintenance efforts. However, considering a complete list of anticipated changes is difficult. What can be easier though is to determine the flexibility of system components to changes. Using sensitivity analysis is useful to measure the flexibility of the different system properties to changes. Furthermore, contracts have been proposed as a means for facilitating the change management process due to their ability to record the dependencies among system’s components. In this paper, we extend a technique that uses a sensitivity analysis to derive safety contracts from Fault Tree Analyses (FTA) and uses these contracts to trace changes in the safety argument. The extension aims to enabling the derivation of hierarchical and correlated safety contracts.We motivate the extension through an illustrative example within which we identify limitations of the technique and discuss potential solutions to these limitations. 

The use of contracts to enhance the maintainability of safety-critical systems has received a significant amount of research effort in recent years. However some key issues have been identified: the difficulty in dealing with the wide range of properties of systems and deriving contracts to capture those properties; and the challenge of dealing with the inevitable incompleteness of the contracts. In this paper, we explore how the derivation of contracts can be performed based on the results of failure analysis. We use the concept of safety kernels to alleviate the issues. Firstly the safety kernel means that the properties of the system that we may wish to manage can be dealt with at a more abstract level, reducing the challenges of representation and completeness of the “safety” contracts. Secondly the set of safety contracts is reduced so it is possible to reason about their satisfaction in a more rigorous manner.

Testing of safety-critical embedded systems is an important and costly endeavor. To date work has been mainly focusing on the design and application of diverse testing strategies. However, they have left an open research issue of when to stop testing a system. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of system tasks, hence, should be stopped. This algorithm was shown to be successful while being applied across task sets having similar characteristics. For the convergence algorithm to become robust, it is important that it holds even if the task set characteristics here called nuisance factors, vary. Generally speaking, there might be either the main factors under analysis, called design factors, or nuisance factors that influence the performance of a process or system. Nuisance factors are not typically of interest in the context of the analysis. However, they vary from system to system and may have large effects on the performance, hence, being very important to be accounted for. Consequently, the current paper looks into a set of nuisance factors that affect our proposed convergence algorithm performance. More specifically, it is interested in situations when the convergence algorithm performance significantly degrades influencing its reliability. The work systematically analyzes each nuisance factor effect using a well-known statistical method, further, derives the most influential factors.

Testing of safety-critical embedded systems is an important and costly endeavor. To date researchers and practitioners have been mainly focusing on the design and application of diverse testing strategies, but leaving the test stopping criteria as an ad hoc decision and an open research issue. In our previous work, we proposed a convergence algorithm that informs the tester when the current testing strategy does not seem to be revealing new insight into the worst-case timing properties of tasks and hence should be stopped. This algorithm was shown to be successful but its trial and error tuning of parameters was an issue. In this paper, we use the Design of Experiment (DOE) approach to optimise the algorithm's performance and to improve its scalability. During our experimental evaluations the optimised algorithm showed improved performance by achieving relatively the same results with 42% less testing cost as compared to our previous work. The algorithm also has better scalability and opens up a new path towards achieving cost effective non-functional testing of real-time embedded systems.

A safety case contains safety arguments together with supporting evidence that together should demonstrate that a system is acceptably safe. System changes pose a challenge to the soundness and cogency of the safety case argument. Maintaining safety arguments is a painstaking process because it requires performing a change impact analysis through interdependent elements. Changes are often performed years after the deployment of a system making it harder for safety case developers to know which parts of the argument are affected. Contracts have been proposed as a means for helping to manage changes. There has been significant work that discusses how to represent and to use them but there has been little on how to derive them. In this paper, we propose a sensitivity analysis approach to derive contracts from Fault Tree Analyses and use them to trace changes in the safety argument, thus facilitating easier maintenance of the safety argument.