https://www.mdu.se/

In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. We present a method for analyzing datasets from a use-case scenario perspective, detecting and quantifying out-of-distribution (OOD) data on dataset level. Our main contribution is the novel use of similarity metrics for the evaluation of the robustness of a model by introducing relative Fréchet Inception Distance (FID) and relative Kernel Inception Distance (KID) measures. These relative measures are relative to a baseline in-distribution dataset and are used to estimate how the model will perform on OOD data (i.e. estimate the model accuracy drop). We find a correlation between our proposed relative FID/relative KID measure and the drop in Average Precision (AP) accuracy on unseen data.

Machine Learning (ML) systems require representative and diverse datasets to accurately learn the objective task. Insupervised learning data needs to be accurately annotated, whichis an expensive and error-prone process. We present a methodfor generating synthetic data tailored to the use-case achievingexcellent performance in a real-world usecase. We provide amethod for producing automatically annotated synthetic visualdata of multirotor unmanned aerial vehicles (UAV) and otherairborne objects in a simulated environment with a high degreeof scene diversity, from collection of 3D models to generation ofannotated synthetic datasets (synthsets). In our data generationframework SynRender we introduce a novel method of usingNeural Radiance Field (NeRF) methods to capture photo-realistichigh-fidelity 3D-models of multirotor UAVs in order to automatedata generation for an object detection task in diverse environments. By producing data tailored to the real-world setting, ourNeRF-derived results show an advantage over generic 3D assetcollection-based methods where the domain gap between thesimulated and real-world is unacceptably large. In the spirit ofkeeping research open and accessible to the research communitywe release our dataset VISER DroneDiversity used in this project,where visual images, annotated boxes, instance segmentation anddepth maps are all generated for each image sample.

In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. In the realm of safety-critical machine learning the used datasets need to reflect the environment in which the system is intended to operate, in order to minimize the generalization gap between trained and real-world inputs. Datasets should be thoroughly prepared and requirements on the properties and characteristics of the collected data need to be specified. In our work we present a case study in which generating a synthetic dataset is accomplished based on real-world flight data from the ADS-B system, containing thousands of approaches to several airports to identify real-world statistical distributions of relevant variables to vary within our dataset sampling space. We also investigate what the effects are of training a model on synthetic data to different extents, including training on translated image sets (using domain adaptation). Our results indicate airport location to be the most critical parameter to vary. We also conclude that all experiments did benefit in performance from pre-training on synthetic data rather than using only real data, however this did not hold true in general for domain adaptation-translated images.

In this paper, we discuss challenges when using neural networks (NNs) in safety-critical applications. We address the challenges one by one, with aviation safety in mind. We then introduce a possible implementation to overcome the challenges. Only a small portion of the solution has been implemented physically and much work is considered as future work. Our current understanding is that a real implementation in a safety-critical system would be extremely difficult. Firstly, to design the intended function of the NN, and secondly, designing monitors needed to achieve a deterministic and fail-safe behavior of the system. We conclude that only the most valuable implementations of NNs should be considered as meaningful to implement in safety-critical systems.