https://www.mdu.se/

Autonomous agents rely on information from the surrounding environment to act upon. In the array of sensors available, the image sensor is perhaps the most versatile, allowing for detection of colour, size, shape, and depth. For the latter, in a dynamic environment, assuming no a priori knowledge, stereo vision is a commonly adopted technique. How to interpret images, and extract relevant information, is referred to as computer vision. Computer vision, and specifically stereo-vision algorithms, are complex and computationally expensive, already considering a single stereo pair, with results that are, in terms of accuracy, qualitatively difficult to compare. Adding to the challenge is a continuous stream of images, of a high frame rate, and the race of ever increasing image resolutions. In the context of autonomous agents, considerations regarding real-time requirements, embedded/resource limited processing platforms, power consumption, and physical size, further add up to an unarguably challenging problem.

This thesis aims to achieve embedded high-resolution stereo-vision of high frame-rate and low latency, by approaching the problem from two different angles, hardware and algorithmic development, in a symbiotic relationship. The first contributions of the thesis are the GIMME and GIMME2 embedded vision platforms, which offer hardware accelerated processing through FGPAs, specifically targeting stereo vision, contrary to available COTS systems at the time. The second contribution, toward stereo vision algorithms, is twofold. Firstly, the problem of scalability and the associated disparity range is addressed by proposing a segment-based stereo algorithm. In segment space, matching is independent of image scale, and similarly, disparity range is measured in terms of segments, indicating relatively few hypotheses to cover the entire range of the scene. Secondly, more in line with the conventional stereo correspondence for FPGAs, the Census Transform (CT) has been identified as a recurring cost metric. This thesis proposes an optimisation of the CT through a Genetic Algorithm (GA) - the Genetic Algorithm Census Transform (GACT). The GACT shows promising results for benchmark datasets, compared to established CT methods, while being resource efficient.

Navigation and orientation in an indoor environment are a challenging task for visually impaired people. This paper proposes a portable vision-based system to provide support for visually impaired persons in their daily activities. Here, machine learning algorithms are used for obstacle avoidance and object recognition. The system is intended to be used independently, easily and comfortably without taking human help. The system assists in obstacle avoidance using cameras and gives voice message feedback by using a pre-trained YOLO Neural Network for object recognition. In other parts of the system, a floor plane estimation algorithm is proposed for obstacle avoidance and fuzzy logic is used to prioritize the detected objects in a frame and generate alert to the user about possible risks. The system is implemented using the Robot Operating System (ROS) for communication on a Nvidia Jetson TX2 with a ZED stereo camera for depth calculations and headphones for user feedback, with the capability to accommodate different setup of hardware components. The parts of the system give varying results when evaluated and thus in future a large-scale evaluation is needed to implement the system and get it as a commercialized product in this area.

The aim of the study presented in this paper is to develop a quantitative evaluation tool of the sense of safety and security for robots in eldercare. By investigating the literature on measurement of safety and security in human-robot interaction, we propose new evaluation tools. These tools are semantic differential scale questionnaires. In experimental validation, we used the Pepper robot, programmed in the way to exhibit social behaviors, and constructed four experimental conditions varying the degree of the robot’s non-verbal behaviors from no gestures at all to full head and hand movements. The experimental results suggest that both questionnaires (for the sense of safety and the sense of security) have good internal consistency.

This paper examines whether social cues, such as facial expressions, can be used to adapt and tailor a robot-assisted training in order to maximize performance and comfort. Specifically, this paper serves as a basis in determining whether key facial signals, including emotions and facial actions, are common among participants during a physical and cognitive training scenario. In the experiment, participants performed basic arm exercises with a social robot as a guide. We extracted facial features from video recordings of participants and applied a recursive feature elimination algorithm to select a subset of discriminating facial features. These features are correlated with the performance of the user and the level of difficulty of the exercises. The long-term aim of this work, building upon the work presented here, is to develop an algorithm that can eventually be used in robot-assisted training to allow a robot to tailor a training program based on the physical capabilities as well as the social cues of the users.

Over the past few decades the use of industrial robots has increased the efficiency as well as the competitiveness of several sectors. Despite this fact, in many cases robot automation investments are considered to be technically challenging. In addition, for most small and medium-sized enterprises (SMEs) this process is associated with high costs. Due to their continuously changing product lines, reprogramming costs are likely to exceed installation costs by a large margin. Furthermore, traditional programming methods of industrial robots are too complex for most technicians or manufacturing engineers, and thus assistance from a robot programming expert is often needed. The hypothesis is that in order to make the use of industrial robots more common within the SME sector, the robots should be reprogrammable by technicians or manufacturing engineers rather than robot programming experts. In this thesis, a novel system for task-level programming is proposed. The user interacts with an industrial robot by giving instructions in a structured natural language and by selecting objects through an augmented reality interface. The proposed system consists of two parts: (i) a multimodal framework that provides a natural language interface for the user to interact in which the framework performs modality fusion and semantic analysis, (ii) a symbolic planner, POPStar, to create a time-efficient plan based on the user's instructions. The ultimate goal of this work in this thesis is to bring robot programming to a stage where it is as easy as working together with a colleague.This thesis mainly addresses two issues. The first issue is a general framework for designing and developing multimodal interfaces. The general framework proposed in this thesis is designed to perform natural language understanding, multimodal integration and semantic analysis with an incremental pipeline. The framework also includes a novel multimodal grammar language, which is used for multimodal presentation and semantic meaning generation. Such a framework helps us to make interaction with a robot easier and more natural. The proposed language architecture makes it possible to manipulate, pick or place objects in a scene through high-level commands. Interaction with simple voice commands and gestures enables the manufacturing engineer to focus on the task itself, rather than the programming issues of the robot. The second issue addressed is due to inherent characteristics of communication with the use of natural language; instructions given by a user are often vague and may require other actions to be taken before the conditions for applying the user's instructions are met. In order to solve this problem a symbolic planner, POPStar, based on a partial order planner (POP) is proposed. The system takes landmarks extracted from user instructions as input, and creates a sequence of actions to operate the robotic cell with minimal makespan. The proposed planner takes advantage of the partial order capabilities of POP to execute actions in parallel and employs a best-first search algorithm to seek the series of actions that lead to a minimal makespan. The proposed planner can also handle robots with multiple grippers, parallel machines as well as scheduling for multiple product types.

This research concerns the information design and visual design of graphical user interfaces (GUI) in the condition monitoring and condition-based maintenance (CBM) of production equipment. It also concerns various communicative aspects of a GUI, which is used to monitor the condition of assets. It applies to one Swedish vendor and its intentions to design information. In addition, it applies to the interaction between the GUI and its individual visual elements, as well as the communication between the GUI and the users (in four Swedish paper mills).

The research is performed as a single case study. Interviews and observations have been the main methods for data collection. Empirical data is analyzed with methods inferred to semiotics, rhetoric and narratology. Theories in information science and regarding remediation are used to interpret the user interface design.

The key conclusion is that there are no less than five different forms of information, all important when determining the conditions of assets. These information forms include the words, images and shapes in the GUI, the machine components and peripherals equipment, the information that takes form when personnel communicate machine conditions, the personnel’s subjective associations, and the information forms that relate to the personnel's actions and interactions.

Preventive technicians interpret the GUI-information individually and collectively in relation to these information forms, which influence their interpretation and understanding of the GUI information. Social media in the GUI makes it possible to represent essential information that takes form when employees communicate a machine’s condition. Photographs may represent information forms as a machine’s components, peripherals, and local environment change over time. Moreover, preventative technicians may use diagrams and photographs in the GUI to change attitudes among the personnel at the mills and convince them, for example, of a machine’s condition or the effectiveness of CBM as maintenance policy.

Designing cutting tools for the turning industry, providing optimal cutting parameters is of importance for both the client, and for the company's own research. By examining the metal chips that form in the turning process, operators can recommend optimal cutting parameters. Instead of doing manual classification of metal chips that come from the turning process, an automated approach of detecting chips and classification is preferred. This thesis aims to evaluate if such an approach is possible using either a Convolutional Neural Network (CNN) or a CNN feature extraction coupled with machine learning (ML).

The thesis started with a research phase where we reviewed existing state of the art CNNs, image processing and ML algorithms. From the research, we implemented our own object detection algorithm, and we chose to implement two CNNs, AlexNet and VGG16. A third CNN was designed and implemented with our specific task in mind. The three models were tested against each other, both as standalone image classifiers and as a feature extractor coupled with a ML algorithm. Because the chips were inside a machine, different angles and light setup had to be tested to evaluate which setup provided the optimal image for classification.

A top view of the cutting area was found to be the optimal angle with light focused on both below the cutting area, and in the chip disposal tray. The smaller proposed CNN with three convolutional layers, three pooling layers and two dense layers was found to rival both AlexNet and VGG16 in terms of both as a standalone classifier, and as a feature extractor. The proposed model was designed with a limited system in mind and is therefore more suited for those systems while still having a high accuracy. The classification accuracy of the proposed model as a standalone classifier was 92.03%. Compared to the state of the art classifier AlexNet which had an accuracy of 92.20%, and VGG16 which had an accuracy of 91.88%. When used as a feature extractor, all three models paired best with the Random Forest algorithm, but the accuracy between the feature extractors is not that significant. The proposed feature extractor combined with Random Forest had an accuracy of 82.56%, compared to AlexNet with an accuracy of 81.93%, and VGG16 with 79.14% accuracy.

MDU, in collaboration with several other universities, plans to join the World Robotic Sailing Championship (WRSC), where in certain sub-challenges some object detection is necessary. Such as for detecting objects such as boats, buoys, and possibly other items. Utilizing a camera system could significantly aid in these tasks, and in this research, an omnidirectional camera is proposed. This is a camera that offers a wide field of view of 360 degrees and could provide comprehensive information about the boat’s surroundings. However, these images use a spherical camera model, which projects the image on a sphere and, when saved to a 2D format, becomes very distorted. To be able to use state-of-the-art vision algorithms for object detection and tracking, this research proposes to project these images to other formats. As such, four systems using object detection and tracking are made that uses different image representation projected from the spherical images. One system uses spherical images and is used as a baseline, while the three remaining systems use some form of projection. The first is cubemap projection, which projects the spherical image to a cube and unfolds this image on a 2D plane. The two other image representations used perspective projections, which are when the spherical image is projected to small sub-images. The two image representations that used perspective projections had 4 and 8 perspective images. None of the systems ultimately performed very well but did have some advantages and disadvantages.

A conventional bicycle is a stable system given adequate forward velocity. However, the velocity region of stability is limited and depends on the geometric parameters of the bicycle. An autonomous bicycle is just not about maintaining the balance but also controlling where the bicycle is heading. Following paths has been accomplished with bicycles and motorcycles in simulation for a while. Car-like vehicles have followed paths in the real world but few bicycles or motorcycles have done so. The goal of this work is to follow a planned path using a physical bicycle without overcoming the dynamic limitations of the bicycle. Using an iterative design process, controllers for direction and position are developed and improved. Kinematic models are also compared in their ability to simulate the bicycle movement and how controllers in simulation translate to outdoors driving. The result shows that the bicycle can follow a turning path on a residential road without human interaction and that some simulation behaviours do not translate to the real world.

Accurate and robust extrinsic calibration is necessary for deploying autonomous systems which need multiple sensors for perception. In this letter, we present a robust system for real-time extrinsic calibration of multiple lidars in vehicle base framewithout the need for any fiducialmarkers or features. We base our approach on matching absolute GNSS (Global Navigation Satellite System) and estimated lidar poses in real-time. Comparing rotation components allows us to improve the robustness of the solution than traditional least-square approach comparing translation components only. Additionally, instead of comparing all corresponding poses, we select poses comprising maximum mutual information based on our novel observability criteria. This allows us to identify a subset of the poses helpful for real-time calibration. We also provide stopping criteria for ensuring calibration completion. To validate our approach extensive tests were carried out on data collected using Scania test vehicles (7 sequences for a total of approximate to 6.5 Km). The results presented in this letter show that our approach is able to accurately determine the extrinsic calibration for various combinations of sensor setups.

Robotic technology helps humans in different areas such as manufacturing, health care and education. Due to the ubiquitous revolution, today's focus is on mobile robots and their applications in a variety of cyber-physical systems. ROS is a wll-known and powerful middleware that facilitates software development for mobile robots. However, this middleware does not support assuring properties such as timeliness and safety of ROS-based software. In this paper we present an integration of Timed Rebeca modeling language with ROS to synthesize verified robotic software. First, a conceptual model of robotic programs is developed using Timed Rebeca. After verifying a set of user-defined correctness properties on this model, it is translated to a ROS program automatically. Experiments on some small-scale case studies illustrates the applicability of the proposed integration method. 

A reliable embedded real-time system has many requirements to fulfil. It must meet target deadlines in a number of situations, most of them in a situation that puts heavy stress on the system. To meet these demands, numerous tests have been created which test the hardware for any possible errors the developers might think of, in order to maximise system reliability and stability. These tests will take a lot of time to execute, and as system complexity grows, more tests are introduced leading to even longer testing times. In this thesis, a method to reduce the testing time of the software and, to a lesser extent, the hardware is examined. By using the full system simulator Simics, an existing industry system from ABB was integrated and tests were performed. A proof of concept test suite for automatic redundancy tests was also implemented. By looking at the test results, it was concluded that the method shows promise. However, problems with the average latency and performance troubles with Simics shows that more work must be put into this research before the system can be run at full speed.

Over the last decade, data has become a highly valuable resource. Electrical power grids deal with large quantities of data, and continuously collect this for analytical purposes. Anomalies that occur within this data is important to identify since they could cause nonoptimal performance within the substations, or in worse cases damage to the substations themselves. However, large datasets in the order of millions are hard or even impossible to gain a reasonable overview of the data manually. When collecting data from electrical power grids, predefined triggering criteria are often used to indicate that an event has occurred within the specific system. This makes it difficult to search for events that are unknown to the operator of the deployed acquisition system. Clustering, an unsupervised machine learning method, can be utilised for fault prediction within systems generating large amounts of multivariate time-series data without labels and can group data more efficiently and without the bias of a human operator. A large number of clustering techniques exist, as well as methods for extracting information from the data itself, and identification of these was of utmost importance. This thesis work presents a study of the methods involved in the creation of such a clustering system which is suitable for the specific type of data. The objective of the study was to identify methods that enables finding the underlying structures of the data and cluster the data based on these. The signals were split into multiple frequency sub-bands and from these features could be extracted and evaluated. Using suitable combinations of features the data was clustered with two different clustering algorithms, CLARA and CLARANS, and evaluated with established quality analysis methods. The results indicate that CLARA performed overall best on all the tested feature sets. The formed clusters hold valuable information such as indications of unknown events within the system, and if similar events are clustered together this can assist a human operator further to investigate the importance of the clusters themselves. A further conclusion from the results is that research into the use of more optimised clustering algorithms is necessary so that expansion into larger datasets can be considered.

In an eye-tracking pilot-project we have asked 6 children, age of 11, to read an opening,

from a geography book for grade five. The aim of the study was to investigate

the role of illustrations in text-books and to what extent they contribute to the learning

process.

The imaging devices sense light reflected from objects and reconstruct images using the 2D-sensor matrix. It is a 2D Cartesian coordinate system where the depth dimension is absent. The absence of a depth axis on 2D images imposes challenges in locating and tracking objects in a 3D environment. Real-time object tracking faces another challenge imposed by network latency. This paper presents the development and analysis of a real-time, real-world object tracker called Trackez, which is capable of tracking within the top hemisphere. It uses Machine Vision at the IoT Edge (Mez) technology to mitigate latency sensitivity. A novel algorithm, Follow-Satisfy-Loop (FSL), has been developed and implemented in this paper that optimally tracks the target. It does not require the depth-axis. The simple and innovative design and incorporation of Mez technology have made the proposed object tracker a latency-insensitive, Z-axis-independent, and effective system. The Trackez reduces the average latency by 85.08% and improves the average accuracy by 81.71%. The object tracker accurately tracks objects moving in regular and irregular patterns at up to 5.4 speed. This accurate, latency tolerant, and Z-axis independent tracking system contributes to developing a better robotics system that requires object tracking.

This thesis aims to create a system using colour markings for Volvo’s autonomous wheel loaders which determines their restricted area and operation using sensors available on the machine. The wheel loader shall be able to interpret and distinguish different colours of spray paint, and depending on the colour, act accordingly. Six different colours are evaluated across two different colour types to find the most suitable ones for the system. Multiple tests are presented throughout the thesis to find the approach with the most optimal performance that meets the system's requirements. The system is evaluated in various weather conditions to determine how the weather affects the performance of the system. The thesis also compares two different line-following approaches, where one is based on edge detection using Canny Edge and Hough transform, and the other uses histogram analysis and sliding window search, to distinguish and track the colour markings. While the wheel loader is in operation, it collects GPS coordinates to create a map of the path taken by the wheel loader and the location of various tasks. The evaluation shows that red, green and blue in fluorescent colour type are the most suitable colours for such a system. The line-following algorithm that utilises perspective warp, histogram and a sliding window search was the most prominent for accurate line detection and tracking. Furthermore, the evaluation showed that the performance of the system was affected depending on the weather condition. 

Abstract

The overall intention of this research project is to increase knowledge regarding design processes in general and artists' design processes specifically. The research is carried out as a case study based on the sketch processes that took place within Projekt Konstpaus (The Art Break Project). The sketches, or suggested designs, are the main objective for analysis and consideration in this study.

Projekt Konstpaus (The Art Break Project) is a development pro­ject partially financed by the European Union (EU). The vision of the project embodied equality, multiculturalism and sustainable community development. The municipality of Strängnäs, Sweden was the leading partner in the project and provided the necessary support for the project idea, financing and infrastructure. The innovative aim of the project was to have various groups of people from different backgrounds working together in the same processes. The project team consisted of several artists and people with university educations, such as archaeologists, cultural geographers, biologists and geologists. The main objective of the project team was to provide the basis for the construction of a culturally inspired walking and bicycle path. Several rest spots/rest stops (“konstpauser”) designed with artistic cha­racter and influenced primarily by the municipality’s extensive nature/cul­tural heritage will be found along the path (which has been approved for construction). One initial task of the project team was to make an inventory of the nature and culture artifacts within the project area as a means of promoting na­ture/culture preservation for the benefit of future generations through information sharing. The walking/bicycle path will be accessible to all, with special provision for physically challenged individuals. The intention is to provide an environment for both quietude and physical recreation.

The artists within the project embedded their artistic interpretations of the inventory and communicated them by suggesting artistic designs (sketches) related to the planned path. A jury then considered the sketches. Sketch, text, models and jury decisions (regarding the designs) are the objective of this research. The analysis of the material (sketches, texts, models and jury decisions, both oral and written) exposes the artistic processes. It is also the key to understanding the messages the artists intend to convey through their suggested designs. It is important to realize the significance (specific characteristics) of different types of sketches to be able to make decisions based on sketch materials. When sketches are examined, this awareness is central to making the right decision. In this study, three different types of sketches are examined, and the reading of each type is discussed.

When studying the suggested designs, insight was gained regarding the differences between using computer-aided design and traditional sketch tools. Knowledge was also increased concerning the development of sketch techniques generally, and when using computers specifically. A dualism of sight and seeing in terms of the visualization of an idea exists, and it is discussed in the light of empirical examples. It is also placed in relation to important technological steps taken earlier in history. The use of Camera Obcura as a helpful tool for composition is one such step. The use of this tool impacted on how the inner view was changed and, with that, manners of expression as well. This is seen in the composition of paintings and the use of language. Our thinking is influenced by what we see, and that, in turn, influences our thoughts. In our contemporary western paradigm, our commonly-held definition of "seeing" is influenced by computer-gained visual representations and the processes used when producing them. The study confirms that while this particular type of sketch exposes the suggested design idea a bit clearer than traditional sketches, it also reveals errors in the suggested designs. I have also found that both written and spoken language routes the interpretation of sketch material. In terms of understanding how the suggested designs are chosen by a jury, this component (the spoken or written language) was seen in the empirical material revealed in this study. It was also theoretically confirmed.

Together with an overall insight into the artistic processes, this study confirms the possibility of using artists in a development process. In this project, the process was to promote na­ture and culture preservation. It is valuable to integrate diverse areas of knowledge in the same process. This is true in both a social and an environmental sense. Finally, findings in this analysis confirm that artists are able to convey messages through their suggested designs (sketches). Those messages include interpretations of place, space, history and findings related to the project area.

Conventional industrial robots are heavily dependent on hard automation that requires pre-specified fixtures and time-consuming (re)programming performed by experienced operators. In this work, teaching by human-only demonstration is used for reducing required time and expertise to setup a robotized assembly station. This is achieved by the proposed framework enhancing the robotic system with advanced perception and cognitive abilities, accessed through a user-friendly Human Robot Interaction interface. The approach is evaluated on a small parts’ assembly use case deployed onto a collaborative industrial robot testbed. Experiments indicate that the proposed approach allows inexperienced users to efficiently teach robots new assembly tasks.

Deep Neural Networks (DNNs) have proven excellent performance and are very successful in image classification and object detection. Safety critical industries such as the automotive and aerospace industry aim to develop autonomous vehicles with the help of DNNs. In order to certify the usage of DNNs in safety critical systems, it is essential to prove the correctness of data within the system. In this thesis, the research is focused on investigating the sources of uncertainty, what effects various sources of uncertainty has on NNs, and how it is possible to reduce uncertainty within an NN. Probabilistic methods are used to implement an NN with uncertainty estimation to analyze and evaluate how the integrity of the NN is affected. By analyzing and discussing the effects of uncertainty in an NN it is possible to understand the importance of including a method of estimating uncertainty. Preventing, reducing, or removing the presence of uncertainty in such a network improves the correctness of data within the system. With the implementation of the NN, results show that estimating uncertainty makes it possible to identify and classify the presence of uncertainty in the system and reduce the uncertainty to achieve an increased level of integrity, which improves the correctness of the predictions. 

More complex and intricate Computer Vision algorithms combined with higher resolution image streams put bigger and bigger demands on processing power. CPU clock frequencies are now pushing the limits of possible speeds, and have instead started growing in number of cores. Most Computer Vision algorithms' performance respond well to parallel solutions. Dividing the algorithm over 4-8 CPU cores can give a good speed-up, but using chips with Programmable Logic (PL) such as FPGA's can give even more.

An interesting recent addition to the FPGA family is a System on Chip (SoC) that combines a CPU and an FPGA in one chip, such as the Zynq-7000 series from Xilinx. This tight integration between the Programmable Logic and Processing System (PS) opens up for designs where C programs can use the programmable logic to accelerate selected parts of the algorithm, while still behaving like a C program.

On that subject, Xilinx has introduced a new High-Level Synthesis Tool (HLST) called Vivado HLS, which has the power to accelerate C code by synthesizing it to Hardware Description Language (HDL) code. This potentially bridges two otherwise very separate worlds; the ever popular OpenCV library and FPGAs.

This thesis will focus on evaluating Vivado HLS from Xilinx primarily with image processing in mind for potential use on GIMME-2; a system with a Zynq-7020 SoC and two high resolution image sensors, tailored for stereo vision.

Emotion detection, also known as Facial expression recognition, is the art of mapping an emotion to some sort of input data taken from a human. This is a powerful tool to extract valuable information from individuals which can be used as data for many different purposes, ranging from medical conditions such as depression to customer feedback. To be able to solve the problem of facial expression recognition, smaller subtasks are required and all of them together form the complete system to the problem. Breaking down the bigger task at hand, one can think of these smaller subtasks in the form of a pipeline that implements the necessary steps for classification of some input to then give an output in the form of emotion. In recent time with the rise of the art of computer vision, images are often used as input for these systems and have shown great promise to assist in the task of facial expression recognition as the human face conveys the subjects emotional state and contain more information than other inputs, such as text or audio. Many of the current state-of-the-art systems utilize computer vision in combination with another rising field, namely AI, or more specifically deep learning. These proposed methods for deep learning are in many cases using a special form of neural network called convolutional neural network that specializes in extracting information from images. Then performing classification using the SoftMax function, acting as the last part before the output in the facial expression pipeline. This thesis work has explored these methods of utilizing convolutional neural networks to extract information from images and builds upon it by exploring a set of machine learning algorithms that replace the more commonly used SoftMax function as a classifier, in attempts to further increase not only the accuracy but also optimize the use of computational resources. The work also explores different techniques for the face detection subtask in the pipeline by comparing two approaches. One of these approaches is more frequently used in the state-of-the-art and is said to be more viable for possible real-time applications, namely the Viola-Jones algorithm. The other is a deep learning approach using a state-of-the-art convolutional neural network to perform the detection, in many cases speculated to be too computationally intense to run in real-time. By applying a state-of-the-art inspired new developed convolutional neural network together with the SoftMax classifier, the final performance did not reach state-of-the-art accuracy. However, the machine-learning classifiers used shows promise and bypass the SoftMax function in performance in several cases when given a massively smaller number of samples as training. Furthermore, the results given from implementing and testing a pure deep learning approach, using deep learning algorithms for both the detection and classification stages of the pipeline, shows that deep learning might outperform the classic Viola-Jones algorithm in terms of both detection rate and frames per second. 

Fog and haze can have a dramatic impact on vision systems for land and sea vehicles. The impact of such conditions on infrared images is not as severe as for standard images. By fusing images from two cameras, one ordinary and one near-infrared camera, a complete dehazing system with colour preservation can be achieved. Applying several different algorithms to an image set and evaluating the results, the most suitable image fusion algoritm has been identified. Using an FPGA, a programmable integrated circuit, a crucial part of the algorithm has been implemented. It is capable of producing processed images 30 times faster than a laptop computer. This implementation lays the foundation of a real-time dehazing system and provides a significant part of the full solution. The results show that such a system can be accomplished with an FPGA.

One goal of mobile robotic telepresence for social interaction is to design robotic units that are easy to operate for novice users and promote good interaction between people. This paper presents an exploratory study on the effect of camera orientation and field of view on the interaction between a remote and local user. Our findings suggest that limiting the width of the field of view can lead to better interaction quality as it encourages remote users to orient the robot towards local users.

Mobile robotic telepresence (MRP) has been introduced to allow communication from remote locations. Modern MRP systems offer rich capabilities for human-human interactions. However, simply driving a telepresence robot can become a burden especially for novice users, leaving no room for interaction at all. In this video we introduce a project which aims to incorporate advanced robotic algorithms into manned telepresence robots in a natural way to allow human-robot cooperation for safe driving. It also shows a very first implementation of cooperative driving based on extracting a safe drivable area in real time using the image stream received from the robot.

In the field of construction equipment, a future is envisioned in which humans and autonomous machines can collaborate seamlessly. An example of this vision is embodied in the Volvo prototype LX03, an autonomous wheel loader engineered to function as a smart and safe partner with collaborative capabilities. In these situations, it is crucial that humans and machines communicate effectively. One critical aspect for machines to consider is the awareness level of humans, as it significantly influences their decision-making processes. This thesis investigates the feasibility of constructing a deep learning model to classify if a human is aware towards the machine or not using computer vision from the machines Point of View. To test this, a state-of-the-art action recognition model was used, namely RGBPose-Conv3D which is a 3D Convolutional Neural Network. This model uses two modalities, namely RGB and Pose, which could be used together or separately. The model was modified and trained to classify aware and unaware behaviour. The dataset used to train and test the model was collected with actors that mimicked aware or unaware behaviour. When using only RGB the model did not perform well, but when using Pose only or Pose and RGB fused, the model performed well in classifying the awareness state. Furthermore, the model exhibited good generalisability to scenarios on which it had not previously been trained. Such as with a machine movement, multiple people or previously not seen scenarios. The thesis highlights the viability of employing deep learning and computer vision for awareness detection, showcasing a novel method that achieves high accuracy despite minimal comparative research.

Differential evolution (DE) represents a class of population-based optimization techniques that uses differences of vectors to search for optimal solutions in the search space. However, promising solutions/ regions are not adequately exploited by a traditional DE algorithm. Memetic computing has been popular in recent years to enhance the exploitation of global algorithms via incorporation of local search. This paper proposes a new memetic framework to enhance DE algorithms using Alopex Local Search (MFDEALS). The novelty of the proposed MFDEALS framework lies in that the behavior of exploitation (by Alopex local search) can be controlled based on the DE global exploration status (population diversity and search stage). Additionally, an adaptive parameter inside the Alopex local search enables smooth transition of its behavior from exploratory to exploitative during the search process. A study of the important components of MFDEALS shows that there is a synergy between them. MFDEALS has been integrated with both the canonical DE method and the adaptive DE algorithm L-SHADE, leading to the MDEALS and ML-SHADEALS algorithms, respectively. Both algorithms were tested on the benchmark functions from the IEEE CEC'2014 Conference. The experiment results show that Memetic Differential Evolution with Alopex Local Search (MDEALS) not only improves the original DE algorithm but also outperforms other memetic DE algorithms by obtaining better quality solutions. Further, the comparison between ML-SHADEALS and L-SHADE demonstrates that applying the MFDEALS framework with Alopex local search can significantly enhance the performance of L-SHADE. 

Enhanced vision and object detection could be useful in the aviation domain in situations of bad weather or cluttered environments. In particular, enhanced vision and object detection could improve situational awareness and aid the pilot in environment interpretation and detection of hazardous objects. The fundamental concept of object detection is to interpret what objects are present in an image with the aid of a prediction model or other feature extraction techniques. Constructing a comprehensive data set that can describe the operational environment and be robust for weather and lighting conditions is vital if the object detector is to be utilised in the avionics domain. Evaluating the accuracy and robustness of the constructed data set is crucial. Since erroneous detection, referring to the object detection algorithm failing to detect a potentially hazardous object or falsely detecting an object, is a major safety issue. Bayesian uncertainty estimations are evaluated to examine if they can be utilised to detect miss-classifications, enabling the use of a Bayesian Neural Network with the object detector to identify an erroneous detection. The object detector Faster RCNN with ResNet-50-FPN was utilised using the development framework Detectron2; the accuracy of the object detection algorithm was evaluated based on obtained MS-COCO metrics. The setup achieved a 50.327 % AP@[IoU=.5:.95] score. With an 18.1 % decrease when exposed to weather and lighting conditions. By inducing artificial artefacts and augmentations of luminance, motion, and weather to the images of the training set, the AP@[IoU=.5:.95] score increased by 15.6 %. The inducement improved the robustness necessary to maintain the accuracy when exposed to variations of environmental conditions, which resulted in just a 2.6 % decrease from the initial accuracy. To fully conclude that the augmentations provide the necessary robustness for variations in environmental conditions, the model needs to be subjected to actual image representations of the operational environment with different weather and lighting phenomena. Bayesian uncertainty estimations show great promise in providing additional information to interpret objects in the operational environment correctly. Further research is needed to conclude if uncertainty estimations can provide necessary information to detect erroneous predictions.

In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. In the realm of safety-critical machine learning the used datasets need to reflect the environment in which the system is intended to operate, in order to minimize the generalization gap between trained and real-world inputs. Datasets should be thoroughly prepared and requirements on the properties and characteristics of the collected data need to be specified. In our work we present a case study in which generating a synthetic dataset is accomplished based on real-world flight data from the ADS-B system, containing thousands of approaches to several airports to identify real-world statistical distributions of relevant variables to vary within our dataset sampling space. We also investigate what the effects are of training a model on synthetic data to different extents, including training on translated image sets (using domain adaptation). Our results indicate airport location to be the most critical parameter to vary. We also conclude that all experiments did benefit in performance from pre-training on synthetic data rather than using only real data, however this did not hold true in general for domain adaptation-translated images.

Despite the growing popularity of swarm-based applications of drones, there is still a lack of approaches to maximize the safety of swarms of drones by minimizing the risks of drone collisions. In this paper, we present an approach that uses optimization, learning, and automatic immediate responses (reflexes) of drones to ensure safe operations of swarms of drones. The proposed approach integrates a high-performance dynamic evolutionary algorithm and a reinforcement learning algorithm to generate safe and efficient drone routes and then augments the generated routes with dynamically computed drone reflexes to prevent collisions with unforeseen obstacles in the flying zone. We also present a parallel implementation of the proposed approach and evaluate it against two benchmarks. The results show that the proposed approach maximizes safety and generates highly efficient drone routes.

As the Automotive industry is heavily regulated from a quality point of view, excellence in pro-duction is obligatory. Due to the fact that removing human error from humans is impossible, new solutions must be found. The transition to more data driven production strategies enables the implantation of automated vision systems for replacing humans in simple classification tasks. As research in the field of artificial intelligence advances, the hardware required to run the algorithms decreases. Concurrently small computing platforms break new performance records and the innovation space converges. This work harnesses state-of-the-art from both domains by implementing a plug-on vision system, driven by a resource-constrained edge device in a production line. The implemented CNN-model based on the MobileNetV2 architecture achieved 97.80, 99.93, and 95.67% in accuracy, precision, and recall respectively. The model was trained using only 100 physical samples, which were expanded by a ratio of 1:15 through innovative real world and digital augmentations. The core of the vision system was a commodity device, the Raspberry Pi 4. The solution fulfilled all the requirements while sparking new development ideas for future work.  

Differential evolution (DE) is widely used for global optimisation problems due to its simplicity and efficiency. L-SHADE is a state-of-the-art variant of DE algorithm that incorporates external archive, success-history-based parameter adaptation, and linear population size reduction. L-SHADE uses a current-to-pbest/1/bin strategy for mutation operator, while all individuals have the same probability to be selected. In this paper, we propose a novel L-SHADE algorithm, RWS-L-SHADE, based on a roulette wheel selection strategy so that better individuals have a higher priority and worse individuals are less likely to be selected. Our extensive experiments on the CEC-2017 benchmark functions and dimensionalities of 30, 50 and 100 indicate that RWS-L-SHADE outperforms L-SHADE. 

For human-robot interaction with the intent to grip a human arm, it is necessary that the ideal gripping location can be identified. In this work, the gripping location is situated on the arm and thus it can be extracted using the position of the wrist and elbow joints. To achieve this human pose estimation is proposed as there exist robust methods that work both in and outside of lab environments. One such example is OpenPose which thanks to the COCO and MPII datasets has recorded impressive results in a variety of different scenarios in real-time. However, most of the images in these datasets are taken from a camera mounted at chest height on people that for the majority of the images are oriented upright. This presents the potential problem that prone humans which are the primary focus of this project can not be detected. Especially if seen from an angle that makes the human appear upside down in the camera frame. To remedy this two different approaches were tested, both aimed at creating a rotation-invariant 2D pose estimation method. The first method rotates the COCO training data in an attempt to create a model that can find humans regardless of orientation in the image. The second approach adds a RotationNet as a preprocessing step to correctly orient the images so that OpenPose can be used to estimate the 2D pose before rotating back the resulting skeletons.

Gimbal mounted camera systems are used for capturing high quality, long-range video from inherently unstable mounts, such as helicopters. The rotational stability of such systems can be veried by mounting the system to a shake machine and using a laser-collimator to perform measurements. This process is, however, both intricate and time consuming, as well as susceptible to measuring errors. The purpose of this thesis, is to investigate the possibility to perform camera rotation measurements in video recordings, by means of image analysis, while maintaining an acceptable level of measurement accuracy.

The method employed is an adaptation of existing target tracking techniques; using a mirror to record an adhesive circular marker attached to the front of the camera, in order to lter out any non-rotational marker movement. Three dierent algorithms for tracking the marker are evaluated; weighted centroid, dual conic ellipsis estimation and DFT registration. Evaluation is performed using a wide range of synthetic videos, simulating the primary error sources. Furthermore, the emergent problem of ltering out gyroscopic camera drift from the measurements is addressed. Finally, a software tool is presented that can be used for testing the rotational stability of the gimbal systems in a controlled environment.

This thesis aims to solve the mathematical inverse problem of characterizing sea waves based on the responses obtained from a marine vessel sailing under certain sea conditions. By researching this problem the thesis contributes to the marine industry by improving products that are using ocean behavior for controlling ship's dynamics. Knowledge about the current state of the sea, such as the wave frequency and height, is important for navigation, control, and for the safety of a vessel. This information can be retrieved from specialized weather reports. However, such information is not at all time possible to obtain during a voyage, and if so usually comes with a certain delay. Therefore this thesis seeks solutions that can estimate on-line the waves' state using methods in the field of Artificial Intelligence. The specific investigation methods are Transfer Functions augmented with Genetic Algorithm, Artificial Neural Networks and Case-Based Reasoning. These methods have been configured and validated using the n-fold cross validation method. All the methods have been tested with an actual implementation. The algorithms have been trained with data acquired from a marine simulation program developed in Simulink. The methods have also been trained and tested using monitored data acquired from an actual ship sailing on the Baltic Sea as well as wave data obtained from a buoy located nearby the vessel's route. The proposed methods have been compared with state-of-the art reports in order evaluate the novelty of the research and its potential applications in industry. The results in this thesis show that the proposed methods can in fact be used for solving the inverse problem. It was also found that among the investigated methods it is the Transfer Function augmented with Genetic Algorithm which yields best results. This Master Thesis is conducted under the Master of Engineering Program in Robotics at Mälardalens högskola in Västerås, Sweden. The thesis was proposed by Q-TAGG R&D AB in Västerås, Sweden, a company which specializes in marine vessel dynamics research.

Caregivers are dissuaded from using deception in dementia care in Swedish policies. However, guidelines in other countries have a more positive approach to deceptive practices when these are used in the best interest of patients. Research also shows that lies and deception are widely used in dementia care. Based on interviews with caregivers and ethnographic visits to nursing homes in Sweden, this paper examines the use of technology in deceptive practices in the care of people with dementia. Three technological devices are in focus: the online patient journal, the security camera, and the robotic animal. The journal allows for a smooth transition of knowledge between caregivers, enabling a person-centered care. However, the patient is often unaware of this circulation of knowledge by which the patient becomes known to everyone without necessarily knowing anyone. The camera makes it possible for caregivers to watch over patients without having to enter their rooms, which means that they do not have to disturb them with unnecessary and inconvenient visits. Paradoxically, the technology thus breaches the patients’ integrity in order to secure it. Last, the robotic animal works particularly well when it is perceived to be a real animal; yet, both policies and previous research caution against using a robot to create an illusion of a living being. The paper argues that to handle these dilemmas, the devices need to be understood in context: deception is not built into healthcare technology but is instead generated in the relationship between caregiver, patient, and technological device.

In the rapidly advancing field of autonomous vehicles, ensuring safety and functionality is of paramount importance. The number of functionalities in autonomous vehicles are increasing everyday, advancing the technology to newer heights. These systems have become ever reliant on the perception of environment and the use of sophisticated sensors, to navigate and interact with their environment. However, this need for situational awareness raises new safety concerns that call for reevaluation of conventional methods. Despite the system being free from any malfunctions, it still might exhibit a hazardous behaviour due to functional insufficiencies or unforeseeable misuse, also referred to as the Safety of the Intended Functionality (SOTIF). This paper applies the STPA method, a novel safety analysis tool based on system theory, as pilot study to gain insights into the effectiveness of the method in addressing these emerging safety concerns. The method is applied to the case of an unsignaled 4-leg intersection with mixed traffic, where an autonomous level 4 vehicle is navigating a left turn. The analysis is narrowed to focus on functional insufficiencies only, with regards to perception in particular, for which corresponding causal factors are generated by the method. The results of the study prove the method to be an excellent tool in systematically identifying factors stemming from both functional insufficiencies and specification gaps, even within complex and challenging settings.

Long-term pedestrian trajectory prediction in crowds is highly valuable for safety driving and social robot navigation. The recent research of trajectory prediction usually focuses on solving the problems of modeling social interactions, physical constraints and multi-modality of futures without considering the generalization of prediction models to other scenes and objects, which is critical for real-world applications. In this paper, we propose a general framework that makes trajectory prediction models able to transfer well across unseen scenes and objects by quickly learning the prior information of trajectories. The trajectory sequences are closely related to the circumstance setting (e.g. exits, roads, buildings, entries etc.) and the objects (e.g. pedestrians, bicycles, vehicles etc.). We argue that those trajectory information varying across scenes and objects makes a trained prediction model not perform well over unseen target data. To address it, we introduce MetaTraj that contains carefully designed sub-tasks and meta-tasks to learn prior information of trajectories related to scenes and objects, which then contributes to accurate long-term future prediction. Both sub-tasks and meta-tasks are generated from trajectory sequences effortlessly and can be easily integrated into many prediction models. Extensive experiments over several trajectory prediction benchmarks demonstrate that MetaTraj can be applied to multiple prediction models and enables them generalize well to unseen scenes and objects.

Safe airspace of airports worldwide is crucial to ensure that passengers, workers, and airplanes are safe from external threats, whether malicious or not. In recent years, several airports worldwide experienced intrusions into their airspace by unmanned aerial vehicles. Based on this observation, there is a need for a reliable detection system capable of detecting unmanned aerial vehicles with high accuracy and integrity. This thesis proposes time redundant triplex parallel diverse convolutional neural network architectures trained to detect unmanned aerial vehicles to address the aforementioned issue. The thesis aims at producing a system capable of real-time performance coupled with previously mentioned networks. The hypothesis in this method will result in lower mispredictions of objects other than drones and high accuracy compared to singular convolutional neural networks. Several improvements to accuracy, lower mispredictions, and faster detection times were observed during the performed experiments with the proposed system. Furthermore, a new way of interpreting the intersection over union results for all neural networks is introduced to ensure the correctness and reliability of results. Lastly, the system produced by this thesis is analyzed from a dependability viewpoint to provide an overview of how this contributes to dependability research.

As the field of Artificial Intelligence continues to evolve into a tool of societal impact, a need of breaking its initial boundaries as a computer science discipline arises to also include different humanistic fields. The work presented in this thesis revolves around the role that explainable artificial intelligence has in human-robot interaction through the study of poetry generators. To better understand the scope of the project, a poetry generators study presents the steps involved in the development process and the evaluation methods. In the algorithmic development of poetry generators, the shift from traditional disciplines to transdisciplinarity is identified. In collaboration with researchers from the Research Institutes of Sweden, state-of-the-art generators are tested to showcase the power of artificially enhanced artifacts. A development plateau is discovered and with the inclusion of Design Thinking methods potential future human-robot interaction development is identified. A physical prototype capable of verbal interaction on top of a poetry generator is created with the new feature of changing the corpora to any given audio input. Lastly, the strengths of transdisciplinarity are connected with the open-sourced community in regards to creativity and self-expression, producing an online tool to address future work improvements and introduce nonexperts to the steps required to self-build an intelligent robotic companion, thus also encouraging public technological literacy. Explainable AI is shown to help with user involvement in the process of creation, alteration and deployment of AI enhanced applications.

Implementing object detection methods for runway detection during landing approaches is limited in the safety-critical aircraft domain. This limitation is due to the difficulty that comes with verification of the design and the ability to understand how the object detection behaves during operation. During operation, object detection needs to consider the aircraft's position, environmental factors, different runways and aircraft attitudes. Training such an object detection model requires a comprehensive dataset that defines the features mentioned above. The feature's impact on the detection capabilities needs to be analysed to ensure the correct distribution of images in the dataset. Gathering images for these scenarios would be costly and needed due to the aviation industry's safety standards. Synthetic data can be used to limit the cost and time required to create a dataset where all features occur. By using synthesised data in the form of generating datasets in a simulated environment, these features could be applied to the dataset directly. The features could also be implemented separately in different datasets and compared to each other to analyse their impact on the object detections capabilities. By utilising this method for the features mentioned above, the following results could be determined. For object detection to consider most landing cases and different runways, the dataset needs to replicate real flight data and generate additional extreme landing cases. The dataset also needs to consider landings at different altitudes, which can differ at a different airport. Environmental conditions such as clouds and time of day reduce detection capabilities far from the runway, while attitude and runway appearance reduce it at close range. Runway appearance did also affect the runway at long ranges but only for darker runways.

The feasibility of deploying object detection to interpret the environment is questioned in several mission-critical applications leading to raised concerns about the ability of object detectors in providing reliable and safe predictions of the operational environment, regardless of weather and light conditions. The lack of a comprehensive dataset, which causes class imbalance and detection difficulties of hard examples, is one of the main reasons of accuracy loss in attitude safe object detection. Data augmentation, as an implicit regularisation technique, has been shown to significantly improve object detection by increasing both the diversity and the size of the training dataset. Despite the success of data augmentation in various computer vision tasks, applying data augmentation techniques to improve safety has not been sufficiently addressed in the literature. In this paper, we leverage a set of data augmentation techniques to improve the safety of object detection. The aircraft in-flight image data is used to evaluate the feasibility of our proposed solution in real-world safety-required scenarios. To achieve our goal, we first generate a training dataset by synthesising the images collected from in-flight recordings. Next, we augment the generated dataset to cover real weather and lighting changes. Introduction of artificially produced distortions is also known as corruptions and has since recently been an approach to enrich the dataset. The introduction of corruptions, as augmentations of weather and luminance in combination with the introduction of artificial artefacts, is done as an approach to achieve a comprehensive representation of an aircraft’s operational environment. Finally, we evaluate the impact of data augmentation on the studied dataset. Faster R-CNN with ResNet-50-FPN was used as an object detector for the experiments. An AP@[IoU=.5:.95] score of 50.327% was achieved with the initial setup, while exposure to altered weather and lighting conditions yielded an 18.1% decrease. The introduction of the conditions into the training set led to a 15.6% increase in comparison to the score achieved from exposure to the conditions. 

In unmanned aerial vehicle photographs, object detection algorithms encounter challenges in enhancing both speed and accuracy for objects of different sizes, primarily due to complex backgrounds and small objects. This study introduces the PDWT-YOLO algorithm, based on the YOLOv7-tiny model, to improve the effectiveness of object detection across all sizes. The proposed method enhances the detection of small objects by incorporating a dedicated small-object detection layer, while reducing the conflict between classification and regression tasks through the replacement of the YOLOv7-tiny model’s detection head (IDetect) with a decoupled head. Moreover, network convergence is accelerated, and regression accuracy is improved by replacing the Complete Intersection over Union (CIoU) loss function with a Wise Intersection over Union (WIoU) focusing mechanism in the loss function. To assess the proposed model’s effectiveness, it was trained and tested on the VisDrone-2019 dataset comprising images captured by various drones across diverse scenarios, weather conditions, and lighting conditions. The experiments show that mAP@0.5:0.95 and mAP@0.5 increased by 5% and 6.7%, respectively, with acceptable running speed compared with the original YOLOv7-tiny model. Furthermore, this method shows improvement over other datasets, confirming that PDWT-YOLO is effective for multiscale object detection. 

In this thesis, the development of a stereo camera system for an intelligent tool is presented. The task of the system is to identify and localise objects so that the tool can guide a robot. Different approaches to object detection have been implemented and evaluated and the systems ability to localise objects has been tested. The results show that the system can achieve a localisation accuracy below 5 mm.

Superpixel segmentation is a preprocessing step for computer vision applications, where an image is split into segments referred to as superpixels. Then running the main algorithm on these superpixels reduces the number of data points processed in comparison to running the algorithm on pixels directly, while still keeping much of the same information. In this thesis, the possibility to run superpixel segmentation on an FPGA is researched. This has resulted in the development of a modified version of the algorithm SLIC, Simple Linear Iterative Clustering. An FPGA implementation of this algorithm has then been built in VHDL, it is designed as a pipeline, unrolling the iterations of SLIC. The designed algorithm shows a lot of potential and runs on real hardware, but more work is required to make the implementation more robust, and remove some visual artefacts.