https://www.mdu.se/

mdu.sePublications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Artificial Intelligence for Non-Contact-Based Driver Health Monitoring
Mälardalen University, School of Innovation, Design and Engineering, Embedded Systems.ORCID iD: 0000-0002-1547-4386
2021 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In clinical situations, a patient’s physical state is often monitored by sensors attached to the patient, and medical staff are alerted if the patient’s status changes in an undesirable or life-threatening direction. However, in unsupervised situations, such as when driving a vehicle, connecting sensors to the driver is often troublesome and wired sensors may not produce sufficient quality due to factors such as movement and electrical disturbance. Using a camera as a non-contact sensor to extract physiological parameters based on video images offers a new paradigm for monitoring a driver’s health and mental state. Due to the advanced technical features in modern vehicles, driving is now faster, safer and more comfortable than before. To enhance transport security (i.e. to avoid unexpected traffic accidents), it is necessary to consider a vehicle driver as a part of the traffic environment and thus monitor the driver’s health and mental state. Such a monitoring system is commonly developed based on two approaches: driving behaviour-based and physiological parameters-based.

This research work demonstrates a non-contact approach that classifies a driver’s cognitive load based on physiological parameters through a camera system and vehicular data collected from control area networks considering image processing, computer vision, machine learning (ML) and deep learning (DL). In this research, a camera is used as a non-contact sensor and pervasive approach for measuring and monitoring the physiological parameters. The contribution of this research study is four-fold: 1) Feature extraction approach to extract physiological parameters (i.e. heart rate [HR], respiration rate [RR], inter-beat interval [IBI], heart rate variability [HRV] and oxygen saturation [SpO2]) using a camera system in several challenging conditions (i.e. illumination, motion, vibration and movement); 2) Feature extraction based on eye-movement parameters (i.e. saccade and fixation); 3) Identification of key vehicular parameters and extraction of useful features from lateral speed (SP), steering wheel angle (SWA), steering wheel reversal rate (SWRR), steering wheel torque (SWT), yaw rate (YR), lanex (LAN) and lateral position (LP); 4) Investigation of ML and DL algorithms for a driver’s cognitive load classification. Here, ML algorithms (i.e. logistic regression [LR], linear discriminant analysis [LDA], support vector machine [SVM], neural networks [NN], k-nearest neighbours [k-NN], decision tree [DT]) and DL algorithms (i.e. convolutional neural networks [CNN], long short-term memory [LSTM] networks and autoencoders [AE]) are used. 

One of the major contributions of this research work is that physiological parameters were extracted using a camera. According to the results, feature extraction based on physiological parameters using a camera achieved the highest correlation coefficient of .96 for both HR and SpO2 compared to a reference system. The Bland Altman plots showed 95% agreement considering the correlation between the camera and the reference wired sensors. For IBI, the achieved quality index was 97.5% considering a 100 ms R-peak error. The correlation coefficients for 13 eye-movement features between non-contact approach and reference eye-tracking system ranged from .82 to .95.

For cognitive load classification using both the physiological and vehicular parameters, two separate studies were conducted: Study 1 with the 1-back task and Study 2 with the 2-back task. Finally, the highest average accuracy achieved in terms of cognitive load classification was 94% for Study 1 and 82% for Study 2 using LR algorithms considering the HRV parameter. The highest average classification accuracy of cognitive load was 92% using SVM considering saccade and fixation parameters. In both cases, k-fold cross-validation was used for the validation, where the value of k was 10. The classification accuracies using CNN, LSTM and autoencoder were 91%, 90%, and 90.3%, respectively. 

This research study shows such a non-contact-based approach using ML, DL, image processing and computer vision is suitable for monitoring a driver’s cognitive state.

Place, publisher, year, edition, pages
Västerås: Mälardalen University , 2021.
Series
Mälardalen University Press Dissertations, ISSN 1651-4238 ; 330
Keywords [en]
Driver Monitoring, Artificial Intelligence, Machine Learning, Deep Learning
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:mdh:diva-53529ISBN: 978-91-7485-499-2 (print)OAI: oai:DiVA.org:mdh-53529DiVA, id: diva2:1531221
Public defence
2021-04-07, Delta + digitalt via Zoom, Mälardalens högskola, Västerås, 13:15 (English)
Opponent
Supervisors
Available from: 2021-02-26 Created: 2021-02-25 Last updated: 2021-04-13Bibliographically approved
List of papers
1. Intelligent Driver Monitoring Based on Physiological Sensor Signals: Application Using Camera
Open this publication in new window or tab >>Intelligent Driver Monitoring Based on Physiological Sensor Signals: Application Using Camera
2015 (English)In: IEEE 18th International Conference on Intelligent Transportation Systems ITSC2015, Canary Islands, Spain, 2015, p. 2637-2642Conference paper, Published paper (Refereed)
Abstract [en]

Recently, there has been increasing interest in low-cost, non-contact and pervasive methods for monitoring physiological information for the drivers. For the intelligent driver monitoring system there has been so many approaches like facial expression based method, driving behavior based method and physiological parameters based method. Physiological parameters such as, heart rate (HR), heart rate variability (HRV), respiration rate (RR) etc. are mainly used to monitor physical and mental state. Also, in recent decades, there has been increasing interest in low-cost, non-contact and pervasive methods for measuring physiological information. Monitoring physiological parameters based on camera images is such kind of expected methods that could offer a new paradigm for driver’s health monitoring. In this paper, we review the latest developments in using camera images for non-contact physiological parameters that provides a resource for researchers and developers working in the area.

Place, publisher, year, edition, pages
Canary Islands, Spain: , 2015
Keywords
Driver monitoringPhysiological signals Camera
National Category
Engineering and Technology Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:mdh:diva-29234 (URN)10.1109/ITSC.2015.424 (DOI)000376668802114 ()2-s2.0-84950298205 (Scopus ID)
Conference
IEEE 18th International Conference on Intelligent Transportation Systems ITSC2015, 15-18 Sep 2015, Canary Islands, Spain
Projects
SafeDriver: A Real Time Driver's State Monitoring and Prediction System
Available from: 2015-10-06 Created: 2015-09-29 Last updated: 2021-02-25Bibliographically approved
2. Non-Contact Physiological Parameters Extraction Using Camera
Open this publication in new window or tab >>Non-Contact Physiological Parameters Extraction Using Camera
2016 (English)In: Internet of Things. IoT Infrastructures: Second International Summit, IoT 360° 2015 Rome, Italy, October 27–29, 2015. Revised Selected Papers, Part I, 2016, Vol. 169, p. 448-453Conference paper, Published paper (Refereed)
Abstract [en]

Physiological parameters such as Heart Rate (HR), Beat-to-Beat Interval (IBI) and Respiration Rate (RR) are vital indicators of people’s physiological state and important to monitor. However, most of the measurements methods are connection based, i.e. sensors are connected to the body which is often complicated and requires personal assistance. This paper proposed a simple, low-cost and non-contact approach for measuring multiple physiological parameters using a web camera in real time. Here, the heart rate and respiration rate are obtained through facial skin colour variation caused by body blood circulation. Three different signal processing methods such as Fast Fourier Transform (FFT), independent component analysis (ICA) and Principal component analysis (PCA) have been applied on the colour channels in video recordings and the blood volume pulse (BVP) is extracted from the facial regions. HR, IBI and RR are subsequently quantified and compared to corresponding reference measurements. High degrees of agreement are achieved between the measurements across all physiological parameters. This technology has significant potential for advancing personal health care and telemedicine. 

Series
Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, ISSN 1867-8211 ; 169
Keywords
Autonomous Car Driver Monitoring Physiological signals Camera Non contact
National Category
Other Engineering and Technologies
Research subject
Computer Science
Identifiers
urn:nbn:se:mdh:diva-30021 (URN)10.1007/978-3-319-47063-4_47 (DOI)000398616500047 ()2-s2.0-85000783926 (Scopus ID)978-331947062-7 (ISBN)
Conference
Second International Summit, IoT 360° 2015 Rome, Italy, October 27–29, 2015. 1st Workshop on Embedded Sensor Systems for Health.
Projects
ESS-H - Embedded Sensor Systems for Health Research ProfileVDM - Vehicle Driver MonitoringSafeDriver: A Real Time Driver's State Monitoring and Prediction System
Available from: 2015-12-20 Created: 2015-12-18 Last updated: 2021-02-25Bibliographically approved
3. Non-contact heart rate monitoring using lab color space
Open this publication in new window or tab >>Non-contact heart rate monitoring using lab color space
2016 (English)In: Studies in Health Technology and Informatics, 2016, Vol. 224, p. 46-53Conference paper, Published paper (Refereed)
Abstract [en]

Research progressing during the last decade focuses more on non-contact based systems to monitor Heart Rate (HR) which are simple, low-cost and comfortable to use. Most of the non-contact based systems are using RGB videos which is suitable for lab environment. However, it needs to progress considerably before they can be applied in real life applications. As luminance (light) has significance contribution on RGB videos HR monitoring using RGB videos are not efficient enough in real life applications in outdoor environment. This paper presents a HR monitoring method using Lab color facial video captured by a webcam of a laptop computer. Lab color space is device independent and HR can be extracted through facial skin color variation caused by blood circulation considering variable environmental light. Here, three different signal processing methods i.e., Fast Fourier Transform (FFT), Independent Component Analysis (ICA) and Principal Component Analysis (PCA) have been applied on the color channels in video recordings and blood volume pulse (BVP) has been extracted from the facial regions. In this study, HR is subsequently quantified and compare with a reference measurement. The result shows that high degrees of accuracy have been achieved compared to the reference measurements. Thus, this technology has significant potential for advancing personal health care, telemedicine and many real life applications such as driver monitoring.

Keywords
Heart rate, Lab color space, Signal processing, blood volume, circulation, driver, face, Fourier transformation, human, independent component analysis, luminance, monitoring, principal component analysis, quantitative study, skin color, telemedicine, videorecording
National Category
Medical Engineering
Identifiers
urn:nbn:se:mdh:diva-32175 (URN)10.3233/978-1-61499-653-8-46 (DOI)000385238500008 ()2-s2.0-84973483708 (Scopus ID)9781614996521 (ISBN)
Conference
13th International Conference on Wearable Micro and Nano Technologies for Personalised Health, pHealth 2016; Heraklion, Crete; Greece; 29 May 2016 through 31 May 2016; Code 121852
Available from: 2016-06-23 Created: 2016-06-23 Last updated: 2021-02-25Bibliographically approved
4. Quality index analysis on camera- A sed R-eak identification considering movements and light illumination
Open this publication in new window or tab >>Quality index analysis on camera- A sed R-eak identification considering movements and light illumination
2018 (English)In: Studies in Health Technology and Informatics, vol 249, IOS Press , 2018, p. 84-92Conference paper, Published paper (Refereed)
Abstract [en]

This paper presents a quality index (QI) analysis on R-peak extracted by a camera system considering movements and light illumination. Here, the proposed camera system is compared with a reference system named Shimmer PPG sensor. The study considers five test subjects with a 15 minutes measurement protocol, where the protocol consists of several conditions. The conditions are: Normal sittings, head movements i.e., up/down/left/right/forward/backword, with light on/off and with moving flash on/off. A percentage of corrected R-peaks are calculated based on time difference in milliseconds (MS) between the R-peaks extracted both from camera-based and sensor-based systems. A comparison results between normal, movements, and lighting condition is presented as individual and group wise. Furthermore, the comparison is extended considering gender and origin of the subjects. According to the results, more than 90% R-peaks are correctly identified by the camera system with ±200 MS time differences, however, it decreases with while there is no light than when it is on. At the same time, the camera system shows more 95% accuracy for European than Asian men. 

Place, publisher, year, edition, pages
IOS Press, 2018
National Category
Medical Engineering
Identifiers
urn:nbn:se:mdh:diva-40196 (URN)10.3233/978-1-61499-868-6-84 (DOI)000492875900009 ()2-s2.0-85049018248 (Scopus ID)9781614998679 (ISBN)
Conference
15th International Conference on Wearable Micro and Nano Technologies for Personalized Health, pHealth 2018; Gjovik; Norway; 12 June 2018 through 14 June 2018
Available from: 2018-07-05 Created: 2018-07-05 Last updated: 2021-02-25Bibliographically approved
5. Non-Contact Physiological Parameters Extraction Using Facial Video Considering Illumination, Motion, Movement and Vibration
Open this publication in new window or tab >>Non-Contact Physiological Parameters Extraction Using Facial Video Considering Illumination, Motion, Movement and Vibration
2020 (English)In: IEEE Transactions on Biomedical Engineering, ISSN 0018-9294, E-ISSN 1558-2531, Vol. 67, no 1, p. 88-98, article id 8715455Article in journal (Refereed) Published
Abstract [en]

Objective: In this paper, four physiological parameters, i.e., heart rate (HR), inter-beat-interval (IBI), heart rate variability (HRV), and oxygen saturation (SpO2), are extracted from facial video recordings. Methods: Facial videos were recorded for 10 min each in 30 test subjects while driving a simulator. Four regions of interest (ROIs) are automatically selected in each facial image frame based on 66 facial landmarks. Red-green-blue color signals are extracted from the ROIs and four physiological parameters are extracted from the color signals. For the evaluation, physiological parameters are also recorded simultaneously using a traditional sensor 'cStress,' which is attached to hands and fingers of test subjects. Results: The Bland Altman plots show 95% agreement between the camera system and 'cStress' with the highest correlation coefficient R = 0.96 for both HR and SpO2. The quality index is estimated for IBI considering 100 ms R-peak error; the accumulated percentage achieved is 97.5%. HRV features in both time and frequency domains are compared and the highest correlation coefficient achieved is 0.93. One-way analysis of variance test shows that there are no statistically significant differences between the measurements by camera and reference sensors. Conclusion: These results present high degrees of accuracy of HR, IBI, HRV, and SpO2 extraction from facial image sequences. Significance: The proposed non-contact approach could broaden the dimensionality of physiological parameters extraction using cameras. This proposed method could be applied for driver monitoring application under realistic conditions, i.e., illumination, motion, movement, and vibration.

Place, publisher, year, edition, pages
IEEE Computer Society, 2020
Keywords
Ambient illumination, driver monitoring, motion, movement, non-contact, physiological parameters, vibration, Cameras, Extraction, Heart, Video recording, Physiological models
National Category
Control Engineering
Identifiers
urn:nbn:se:mdh:diva-46689 (URN)10.1109/TBME.2019.2908349 (DOI)000505526300009 ()2-s2.0-85077175941 (Scopus ID)
Available from: 2020-01-09 Created: 2020-01-09 Last updated: 2021-02-25Bibliographically approved
6. Non-contact-based driver's cognitive load classification using physiological and vehicular parameters
Open this publication in new window or tab >>Non-contact-based driver's cognitive load classification using physiological and vehicular parameters
2020 (English)In: Biomedical Signal Processing and Control, ISSN 1746-8094, E-ISSN 1746-8108, Vol. 55, article id 101634Article in journal (Refereed) Published
Abstract [en]

Classification of cognitive load for vehicular drivers is a complex task due to underlying challenges of the dynamic driving environment. Many previous works have shown that physiological sensor signals or vehicular data could be a reliable source to quantify cognitive load. However, in driving situations, one of the biggest challenges is to use a sensor source that can provide accurate information without interrupting diverging tasks. In this paper, instead of traditional wire-based sensors, non-contact camera and vehicle data are used that have no physical contact with the driver and do not interrupt driving. Here, four machine learning algorithms, logistic regression (LR), support vector machine (SVM), linear discriminant analysis (LDA) and neural networks (NN), are investigated to classify the cognitive load using the collected data from a driving simulator study. In this paper, physiological parameters are extracted from facial video images, and vehicular parameters are collected from controller area networks (CAN). The data collection was performed in close collaboration with industrial partners in two separate studies, in which study-1 was designed with a 1-back task and study-2 was designed with both 1-back and 2-back task. The goal of the experiment is to investigate how accurately the machine learning algorithms can classify drivers' cognitive load based on the extracted features in complex dynamic driving environments. According to the results, for the physiological parameters extracted from the facial videos, the LR model with logistic function outperforms the other three classification methods. Here, in study-1, the achieved average accuracy for the LR classifier is 94% and in study-2 the average accuracy is 82%. In addition, the classification accuracy for the collected physiological parameters was compared with reference wire-sensor signals. It is observed that the classification accuracies between the sensor and the camera are very similar; however, better accuracy is achieved with the camera data due to having lower artefacts than the sensor data. 

Place, publisher, year, edition, pages
ELSEVIER SCI LTD, 2020
Keywords
Non-contact, Physiological parameters, Vehicular parameters, Cognitive load, Classification, Logistic regression, Support vector machine, Decision tree
National Category
Computer Systems
Identifiers
urn:nbn:se:mdh:diva-46634 (URN)10.1016/j.bspc.2019.101634 (DOI)000502893200022 ()2-s2.0-85071533851 (Scopus ID)
Available from: 2020-01-02 Created: 2020-01-02 Last updated: 2021-02-25Bibliographically approved
7. Driver’s Cognitive Load Classification based on Eye Movement through Facial Image using Machine Learning
Open this publication in new window or tab >>Driver’s Cognitive Load Classification based on Eye Movement through Facial Image using Machine Learning
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

The driver's cognitive load is considered a good indication if the driver is alert or distracted but determing cognitive load is challenging and the acceptance of wire sensor solutions like EEG and ECG are not not preferred in real-world driving scenario. The recent development of image processing, machine learning, and decreasing hardware prices enables new solutions and there are several interesting features related to the driver’s eyes that are currently explored in research. Two different wireless sensor systems, one commercial giving eye position (SmartEye) and one Microsoft LifeCam Studio with resolution 1920 x 1080 were used for data collection. In this paper, two eye movement parameters, saccade, and fixation are investigated through facial images and 13 features are manually extracted. Five machine learning algorithms, support vector machine (SVM), logistic regression (LR), linear discriminant analysis (LDA), k-nearest neighbors (k-NN), and decision tree (DT), are investigated to classify the cognitive load. According to the results, the SVM model with linear kernel function outperforms the other four classification methods. Here, the achieved average accuracy is 92% using SVM. Again, three deep learning architectures, convolutional neural networks (CNN),  long short-term memory (LSTM), and autoencoder (AE) are designed both for automatic feature extraction and cognitive load classification. The results show that CNN architecture achieves the highest classification accuracy which is 91%.  Besides, the classification accuracy for the extracted eye movement parameters is compared with reference eye tracker signals. It is observed that the classification accuracies between the eye tracker and the camera are very similar. 

National Category
Computer Sciences
Research subject
Computer Science; Computer Science
Identifiers
urn:nbn:se:mdh:diva-53510 (URN)
Available from: 2021-02-23 Created: 2021-02-23 Last updated: 2021-07-07Bibliographically approved

Open Access in DiVA

fulltext(4202 kB)735 downloads
File information
File name FULLTEXT02.pdfFile size 4202 kBChecksum SHA-512
dc6534481a379559cac4a1b4302c13ea9dc581a1ac091580fe0bd43c55f39ec0851e688ae813a2fc474cd7b7f5ecc0c86a6ee813fdd54d6d05891c660bfb5098
Type fulltextMimetype application/pdf
errata(93 kB)73 downloads
File information
File name FULLTEXT03.pdfFile size 93 kBChecksum SHA-512
b3ea63f8c985e122b7207f02b806bffc330331272ed66c9a590c2c150191debb287b6ec21c7553f5ca6cb549957d4e869cceffb2b2ef427f1f0e894b886c3b7f
Type errataMimetype application/pdf

Authority records

Rahman, Hamidur

Search in DiVA

By author/editor
Rahman, Hamidur
By organisation
Embedded Systems
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 812 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 2941 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf