International Journal of Web Research

International Journal of Web Research

Multi-Modal Driver Drowsiness Detection in ADAS via Attention-Guided Siamese Network with Temporal Modeling

Document Type : Original Article

Authors
Mahdi Seyfipoor 1
Mohadese Parvizib 2
Siamak Mohammadi 1
1 Department of Computer Engineering, School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran;
2 Department of Computer Engineering, Faculty of Engineering, Alzahra University, Tehran, Iran;
10.22133/ijwr.2025.525802.1289
Abstract
Driver drowsiness detection plays a critical role in improving road safety, as drowsiness substantially increases the likelihood of traffic accidents. In this study, we propose a novel multi-modal framework within Advanced Driver Assistance Systems (ADAS) that leverages an Attention-Guided Siamese Network coupled with temporal modeling to accurately capture both spatial and temporal patterns of driver fatigue. The Siamese network processes paired facial images, enabling the extraction of discriminative features that highlight subtle changes in driver state. The attention mechanism is explicitly applied to the spatial feature maps within each branch of the Siamese network, allowing the model to focus selectively on key facial regions—such as eyes and mouth—that are most indicative of drowsiness, while also weighting complementary sensor modalities dynamically. Temporal modeling is incorporated through a sequential module (e.g., LSTM or temporal convolution) that analyzes the extracted features over time, capturing gradual and evolving signs of drowsiness that static frame-based methods often overlook. Extensive evaluations on benchmark datasets (YawDD, NTHUDDD) and a novel real-world driving dataset demonstrate superior accuracy exceeding 98.8%, along with strong cross-subject generalization. Ablation studies confirm the critical contributions of the attention mechanism in improving feature discrimination, and the temporal modeling module in enhancing sensitivity to progressive drowsiness. The proposed method surpasses traditional approaches in temporal awareness, data efficiency, and resilience to inter-subject and environmental variations, offering a robust and interpretable solution for real-time driver drowsiness monitoring in intelligent vehicles.
Keywords
Driver Drowsiness Detection
Multi-modal Fusion
Attention-Guided Siamese Network
Temporal Modeling
ADAS
Contrastive Learning
Facial Feature

Subjects

[1]     H. Fu, S. Ye, X. Fu, T. Chen and J. Zhao, “New insights into factors affecting the severity of autonomous vehicle crashes from two sources of AV incident records,” Travel Behav. Soc., vol. 38, p. 100934, 2025. https://doi.org/10.1016/j.tbs.2024.100934
[2]     H. Hayashi, N. Oka, S. Sugano and M. Kamezaki, “TUN-DAS: Time- Series Analysis and Unsupervised Learning Based Driving Behavior Assessment System,” in IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 1, pp. 604-619, Jan. 2025. https://doi.org/10.1109/TITS.2024.3484452
[3]     M. Seyfipoor, M. Parvizi and S. Mohammadi, “Collision-Aware Autonomous Driving System,” in Proc. 7th Int. Conf. Pattern Recognition and Image Analysis (IPRIA), Lahijan, Iran, Feb. 2025, pp. 1-7. https://doi.org/10.1109/IPRIA68579.2025.11263550  
[4]     J. Ju and H. Li, “A Survey of EEG-Based Driver State and Behavior Detection for Intelligent Vehicles,” in IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 6, no. 3, pp. 420 434, July 2024. https://doi.org/10.1109/TBIOM.2024.3400866
[5]     S. Yaqoob, G. Morabito, S. Cafiso, G. Pappalardo and A. Ullah, “AI- Driven Driver Behavior Assessment Through Vehicle and Health Monitoring for Safe Driving—A Survey,” in IEEE Access, vol. 12, pp. 48044-48056, 2024. https://doi.org/10.1109/ACCESS.2024.3383775
[6]     F. Qu, N. Dang, B. Furht and M. Nojoumian, “Comprehensive study of driver behavior monitoring systems using computer vision and machine learning techniques,” J. Big Data, vol. 11, no. 1, p. 32, 2024. https://doi.org/10.1186/s40537-024-00890-0
[7]     S. Cao, B. Jiang and C. Wang, “Optimized driver fatigue detection method using multimodal neural networks,” Scientific Reports, vol. 15, no. 1, p. 12240, 2025. https://doi.org/10.1038/s41598-025-86709-1
[8]     . S. Cui, H. Li, X. Liu, C. Yang, H. Gao, J. Yang, and B. Yang, “Driver fatigue detection based on facial multi-feature fusion computing,” Computing, vol. 107, no. 12, p. 1-23, 2025. https://doi.org/10.1007/s00607-025-01557-1
[9]      O. F. Hassan, A. F. Ibrahim, A. Gomaa, M. A. Makhlouf and B. Hafiz, “Real-time driver drowsiness detection using transformer architectures: a novel deep learning approach,” Scientific Reports, vol. 15, no. 1, 17493, 2025. https://doi.org/10.1038/s41598-025-02111-x
[10]   . L. Yu, X. Yang, H. Wei, J. Liu, B. Li, “Driver fatigue detection using PPG signal, facial features, head postures with an LSTM model,” Heliyon, vol. 10, no. 21, 2024. https://doi.org/10.1016/j.heliyon.2024.e39479
[11]   L. Lin, S. Wang, J. Yang, F. Wei, “A multi-aware graph convolutional network for driver drowsiness detection,” Knowledge-Based Systems, vol. 305, p. 112643, 2024. https://doi.org/10.1016/j.knosys.2024.112643
[12]   T. V. Khoa, D. H. Son, M. A. Alsheikh, Y. F. Alem and D. T. Hoang, “Privacy-preserving driver drowsiness detection with spatial self-attention and federated learning,” arXiv preprint, arXiv:2508.00287, Aug. 2025. https://doi.org/10.48550/arXiv.2508.00287
[13]   Y. You, S. Chang, Z. Yang and Q. Sun, “PSNSleep: a self-supervised learning method for sleep staging based on Siamese networks with only positive sample pairs.” Frontiers in Neuroscience, vol. 17, p. 1167723, 2023. https://doi.org/10.3389/fnins.2023.1167723
[14]   L. J. Chang, H. A. Chen, , C. Chang and C. S. Wei, “SiamEEGNet: Siamese Neural Network-Based EEG Decoding for Drowsiness Detection.” bioRxiv 2023-10, 2023. https://doi.org/10.1101/2023.10.23.563513
[15]   M. Seyfipoor, M. Parvizi, and S. Mohammadi, “A Novel Method for Facial Expression-Based Driver Drowsiness Detection Leveraging Siamese Network in ADAS Applications,” 11th International Conference on Web Research, Tehran, Iran, 2025, pp. 281-287, https://doi.org/10.1109/ICWR65219.2025.11006213
[16]   M. Ahmed, S. Masood, M. Ahmad and A. A. Abd El-Latif, "Intelligent Driver Drowsiness Detection for Traffic Safety Based on Multi CNN Deep Model and Facial Subsampling," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 10, pp. 19743-19752, Oct. 2022. https://doi.org/10.1109/TITS.2021.3134222
[17]   Y. Peng et al., “ A multi-source fusion approach for driver fatigue detection using physiological signals and facial image,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, 11, pp. 16614-16624, Nov. 2024, https://doi.org/10.1109/TITS.2024.3420409
[18]   A. Lemkaddem et al., “Multi-modal driver drowsiness detection: A feasibility study,” IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, USA, 2018, pp. 9-12, https://doi.org/10.1109/BHI.2018.8333357
[19]   R. Malik, M. Vijarania and M. Malik, “A Comprehensive Review of Deep Learning and IoT in Driver Drowsiness Detection for Safer Roads,” 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON), New Delhi, India, 2024, pp. 1-7, https://doi.org/10.1109/DELCON64804.2024.10866889
[20]   Y. Qian, G. Zheng, Y. Xie, X. Lv and W. Zhang, “Fatigue driving detection based on driver facial temporal sequences,” Academic Journal of Science and Technology, vol. 10, no. 3, pp. 37–41, 2024. https://doi.org/10.54097/t3pmh008
[21]   Z. Jin and L. Dong, “YOLO11-CR: A lightweight convolution-and-attention framework for accurate fatigue driving detection,” arXiv preprint, arXiv:2508.13205, Aug. 2025.
https://doi.org/10.48550/arXiv.2508.13205
[22]   J. Ren et al., “LiteFat: Lightweight spatio-temporal graph learning for real-time driver fatigue detection,” arXiv preprint, arXiv:2507.21756, Jul. 2025.
https://doi.org/10.48550/arXiv.2507.21756
[23]    S. Qu, Z. Gao, X. Chen, N. Li, Y. Wang and X. Wu, “Multi-task learning for fatigue detection and face recognition of drivers via tree-style space-channel attention fusion network,” arXiv preprint, arXiv:2405.07845, May 2024.
https://doi.org/10.48550/arXiv.2405.07845
[24]   Y. Albadawi, A. AlRedhaei and M. Takruri. “Real-time machine learning-based driver drowsiness detection using visual features. ” Journal of imaging, vol. 9, no. 5, p. 91, 2023. https://doi.org/10.3390/jimaging9050091
[25]   E. Yang and O. Yi, “Enhancing road safety: Deep learning-based intelligent driver drowsiness detection for advanced driver-assistance systems.” Electronics, vol. 13, no. 4, p. 708, 2024.  https://doi.org/10.3390/electronics13040708
[26]   S. Arif, S. Munawar and H. Ali, “Driving drowsiness detection using spectral signatures of EEG-based neurophysiology,” Sec. Computational Physiology and Medicine., vol. 14, p. 1153268, 2023. https://doi.org/10.3389/fphys.2023.1153268
[27]   A. C. Phan, N. H. Q. Nguyen, T. N.Trieu and T. C. Phan, “An efficient approach for d etecting driver drowsiness based on deep learning,” Applied Sciences, vol. 11, no. 18, p. 8441, 2021. https://doi.org/10.3390/app11188441
[28]   L. Sharara et al., “A Real-Time Automotive Safety System Based on Advanced AI Facial Detection Algorithms,” in IEEE Transactions on Intelligent Vehicles, vol. 9, no. 6, pp. 5080-5100, June 2024. https://doi.org/10.1109/TIV.2023.3272304
[29]   L. Yang, H. Yang, H. Wei, Z. Hu and C. Lv, “Video-Based Driver Drowsiness Detection with Optimised Utilization of Key Facial Features,” in IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 7, pp. 6938-6950, July 2024. https://doi.org/10.1109/TITS.2023.3346054
[30]   H. A. Khalil, S. A. Hammad, H. E. Abd El Munim and S. A. Maged, “Low-Cost Driver Monitoring System Using Deep Learning,” in IEEE Access, vol. 13, pp. 14151-14164, 2025. https://doi.org/10.1109/ACCESS.2025.3530296
[31]   Q. Wu, N. Li, L. Zhang and F. R. Yu, “Driver Drowsiness Detection Based on Joint Human Face and Facial Landmark Localization with Cheap Operations,” in IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 12, pp. 19633 19645, Dec. 2024. https://doi.org/10.1109/TITS.2024.3443832
[32]   K. Fujiwara, H. Iwamoto, K. Hori and M. Kano, “Driver Drowsiness Detection Using R-R Interval of Electrocardiogram and Self- Attention Autoencoder,” in IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 2956-2965, Jan. 2024. https://doi.org/10.1109/TIV.2023.3308575
[33]   D. L. Nguyen, M. D. Putro and K. H. Jo, “Lightweight CNN-Based Driver Eye Status Surveillance for Smart Vehicles,” in IEEE Transactions on Industrial Informatics, vol. 20, no. 3, pp. 3154-3162, March 2024. https://doi.org/10.1109/TII.2023.3296921
[34]   T. Debbarma, T. Pal, A. Saha and N. Debbarma, “HCNNet: A hybrid convolutional neural network for abnormal human driver behaviour detection,” Sādhanā, vol. 50, no. 1, p. 9, 2025. https://doi.org/10.1007/s12046-024-02656-z
[35]   S. Dey, A. Dutta, J. I. Toledo, S. K. Ghosh, J. Lladós and U. Pal, “Signet: Convolutional siamese network for writer independent offline signature verification,” arXiv preprint arXiv:1707.02131, 2017.
https://doi.org/10.48550/arXiv.1707.02131
[36]   L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, vol. 8, pp. 1–74, 2021. https://doi.org/10.1186/s40537-021-00444-8
[37]    Y. Albadawi, M. Takruri and M. Awad, “A review of recent developments in driver drowsiness detection systems,” Sensors, 22, no. 5, 2022.  https://doi.org/10.3390/s22052069 
[38]   Q. Wang, J. Li, X. Tong, and P. M. Atkinson, “TSI Siamnet: A Siamese network for cloud and shadow detection based on time-series cloudy images,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 213, pp. 107-123, 2024. https://doi.org/10.1016/j.isprsjprs.2024.05.022
[39]   A. Fedele, G. Riccardo and P. Dino, “Explaining Siamese networks in few-shot learning,” Machine Learning, vol. 113, pp. 7723-7760, 2024. https://doi.org/10.1007/s10994-024-06529-8
[40]   S. A. El-Nabi, W. El-Shafai, E. S. M. El-Rabaie, K. F. Ramadan, F. A. Abd El-Samie and S. Mohsen, “Machine learning and deep learning techniques for driver fatigue and drowsiness detection: A review,” Multimedia Tools and Applications, vol. 83, no. 3, pp. 9441-9477, 2024. https://doi.org/10.1007/s11042-023-15054-0
[41]   Y. Peng et al., “A Multi-Source Fusion Approach for Driver Fatigue Detection Using Physiological Signals and Facial Image” in IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 11, pp. 16614-16624, Nov. 2024. https://doi.org/10.1109/TITS.2024.3420409
[42]   D. Gusak, G. Mezentsev, I. Oseledets, and E. Frolov, “RECE: Reduced Cross-Entropy Loss for Large Catalogue Sequential Recommenders,” arXiv preprint, arXiv:2408.02354, Aug 2024 https://doi.org/10.1145/3627673.3679986
[43]   B. Kulambayev, G. Astaubayeva, G. Tleuberdiyeva, J. Alimkulova, G. Nussupbekova and O. Kisseleva, “Deep CNN Approach with Visual Features for Real Time Pavement Crack Detection,” International Journal of Advanced Computer Science and Applications (IJACSA), vol. 15, no. 3, 2024. https://doi.org/10.14569/IJACSA.2024.0150333
[44]   M. Venkateswarlu and V. Rami Reddy Ch, “DrowsyDetectNet: Driver Drowsiness Detection Using Lightweight CNN With Limited Training Data,” in IEEE Access, vol. 12, pp. 110476-110491, 2024. https://doi.org/10.1109/ACCESS.2024.3440585
[45]   E. Agliari, F. Alemanno, M. Aquaro and A. Fachechi, “Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting,” Neural Networks, vol. 177, p. 106389, 2024. https://doi.org/10.1016/j.neunet.2024.106389
[46]   T. Miseta, A. Fodor and A. Vathy-Fogarassy, “Surpassing early stopping: A novel correlation-based stopping criterion for neural networks,” Neurocomputing, vol. 567, p. 127028, 2024. https://doi.org/10.1016/j.neucom.2023.127028
[47]   A. Cutkosky, A. Defazio and H. Mehta, “Mechanic: A learning rate tuner,” arXiv preprint, arXiv: 2306.00144, 2023.
https://doi.org/10.48550/arXiv.2306.00144
[48]    M. V. Ferro, Y. D. Mosquera, F. J. R. Pena and V. M. D. Bilbao, “Early stopping by correlating online indicators in neural networks,” Neural Networks, vol. 159, pp.109-124, 2023. https://doi.org/10.1016/j.neunet.2022.11.035
[49]   A. Jami, M. Razzaghpour, H. Alnuweiri and Y. P. Fallah, “Augmented driver behavior models for high‐fidelity simulation study of crash detection algorithms,” IET Intelligent Transport Systems, vol. 18, no. 3, pp. 436-449, 2024. https://doi.org/10.1049/itr2.12373
[50]    R. Sun et al., “YawnNet: A Visual-Centric Approach for Yawning Detection,” In Proceedings of the International Conference on Multimedia Retrieval, pp. 1140-1144, 2024. https://doi.org/10.1145/3652583.3657618
[51]   M. Simon et al., “EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions,” Clin. Neurophysiol., vol. 122, no. 6, pp. 1168–1178, Jun. 2011. https://doi.org/10.1016/j.clinph.2010.10.044
International Journal of Web Research
Volume 9, Issue 1
Winter 2026
Pages 39-56