International Journal of Web Research

International Journal of Web Research

A Scalable Method for Real-Time Facial Emotion Recognition using an Artificial Neural Network and Polynomial Equation

Document Type : Original Article

Authors
Omid Ghadami
Alireza Rezvanian
Department of Computer Engineering, University of Science and Culture, Tehran, Iran
10.22133/ijwr.2024.459595.1224
Abstract
Facial emotion recognition has recently attracted considerable interest due to its wide range of applications. It plays a crucial role in supporting individuals with autism spectrum disorders and improving interactions between humans and computers. The ability to execute these applications in real-time is essential. The architecture of the model and the computational resources available are the key determinants of inference time. Consequently, the development of a real-time solution requires a concentrated effort on these elements. In this paper, we present a scalable approach that utilizes EfficientNetV2, chosen for its operational efficiency. Our methodology involves resolution scaling based on a polynomial equation, which ensures real-time performance across various computational resources and model configurations. This scalable technique employs a polynomial equation to identify the optimal resolution for designated inference times, specifically adapted to our hardware and model specifications. By implementing the polynomial equation for resolution scaling, we created two variants of EfficientNetV2. Our findings from the KDEF dataset indicate that the proposed EfficientNetV2 can accurately classify images in real time on our hardware.
Keywords
Real-time facial emotion recognition
Deep Learning
EfficientNetV2
Imbalanced Datasets
Resolution scaling

Subjects

  • Ghadami, A. Rezvanian, and S. Shakuri, “Scalable Real-time Emotion Recognition using EfficientNetV2 and Resolution Scaling,” in 2024 10th International Conference on Web Research (ICWR), Tehran, Iran, IEEE, 2024, pp. 7–12. https://doi.org/10.1109/ICWR61162.2024.10533360
  • K. Chowdary, T. N. Nguyen, and D. J. Hemanth, “Deep learning-based facial emotion recognition for human–computer interaction applications,” Neural Comput & Applic, vol. 35, no. 32, pp. 23311–23328, Nov. 2023. https://doi.org/10.1007/s00521-021-06012-8
  • R. H. Lee and A. Wong, “Timeconvnets: A deep time windowed convolution neural network design for real-time video facial expression recognition,” in 2020 17th Conference on Computer and Robot Vision (CRV), Ottawa, ON, Canada, IEEE, 2020, pp. 9–16, https://doi.org/10.1109/CRV50864.2020.00010.
  • Jaiswal and G. C. Nandi, “Robust real-time emotion detection system using CNN architecture,” Neural Comput & Applic, vol. 32, no. 15, pp. 11253–11262, Aug. 2020. https://doi.org/10.1007/s00521-019-04564-4
  • Kratzwald, S. Ilić, M. Kraus, S. Feuerriegel, and H. Prendinger, “Deep learning for affective computing: Text-based emotion recognition in decision support,” Decision support systems, vol. 115, pp. 24–35, 2018. https://doi.org/10.1016/j.dss.2018.09.002
  • Wang, J. Huang, J. Zhu, M. Yang, and F. Yang, “Facial expression recognition with deep learning,” in Proceedings of the 10th International Conference on Internet Multimedia Computing and Service, Nanjing China: ACM, Aug. 2018, pp. 1–4. https://doi.org/10.1145/3240876.3240908
  • Tan and Q. Le, “Efficientnetv2: Smaller models and faster training,” in International Conference on Machine Learning, PMLR, 2021, pp. 10096–10106. https://proceedings.mlr.press/v139/tan21a.html
  • T. Lopes, E. De Aguiar, A. F. De Souza, and T. Oliveira-Santos, “Facial expression recognition with convolutional neural networks: coping with few data and the training sample order,” Pattern recognition, vol. 61, pp. 610–628, 2017. https://doi.org/10.1016/j.patcog.2016.07.026
  • -Y. Huang et al., “A study on computer vision for facial emotion recognition,” Scientific Reports, vol. 13, no. 1, p. 8425, 2023. https://doi.org/10.1038/s41598-023-35446-4
  • Al Chanti and A. Caplier, “Deep learning for spatio-temporal modeling of dynamic spontaneous emotions,” IEEE Transactions on Affective Computing, vol. 12, no. 2, pp. 363–376, 2018. https://doi.org/10.1109/TAFFC.2018.2873600
  • Vignesh, M. Savithadevi, M. Sridevi, and R. Sridhar, “A novel facial emotion recognition model using segmentation VGG-19 architecture,” Int. j. inf. tecnol., vol. 15, no. 4, pp. 1777–1787, Apr. 2023. https://doi.org/10.1007/s41870-023-01184-z
  • Pramerdorfer and M. Kampel, “Facial Expression Recognition using Convolutional Neural Networks: State of the Art,” Dec. 09, 2016, arXiv: arXiv:1612.02903. https://doi.org/10.48550/arXiv.1612.02903
  • Yao, Y. Wan, H. Ni, and B. Xu, “Action unit classification for facial expression recognition using active learning and SVM,” Multimed Tools Appl, vol. 80, no. 16, pp. 24287–24301, Jul. 2021. https://doi.org/10.1007/s11042-021-10836-w
  • Barra, L. De Maio, and S. Barra, “Emotion recognition by web-shaped model,” Multimed Tools Appl, vol. 82, no. 8, pp. 11321–11336, Mar. 2023. https://doi.org/10.1007/s11042-022-13361-6
  • Arriaga, M. Valdenegro-Toro, and P. Plöger, “Real-time Convolutional Neural Networks for emotion and gender classification,” in 27th European Symposium on Artificial Neural Networks, ESANN 2019, Bruges, Belgium, April 24-26, 2019, 2019, pp. 221–226. https://www.esann.org/sites/default/files/proceedings/legacy/es2019-157.pdf
  • Bakariya, A. Singh, H. Singh, P. Raju, R. Rajpoot, and K. K. Mohbey, “Facial emotion recognition and music recommendation system using CNN-based deep learning techniques,” Evolving Systems, vol. 15, no. 2, pp. 641–658, Apr. 2024. https://doi.org/10.1007/s12530-023-09506-z
  • Rosenbrock, Deep Learning for Computer Vision with Python Practitioner bundle, PyImageSearch. com, 2018.
  • G. Howard et al., “MobileNets: efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, vol. 126, 2017.https://doi.org/10.48550/arXiv.1704.04861
International Journal of Web Research
Volume 7, Issue 4
Autumn 2024
Pages 39-49