Multimodal Sentiment Analysis of Social Media Posts Using Deep Neural Networks

Document Type : Original Article

Authors

1 Department of Computer Eng., Shahrekord University, Shahrekord, Iran

2 Department of Computer Engineering, Faculty of Engineering Shahrekord University Shahrekord, Iran

3 Computer Engineering Dept., Shahrekord University, Shahrekord, Iran

Abstract

With the fast growth of social media, they have become the most important platform for posting multimodal content generated by users. Much of the data on social networks such as Instagram and Telegram is multimodal data. With the aim of analyzing such multimodal data in social networks, multimodal sentiment analysis has become one of the most significant subjects for researchers in the field of emotion recognition and data mining. Although multimodal sentiment analysis of social media data for English language has been addressed in several researches recently, few studies addressed the problem for the Persian language which is the official language of more than 120 million of people around the word. In this study, a multimodal deep learning model is proposed to address this problem. The proposed method utilizes a bi-directional long short-term memory (bi-LSTM) for processing text posts and a VGG16 convolutional network for analyzing images. A new dataset of Instagram and Telegram posts, MPerSocial, containing 1000 pairs of images and Persian comments is introduced in the current study and used for evaluating the proposed method. The results of experiments show that using the fusion of textual and image modalities improves sentiment polarity detection accuracy by 20% and 8% compared with the scenario in which image and text modalities in isolation. Also, the performance of the proposed model is better than three similar deep and four traditional machine learning models. All codes and dataset used in the current study are publicly available at GitHub.  

Keywords

Main Subjects

References

  • Rhee, Bo‐A., Federico Pianzola, and Gang‐Ta Choi. "Analyzing the museum experience through the lens of Instagram posts." Curator: The Museum Journal (2021).
  • Ershov, Daniel, and Matthew Mitchell. "The Effects of Influencer Advertising Disclosure Regulations: Evidence From Instagram." In Proceedings of the 21st ACM Conference on Economics and Computation, pp. 73-74. 2020.
  • Romero-Rodríguez, José-María, Carmen Rodríguez-Jiménez, Magdalena Ramos Navas-Parejo, José-Antonio Marín-Marín, and Gerardo Gómez-García. "Use of Instagram by Pre-Service Teacher Education: Smartphone Habits and Dependency Factors." International Journal of Environmental Research and Public Health 17, no. 11 (2020): 4097.
  • Ferwerda, Bruce, Markus Schedl, and Marko Tkalcic. "Predicting personality traits with instagram pictures." In Proceedings of the 3rd Workshop on Emotions and Personality in Personalized Systems 2015, pp. 7-10. 2015.
  • Xing, Frank, Lorenzo Malandri, Yue Zhang, and Erik Cambria. "Financial Sentiment Analysis: An Investigation into Common Mistakes and Silver Bullets." In Proceedings of the 28th International Conference on Computational Linguistics, pp. 978-987. 2020.
  • Khatua, Aparup, Apalak Khatua, and Erik Cambria. "Predicting political sentiments of voters from Twitter in multi-party contexts." Applied Soft Computing 97 (2020): 106743.
  • Basiri, Mohammad Ehsan, and Arman Kabiri. "HOMPer: A new hybrid system for opinion mining in the Persian language." Journal of Information Science 46, no. 1 (2020): 101-117.
  • Abdar, Moloud, Mohammad Ehsan Basiri, Junjun Yin, Mahmoud Habibnezhad, Guangqing Chi, Shahla Nemati, and Somayeh Asadi. "Energy choices in Alaska: Mining people's perception and attitudes from geotagged tweets." Renewable and Sustainable Energy Reviews 124 (2020): 109781.
  • Ma, Yukun, Haiyun Peng, Tahir Khan, Erik Cambria, and Amir Hussain. "Sentic LSTM: a hybrid network for targeted aspect-based sentiment analysis." Cognitive Computation 10, no. 4 (2018): 639-650.
  • Tran, Ha-Nguyen, and Erik Cambria. "Ensemble application of ELM and GPU for real-time multimodal sentiment analysis." Memetic Computing 10, no. 1 (2018): 3-13.
  • Poria, Soujanya, Erik Cambria, Newton Howard, Guang-Bin Huang, and Amir Hussain. "Fusing audio, visual and textual clues for sentiment analysis from multimodal content." Neurocomputing 174 (2016): 50-59.
  • Cambria, Erik, Devamanyu Hazarika, Soujanya Poria, Amir Hussain, and R. B. V. Subramanyam. "Benchmarking multimodal sentiment analysis." In International Conference on Computational Linguistics and Intelligent Text Processing, pp. 166-179. Springer, Cham, 2017.
  • Nemati, R. Rohani, M. E. Basiri, M. Abdar, N. Y. Yen, and V. Makarenkov, “A Hybrid Latent Space Data Fusion Method for Multimodal Emotion Recognition,” IEEE Access, vol. 7, 2019.
  • M. Correa, T. Adali, Y.-O. Li, and V. D. Calhoun, “Canonical correlation analysis for data fusion and group inferences,” IEEE signal processing magazine, vol. 27, no. 4, pp. 39–50, 2010.
  • Poria, A. Hussain, and E. Cambria, Multimodal sentiment analysis. 2018.
  • Poria, N. Majumder, D. Hazarika, E. Cambria, A. Gelbukh, and A. Hussain, “Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines,” IEEE Intelligent Systems, vol. 33, no. 6, pp. 17–25, 2018.
  • Dashtipour, “Novel symbolic and sub-symbolic approaches for text based and multimodal sentiment analysis,” 2019.
  • E. Basiri, A. Kabiri, M. Abdar, W. K. Mashwani, N. Y. Yen, and J. C. Hung, “The effect of aggregation methods on sentiment classification in Persian reviews,” Enterprise Information Systems, vol. 14, no. 9–10, 2020.
  • Dashtipour, A. Hussain, Q. Zhou, A. Gelbukh, A. Y. A. Hawalah, and E. Cambria, “PerSent: A Freely Available Persian Sentiment Lexicon,” 2016, pp. 310–320.
  • E. Basiri, N. Ghasem-Aghaee, and A. R. Naghsh-nilchi, “Lexicon-based Sentiment Analysis in Persian,” Current and Future Developments in Artificial Intelligence, p. 154, 2017.
  • Amini, S. Karimi, and A. Shakery, “Cross-lingual Subjectivity Detection for Resource Lean Languages,” 2019.
  • Ghasemi, S. A. Ashrafi Asli, and S. Momtazi, “Deep Persian sentiment analysis: Cross-lingual training for low-resource languages,” Journal of Information Science, 2020.
  • Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor Fusion Network for Multimodal Sentiment Analysis,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 1103–1114.
  • E. Basiri et al., “Improving Sentiment Polarity Detection through Target Identification,” IEEE Transactions on Computational Social Systems, vol. 7, no. 1, 2020.
  • E. Basiri, A. R. Naghsh-Nilchi, and N. Ghassem-Aghaee, “A framework for sentiment analysis in persian,” Open Transactions on Information Processing, vol. 1, no. 3, pp. 1–14, 2014.
  • Asgarian, M. Kahani, and S. Sharifi, “The Impact of Sentiment Features on the Sentiment Polarity Classification in Persian Reviews,” Cognitive Computation, vol. 10, no. 1, pp. 117–135, Feb. 2018.
  • M. Bagheri A., “Persian sentiment analyzer: A framework based on a novel feature selection method,” International Journal of Artificial Intelligence, vol. 12,2,,115, no. pp. 115–129, 2014.
  • B. Dastgheib, S. Koleini, and F. Rasti, “The application of Deep Learning in Persian Documents Sentiment Analysis,” International Journal of Information Science and Management (IJISM), vol. 18, no. 1, pp. 1–15, 2020.
  • Roshanfekr, S. Khadivi, and M. Rahmati, “Sentiment analysis using deep learning on Persian texts,” in 2017 Iranian Conference on Electrical Engineering (ICEE), 2017, pp. 1503–1508.
  • Heidari and P. Shamsinejad, “Producing An Instagram Dataset For Persian Language Sentiment Analysis Using Crowdsourcing Method.”
  • Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
  • Kia Dashtipour, Mandar Gogate, Ahsan Adeel, Cosimo Ieracitano, Hadi Larijani, Amir Hussain. Exploiting Deep Learning for Persian Sentiment Analysis. arXiv:1808.05077. 2018 Aug
  • Zahra Bokaee Nezhad, Mohammad Ali Deihimi. A Combined Deep Learning Model for Persian Sentiment Analysis. IIUM Engineering Journal. 2019.
  • Seyed Arad Ashrafi Asli, Behnam Sabeti, Zahra Majdabadi, Preni Golazizian, Reza Fahmi, Omid Momenzadeh. Optimizing Annotation Effort Using Active Learning Strategies: A Sentiment Analysis Case Study in Persian. Proceedings of the 12th Language Resources and Evaluation Conference. 2020.
  • Javad PourMostafa Roshan Sharami, Parsa Abbasi Sarabestani, Seyed Abolghasem Mirroshandel. DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus. arXiv:2004.05328. 2020.
  • Nazanin Sabri, Ali Edalat, Behnam Bahrak. Sentiment Analysis of Persian-English Code-mixed Texts. arXiv:2102.12700. 2021.
  • Kia Dashtipour, Cosimo Ieracitano, Francesco Carlo Morabito, Ali Raza. An Ensemble Based Classification Approach for Persian Sentiment Analysis. In book: Progresses in Artificial Intelligence and Neural Systems (pp.207-215). 2021.
  • Nan Xu, Wenji Mao. A residual merged neutral network for multimodal sentiment analysis. 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA).
  • Quoc-Tuan Truong, Hady Wirawan Lauw. VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis. Proceedings of the AAAI Conference on Artificial Intelligence 33:305-312. 2019.
  • Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, Ajay Divakaran. Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts. arXiv:1904.09073. 2019.
  • Mathieu Fortin, Brahim Chaib-Draa. Multimodal Sentiment Analysis: A Multitask Learning Approach. 8th International Conference on Pattern Recognition Applications and Methods. 2019.
  • Akshi Kumar, Geetanjali Garg. Sentiment analysis of multimodal twitter data. Multimedia Tools and Applications. 2019.
  • Ferda Ofli, Firoj Alam, Muhammad Imran. Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response. arXiv:2004.11838. 2020.
  • Manaswi, Navin Kumar. "Understanding and working with Keras." In Deep Learning with Applications Using Python, pp. 31-43. Apress, Berkeley, CA, 201

 

 Aria Naseri Karimvand received his B.S. degree in software engineering from MJDKH university in 2019 and his M.S. from Shahrekord University in 2021. His research interest includes natural language processing, deep learning, and social media data mining.

 Shahla Nemati was born in Shiraz, Iran in 1982. She received the B.S. degree in hardware engineering from Shiraz University, Shiraz, Iran, in 2005, the M.S. degree from Isfahan University of Technology, Isfahan, Iran, in 2008, and the Ph.D. degree in computer engineering from Isfahan University, Isfahan, Iran, in 2016. Since 2017, she has been an Assistant Professor with the Computer Engineering Department, Shahrekord University, Shahrekord, Iran. Her research interests include data fusion, affective computing, and data mining.

 Reza Salehi Chegeni received his B.S. degree in software engineering from Lorestan university in 2018 and his M.S. from Shahrekord University in 2021. His research interest includes evolutionry algorithms, natural language processing, deep learning, and data mining.

 Mohammad Ehsan Basiri received the B.S. degree in software engineering from Shiraz University, Shiraz, Iran, in 2006 and the M.S. and Ph.D. degrees in Artificial Intelligence from Isfahan University, Isfahan, Iran, in 2009 and 2014. Since 2014, he has been an Assistant Professor with the Computer Engineering Department, Shahrekord University, Shahrekord, Iran. He is the author of three books and more than 35 articles. His research interests include sentiment analysis, natural language processing, deep learning, and data mining

 

 

 

 

 

International Journal of Web Research
Volume 4, Issue 1
Winter-Spring
June 2021
Pages 1-9

Files

Share

How to cite

Statistics