Abstract
In this work, an attempt has been made to classify emotional states using electrodermal activity (EDA) signals and multiscale convolutional neural networks. For this, EDA signals are considered from a publicly available “A Dataset for Emotion Analysis using Physiological Signals” (DEAP) database. These signals are decomposed into multiple-scales using the coarse-grained method. The multiscale signals are applied to the Multiscale Convolutional Neural Network (MSCNN) to automatically learn robust features directly from the raw signals. Experiments are performed with the MSCNN approach to evaluate the hypothesis (i) improved classification with electrodermal activity signals, and (ii) multiscale learning captures robust complementary features at a different scale. Results show that the proposed approach is able to differentiate various emotional states. The proposed approach yields a classification accuracy of 69.33% and 71.43% for valence and arousal states, respectively. It is observed that the number of layers and the signal length are the determinants for the classifier performance. The performance of the proposed approach outperforms the single-layer convolutional neural network. The MSCNN approach provides end-to-end learning and classification of emotional states without additional signal processing. Thus, it appears that the proposed method could be a useful tool to assess the difference in emotional states for automated decision making.





Similar content being viewed by others
References
Anusha AS, Sukumaran P, Sarveswaran V, et al Electrodermal activity based pre-surgery stress detection using a wrist wearable. IEEE J. Biomed. Health Inform. 1–1. https://doi.org/10.1109/JBHI.2019.2893222, (2019)
Affanni A, Bernardini R, Piras A, et al (2018) Driver’s stress detection using skin potential response signals. Measurement 122:264–274. https://doi.org/10.1016/j.measurement.2018.03.040
Nazarov AI (2018) Comparison of frequency components of EEG and skin potential (quantitative analysis). Exp. Psychol. 11:95–109. https://doi.org/10.17759/exppsy.2018110207
Akay A, Hess H (2019) Deep learning: current and emerging applications in medicine and technology. IEEE J. Biomed. Health Inform. 23:906–920. https://doi.org/10.1109/JBHI.2019.2894713, (2019)
Alarcao SM, Fonseca MJ (2017) Emotions recognition using EEG signals: a survey. IEEE Trans. Affect. Comput. 10:374–393. https://doi.org/10.1109/TAFFC.2017.2714671
Albasri A, Abdali-Mohammadi F, Fathi A (2019) EEG electrode selection for person identification thru a genetic-algorithm method. J. Med. Syst. 43:297. https://doi.org/10.1007/s10916-019-1364-8
Amin MR, Faghih RT (2019) Sparse deconvolution of electrodermal activity via continuous-time system identification. IEEE. Trans. Biomed. Eng. 66:2585–95. https://doi.org/10.1109/TBME.2019.2892352
Arapakis I, Barreda-Ángeles M, Pereda-Baños A (2019) Interest as a proxy of engagement in news reading: spectral and entropy analyses of EEG activity patterns. IEEE Trans. Affect. Comput. 10:100–114. https://doi.org/10.1109/TAFFC.2017.2682089
Boucsein W (2012) Electrodermal Activity, 2nd ed. Springer, New York
Brishtel I, Khan AA, Schmidt T, et al (2020) Mind wandering in a multimodal reading setting: behavior analysis; automatic detection using eye-tracking and an EDA sensor. Sensors 20:2546. https://doi.org/10.3390/s20092546
Charlson F, van Ommeren M, Flaxman A, et al. (2019) New WHO prevalence estimates of mental disorders in conflict settings: a systematic review and meta-analysis. The Lancet S0140673619309341. https://doi.org/10.1016/S0140-6736(19)30934-1
Daly I, Williams D, Hwang F, et al (2019) Electroencephalography reflects the activity of sub-cortical brain regions during approach-withdrawal behaviour while listening to music. Sci. Rep. 9:9415. https://doi.org/10.1038/s41598-019-45105-2
Fang WC, Wang KY, Fahier N, et al (2019) Development and validation of an EEG-based real-time emotion recognition system using edge AI Computing Platform With Convolutional Neural Network System-on-chip design. IEEE J. Emerg. Sel. Top. Circuits. Syst. 9:645–657. https://doi.org/10.1109/JETCAS.2019.2951232
Feng H, Golshan HM, Mahoor MH (2018) A wavelet-based approach to emotion classification using EDA signals. Expert. Syst. Appl. 112:77–86. https://doi.org/10.1016/j.eswa.2018.06.014
Ganapathy N, Swaminathan R, Deserno TM (2018) Deep Learning on 1-D Biosignals: a taxonomy-based survey. Yearb. Med. Inform. 27:98–109. https://doi.org/10.1055/s-0038-1667083
Gravina R, Li Q (2019) Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion. Inf. Fusion 48:1–10. https://doi.org/10.1016/j.inffus.2018.08.001
Greco A, Valenza G, Lanata A, et al (2014) Electrodermal activity in bipolar patients during affective elicitation. IEEE J. Biomed. Health Inform. 18:1865–1873. https://doi.org/10.1109/JBHI.2014.2300940
Greco A, Valenza G, Scilingo EP (2016) Advances in electrodermal activity processing with applications for mental health. Springer International Publishing, Cham
Greco A, Valenza G, Lanata A, et al (2016) cvxEDA: a convex optimization approach to electrodermal activity processing. IEEE Trans. Biomed. Eng. 63:797–804. https://doi.org/10.1109/TBME.2015.2474131
Greco A, Valenza G, Scilingo EP Brain dynamics during arousal-dependent pleasant/unpleasant visual elicitation: an electroencephalographic study on the circumplex model of affect. IEEE Trans. Affect. Comput. 1–1. https://doi.org/10.1109/TAFFC.2018.2879343, (2018)
Hamada M, Zaidan BB, Zaidan AA (2018) A systematic review for human EEG brain signals based emotion classification, feature extraction, brain condition, group comparison. J. Med. Syst. 42:162. https://doi.org/10.1007/s10916-018-1020-8
Hossain MS, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Inf. Fusion 49:69–78. https://doi.org/10.1016/j.inffus.2018.09.008
Jiang G, He H, Yan J, Xie P (2019) Multiscale convolutional neural networks for fault diagnosis of wind turbine Gearbox. IEEE Trans. Ind. Electron. 66:3196–3207 https://doi.org/10.1109/TIE.2018.2844805
Koelstra S, Muhl C, Soleymani M, et al (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3:18–31. https://doi.org/10.1109/T-AFFC.2011.15
Kroupi E, Vesin JM, Ebrahimi T Phase-amplitude coupling between EEG and EDA while experiencing multimedia content. In: Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction. Geneva: IEEE, pp. 865–870, (2013)
Kroupi E, Vesin JM, Ebrahimi T (2014) Implicit affective profiling of subjects based on physiological data coupling. Brain-Comput. Interfaces 1:85–98. https://doi.org/10.1080/2326263X.2014.912882
Liu G (2014) An entropy measure of emotional arousal via skin conductance response. J. Fiber. Bioeng. Inform. 7:67–80. https://doi.org/10.3993/jfbi03201406
Liu Z, Xie Q, Wu M, et al (2018) Electroencephalogram emotion recognition based on empirical mode decomposition and optimal feature selection. IEEE Trans. Cogn. Dev. Syst. 11:517–26. https://doi.org/10.1109/TCDS.2018.2868121
Liu M, Xu C, Luo Y, et al (2018) Cost-sensitive feature selection by optimizing F-measures. IEEE Trans. Image. Process. 27:1323–1335. https://doi.org/10.1109/TIP.2017.2781298
Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern. Anal. Mach. Intell. 23:1175–1191. https://doi.org/10.1109/34.954607
Posada-Quintero HF, Bolkhovsky JB (2019) Machine Learning models for the identification of cognitive tasks using autonomic reactions from heart rate variability and electrodermal activity. Behav. Sci. 9:45. https://doi.org/10.3390/bs9040045
Posada-Quintero HF, Chon KH Phasic component of electrodermal activity is more correlated to brain activity than tonic component. In: 2019 IEEE EMBS International Conference on Biomedical Health Informatics (BHI). pp. 1–4, (2019)
Rahman JS, Hossain MZ, Gedeon T Measuring observers’ EDA responses to emotional videos. In: Proceedings of the 31st Australian Conference on Human-Computer-Interaction. Fremantle, WA, Australia: Association for Computing Machinery, pp. 457–461, (2019)
Ringeval F, Sonderegger A, Sauer J, Lalanne D Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). Shanghai, China, IEEE, pp. 1–8, (2013)
Sarabadani S, Schudlo LC, Samadani A, Kushki A (2018) Physiological detection of affective states in children with autism spectrum disorder. IEEE Trans. Affect. Comput. 11:588–600. https://doi.org/10.1109/TAFFC.2018.2820049
Sarchiapone M, Gramaglia C, Iosue M, et al (2018) The association between electrodermal activity (EDA), depression and suicidal behaviour: a systematic review and narrative synthesis. BMC Psychiatry 18: https://doi.org/10.1186/s12888-017-1551-4
Sharma V, Prakash NR, Kalra P (2019) Audio-video emotional response mapping based upon electrodermal activity. Biomed. Signal. Process. Control. 47:324–333. https://doi.org/10.1016/j.bspc.2018.08.024
Shukla J, Barreda-Angeles M, Oliver J, et al Feature extraction and selection for emotion recognition from electrodermal activity. IEEE Trans. Affect. Comput. 1–1. https://doi.org/10.1109/TAFFC.2019.2901673, (2019)
Soleymani M, Pantic M, Pun T (2012) Multimodal emotion recognition in response to videos. IEEE Trans. Affect. Comput. 3:211–223. https://doi.org/10.1109/T-AFFC.2011.37
Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3:42–55. https://doi.org/10.1109/T-AFFC.2011.25
Song J, Davey C, Poulsen C, et al (2015) EEG source localization: sensor density and head surface coverage. J. Neurosci. Methods 256:9–21. https://doi.org/10.1016/j.jneumeth.2015.08.015
Susam BT, Akcakaya M, Nezamfar H, et al (2018) Automated pain assessment using electrodermal activity data and machine learning. Conf. Proc. Annu. Int. Conf. IEEE. Eng. Med. Biol. Soc. IEEE. Eng. Med. Biol. Soc. Annu. Conf. 2018:372–375. https://doi.org/10.1109/EMBC.2018.8512389
Syrjälä E, Jiang M, Pahikkala T, et al Skin conductance response to gradual-increasing experimental pain. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 3482–3485, (2019)
Torres CA, Orozco ÁA, Álvarez MA Feature selection for multimodal emotion recognition in the arousal-valence space. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp 4330–4333, (2013)
Urtnasan E, Park JU, Joo EY, Lee KJ (2018) Automated detection of obstructive sleep apnea events from a single-lead electrocardiogram using a convolutional neural network. J. Med. Syst. 42:1–8. https://doi.org/10.1007/s10916-018-0963-0
Vavrinsky E, Subjak J, Donoval M, et al (2020) Application of modern multi-sensor Holter in diagnosis and treatment. Sensors 20:2663. https://doi.org/10.3390/s20092663
Vieluf S, Reinsberger C, El Atrache R, et al (2020) Autonomic nervous system changes detected with peripheral sensors in the setting of epileptic seizures. Sci. Rep. 10:11560. https://doi.org/10.1038/s41598-020-68434-z
Walker ER, McGee RE, Druss BG (2015) Mortality in mental disorders and global disease burden implications: A systematic review and meta-analysis. JAMA Psychiatry 72:334–341. https://doi.org/10.1001/jamapsychiatry.2014.2502
Yang S, Yin Z, Wang Y, et al (2019) Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders. Comput. Biol. Med. 109:159–170. https://doi.org/10.1016/j.compbiomed.2019.04.034
Yasemin M, Sarıkaya MA, Ince G Emotional state estimation using sensor fusion of EEG and EDA. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 5609–5612, (2019)
Yeo M, Lim S, Yoon G (2018) Analysis of biosignals during immersion in computer games. J. Med. Syst. 42:3. https://doi.org/10.1007/s10916-017-0860-y
Zhang D, Yao L, Chen K, et al (2019) Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE Trans. Cybern. 50:3033–3044. https://doi.org/10.1109/TCYB.2019.2905157
Wei-Long Zheng, Bao-Liang Lu (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7:162–175. https://doi.org/10.1109/TAMD.2015.2431497
Zheng WL, Zhu JY, Peng Y, Lu BL EEG-based emotion classification using deep belief networks. In: Proceedings of the International Conference on Multimedia and Expo. pp. 1–6, (2014)
Zhu J, Chen N, Peng W (2019) Estimation of bearing remaining useful life based on multiscale convolutional neural network. IEEE Trans. Ind. Electron. 66:3208–3216. https://doi.org/10.1109/TIE.2018.2844856
Zheng WL, Zhu JY, Lu BL (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans. Affect. Comput.10:417–429. https://doi.org/10.1109/TAFFC.2017.2712143
Acknowledgments
The authors extend sincere thanks to the Ministry of Human Resource Development, (Government of India) for supporting this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical Approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
EFMI STC 2019 on ICT for Health Science Research, Thomas Deserno
Rights and permissions
About this article
Cite this article
Ganapathy, N., Veeranki, Y.R., Kumar, H. et al. Emotion Recognition Using Electrodermal Activity Signals and Multiscale Deep Convolutional Neural Network. J Med Syst 45, 49 (2021). https://doi.org/10.1007/s10916-020-01676-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10916-020-01676-6