Skip to main content

Advertisement

Springer Nature Link
Log in

Emotion Recognition Using Electrodermal Activity Signals and Multiscale Deep Convolutional Neural Network

  • Systems-Level Quality Improvement
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

In this work, an attempt has been made to classify emotional states using electrodermal activity (EDA) signals and multiscale convolutional neural networks. For this, EDA signals are considered from a publicly available “A Dataset for Emotion Analysis using Physiological Signals” (DEAP) database. These signals are decomposed into multiple-scales using the coarse-grained method. The multiscale signals are applied to the Multiscale Convolutional Neural Network (MSCNN) to automatically learn robust features directly from the raw signals. Experiments are performed with the MSCNN approach to evaluate the hypothesis (i) improved classification with electrodermal activity signals, and (ii) multiscale learning captures robust complementary features at a different scale. Results show that the proposed approach is able to differentiate various emotional states. The proposed approach yields a classification accuracy of 69.33% and 71.43% for valence and arousal states, respectively. It is observed that the number of layers and the signal length are the determinants for the classifier performance. The performance of the proposed approach outperforms the single-layer convolutional neural network. The MSCNN approach provides end-to-end learning and classification of emotional states without additional signal processing. Thus, it appears that the proposed method could be a useful tool to assess the difference in emotional states for automated decision making.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Anusha AS, Sukumaran P, Sarveswaran V, et al Electrodermal activity based pre-surgery stress detection using a wrist wearable. IEEE J. Biomed. Health Inform. 1–1. https://doi.org/10.1109/JBHI.2019.2893222, (2019)

  2. Affanni A, Bernardini R, Piras A, et al (2018) Driver’s stress detection using skin potential response signals. Measurement 122:264–274. https://doi.org/10.1016/j.measurement.2018.03.040

    Article  Google Scholar 

  3. Nazarov AI (2018) Comparison of frequency components of EEG and skin potential (quantitative analysis). Exp. Psychol. 11:95–109. https://doi.org/10.17759/exppsy.2018110207

  4. Akay A, Hess H (2019) Deep learning: current and emerging applications in medicine and technology. IEEE J. Biomed. Health Inform. 23:906–920. https://doi.org/10.1109/JBHI.2019.2894713, (2019)

  5. Alarcao SM, Fonseca MJ (2017) Emotions recognition using EEG signals: a survey. IEEE Trans. Affect. Comput. 10:374–393. https://doi.org/10.1109/TAFFC.2017.2714671

  6. Albasri A, Abdali-Mohammadi F, Fathi A (2019) EEG electrode selection for person identification thru a genetic-algorithm method. J. Med. Syst. 43:297. https://doi.org/10.1007/s10916-019-1364-8

    Article  PubMed  Google Scholar 

  7. Amin MR, Faghih RT (2019) Sparse deconvolution of electrodermal activity via continuous-time system identification. IEEE. Trans. Biomed. Eng. 66:2585–95. https://doi.org/10.1109/TBME.2019.2892352

  8. Arapakis I, Barreda-Ángeles M, Pereda-Baños A (2019) Interest as a proxy of engagement in news reading: spectral and entropy analyses of EEG activity patterns. IEEE Trans. Affect. Comput. 10:100–114. https://doi.org/10.1109/TAFFC.2017.2682089

    Article  Google Scholar 

  9. Boucsein W (2012) Electrodermal Activity, 2nd ed. Springer, New York

    Book  Google Scholar 

  10. Brishtel I, Khan AA, Schmidt T, et al (2020) Mind wandering in a multimodal reading setting: behavior analysis; automatic detection using eye-tracking and an EDA sensor. Sensors 20:2546. https://doi.org/10.3390/s20092546

    Article  Google Scholar 

  11. Charlson F, van Ommeren M, Flaxman A, et al. (2019) New WHO prevalence estimates of mental disorders in conflict settings: a systematic review and meta-analysis. The Lancet S0140673619309341. https://doi.org/10.1016/S0140-6736(19)30934-1

  12. Daly I, Williams D, Hwang F, et al (2019) Electroencephalography reflects the activity of sub-cortical brain regions during approach-withdrawal behaviour while listening to music. Sci. Rep. 9:9415. https://doi.org/10.1038/s41598-019-45105-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Fang WC, Wang KY, Fahier N, et al (2019) Development and validation of an EEG-based real-time emotion recognition system using edge AI Computing Platform With Convolutional Neural Network System-on-chip design. IEEE J. Emerg. Sel. Top. Circuits. Syst. 9:645–657. https://doi.org/10.1109/JETCAS.2019.2951232

  14. Feng H, Golshan HM, Mahoor MH (2018) A wavelet-based approach to emotion classification using EDA signals. Expert. Syst. Appl. 112:77–86. https://doi.org/10.1016/j.eswa.2018.06.014

    Article  Google Scholar 

  15. Ganapathy N, Swaminathan R, Deserno TM (2018) Deep Learning on 1-D Biosignals: a taxonomy-based survey. Yearb. Med. Inform. 27:98–109. https://doi.org/10.1055/s-0038-1667083

    Article  PubMed  PubMed Central  Google Scholar 

  16. Gravina R, Li Q (2019) Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion. Inf. Fusion 48:1–10. https://doi.org/10.1016/j.inffus.2018.08.001

    Article  Google Scholar 

  17. Greco A, Valenza G, Lanata A, et al (2014) Electrodermal activity in bipolar patients during affective elicitation. IEEE J. Biomed. Health Inform. 18:1865–1873. https://doi.org/10.1109/JBHI.2014.2300940

    Article  PubMed  Google Scholar 

  18. Greco A, Valenza G, Scilingo EP (2016) Advances in electrodermal activity processing with applications for mental health. Springer International Publishing, Cham

    Book  Google Scholar 

  19. Greco A, Valenza G, Lanata A, et al (2016) cvxEDA: a convex optimization approach to electrodermal activity processing. IEEE Trans. Biomed. Eng. 63:797–804. https://doi.org/10.1109/TBME.2015.2474131

    Article  PubMed  Google Scholar 

  20. Greco A, Valenza G, Scilingo EP Brain dynamics during arousal-dependent pleasant/unpleasant visual elicitation: an electroencephalographic study on the circumplex model of affect. IEEE Trans. Affect. Comput. 1–1. https://doi.org/10.1109/TAFFC.2018.2879343, (2018)

  21. Hamada M, Zaidan BB, Zaidan AA (2018) A systematic review for human EEG brain signals based emotion classification, feature extraction, brain condition, group comparison. J. Med. Syst. 42:162. https://doi.org/10.1007/s10916-018-1020-8

    Article  PubMed  Google Scholar 

  22. Hossain MS, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Inf. Fusion 49:69–78. https://doi.org/10.1016/j.inffus.2018.09.008

    Article  Google Scholar 

  23. Jiang G, He H, Yan J, Xie P (2019) Multiscale convolutional neural networks for fault diagnosis of wind turbine Gearbox. IEEE Trans. Ind. Electron. 66:3196–3207 https://doi.org/10.1109/TIE.2018.2844805

    Article  Google Scholar 

  24. Koelstra S, Muhl C, Soleymani M, et al (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3:18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  25. Kroupi E, Vesin JM, Ebrahimi T Phase-amplitude coupling between EEG and EDA while experiencing multimedia content. In: Proceedings of the Humaine Association Conference on Affective Computing and Intelligent Interaction. Geneva: IEEE, pp. 865–870, (2013)

  26. Kroupi E, Vesin JM, Ebrahimi T (2014) Implicit affective profiling of subjects based on physiological data coupling. Brain-Comput. Interfaces 1:85–98. https://doi.org/10.1080/2326263X.2014.912882

  27. Liu G (2014) An entropy measure of emotional arousal via skin conductance response. J. Fiber. Bioeng. Inform. 7:67–80. https://doi.org/10.3993/jfbi03201406

    Article  Google Scholar 

  28. Liu Z, Xie Q, Wu M, et al (2018) Electroencephalogram emotion recognition based on empirical mode decomposition and optimal feature selection. IEEE Trans. Cogn. Dev. Syst. 11:517–26. https://doi.org/10.1109/TCDS.2018.2868121

  29. Liu M, Xu C, Luo Y, et al (2018) Cost-sensitive feature selection by optimizing F-measures. IEEE Trans. Image. Process. 27:1323–1335. https://doi.org/10.1109/TIP.2017.2781298

    Article  Google Scholar 

  30. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern. Anal. Mach. Intell. 23:1175–1191. https://doi.org/10.1109/34.954607

    Article  Google Scholar 

  31. Posada-Quintero HF, Bolkhovsky JB (2019) Machine Learning models for the identification of cognitive tasks using autonomic reactions from heart rate variability and electrodermal activity. Behav. Sci. 9:45. https://doi.org/10.3390/bs9040045

    Article  Google Scholar 

  32. Posada-Quintero HF, Chon KH Phasic component of electrodermal activity is more correlated to brain activity than tonic component. In: 2019 IEEE EMBS International Conference on Biomedical Health Informatics (BHI). pp. 1–4, (2019)

  33. Rahman JS, Hossain MZ, Gedeon T Measuring observers’ EDA responses to emotional videos. In: Proceedings of the 31st Australian Conference on Human-Computer-Interaction. Fremantle, WA, Australia: Association for Computing Machinery, pp. 457–461, (2019)

  34. Ringeval F, Sonderegger A, Sauer J, Lalanne D Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). Shanghai, China, IEEE, pp. 1–8, (2013)

  35. Sarabadani S, Schudlo LC, Samadani A, Kushki A (2018) Physiological detection of affective states in children with autism spectrum disorder. IEEE Trans. Affect. Comput. 11:588–600. https://doi.org/10.1109/TAFFC.2018.2820049

  36. Sarchiapone M, Gramaglia C, Iosue M, et al (2018) The association between electrodermal activity (EDA), depression and suicidal behaviour: a systematic review and narrative synthesis. BMC Psychiatry 18: https://doi.org/10.1186/s12888-017-1551-4

  37. Sharma V, Prakash NR, Kalra P (2019) Audio-video emotional response mapping based upon electrodermal activity. Biomed. Signal. Process. Control. 47:324–333. https://doi.org/10.1016/j.bspc.2018.08.024

    Article  Google Scholar 

  38. Shukla J, Barreda-Angeles M, Oliver J, et al Feature extraction and selection for emotion recognition from electrodermal activity. IEEE Trans. Affect. Comput. 1–1. https://doi.org/10.1109/TAFFC.2019.2901673, (2019)

  39. Soleymani M, Pantic M, Pun T (2012) Multimodal emotion recognition in response to videos. IEEE Trans. Affect. Comput. 3:211–223. https://doi.org/10.1109/T-AFFC.2011.37

    Article  Google Scholar 

  40. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3:42–55. https://doi.org/10.1109/T-AFFC.2011.25

    Article  Google Scholar 

  41. Song J, Davey C, Poulsen C, et al (2015) EEG source localization: sensor density and head surface coverage. J. Neurosci. Methods 256:9–21. https://doi.org/10.1016/j.jneumeth.2015.08.015

    Article  PubMed  Google Scholar 

  42. Susam BT, Akcakaya M, Nezamfar H, et al (2018) Automated pain assessment using electrodermal activity data and machine learning. Conf. Proc. Annu. Int. Conf. IEEE. Eng. Med. Biol. Soc. IEEE. Eng. Med. Biol. Soc. Annu. Conf. 2018:372–375. https://doi.org/10.1109/EMBC.2018.8512389

    Article  Google Scholar 

  43. Syrjälä E, Jiang M, Pahikkala T, et al Skin conductance response to gradual-increasing experimental pain. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 3482–3485, (2019)

  44. Torres CA, Orozco ÁA, Álvarez MA Feature selection for multimodal emotion recognition in the arousal-valence space. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp 4330–4333, (2013)

  45. Urtnasan E, Park JU, Joo EY, Lee KJ (2018) Automated detection of obstructive sleep apnea events from a single-lead electrocardiogram using a convolutional neural network. J. Med. Syst. 42:1–8. https://doi.org/10.1007/s10916-018-0963-0

  46. Vavrinsky E, Subjak J, Donoval M, et al (2020) Application of modern multi-sensor Holter in diagnosis and treatment. Sensors 20:2663. https://doi.org/10.3390/s20092663

    Article  CAS  Google Scholar 

  47. Vieluf S, Reinsberger C, El Atrache R, et al (2020) Autonomic nervous system changes detected with peripheral sensors in the setting of epileptic seizures. Sci. Rep. 10:11560. https://doi.org/10.1038/s41598-020-68434-z

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Walker ER, McGee RE, Druss BG (2015) Mortality in mental disorders and global disease burden implications: A systematic review and meta-analysis. JAMA Psychiatry 72:334–341. https://doi.org/10.1001/jamapsychiatry.2014.2502

    Article  PubMed  PubMed Central  Google Scholar 

  49. Yang S, Yin Z, Wang Y, et al (2019) Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders. Comput. Biol. Med. 109:159–170. https://doi.org/10.1016/j.compbiomed.2019.04.034

    Article  PubMed  Google Scholar 

  50. Yasemin M, Sarıkaya MA, Ince G Emotional state estimation using sensor fusion of EEG and EDA. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 5609–5612, (2019)

  51. Yeo M, Lim S, Yoon G (2018) Analysis of biosignals during immersion in computer games. J. Med. Syst. 42:3. https://doi.org/10.1007/s10916-017-0860-y

    Article  Google Scholar 

  52. Zhang D, Yao L, Chen K, et al (2019) Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE Trans. Cybern. 50:3033–3044. https://doi.org/10.1109/TCYB.2019.2905157

  53. Wei-Long Zheng, Bao-Liang Lu (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Ment. Dev. 7:162–175. https://doi.org/10.1109/TAMD.2015.2431497

    Article  Google Scholar 

  54. Zheng WL, Zhu JY, Peng Y, Lu BL EEG-based emotion classification using deep belief networks. In: Proceedings of the International Conference on Multimedia and Expo. pp. 1–6, (2014)

  55. Zhu J, Chen N, Peng W (2019) Estimation of bearing remaining useful life based on multiscale convolutional neural network. IEEE Trans. Ind. Electron. 66:3208–3216. https://doi.org/10.1109/TIE.2018.2844856

    Article  Google Scholar 

  56. Zheng WL, Zhu JY, Lu BL (2019) Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans. Affect. Comput.10:417–429. https://doi.org/10.1109/TAFFC.2017.2712143

Download references

Acknowledgments

The authors extend sincere thanks to the Ministry of Human Resource Development, (Government of India) for supporting this study.

Author information

Authors and Affiliations

  1. Biomedical Engineering Group, Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai, India

    Nagarajan Ganapathy, Yedukondala Rao Veeranki, Himanshu Kumar & Ramakrishnan Swaminathan

Authors
  1. Nagarajan Ganapathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yedukondala Rao Veeranki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Himanshu Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ramakrishnan Swaminathan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nagarajan Ganapathy.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

EFMI STC 2019 on ICT for Health Science Research, Thomas Deserno

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ganapathy, N., Veeranki, Y.R., Kumar, H. et al. Emotion Recognition Using Electrodermal Activity Signals and Multiscale Deep Convolutional Neural Network. J Med Syst 45, 49 (2021). https://doi.org/10.1007/s10916-020-01676-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-020-01676-6

Keywords