Skip to main content
Springer Nature Link
Log in

Human behaviour in multimodal interaction: main effects of civic action and interpersonal and problem-solving skills

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Metacognitive skill training may rest within any kind of social interaction that requires awareness of what an individual and others think, in social, educational and organizational settings alike. This work is an extensive study of multimodal application interaction (virtual agent, spoken dialogue, visual communication of progress) for metacognitive skill training via negotiation skill training scenarios. Human behaviour, as effected by civic action and interpersonal and problem-solving skill training, is investigated through interaction sessions with a virtual agent on multimodal multiparty negotiation. This work reports on the results of the user-system evaluation sessions involving 41 participants before and after interaction with the system, integrating macro- (dialogue system performance) and micro- (metacognitive-related and individual- and community-level-related attitudes and skills) factors. Findings indicate significant and positive relationships between user and system evaluation questions after interaction with the dialogue system and between self-efficacy, self-regulation, individual readiness to change, mastery goal orientation, interpersonal and problem-solving skills and civic action before and after the interaction experience. Implications, limitations and further research issues are discussed in light of context of the multimodal interaction and its effects on the human behaviour during metacognitive skill training.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Agarwal R, Viswanathan S, Animesh A (2012) The “new” world of negotiations: interactions mediated by information technology. In: The psychology of negotiations in the 21st century workplace, new challenges and new solutions, vol 14, pp 399–440

  • Alexandersson J, Girenko A, Spiliotopoulos D, Petukhova V, Klakow D, Koryzis D, Taatgen N, Specht M, Campbell N, Aretoulaki M, Stricker A, Gardner M (2014) Metalogue: a multiperspective multimodal dialogue system with metacognitive abilities for highly adaptive and flexible dialogue management. In: 2014 10th international conference on intelligent environments (IE14), pp 365–368

  • Ames C (1992) Classrooms: goals, structures, and student motivation. J Educ Psychol 84(3):261–271

    Google Scholar 

  • Anglin AE, Johnson-Pynn JS, Johnson LR (2012) Ethnic identity and civic attitudes in Latino and Caucasian youth. J Youth Stud 15(5):621–643

    Google Scholar 

  • Armenakis A, Harris S, Mossholder K (1993) Creating readiness for organizational change. Hum Relat 46(6):681–702

    Google Scholar 

  • Aspinwall LG (2004) Dealing with adversity: self-regulation, coping, adaptation and health. Appl Soc Psychol Perspect Soc Psychol Ser 4(1):3–27

    Google Scholar 

  • Bandura A (1991) Social cognitive theory of self-regulation. Organ Behav Hum Decis Process 50:248–287

    Google Scholar 

  • Bandura A (1997) Self-efficacy: towards a unifying theory of behavioural change. Psychol Rev 84:191–215

    Google Scholar 

  • Beaumont E (2010) Political agency and empowerment: pathways for developing a sense of political efficacy in young adults. In: Handbook of research on civic engagement in youth, pp 525–558

  • Bolt R (1980) Put that there: voice and gesture at the graphics interface. ACM Comput Gr 14(3):262–270

    MathSciNet  Google Scholar 

  • Bringle R, Velo P (1998) Attributions about misery: a social psychological analysis. In: With service in mind: concepts and models for service-learning in psychology, pp 51–67

  • Caridakis G, Raouzaiou A, Bevacqua E, Mancini M, Karpouzis K, Malatesta L, Pelachaud C (2007) Virtual agent multimodal mimicry of humans. Lang Resour Eval 41(3–4):367–388

    Google Scholar 

  • Carver C, Scheier MF (1999) Themes and issues in the self-regulation of behaviour. Adv Soc Cognit Perspect Behav Self-Regulation 12:1–105

    Google Scholar 

  • Cassell J (2001) Embodied conversational agents: representation and intelligence in user interfaces. AI Mag 22(4):67–83

    Google Scholar 

  • Cassell J, Bickmore T (2000) External manifestations of trustworthiness of the interface. Commun ACM 43(12):50–56

    Google Scholar 

  • Chae SW, Lee KC, Seo YW (2016) Exploring the effect of avatar trust on learners’ perceived participation intentions in an e-learning environment. Int J Hum Comput Interact 32:373–393

    Google Scholar 

  • Cohen A, Chaffee B (2013) The relationship between adolescents’ civic knowledge, civic attitude, and civic behaviour and their self-reported future likelihood of voting. Educ Citizsh Soc Justice 8(1):43–57

    Google Scholar 

  • Diehl M, Semegon A, Schwarzer R (2006) Assessing attention control in goal pursuit: a component of dispositional self-regulation. J Pers Assess 86(3):306–317

    Google Scholar 

  • Domagk S (2010) Do pedagogical agents facilitate learner motivation and learning outcomes? The role of the appeal of agent’s appearance and voice. J Med Psychol 22(2):84–97

    Google Scholar 

  • Flavell H (1987) Speculations about the nature and development of metacognition. Metacognition Motiv Underst 52:21–29

    Google Scholar 

  • Gal Y, Kraus S, Gelfand M, Khashan H, Salmon E (2011) An adaptive agent for negotiating with people in different cultures. ACM Trans Intell Syst Technol 3(1):1–24

    Google Scholar 

  • Giles D, Eyler J (1994) The impact of a college community service laboratory on students’ personal, social and cognitive outcomes. J Adolesc 17:327–339

    Google Scholar 

  • Gama S, Barata G, Gonçalves D, Prada R, Paiva A (2011) SARA: social affective relational agent: a study on the role of empathy in artificial social agents. In: D’Mello S, Graesser A, Schuller B, Martin JC (eds) Affective computing and intelligent interaction. ACII 2011. Lecture Notes in Computer Science, vol 6974. Springer, Berlin, Heidelberg, pp 507–516

  • Goya-Martinez M (2016) The emulation of emotions in artificial intelligence: another step into anthropomorphism. Emot Technol Des 8:171–185

    Google Scholar 

  • Gratch J, Rickel J, André E, Cassell J, Petajan E, Badler N (2002) Creating interactive virtual humans: some assembly required. IEEE Intell Syst 17(4):54–63

    Google Scholar 

  • Grivokostopoulou F, Paraskevas M, Perikos I, Nicolic S, Kovas K, Hatzilygeroudis I (2018) Examining the impact of pedagogical agents on students learning experience in virtual worlds. In: 2018 international conference on teaching, assessment and learning for engineering (TALE), IEEE, pp 602–607

  • Grosssman R, Thayer A, Shuffler M, Shawn C, Salas E (2015) Critical social thinking: A conceptual model and insights for training. Organ Psychol Rev 5(2):99–125

    Google Scholar 

  • Guo Y, Goh D (2015) Affect in embodied pedagogical agents: meta-analytic review. J Educ Comput Res 53(1):124–149

    Google Scholar 

  • Heidig S, Clarebout G (2011) Do pedagogical agents make a difference to student motivation and learning? Educ Res Rev 6(1):27–54

    Google Scholar 

  • Henderson M, Huang H, Grant L, Henderson L (2009) Language acquisition in second life: improving self-efficacy beliefs. In: 26th international conference on innovation, practice and research in the use of educational technologies in tertiary education, ASCILITE, pp 464–474

  • Hollingworth R, McLoughlin C (2001) Developing science students’ metacognitive problem-solving skills online. Aust J Educ Technol 17(1):50–63

    Google Scholar 

  • Holmes J (2007) Designing agents to support learning by explaining. Comput Educ 48:523–547

    Google Scholar 

  • Holt D, Armenakis A, Field H, Harris S (2007) Readiness for organizational change. J Appl Behav Sci 43(2):232–255

    Google Scholar 

  • Howell AJ, Watson DC (2007) Procrastination: associations with achievement goal orientation and learning strategies. Personal Individ Differ 43:167–178

    Google Scholar 

  • Kahne J, Middaugh E (2008) Democracy for some: the civic opportunity gap in high school. In: Engaging young people in civic life, pp 29–58

  • Kaptein M, Markopoulos P, deRuyter B, Aarts E (2011) Two acts of social intelligence: the effects of mimicry and social praise on the evaluation of an artificial agent. AI Soc 26:261–273

    Google Scholar 

  • Khan R, DeAngeli A (2009) The attractiveness stereotype in the evaluation of embodied conversational agents. In: 2009 international conference on human-computer interaction, Springer, pp 405–417

  • Khan R, Satcliffe A (2014) Attractive agents are more persuasive”. Int J Hum Comput Interact 30(2):142–150

    Google Scholar 

  • King K (1999) Metacognition in the composition classroom. Idaho State University, Idaho

    Google Scholar 

  • Klassen R, Ang R, Chong W, Krawchuk L, Huan V, Wong I, Yeo L (2010) Academic procrastination in two settings: Motivation correlates, behavioral patterns, and negative impact of procrastination in Canada and Singapore. Appl Psychol 59(3):361–379

    Google Scholar 

  • Kolb D (1984) Experiential learning: experience as the source of learning and development. Prentice-Hall Inc., Englewood Cliffs

    Google Scholar 

  • Kolb A, Kolb D (2006) A review of multidisciplinary application of experiential learning theory in higher education. In: Learning styles and learning: a key to meeting the accountability demands in education, pp 45–91

  • Komlao P. (2015) Gap between intention and action: the role of self-efficacy, self-regulation and self-continuity in predicting academic procrastination. Psychology Ph.D. thesis, National College of Ireland

  • Koryzis D, Svolopoulos V, Spiliotopoulos D (2016) Metalogue: a multimodal learning journey. In: 2016 ACM 9th international conference on pervasive technologies related to assistive environments, ACM, pp 1–4

  • Kraus S (2001) Automated negotiation and decision making in multiagent environments. Multi Ag Syst Appl 2086:150–172

    MATH  Google Scholar 

  • Kurzweil R (1999) The age of spiritual machines: When computers exceeded human intelligence. Penguin, New York

    Google Scholar 

  • Lester J, Converse C, Kahler S, Barlow S, Stone B, Bhogal R (1997) The persona effect: affective impact of animated pedagogical agents. In: 1997 ACM SIGCHI conference on human factors in computing systems, pp 359–366

  • Levinson M (2010) The civic empowerment gap: defining the problem and locating solutions. In: Handbook of research on civic engagement in youth, pp 331–361

  • Liew TW, Tan SM, Jayothisa C (2013) The effects of peer-like and expert-like pedagogical agents on learners’ agent perceptions, task-related attitudes, and learning achievement. Educ Technol Soc 16(4):275–286

    Google Scholar 

  • Lin R, Kraus S (2010) Can automated agents proficiently negotiate with humans? Commun ACM 53(1):79–88

    Google Scholar 

  • Liu K, Tolins J, Fox J, Neff M, Walker M (2016) Two techniques for assessing virtual agent personality. IEEE Trans Affect Comput 7(1):94–105

    Google Scholar 

  • Low P, Ang S (2011) Information communication technology (ICT) for negotiations. J Res Int Bus Manag 1(6):183–196

    Google Scholar 

  • Maes S, Karoly P (2005) Self-regulation assessment and intervention in physical health and illness: a review. Appl Psychol Int Rev 54(2):267–329

    Google Scholar 

  • Markopoulos P, de Ruyter B, Saini P, van Breemen A (2005) Case study: bringing social intelligence into home dialogue systems. ACM Interact 12(4):37–44

    Google Scholar 

  • Moely B, Santulli K, Obach M (1995) Strategy instruction, metacognition, and motivation in the elementary school classroom. In: Memory performance and competencies: issues in growth and development, Erlbaum, Hillsdale

  • Moely B, Mercer S, Illustre V, Miron D, McFarland M (2002) Psychometric properties and correlates of the civic attitudes and skills questionnaire (CASQ): a measure of students’ attitudes related to service-learning. Mich J Community Serv Learn 8:15–26

    Google Scholar 

  • Morrison E, Phelps C (1999) Taking charge at work: extra role efforts to initiate workplace change. Acad Manag J 42:403–419

    Google Scholar 

  • Oviatt S (1999) Moving from traditional interfaces toward interfaces offering users greater expressive power, naturalness, and portability. In: Communications of the ACM Ten Myths of multimodal interaction, vol 42, no 11, pp 74–81

  • Pascarella E, Ethington C, Smart J (1988) The influence of college on humanitarian/civic involvement values. J Higher Educ 59:412–437

    Google Scholar 

  • Pelachaud C (2009) Modelling multimodal expression of emotion in a virtual agent. Philos Trans R Soc 364:3539–3548

    Google Scholar 

  • Pickard MD, Burgoon JK, Derrick DC (2014) Toward an objective linguistic-based measure of perceived embodied conversational agent power and likeability. Int J Hum Comput Interact 30:495–516

    Google Scholar 

  • Pintrich P (2003) A motivational science perspective on the role of student motivation in learning and teaching contexts. J Educ Psychol 95(4):667–686

    Google Scholar 

  • Prat-Sala M, Redford P (2010) The interplay between motivation, self-efficacy, and approaches to studying. Br J Educ Psychol 80(2):283–305

    Google Scholar 

  • Pressley M, Ghatala E (1990) Self-regulated learning: monitoring learning from text. Educ Psychol 25:19–33

    Google Scholar 

  • Qiu L, Benbasat I (2009) Evaluating anthropomorphic product recommendation agents: a social relationship perspective to designing information systems. J Manag Inf Syst 25(4):145–181

    Google Scholar 

  • Reeves B, Nass C (1996) The media equation: how people treat computers, television and new media like real people and places. Cambridge University Press, Cambridge

    Google Scholar 

  • Schraw G, Dennison R (1994) Assessing metacognitive awareness. Contemp Educ Psychol 19:460–475

    Google Scholar 

  • Schoenfeld A (1987) Cognitive science and mathematics education. Erlbaum, Hillsdale

    Google Scholar 

  • Schwarzer R, Diehl M, Schmitz G (1999) Self-regulation scale. https://www.fu-berlin.de/gesund/skalen. Accessed 5 Oct 2019

  • Schwienhorst K (2002) Why virtual, why environments? Implementing virtual reality concepts in computer-assisted language learning. Simul Gaming 33(2):196–209

    Google Scholar 

  • Short J, Williams E, Christie B (1976) The social psychology of telecommunications. Willey, London

    Google Scholar 

  • Siegel DJ (2007) The mindful brain: reflection and attunement in the cultivation of well-being. Norton and Company, New York

    Google Scholar 

  • Soliman M, Guetl C (2014) Evaluation study and results of intelligent pedagogical agent-led learning scenarios in a virtual world. In: 2014 international convention on information and communication technology, electronics and microelectronics, IEEE, pp 914–918

  • Speier C, Frese M (1997) Generalized self-efficacy as a mediator and moderator between control and complexity at work and personal initiative: a longitudinal field study in East Germany. Hum Perform 10:171–192

    Google Scholar 

  • Sproull L, Subramani M, Kiesler S, Walker J, Waters K (2004) When the interface is a face. Hum Comput Interact 17:71–88

    Google Scholar 

  • Sumi K (2016) Spoken dialog agent applications using emotional expressions. Emot Technol Des 5:83–102

    Google Scholar 

  • Theodosiou A, Papaioannou A (2006) Motivational climate, achievement goals and metacognitive activity in physical education and exercise involvement in out-of-school settings. Psychol Sport Exerc 7:361–379

    Google Scholar 

  • Tims M, Bakker A (2010) Job crafting: Towards a new model of individual job redesign. SA J Ind Psychol 36(2):1–9

    Google Scholar 

  • Vakola M (2014) What’s in there for me? Individual readiness to change and the perceived impact of organizational change. Leadersh Organ Dev J 35(3):195–209

    Google Scholar 

  • Walker J, Sproull J, Subramani R (1994) Using a human face in the interface. In: 1994 SIGCHI conference on human factors in computing systems, ACM, pp 85–91

  • Wang TJ (2011) Educating avatars: on virtual worlds and pedagogical intent. Teach High Educ 16(6):617–628

    Google Scholar 

  • Wilson K, Narayan A (2016) Relationships among individual task self-efficacy, self-regulated learning strategy use and academic performance in a computer supported collaborative learning environment. Educ Psychol 36(2):236–253

    Google Scholar 

  • Yee N, Bailenson J, Rickertsen K (2007) A meta-analysis of the impact of the inclusion and realism of human-like faces on user experience in interfaces. In: 2007 SIGCHI conference on human factors in computing systems, ACM, pp 1–10

  • Yung H, Paas F (2015) Effects of cueing by a pedagogical agent in an instructional animation: a cognitive load approach. Educ Technol Soc 18(3):153–160

    Google Scholar 

  • Zanbaka C, Goolkasian P, Hodges L (2006) Can a virtual persuade you? The role of gender and realism in speaker persuasiveness. In: 2006 SIGCHI conference on human factors in computing systems, ACM, pp 1153–1162

  • Zimmerman B (1989) A social cognitive view of self-regulated academic learning. J Educ Psychol 81:329–339

    Google Scholar 

  • Zimmerman B (2006) Development and adaptation of expertise: The role of self-regulatory processes and beliefs. In: The Cambridge handbook of expertise and expert performance, pp 705–722

Download references

Author information

Authors and Affiliations

  1. Department of Informatics and Telecommunications, University of the Peloponnese, Tripoli, Greece

    Eleni Makri, Dimitris Spiliotopoulos & Costas Vassilakis

  2. Department of Informatics and Telecommunications, University of Athens, Athens, Greece

    Dionisis Margaris

Authors
  1. Eleni Makri
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Dimitris Spiliotopoulos
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Costas Vassilakis
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Dionisis Margaris
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dimitris Spiliotopoulos.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

The Table 2 questions are presented below:

Q1::

Do you think the actions of the system were correct?

Q2::

Did the interaction with the system made sense to you?

Q3::

Did the system communicate enough information to you?

Q4::

Did the system communicate too much information to you?

Q5::

Was the information provided by the system to you useful?

Q6::

Was the system communication to you timely?

Q7::

Was it easy to complete tasks in your interaction?

Q8::

Was the pace of interaction fast enough to feel right?

Q9::

Would you say that the interaction with the system was natural?

Q10::

Did you know what you could say at each point of the dialogue?

Q11::

Would you say that your interaction with the system was natural?

Q12::

Are you confident you know enough about the functionalities and the information found in Metalogue so you would be able to use it on your own?

Q13::

How easy was to interact with the system?

Q14::

How natural was to interact with system?

Q15::

Do you think that the concept is an interesting idea?

Q16::

Do you find the setup of the setup of the system intimidating?

Q17::

Would you use the system again if it was an integral part of your training routine?

Q18::

Do you think that the system has the potential to become a great skills training application?

Q19::

Would you use a simplified version of the system with only the content or functionality you find it interesting?

Q20::

Was the feedback provided “during” the interaction valuable to you?

Q21::

Was the feedback provided “after” the interaction valuable to you?

Q22::

Did the feedback that was provided “during” the interaction help you to become more aware of your performance?

Q23::

Did the feedback that was provided “after” the interaction help you to become more aware of your performance?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Makri, E., Spiliotopoulos, D., Vassilakis, C. et al. Human behaviour in multimodal interaction: main effects of civic action and interpersonal and problem-solving skills. J Ambient Intell Human Comput 11, 5991–6006 (2020). https://doi.org/10.1007/s12652-020-01846-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-01846-x

Keywords