Abstract
We present the i-Walk system, a novel framework for intelligent mobility assistance applications. The proposed system is capable of automatically understanding human activity, assessing mobility and rehabilitation progress, recognizing human intentions and communicating with the patients by giving meaningful feedback. To this end, multiple sensors, i.e. cameras, microphones, lasers, provide multimodal data in order to allow for user monitoring, while state-of-the-art and beyond algorithms have been developed and integrated into the system to enable recognition, interaction and assessment. More specifically, i-Walk performs in real-time and consists of four main sub-modules that interact automatically to provide speech understanding, activity recognition, mobility analysis and multimodal communication for seamless HRI. The i-Walk assessment system is evaluated on a database of healthy subjects and patients, who participated in carefully designed experimental scenarios that cover essential needs of rehabilitation. The presented results highlight the efficacy of the proposed framework to endow personal assistants with intelligence.
G. Chalvatzaki, P. Koutras, A. Tsiami—Equal Contribution
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
DIAPLASIS Rehabilitation Center. https://www.diaplasis.eu/
Google cloud speech-to-text. https://cloud.google.com/speech-to-text/
RASA. http://rasa.com
RASA. http://github.com/RasaHQ
Robot Operating System (ROS). http://www.ros.org/about-ros/
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2D human pose estimation: new benchmark and state of the art analysis. In: CVPR (2014)
Bocklisch, T., Faulkner, J., Pawlowski, N., Nichol, A.: Rasa: open source language understanding and dialogue management. In: NIPS (2017)
Cao, Z., Simon, T., Wei, S., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: CVPR (2017)
Chalvatzaki, G., Koutras, P., Hadfield, J., Papageorgiou, X.S., Tzafestas, C.S., Maragos, P.: LSTM-based network for human gait stability prediction in an intelligent robotic rollator. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 4225–4232. IEEE (2019)
Chalvatzaki, G., Papageorgiou, X.S., Maragos, P., Tzafestas, C.S.: User-adaptive human-robot formation control for an intelligent robotic walker using augmented human state estimation and pathological gait characterization. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6016–6022. IEEE (2018)
Chalvatzaki, G., Papageorgiou, X.S., Maragos, P., Tzafestas, C.S.: Learn to adapt to human walking: a model-based reinforcement learning approach for a robotic assistant rollator. IEEE Rob. Autom. Lett. 4(4), 3774–3781 (2019)
Chalvatzaki, G., Papageorgiou, X.S., Tzafestas, C.S., Maragos, P.: Augmented human state estimation using interacting multiple model particle filters with probabilistic data association. IEEE Rob. Autom. Lett. 3(3), 1872–1879 (2018)
Chang, M., Mou, W., Liao, C., Fu, L.: Design and implementation of an active robotic walker for Parkinson’s patients. In: Proceedings of SICE, pp. 2068–2073 (2012)
Chen, Y., Tian, Y., He, M.: Monocular human pose estimation: a survey of deep learning-based methods. Comput. Vis. Image Underst. 192, 102897 (2020)
Deriu, J., et al.: Survey on evaluation methods for dialogue systems. Artif. Intell. Rev. 1–56 (2020). https://doi.org/10.1007/s10462-020-09866-x
Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: CVPR (2015)
Dubowsky, S., et al.: PAMM-A robotic aid to the elderly for mobility assistance and monitoring: a“ helping-hand” for the elderly. In: ICRA (2000)
Efthymiou, N., Koutras, P., Filntisis, P.P., Potamianos, G., Maragos, P.: Multi-view fusion for action recognition in child-robot interaction. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 455–459. IEEE (2018)
Frizera-Neto, A., Ceres, R., Rocon, E., Pons, J.: Empowering and assisting natural human mobility: the Simbiosis walker. Int. J. Adv. Rob. Syst. 8(3), 29 (2011)
Guler, A., et al.: Human joint angle estimation and gesture recognition for assistive robotic vision. In: ECCV (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jenkins, S., Draper, H.: Care, monitoring, and companionship: views on care robots from older people and their carers. Int. J. Soc. Rob. 7(5), 673–683 (2015)
Kulyukin, V., Kutiyanawala, A., LoPresti, E., Matthews, J., Simpson, R.: iWalker: toward a rollator-mounted wayfinding system for the elderly. In: IEEE International Conference on RFID (2008)
Leo, M., Furnari, A., Medioni, G.G., Trivedi, M., Farinella, G.M.: Deep learning for assistive computer vision. In: ECCV (2018)
Lin, T., et al.: Microsoft COCO: common objects in context. In: ECCV (2014)
Liu, J., Wang, G., Hu, P., Duan, L., Kot, A.: Global context-aware attention LSTM networks for 3D action recognition. In: CVPR (2017)
Morris, A., et al.: A robotic walker that provides guidance. In: ICRA (2003)
Muro-De-La-Herran, A., Garcia-Zapirain, B., Mendez-Zorrilla, A.: Gait analysis methods: an overview of wearable and non-wearable systems, highlighting clinical applications. Sensors 14(2), 3362–3394 (2014)
Papageorgiou, X.S., Chalvatzaki, G., Dometios, A.C., Tzafestas, C.S., Maragos, P.: Intelligent assistive robotic systems for the elderly: two real-life use cases. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 360–365 (2017)
Papageorgiou, X.S., et al.: Advances in intelligent mobility assistance robot integrating multimodal sensory processing. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014. LNCS, vol. 8515, pp. 692–703. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07446-7_66
Paulo, J., Peixoto, P., Nunes, U.: ISR-AIWALKER: robotic walker for intuitive and safe mobility assistance and gait analysis. IEEE Trans. Hum. Mach. Syst. 47(6), 1110–1122 (2017)
Perry, J.: Gait Analysis: Normal and Pathological Function. Slack Incorporated (1992)
Piyathilaka, L., Kodagoda, S.: Human activity recognition for domestic robots. In: Field and Service Robotics, pp. 395–408 (2015)
Rezazadegan, F., Shirazi, S., Upcrofit, B., Milford, M.: Action recognition: from static datasets to moving robots. In: ICRA (2017)
Rodomagoulakis, I., et al.: Multimodal human action recognition in assistive human-robot interaction. In: ICASSP (2016)
Rodriguez-Losada, D., Matia, F., Jimenez, A., Galan, R., Lacey, G.: Implementing map based navigation in guido, the robotic SmartWalker. In: ICRA (2005)
Roitberg, A., Perzylo, A., Somani, N., Giuliani, M., Rickert, M., Knoll, A.: Human activity recognition in the context of industrial human-robot interaction. In: Signal and Information Processing Association Annual Summit and Conference (2014)
Shahroudy, A., Liu, J., Ng, T., Wang, G.: NTU RGB+ D: a large scale dataset for 3D human activity analysis. In: CVPR (2016)
Sharkey, A., Sharkey, N.: Children, the elderly, and interactive robots. IEEE Robot. Autom. Mag. 18(1), 32–38 (2011)
Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In: AAAI Conference on Artificial Intelligence (2017)
Stavropoulos, G., Giakoumis, D., Moustakas, K., Tzovaras, D.: Automatic action recognition for assistive robots to support mci patients at home. In: PETRA (2017)
Tinetti, M., Williams, T., Mayewski, R.: Fall risk index for elderly patients based on number of chronic disabilities. Am. J. Med. 80(3), 429–434 (1986)
Tsiami, A., Filntisis, P.P., Efthymiou, N., Koutras, P., Potamianos, G., Maragos, P.: Far-field audio-visual scene perception of multi-party human-robot interaction for children and adults. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6568–6572. IEEE (2018)
Tsiami, A., Koutras, P., Efthymiou, N., Filntisis, P.P., Potamianos, G., Maragos, P.: Multi3: multi-sensory perception system for multi-modal child interaction with multiple robots. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–8. IEEE (2018)
Veeriah, V., Zhuang, N., Qi, G.: Differential recurrent neural networks for action recognition. In: ICCV (2015)
Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: CVPR (2014)
Vrigkas, M., Nikou, C., Kakadiaris, I.A.: A review of human activity recognition methods. Front. Rob. AI 2, 28 (2015)
Wang, J., Liu, Z., Wu, Y., Yuan, J.: Learning actionlet ensemble for 3D human action recognition. PAMI 36(5), 914–927 (2013)
Wang, P., Yuan, C., Hu, W., Li, B., Zhang, Y.: Graph based skeleton motion representation and similarity measurement for action recognition. In: ECCV (2016)
Zanfir, M., Leordeanu, M., Sminchisescu, C.: The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. In: ICCV (2013)
Zhu, W., et al.: Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In: AAAI Conference on Artificial Intelligence (2016)
Zimmermann, C., Welschehold, T., Dornhege, C., Burgard, W., Brox, T.: 3D human pose estimation in RGBD images for robotic task learning. In: ICRA (2018)
Acknowledgment
This research has been co-financed by the European Union and Greek national funds (project code:T1EDK-01248, acronym: i-Walk).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 37206 KB)
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chalvatzaki, G., Koutras, P., Tsiami, A., Tzafestas, C.S., Maragos, P. (2020). i-Walk Intelligent Assessment System: Activity, Mobility, Intention, Communication. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12538. Springer, Cham. https://doi.org/10.1007/978-3-030-66823-5_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-66823-5_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66822-8
Online ISBN: 978-3-030-66823-5
eBook Packages: Computer ScienceComputer Science (R0)