Skeleton-Based Human Action Recognition via Multi-Knowledge Flow Embedding Hierarchically Decomposed Graph Convolutional Network

Abstract

Skeleton-based action recognition has great potential and extensive application scenarios such as virtual reality and human-robot interaction due to its robustness under complex background and different viewing angles. Recent approaches converted skeleton sequences into spatial-temporal graphs and adopted graph convolutional networks to extract features. Multi-modality recognition and attention mechanisms have also been proposed to boost accuracy. However, the complex feature extraction modules and multi-stream ensemble have increased computational complexity significantly. Thus, most existing methods failed to meet lightweight industrial requirements and lightweight methods were unable to output sufficiently accurate results. To tackle the problem, we propose multi-knowledge flow embedding graph convolutional network, which can achieve high accuracy while maintaining lightweight. We first construct multiple knowledge flows by extracting diverse features from different hierarchically decomposed graphs. Each knowledge flow not only contains information on target class, but also stores profound information for non-target class. Inspired by knowledge distillation, we designed a novel multi-knowledge flow embedding module, which can effectively embed the knowledge into a student model without increasing model complexity. Moreover, student model can be enhanced dramatically by learning simultaneously from complementary knowledge flows. Extensive experiments on authoritative datasets demonstrate that our approach outperforms state-of-the-art with significantly lower computational complexity.