Face recognition is a hot research topic with a variety of application possibilities, including video surveillance and mobile payment. It has been well researched in traditional computer vision community. However, new research issues arise when it comes to resource constrained devices, such as smart glasses, due to the overwhelming computation and energy requirements of the accurate face recognition methods. In this paper, we propose a robust and efficient sensor-assisted face recognition system on smart glasses by exploring the power of multimodal sensors including the camera and Inertial Measurement Unit (IMU) sensors. The system is based on a novel face recognition algorithm, namely Multi-view Sparse Representation Classification (MVSRC), by exploiting the prolific information among multi-view face images. To improve the efficiency of MVSRC on smart glasses, we propose two novel sampling optimization strategies using the less expensive inertial sensors. Our evaluations on public and private datasets show that the proposed method is up to 10 percent more accurate than the state-of-the-art multi-view face recognition methods while its computation cost is the same order as an efficient benchmark method (e.g., Eigenfaces). Finally, extensive real-world experiments show that our proposed system improves recognition accuracy by up to 15 percent while achieving the same level of system overhead compared to the existing face recognition system (OpenCV algorithms) on smart glasses.