Videos spread over the Internet contain a huge knowledge of human society. Diversified knowledge is demonstrated as the storyline of the video unfolds. Therefore, realization of automatically constructing social relation network from massive video data facilitates the deep semantics of mining big data, which includes face recognition and social relation recognition. For face recognition, previous studies are focus on high-level features of face and multiple body cues. However, these methods are mostly based on supervised learning and clustering need to specify clusters k, which cannot recognize characters when new video data is input and individual and its numbers are unknown. For social relation recognition, previous studies are concentrated on images and videos. However, these methods are only concentrated on social relations in same frame and incapable of extracting social relation of characters that are not present in the same frame. In this paper, a model named SRE-Net is proposed for building social relation network to address these challenges. First, MoCNR algorithm is introduced by clustering similar-appearing faces from different keyframes of video. As far as we know, it is the first algorithm to identify character nodes using unsupervised double-clustering methods. Second, we propose a scene based social relation recognition method to solve challenges that cannot recognize social relations of characters in different frames. Finally, comprehensive evaluations demonstrate that our model is effective for social relation network construction.