In this paper, we propose a novel approach for dominant color region detection using dominant sets clustering and apply it to soccer video shot classification. Compared with the widely used histogram based dominant color extraction methods which require appropriate thresholds and sufficient training samples, the proposed method can automatically extract dominant color region without any threshold setting. Moreover, the dominant color distribution can be sufficiently characterized by the use of dominant sets clustering which naturally provides a principled measure of a cluster’s cohesiveness as well as a measure of a vertex participation to each group. The Earth Mover’s Distance (EMD) is employed to measure the similarity between dominant color regions of two frames, which is incorporated into the kernel function of SVM. Experimental results have shown the proposed method is much more effective.