Selective Pseudo-Label Clustering

Abstract

Deep neural networks (DNNs) offer a means of addressing the challenging task of clustering high-dimensional data. DNNs can extract useful features, and so produce a lower dimensional representation, which is more amenable to clustering techniques. As clustering is typically performed in a purely unsupervised setting, where no training labels are available, the question then arises as to how the DNN feature extractor can be trained. The most accurate existing approaches combine the training of the DNN with the clustering objective, so that information from the clustering process can be used to update the DNN to produce better features for clustering. One problem with this approach is that these “pseudo-labels” produced by the clustering algorithm are noisy, and any errors that they contain will hurt the training of the DNN. In this paper, we propose selective pseudo-label clustering, which uses only the most confident pseudo-labels for training the DNN. We formally prove the performance gains under certain conditions. Applied to the task of image clustering, the new approach achieves a state-of-the-art performance on three popular image datasets.

Appendices

A Appendix A: Full Proofs

This appendix contains the full proofs of the results in Sect. 4.

1.1 A.1 More Accurate Pseudo-Labels Supplement

The only part omitted from the argument in the main paper is a proof for the claim about the entropy of the random variable X. This is supplied by the following proposition.

Proposition 1

Given a categorical random variable X of the form

$$\begin{aligned} p(X=c_0) = t \\ \forall c \ne c_0, p(X=c) = \frac{1-t}{C-1}, \end{aligned}$$

for some $1/C \le t \le 1$, the entropy H(X) is a strictly decreasing function of t.

Proof

$$\begin{aligned} H(X) =&- t\log t - (1-t)\log \frac{1-t}{C-1} \\ \frac{d(H(X))}{dt} =&- \log t - 1 - \frac{1}{1-t} + \log \frac{1}{C-1}\\&+ \frac{t}{1-t} + \log 1-t \\ =&-2 - \log t - \log C-1 + \log t-1 \\ =&-2 - \log \left( \frac{t}{1-t}(C-1)\right) . \end{aligned}$$

The argument to the $\log$ is clearly an increasing function of t for $t>1$. Therefore, for $1/C \le t < 1$, it is lower-bounded by setting $t = 1/C$. This gives

$$\begin{aligned} \frac{d(H(X))}{dt}&\le -2 - \log \left( \frac{1/C}{1-1/C}(C-1)\right) \\&< -\log \left( \frac{1/C}{1-1/C}(C-1)\right) = -\log 1 = 0. \end{aligned}$$

The derivative is always strictly negative with respect to t, so, as a function of t, H(X) is always strictly decreasing.

1.2 A.2 Lemma 1 Supplement

The following is a proof for the claim that $u_{ same } < u_{ diff }$, as stated in Sect. 4.

Decomposing $u_{ same }$ according to the definition of variance (as the expectation of the square minus the square of the expectation) gives

$$\begin{aligned} \underset{x, x' \sim T}{\mathbb {E}}[w^T(x - x') - \eta w'(||x||^2-||x'||^2)]^2 \\ +\,\text {Var}(w^T(x - x') - \eta w'(||x||^2-||x'||^2)). \end{aligned}$$

The expectation term equals 0, as

$$\begin{aligned} w^T\underset{x, x' \sim T}{\mathbb {E}}[(x - x')] - \eta w'\underset{x, x' \sim T}{\mathbb {E}}[(||x||^2-||x'||^2)] \\ =\,(w\mathbb {E}[T] - \mathbb {E}[T]) - \eta w'(\mathbb {E}[||T||^2]-\mathbb {E}[||T||^2]) = 0 . \end{aligned}$$

By symmetry, we can replace covariances involving $x'$ with the same involving x. The remaining term can then be rearranged to give

$$\begin{aligned} u_{same} = 2\text {Var}(w^Tx - \eta w'||x||^2) \\ = 2w^TCov(T)w + 2\eta w' Var(||x||^2) - 4\text {Cov}(w^Tx, \eta w'||x||^2). \end{aligned}$$

Now rewrite $u_{ diff }$. Decomposing as above gives

$$\begin{aligned} \underset{x, x' \sim T}{\mathbb {E}}[w^T(x - x') - \eta w'(||x-x'||^2)]^2 \\ +\,\text {Var}(w^T(x - x') - \eta w'(||x-x'||^2))\,, \end{aligned}$$

and here the expectation term does not equal 0:

$$\begin{aligned} (w^T\underset{x, x' \sim T}{\mathbb {E}}[(x - x')] - \eta w' \underset{x, x' \sim T}{\mathbb {E}}[(||x-x'||^2)])^2 \\ =\,(\eta w')^2\underset{x, x' \sim T}{\mathbb {E}}[||x-x'||^2]^2. \end{aligned}$$

The variance term can be expanded to give:

$$\begin{aligned} \text {Var}(w^T(x - x') - \eta w'(||x-x'||^2)) \\ =\,2w^TCov(T)w + 2\eta w'\text {Var}(||x - x'||^2) \\ -\,4\text {Cov}(w^Tx, \eta w'||x - x'||^2). \end{aligned}$$

By comparing terms, we can see that this expression is at least as large as $u_{same}$. First, consider the covariance terms.

Claim. $\text {Cov}(w^Tx, \eta w'||x - x'||^2) = \text {Cov}(w^Tx,\eta w'||x||^2)$.

$$\begin{aligned}&\text {Cov}(w^Tx, \eta w'||x - x'||^2) \\&=\mathbb {E}[w^Tx\eta w'||x-x'||^2] - \mathbb {E}[w^Tx]\mathbb {E}[\eta w'||x - x'||^2] \\&= \eta w'\mathbb {E}[w^Tx||x-x'||^2] - 0\mathbb {E}[\eta w'||x - x'||^2]\\&= \eta w'\mathbb {E}[w^Tx||x-x'||^2] \\&= \eta w'\mathbb {E}[w^Tx\sum _{k}x^2 - 2xx' + x^{'2}] \\&= \eta w'\sum _{k}\mathbb {E}[w^Txx_k^2] - 2\mathbb {E}[w^Txx_k]\mathbb {E}[x'] + \mathbb {E}[w^Tx]\mathbb {E}[x^{'2}] \\&= \eta w'\sum _{k}\mathbb {E}[w^Txx_k^2] - 2\mathbb {E}[w^Txx_k]\vec{0} + 0\mathbb {E}[x^{'2}] \\&= \eta w'\sum _{k}\mathbb {E}[w^Txx_k^2] \\&= \eta w'\mathbb {E}[w^Tx\sum _{k}x_k^2] \\&= \eta w'\mathbb {E}[w^Tx||x||^2] \\&= \eta w'\mathbb {E}[w^Tx||x||^2] - 0\mathbb {E}[\eta w'||x||^2] \\&= \mathbb {E}[w^Tx\eta w'||x||^2] - \mathbb {E}[w^Tx]\mathbb {E}[\eta w'||x||^2] \\&= \text {Cov}(w^Tx,\eta w'||x||^2). \end{aligned}$$

So, we see the covariance terms are equal.

Next, compare the second variance terms

Claim. $\text {Var}(||x - x'||^2) \ge \text {Var}(||x||^2)$.

$$\begin{aligned}&\text {Var}(||x - x'||^2) \\&=\, \text {Var}\left( \sum _{k=0}^{nz}(x)_k^2 + (x')_k^2 - 2(x)_k(x')_k\right) \\&=\, \text {Var}\left( \sum _{k=0}^{nz}(x)_k^2\right) + \text {Var}\left( \sum _{k=0}^{nz}(x')_k^2\right) + 2{{\,\mathrm{Var}\,}}\left( \sum _{k=0}^{nz} x_kx'_k\right) \\&=\, 2\text {Var}\left( \sum _{k=0}^{nz}(x)_k^2\right) + 2{{\,\mathrm{Var}\,}}\left( \sum _{k=0}^{nz} x_kx'_k\right) \\&=\, 2(\text {Var}(||x||^2) + {{\,\mathrm{Var}\,}}(x^Tx'))\\&\ge \, \text {Var}(||x||^2). \end{aligned}$$

Assuming that the data are not all identical, this implies that $u_{ diff }$ is strictly greater than $u_{same}$.

$$\begin{aligned}&u_{ diff } - u_{same} \\ =\,&(\eta w')^2\underset{x, x' \sim T}{\mathbb {E}}[||x-x'||^2]^2 + 2w^T{{\,\mathrm{Cov}\,}}(T)w \\&+2\eta w'\text {Var}(||x - x'||^2) - 4{{\,\mathrm{Cov}\,}}(w^Tx, \eta w'||x - x'||^2) \\&- ((2w^T{{\,\mathrm{Cov}\,}}(T)w + 2\eta w' Var(||x||^2) \\&- 4{{\,\mathrm{Cov}\,}}(w^Tx, \eta w'||x||^2))) \\ =\,&(\eta w')^2\underset{x, x' \sim T}{\mathbb {E}}[||x-x'||^2]^2 \\ \quad&+ 2\eta w'\left( \text {Var}(||x - x'||^2) - \text {Var}(||x||^2)\right) \\&- 4\left( {{\,\mathrm{Cov}\,}}(w^Tx, \eta w'||x - x'||^2) - {{\,\mathrm{Cov}\,}}(w^Tx, \eta w'||x||^2)\right) \\ =\,&(\eta w')^2\underset{x, x' \sim T}{\mathbb {E}}[||x-x'||^2]^2 \\ \quad&+ 2\eta w'\left( \text {Var}(||x - x'||^2) - \text {Var}(||x||^2)\right) \\ \ge \,&(\eta w')^2\underset{x, x' \sim T}{\mathbb {E}}[||x-x'||^2]^2 > 0. \end{aligned}$$

1.3 A.3 Lemma 2 Supplement

The following is the complete proof of Lemma 2, which was omitted from the main paper.

Proof

$v_{ diff } - v_{same}$

$$\begin{aligned} =\,&\mathbb {E}[(w^T(x-z) - w'(x-x')(x-z))^2] \\&-\mathbb {E}[(w^T(x-z) - w'(x+x')(x-z))^2] \\ =\,&\mathbb {E}[(w^T(x-z) - w'(x-x')^T(x-z))^2 \\&-(w^T(x-z) - w'(x+x')^T(x-z))^2] \\ =\,&\mathbb {E}[(w^T(x-z) - w'(x-x')^T(x-z) \\&+w^T(x-z) - w'(x+x')^T(x-z)) \\&(w^T(x-z) - w'(x-x')^T(x-z) \\&-w^T(x-z) - w'(x+x')^T(x-z))] \\ =\,&\mathbb {E}[(2w^T(x-z) - w'(x-z)^T(x-x'+x+x'))\\&(- w'(x-z)^T(x-x'-x-x'))] \\ =\,&\mathbb {E}[(2w^T(x-z) - 2w'(x-z)^T(x))(2w'(x-z)^T(x'))] \\ =\,&2\mathbb {E}[(w^T(x-z) - w'(x-z)^T(x))w'(x-z)^T]\mathbb {E}[x'] \\ =\,&2\mathbb {E}[(w^T(x-z) - w'(x-z)(x))w'(x-z)^T]\overrightarrow{0} = 0\,. \end{aligned}$$

1.4 A.4 Lemma 3 Supplement

The following is the complete proof of Lemma 3, which was omitted from the main paper.

Proof

$$\begin{aligned} {{\,\mathrm{Var}\,}}(T) =\,&\tfrac{1}{2}\underset{x, x' \sim T}{\mathbb {E}}[(x-x')^2] \\ =\,&\tfrac{1}{2}(\underset{x, x' \sim T}{\mathbb {E}}[(x-x')^2|y(x) = y(x')]P(y(x) = y(x')) \\&+\underset{x, x' \sim T}{\mathbb {E}}[(x-x')^2|y(x) \ne y(x')]P(y(x) \ne y(x'))\\ =\,&\tfrac{1}{2}(sP(y(x) = y(x')) + rP(y(x) \ne y(x'))\\ =\,&\frac{1}{2}\left( s\frac{1}{C} + r\frac{C-1}{C}\right) . \end{aligned}$$

Noting that $s = 2\mathbb {E}[{{\,\mathrm{Var}\,}}(T|C)]$, and using Eve’s law, we have

$$\begin{aligned} d =\,&{{\,\mathrm{Var}\,}}(T) -s \\ =\,&\frac{1}{2}\left( s\frac{1}{C} + r\frac{C-1}{C}\right) -s \\ =\,&\frac{C-1}{2C}r - \frac{2C -1}{2C}s. \end{aligned}$$

1.5 A.5 Theorem 5 Supplement

The following is a more detailed version of the argument given in the main paper.

If $y(x) = y(x')$, then Lemma 2 means that the expected distance of the encodings of x and $x'$ to any data point from another cluster is unchanged by whether the update was from points with the same or with different labels. Similarly, the distance between any two other points is unchanged by whether the update was from points with the same or with different labels. This establishes that $r_T = r_F$. As for the intra-cluster variance, it is smaller after the update with the same labels than with different labels. Lemma 1 shows that the expected distance between the encodings of the two points themselves is smaller if the labels were the same, and the same argument as above shows that all other expected distances within clusters are unchanged.

If $y(x) \ne y(x')$, then Lemma 2 means that the expected distance of the encodings of x and any data point from the same cluster is unchanged by whether the update was from points with the same or with different labels (and the same for $x'$). Similarly, the distance between any two other points is unchanged by whether the update was from points with the same or with different labels. This establishes that $s_T = s_F$. As for the inter-cluster variance, it is larger after the update with the same labels than with different labels. Lemma 1 shows that the expected distance between the encodings of the two points themselves is larger if the labels were different, and the same argument as above shows that all other expected distances within clusters are unchanged.

Table 4. Sizes of predicted clusters for MNIST.

Table 5. Sizes of predicted clusters for USPS.

Table 6. Sizes of predicted clusters for FashionMNIST.

B Appendix C: Extended Results

The results in the main paper report the central tendency of five different training runs for each dataset. Tables 4, 5, and 6 show the sizes of the clusters predicted by SPC for one randomly selected run out of these five. On MNIST and USPS, where the accuracy of SPC is ${>}98\%$, the predicted sizes are close to the true sizes. On FashionMNIST, where the accuracy is ${\sim }65\%$, there is a much greater variance. This accounts for the discrepancy in ACC and NMI for FashionMNIST. Most of the errors are put into one large cluster, specifically the cluster that was aligned to ‘coat’ is over three times larger than it should be. This hurts accuracy more than NMI, because the incorrect data points in the ‘coat’ cluster count for zero when calculating the accuracy, but they are not randomly distributed among the other classes, so the conditional entropy of a data point that was mis-clustered as a coat is $<\log (10)$. Actually, most of the mistakes in the ‘coat’ cluster are pullovers or shirts, and almost none of them are, for examples, boots or tops. Comparing the cluster sizes for SPC-HDBSCAN and SPC-GMM also accounts for the differences across ACC and NMI between these two settings on FashionMNIST: SPC-GMM produces more uniformly-sized clusters, so the difference between ACC and NMI is smaller.