Deep semantic hashing with dual attention for cross-modal retrieval

Abstract

With the explosive growth of multimodal data, cross-modal retrieval has drawn increasing research interests. Hashing-based methods have made great advancements in cross-modal retrieval due to the benefits of low storage cost and fast query speed. However, there still exists a crucial challenge to improve the accuracy of cross-modal retrieval due to the heterogeneity gap between modalities. To further tackle this problem, in this paper, we propose a new two-staged cross-modal retrieval method, called Deep Semantic Hashing with Dual Attention (DSHDA). In the first stage of DSHDA, a Semantic Label Network (SeLabNet) is designed to extract label semantic features and hash codes by training the multi-label annotations, which can make the learning of different modalities in a common semantic space and bridge the modality gap effectively. In the second stage of DSHDA, we propose a deep neural network to simultaneously integrate feature and hash code learning for each modality into the same framework, the training of the framework is guided by the label semantic features and hash codes generated from SeLabNet to maximize the cross-modal semantic relevance. Moreover, dual attention mechanisms are used in our neural networks: (1) Lo-attention is used to extract the local key information of each modality and improve the quality of modality features. (2) Co-attention is used to strengthen the relationship between different modalities to produce more consistent and accurate hash codes. Extensive experiments on two real datasets with image-text modalities demonstrate the superiority of the proposed method in cross-modal retrieval tasks.

Appendices

Optimization algorithm

It is intractable to optimize Eq. (5) directly since it is non-convex with variables $\omega ^p$, $\omega ^t$, $\mathbf{B }$. However, it is convex when taking one variable with the other two variables fixed. Therefore, we use an alternating learning strategy that fixing two parameters and updating the left one at a time until convergence. The whole alternating learning procedure is shown in Algorithm 1 and the detailed optimization steps are listed as follows:

Step 1: Optimize $\omega ^p$ with $\omega ^t$ and $\mathbf{B }$ fixed. We use SGD with a BP algorithm to optimize the CNN parameter $\omega ^p$ of the image modality. For each sampled point $\mathbf{p }_j$, we can compute the gradient for $\hat{\mathbf{F }}_{*i}^p$ as follows:

$$\begin{aligned} \frac{\partial {J}}{\partial {\hat{\mathbf{F }}_{*i}^p}}=\frac{1}{512}{\sum _{j=1}^n{\left( {\left( \frac{1}{2\cdot 512}(\hat{\mathbf{F }}^p_{*i})^\top (\hat{\mathbf{F }}^l_{*j})+\frac{1}{2}\right) -{\hat{S}}_{ij}}\right) \cdot {\hat{\mathbf{F }}^l_{*j}}}} \end{aligned}$$

(12)

Compute the gradient for $\hat{\mathbf{H }}_{*i}^p$ as follows:

$$\begin{aligned} \begin{aligned} \frac{\partial {J}}{\partial {\hat{\mathbf{H }}_{*i}^p}}&=\frac{\gamma }{c}{\sum _{j=1}^n{\left( {\left( \frac{1}{2c}(\hat{\mathbf{H }}^p_{*i})^\top (\hat{\mathbf{H }}^l_{*j})+\frac{1}{2}\right) -{\hat{S}}_{ij}}\right) \cdot {\hat{\mathbf{H }}^l_{*j}}}} \\&\quad +2\mu {(\hat{\mathbf{H }}_{*i}^p-\mathbf{B }_{*i})}+2\tau \hat{\mathbf{H }}^p\cdot \mathbf{1 } \end{aligned} \end{aligned}$$

(13)

Then $\frac{\partial {J}}{\partial {\omega ^p}}$ can be computed with $\frac{\partial {J}}{\partial {\hat{\mathbf{F }}_{*i}^p}}$ and $\frac{\partial {J}}{\partial {\hat{\mathbf{H }}_{*i}^p}}$ by using the chain rule, based on which BP can be used to update the parameter $\omega ^p$.

Step 2: Optimize $\omega ^t$ with $\omega ^p$ and $\mathbf{B }$ fixed. We use SGD with a BP algorithm to optimize deep neural network parameter $\omega ^t$ of the text modality. For each sampled point $\mathbf{t }_j$, we can compute the gradient for $\hat{\mathbf{F }}_{*i}^t$ as follows:

$$\begin{aligned} \frac{\partial {J}}{\partial {\hat{\mathbf{F }}_{*i}^t}}=\frac{1}{512}{\sum _{j=1}^n{\left( {\left( \frac{1}{2\cdot 512}(\hat{\mathbf{F }}^t_{*i})^\top (\hat{\mathbf{F }}^l_{*j})+\frac{1}{2}\right) -{\hat{S}}_{ij}}\right) \cdot {\hat{\mathbf{F }}^l_{*j}}}} \end{aligned}$$

(14)

Compute the gradient for $\hat{\mathbf{H }}_{*i}^t$ as follows:

$$\begin{aligned} \begin{aligned} \frac{\partial {J}}{\partial {\hat{\mathbf{H }}_{*i}^t}}&=\frac{\gamma }{c}{\sum _{j=1}^n{\left( {\left( \frac{1}{2c}(\hat{\mathbf{H }}^t_{*i})^\top (\hat{\mathbf{H }}^l_{*j})+\frac{1}{2}\right) -{\hat{S}}_{ij}}\right) \cdot {\hat{\mathbf{H }}^l_{*j}}}} \\&\quad +2\mu {(\hat{\mathbf{H }}_{*i}^t-\mathbf{B }_{*i})}+2\tau \hat{\mathbf{H }}^t\cdot \mathbf{1 } \end{aligned} \end{aligned}$$

(15)

Then $\frac{\partial {J}}{\partial {\omega ^t}}$ can be computed with $\frac{\partial {J}}{\partial {\hat{\mathbf{F }}_{*i}^t}}$ and $\frac{\partial {J}}{\partial {\hat{\mathbf{H }}_{*i}^t}}$ by using the chain rule, based on which BP can be used to update the parameter $\omega ^t$.

Step 3: Optimize $\mathbf{B }$ with $\omega ^p$ and $\omega ^t$ fixed. The objective function shown in Eq. (5) can be reformulated as follows:

$$\begin{aligned} \begin{aligned} \min \limits _{\mathbf{B }}J&=\mu {({\Vert \mathbf{B }-\hat{\mathbf{H }}^p\Vert }_F^2+{\Vert \mathbf{B }-\hat{\mathbf{H }}^t\Vert }_F^2)}\\ s.t. \quad&\mathbf{B }\in {\{-1,1\}}^{c \times n} \end{aligned} \end{aligned}$$

(16)

which is rewritten as follows:

$$\begin{aligned} \begin{aligned} \max \limits _{\mathbf{B }}J&=Tr{(\mathbf{B }^\top (\mu (\hat{\mathbf{H }}^p+\hat{\mathbf{H }}^t)))} \\ s.t. \quad&\mathbf{B }\in {\{-1,1\}}^{c \times n} \end{aligned} \end{aligned}$$

(17)

To maximize the above formulation, we need to keep the two values of the product the same sign. Therefore, the following formulation can be obtained:

$$\begin{aligned} \mathbf{B }=sign(\mu (\hat{\mathbf{H }}^p+\hat{\mathbf{H }}^t)) \end{aligned}$$

(18)

Time complexity of DSHDA

As Fig. 1, DSHDA consists of two stages, each of which consists of different modules further. We will analyze the time complexity of these modules one by one as follows.

The first stage of DSHDA is the SeLabNet, which is a network model with multiple full connected layers. The time complexity of the network model can be expressed as follows:

$$\begin{aligned} O\left( \sum _{l=1}^{d_F} {w_{l-1}\cdot w_l}\right) \end{aligned}$$

(19)

where $w_l$ is the number of neurons in the l-th layer and $d_F$ is the total number of full connected layers.

Let $w_0=k$ be the dimension of the input multi-label annotation. Then, according to Eq. (19) and Table 1, we can obtain the time complexity of the semantic feature generation module $G^l$ and semantic hash code generation mobule $D^l$ as following Eqs. (20) and (21), respectively.

$$\begin{aligned}&O(k \cdot k + 2048 \cdot k + 2048 \cdot 512 ) \nonumber \\&\quad \sim O(k^2+2048 \cdot k + 1.05 \times 10^6) \end{aligned}$$

(20)

$$\begin{aligned}&O(512 \cdot c ) \end{aligned}$$

(21)

The second stage of DSHDA is the image and text network, which consists of five modules, including feature learning module ($E^p$ and $E^t$), Lo-attention ($A^p$ and $A^t$), semantic feature generation ($G^p$ and $G^t$), B-structure ($V^p$ and $V^t$) and semantic hash code generation ($D^p$ and $D^t$).

For the image network, the image feature learning module $E^p$ is a five-layer convolutional network. The time complexity of the network model with multiple convolutional layers is given by [37]:

$$\begin{aligned} O\left( \sum _{l=1}^{d_L} {n_{l-1} \cdot s_l^2 \cdot n_l \cdot m_l^2}\right), \end{aligned}$$

(22)

where $d_L$ is the number of convolutional layers, $s_l$ is the spatial size (length) of the filter in the l-th layer, $n_l$ is the number of filters in the l-th layer, $m_l$ is the spatial size of the output feature map in the l-th layer, and $n_{l-1}$ is also known as the number of input channels of the l-th layer.

Let $t_l$ be the stride of the filter in the l-th layer, we have

$$\begin{aligned} m_l \approx \frac{m_{l-1}}{t_l} \end{aligned}$$

(23)

Besides, the length of the original image feature $d_p$ can be expressed as:

$$\begin{aligned} d_p = m_0^2 \cdot n_0, \end{aligned}$$

(24)

where $m_0$ and $n_0$ can be considered as the spatial size and the number of channels of the input feature map for the first convolutional layer, respectively.

Then, Eq. (22) can be transformed as:

$$\begin{aligned} O\left( \sum _{l=1}^{d_L} \frac{n_{l-1} \cdot s_l^2 \cdot n_l}{n_0 \cdot \prod _{j=1}^{l}{t_j}} \cdot d_p\right) \end{aligned}$$

(25)

Now, Let $n_0=3$ represent the three color channels of the input image. According to Eq. (25) and Table 2, we can obtain the time complexity of module $E^p$ as follows:

$$\begin{aligned} \begin{aligned}&O\left( \frac{11\cdot 11\cdot 64}{4^2}\cdot d_p +\frac{64\cdot 5\cdot 5\cdot 265}{3\cdot 4^2}\cdot d_p \right. \\&\left. \quad +3\cdot \frac{64\cdot 3\cdot 3\cdot 265}{3\cdot 4^2}\cdot d_p \right) \\&\quad \sim O(4.88 \times 10^4 \cdot d_p) \end{aligned} \end{aligned}$$

(26)

Since the dimension of the output image feature maps by module $E^p$ is $d_C \cdot d_H \cdot d_W$, we have $d_C=265$, $d_H \cdot d_W=m_5^2=d_p/48$. These feature maps will be the input of the Lo-attention module $A^p$, which contains is a convolutional layer with a $1 \times 1$ kernel size. Then, the time complexity of module $A^p$ is:

$$\begin{aligned} O(33 \cdot d_p ) \end{aligned}$$

(27)

With the same approach, we can obtain the time complexity of other modules in the image and text network of DSHDA. Due to space limitations, we omit the process of the analysis and just list the time complexity of DSHDA modules in Table 11.

Table 11 Time complexity of DSHDA modules

Thus, the total time complexity of DSHDA is the sum of all modules in Table 11. Besides, noticing that the values of k and c are $O(10^2)$, $d_p$ and $d_t$ are $O(10^3)$ in practice, we can obtain the time complexity of DSHDA is:

$$\begin{aligned} O(\epsilon \times 10^8), \end{aligned}$$

(28)

where $\epsilon >1$ is a scale factor of the time complexity.