Computing Dense Tensor Decompositions with Optimal Dimension Trees

Abstract

Dense tensor decompositions have been widely used in many signal processing problems including analyzing speech signals, identifying the localization of signal sources, and many other communication applications. Computing these decompositions poses major computational challenges for big datasets emerging in these domains. CANDECOMP/PARAFAC (CP) and Tucker formulations are the prominent tensor decomposition schemes heavily used in these fields, and the algorithms for computing them involve applying two core operations, namely tensor-times–matrix and tensor-times–vector multiplication, which are executed repetitively within an iterative framework. In the recent past, efficient computational schemes using a data structure called dimension tree, are employed to significantly reduce the cost of these two operations, through storing and reusing partial results that are commonly used across different iterations of these algorithms. This framework has been introduced for sparse CP and Tucker decompositions in the literature, and a recent work investigates using an optimal binary dimension tree structure in computing dense Tucker decompositions. In this paper, we investigate finding an optimal dimension tree for both CP and Tucker decompositions. We show that finding an optimal dimension tree for an N-dimensional tensor is NP-hard for both decompositions, provide faster exact algorithms for finding an optimal dimension tree in $O(3^N)$ time using $O(2^N)$ space for the Tucker case, and extend the algorithm to the case of CP decomposition with the same time and space complexities.

Appendix: Counter-Example: Tensor Having No Optimal BDT

Here, we provide a counter-example using a 6-dimensional tensor $\varvec{\mathcal {X}} \in \mathbb {R}^{I_1 \times \dots \times I_6}$ to show that using a BDT is not necessarily optimal to compute Tucker decomposition. The first three and the last three dimensions of $\varvec{\mathcal {X}}$ have identical sizes and ranks of approximation. Specifically, we let $I_1 = I_2 = I_3 = k$ and $R_1 = R_2 = R_3 = R < k$ in the first three dimensions, and $I_4 = I_5 = I_6 = R_4 = R_5 = R_6 = c$ in the last three dimensions. We call the first three and the last tree dimensions type-1 and type-2 dimensions, respectively. Note that $\alpha = \alpha _1 = \alpha _2 = \alpha _3 < \alpha _4 = \alpha _5 = \alpha _6 = 1$, and similarly $\beta _1 = \beta _2 = \beta _3 < \beta _4 = \beta _5 = \beta _6 = \infty$, therefore, type-1 dimensions are to be multiplied before type-2 dimensions according to Theorem 1. In Fig. 3, we provide a ternary dimension tree with a total cost of $(3 + 3\alpha + 3\alpha ^2)k + 9\alpha ^2c$. We can choose c arbitrarily large so that the term $9\alpha ^2c$ dominates the cost, and $\alpha$ small enough so that $\alpha ^0 c \gg 9\alpha ^2c$ and $\alpha ^1c \gg 9\alpha ^2c$. In this case, any BDT whose cost involves a term of order $\alpha ^0 c$ or $\alpha ^1 c$ cannot be optimal, having a greater cost than the provided ternary tree.

Fig. 3

A ternary dimension tree for $\varvec{\mathcal {X}}$. The $\mu$ set of the node is provided on the left, and the tensor size coefficient is provided on the right of each tree node. The TTM cost of each level is provided on the right

We now show that any BDT using $\varvec{\mathcal {X}}$ either has a cost with a term of order $\alpha ^0 c$ or $\alpha ^1c$, or with a term $9\alpha ^2c + d\alpha ^3c$ with $d \ge 1$; hence, cannot be optimal with a sufficiently large c and a sufficiently small $\alpha$. We do this by exhaustively considering all possible BDTs and analyzing their costs, while aggressively pruning tree configurations that cannot provide optimality. Luckily, most non-optimal configurations can easily be pruned due to symmetry (as we only have two types of dimensions), leaving us with only a handful of instances to consider. We begin by partitioning type-2 dimensions to the children $t_1$ and $t_2$ of the root. There are only two possibilities: $4 \in \mu (t_1)$ and $5, 6 \in \mu (t_2)$, or $4, 5, 6 \in \mu (t_2)$. Since we have only three type-1 dimensions, in the former case either $\mu (t_1)$ or $\mu (t_2)$ will have one or zero type-1 dimension, while the other set having two or three of them. In this case, the TTM cost of the other vertex involves a term $\alpha ^0c$ or $\alpha ^1c$, which already renders this configuration non-optimal. Therefore, we only consider the partition $4, 5, 6 \in \mu (t_2)$, which is the only one that can possibly provide an optimal solution. In this case, there are three possible configurations after partitioning type-1 dimensions: $\mu (t_1) = \{ 1 \}$ and $\mu (t_2) = \{ 2, 3, 4, 5, 6 \}$, $\mu (t_1) = \{ 1, 2 \}$ and $\mu (t_2) = \{ 3, 4, 5, 6 \}$, or $\mu (t_1) = \{ 1, 2, 3 \}$ and $\mu (t_2) = \{ 4, 5, 6 \}$. Note that in the second and third configurations, computing TTM for $t_1$ involves a term $\alpha ^1c$ and $\alpha ^0c$, respectively, which prevents optimality. Hence, in the rest of the discussion we only consider the first configuration, in which $t_1$ incurs the TTM cost $3\alpha ^2c$. We focus on the cost of the sub-tree rooted at $t_2$, and count only the cost due to terms with the coefficient c. Note that if the remaining type-1 dimensions, namely 2 and 3, reside in the same child of $t_2$, the other child of $t_2$ incurs a cost of at least $\alpha ^1c$; therefore, 2 and 3 must reside in different children of $t_2$. In Fig. 4 we provide six such possibilities, all of which incur a cost of $9\alpha ^2c + c\alpha ^3c$ with $c \ge 1$. Therefore, we conclude that a BDT cannot be optimal for the given $\varvec{\mathcal {X}}$ for sufficiently small $\alpha$ and large c.

Fig. 4

All possible configurations and associated TTM costs at each level of BDTs rooted at $t_2$. $\mu (t)$ is given on the left of each tree node t, On the right of a tree node, the size coefficient of its tensor is provided, i.e., $t_2$ has a tensor of size $(I_1 I_2 I_3 I_4 I_5 I_6)\alpha$