Optimization of generalized mean square error in signal processing and communication

Two matrix optimization problems are analyzed. These problems arise in signal processing and communication. In the first problem, the trace of the mean square error matrix is minimized, subject to a power constraint. The solution is the training sequence, which yields the best estimate of a communication channel. The solution is expressed in terms of the eigenvalues and eigenvectors of correlation and covariance matrices describing the communication, and an unknown permutation. Our analysis exhibits the optimal permutation when the power is either very large or very small. Based on the structure of the optimal permutation in these limiting cases, we propose a small class of permutations to focus on when computing the optimal permutation for arbitrary power. In numerical experiments, with randomly generated matrices, the optimal solution is contained in the proposed permutation class with high probability. The second problem is connected with the optimization of the sum capacity of a communication channel. The second problem, which is obtained from the first by replacing the trace operator in the objective function by the determinant, minimizes the product of eigenvalues, while the first problem minimizes the sum of eigenvalues. For small values of the power, both problems have the same solution. As the power increases, the solutions are different, since the permutation matrix appearing in the solution of the trace problem is not present in the solution of the determinant problem. For large power, the ordering of the eigenvectors in the solution of the trace problem is the opposite of the ordering in the determinant problem.