On the Influence of Ill-conditioned Regression Matrix on Hyper-parameter Estimators for Kernel-based Regularization Methods

In this paper, we study the influence of ill-conditioned regression matrix on two hyper-parameter estimation methods for the kernel-based regularization method: the empirical Bayes (EB) and the Steins unbiased risk estimator (SURE). First, we consider the convergence rate of the cost functions of EB and SURE, and we find that they have the same convergence rate but the influence of the ill-conditioned regression matrix on the scale factor are different: for upper bounds, the scale factor for SURE contains one more factor cond(Phi(T)Phi) than that of EB, where Phi is the regression matrix and cond(.) denotes the condition number of a matrix. This finding indicates that when Phi is ill-conditioned, i.e., cond(Phi(T)Phi) is large, the cost function of SURE converges slower than that of EB. Then we consider the convergence rate of the optimal hyper-parameters of EB and SURE, and we find that they are both asymptotically normally distributed and have the same convergence rate, but the influence of the ill-conditioned regression matrix on the scale factor are different. In particular, for the ridge regression case, we show that the optimal hyper-parameter of SURE converges slower than that of EB with a factor of 1/n(2), as cond(Phi(T)Phi) goes to infinity, where n is the FIR model order.
Funding Agencies|Thousand Youth Talents Plan funded by the central government of China - NSFCNational Natural Science Foundation of China (NSFC) [61773329]; Shenzhen Science and Technology Innovation Council [Ji-20170189 (J-CY20170411102101881)]; Robotic Discipline Development Fund from Shenzhen Government [20161418]; CUHKSZ [2014.0003.23]; [PF. 01.000249]