Search

We investigate the expressive power of depth-2 bandlimited random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random assignment, and only the output layer parameters are trained by loss minimization. Using random weights for a hidden layer is an effective method to avoid non-convex optimization in standard gradient descent learning. It has also been adopted in recent deep learning theories. Despite the well-known fact that a neural network is a universal approximator, in this study, we mathematically show that when hidden parameters are distributed in a bounded domain, the network may not achieve zero approximation error. In particular, we derive a new nontrivial approximation error lower bound. The proof utilizes the technique of ridgelet analysis, a harmonic analysis method designed for neural networks. This method is inspired by fundamental principles in classical signal processing, specifically the idea that signals with limited bandwidth may not always be able to perfectly recreate the original signal. We corroborate our theoretical results with various simulation studies, and generally, two main take-home messages are offered: (i) Not any distribution for selecting random weights is feasible to build a universal approximator; (ii) A suitable assignment of random weights exists but to some degree is associated with the complexity of the target function., Comment: Published as a conference paper at ICML 2023

This paper concerns the approximation by the Boolean sums of Jackson operators $\oplus^rJ_{k,s}(f)$ on the unit sphere $\mathbb S^{n-1}$ of $\mathbb{R}^{n}$. We prove the following the direct and inverse theorem for $\oplus^rJ_{k,s}(f)$: there are constants $C_1$ and $C_2$ such that \begin{equation*} C_1\|\oplus^rJ_{k,s}f-f\|_p \leq \omega^{2r}(f,k^{-1})_p \leq C_2 \max_{v\geq k}\|\oplus^rJ_{k,s}f-f\|_p \end{equation*} for any positive integer $k$ and any $p$th Lebesgue integrable functions $f$ defined on $\mathbb S^{n-1}$, where $\omega^{2r}(f,t)_p$ is the modulus of smoothness of degree $2r$ of $f$. We also prove that the saturation order for $\oplus^rJ_{k,s}$ is $k^{-2r}$.

Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.

Approximation on the spherical cap is different from that on the sphere which requires us to construct new operators. This paper discusses the approximation on the spherical cap. That is, so called Jackson-type operator $\{J_{k,s}^m\}_{k=1}^{\infty}$ is constructed to approximate the function defined on the spherical cap $D(x_0,\gamma)$. We thus establish the direct and inverse inequalities and obtain saturation theorems for $\{J_{k,s}^m\}_{k=1}^{\infty}$ on the cap $D(x_0,\gamma)$. Using methods of $K$-functional and multiplier, we obtain the inequality \begin{eqnarray*} C_1\:\| J_{k,s}^m(f)-f\|_{D,p}\leq \omega^2\left(f,\:k^{-1}\right)_{D,p} \leq C_2 \max_{v\geq k}\| J_{v,s}^m(f) - f\|_{D,p} \end{eqnarray*} and that the saturation order of these operators is $O(k^{-2})$, where $\omega^2\left(f,\:t\right)_{D,p}$ is the modulus of smoothness of degree 2, the constants $C_1$ and $C_2$ are independent of $k$ and $f$.

This paper discusses the approximation by de la Vall\'{e}e Poussin means $V_nf$ on the unit sphere. Especially, the lower bound of approximation is studied. As a main result, the strong converse inequality for the means is established. Namely, it is proved that there are constants $C_1$ and $C_2$ such that \begin{eqnarray*} C_1\omega(f,\frac{1}{\sqrt n})_p \leq \|V_{n}f-f\|_p \leq C_2\omega(f,\frac{1}{\sqrt n})_p \end{eqnarray*} for any $p$-th Lebesgue integrable or continuous function $f$ defined on the sphere, where $\omega(f,t)_p$ is the modulus of smoothness of $f$., Comment: 14 pages

This paper discusses the approximation by %semigroups of operators of class ($\mathscr{C}_0$) on the sphere and focuses on a class of so called exponential-type multiplier operators. It is proved that such operators form a strongly continuous semigroup of contraction operators of class ($\mathscr{C}_0$), from which the equivalence between approximation for these operators and $K$-functionals introduced by the operators is given. As examples, the constructed $r$-th Boolean of generalized spherical Abel-Poisson operator and $r$-th Boolean of generalized spherical Weierstrass operator denoted by $\oplus^r V_t^{\gamma}$ and $\oplus^r W_t^{\kappa}$ separately ($r$ is any positive integer, $0<\gamma,\kappa\leq1$ and $t>0$) satisfy that $\|\oplus^r V_t^{\gamma}f - f\|_{\mathcal{X}}\approx \omega^{r\gamma}(f,t^{1/\gamma})_{\mathcal{X}}$ and $\|\oplus^r W_t^{\kappa}f - f\|_{\mathcal{X}}\approx \omega^{2r\gamma}(f,t^{1/(2\kappa)})_{\mathcal{X}}$, for all $f\in \mathcal{X}$, where $\mathcal{X}$ is a Banach space of continuous functions or $\mathcal{L}^p$-integrable functions ($1\leq p<\infty$) and $\|\cdot\|_{\mathcal{X}}$ is the norm on $\mathcal{X}$ and $\omega^s(f,t)_{\mathcal{X}}$ is the moduli of smoothness of degree $s>0$ for $f\in \mathcal{X}$. The saturation order and saturation class of the regular exponential-type multiplier operators with positive kernels are also obtained. Moreover, it is proved that $\oplus^r V_t^{\gamma}$ and $\oplus^r W_t^{\kappa}$ have the same saturation class if $\gamma=2\kappa$.