Search

We present first-order perturbation analysis of a simple eigenvalue and the corresponding right and left eigenvectors of a general square matrix, not assumed to be Hermitian or normal. The eigenvalue result is well known to a broad scientific community. The treatment of eigenvectors is more complicated, with a perturbation theory that is not so well known outside a community of specialists. We give two different proofs of the main eigenvector perturbation theorem. The first, a block-diagonalization technique inspired by the numerical linear algebra research community and based on the implicit function theorem, has apparently not appeared in the literature in this form. The second, based on complex function theory and on eigenprojectors, as is standard in analytic perturbation theory, is a simplified version of well-known results in the literature. The second derivation uses a convenient normalization of the right and left eigenvectors defined in terms of the associated eigenprojector, but although this dates back to the 1950s, it is rarely discussed in the literature. We then show how the eigenvector perturbation theory is easily extended to handle other normalizations that are often used in practice. We also explain how to verify the perturbation results computationally. We conclude with some remarks about difficulties introduced by multiple eigenvalues and give references to work on perturbation of invariant subspaces corresponding to multiple or clustered eigenvalues. Throughout the paper we give extensive bibliographic commentary and references for further reading.

We review the basic outline of the highly successful diffusion Monte Carlo technique commonly used in contexts ranging from electronic structure calculations to rare event simulation and data assimilation, and propose a new class of randomized iterative algorithms based on similar principles to address a variety of common tasks in numerical linear algebra. From the point of view of numerical linear algebra, the main novelty of the Fast Randomized Iteration schemes described in this article is that they work in either linear or constant cost per iteration (and in total, under appropriate conditions) and are rather versatile: we will show how they apply to solution of linear systems, eigenvalue problems, and matrix exponentiation, in dimensions far beyond the present limits of numerical linear algebra. While traditional iterative methods in numerical linear algebra were created in part to deal with instances where a matrix (of size $\mathcal{O}(n^2)$) is too big to store, the algorithms that we propose are effective even in instances where the solution vector itself (of size $\mathcal{O}(n)$) may be too big to store or manipulate. In fact, our work is motivated by recent DMC based quantum Monte Carlo schemes that have been applied to matrices as large as $10^{108} \times 10^{108}$. We provide basic convergence results, discuss the dependence of these results on the dimension of the system, and demonstrate dramatic cost savings on a range of test problems., 44 pages, 7 figures

Every student of numerical linear algebra is familiar with block matrices and vectors. The same ideas can be applied to the continuous analogues of operators, functions, and functionals. It is shown here how the explicit consideration of block structures at the continuous level can be a useful tool. In particular, block operator diagrams lead to templates for spectral discretization of differential and integral equation boundary-value problems in one space dimension by the rectangular differentiation, identity, and integration matrices introduced recently by Driscoll and Hale. The templates are so simple that we are able to present them as executable Matlab codes just a few lines long, developing ideas through a sequence of 12 increasingly advanced examples. The notion of the rectangular shape of a linear operator is made mathematically precise by the theory of Fredholm operators and their indices, and the block operator formulations apply to nonlinear problems too. We propose the convention of representing nonlinear blocks as shaded. At each step of a Newton iteration for a nonlinear problem, the structure is linearized and the blocks become unshaded, representing Fréchet derivative operators, square or rectangular.

The growth factor of a matrix quantifies the amount of potential error growth possible when a linear system is solved using Gaussian elimination with row pivoting. While it is an easy matter [N. J. Higham and D. J. Higham, SIAM J. Matrix Anal. Appl., 10 (1989), pp. 155-164] to construct examples of $n\times n$ matrices having any growth factor up to the maximum of $2^{n-1}$, the weight of experience and analysis [N. J. Higham, Accuracy and Stability of Numerical Algorithms, SIAM, Philadelphia, 1996], [L. N. Trefethen and R. S. Schreiber, SIAM J. Matrix Anal. Appl., 11 (1990), pp. 335-360], [L. N. Trefethen and I. D. Bau, Numerical Linear Algebra, SIAM, Philadelphia, 1997] suggest that matrices with exponentially large growth factors are exceedingly rare. Here we show how to conduct numerical experiments on random matrices using a multicanonical Monte Carlo method to explore the tails of growth factor probability distributions. Our results suggest, for example, that the occurrence of an $8\times 8$ matrix with a growth factor of 40 is on the order of a once-in-the-age-of-the-universe event.

When solving large nonsymmetric systems of linear equations with the restarted GMRES algorithm, one is inclined to select a relatively large restart parameter in the hope of mimicking the full GMRES process. Surprisingly, cases exist where small values of the restart parameter yield convergence in fewer iterations than larger values. Here, two simple examples are presented where GMRES(1) converges exactly in three iterations, while GMRES(2) stagnates. One of these examples reveals that GMRES(1) convergence can be extremely sensitive to small changes in the initial residual.

This paper is an updated and extended version of the paper "The QR Decomposition and the Singular Value Decomposition in the Symmetrized Max-Plus Algebra" (B. De Schutter and B. De Moor, SIAM J. Matrix Anal. Appl., 19 (1998), pp. 378--406). The max-plus algebra, which has maximization and addition as its basic operations, can be used to describe and analyze certain classes of discrete-event systems, such as flexible manufacturing systems, railway networks, and parallel processor systems. In contrast to conventional algebra and conventional (linear) system theory, the max-plus algebra and the max-plus-algebraic system theory for discrete-event systems are at present far from fully developed, and many fundamental problems still have to be solved. Currently, much research is going on to deal with these problems and to further extend the max-plus algebra and to develop a complete max-plus-algebraic system theory for discrete-event systems. In this paper we address one of the remaining gaps in the max-plus algebra by considering matrix decompositions in the symmetrized max-plus algebra. The symmetrized max-plus algebra is an extension of the max-plus algebra obtained by introducing a max-plus-algebraic analogue of the $-$-operator. We show that we can use well-known linear algebra algorithms to prove the existence of max-plus-algebraic analogues of basic matrix decompositions from linear algebra such as the QR decomposition, the singular value decomposition, the Hessenberg decomposition, the LU decomposition, and so on. These max-plus-algebraic matrix decompositions could play an important role in the max-plus-algebraic system theory for discrete-event systems.

Matrix factorization in numerical linear algebra (NLA) typically serves the purpose of restating some given problem in such a way that it can be solved more readily; for example, one major application is in the solution of a linear system of equations. In contrast, within applied statistics/psychometrics (AS/P), a much more common use for matrix factorization is in presenting, possibly spatially, the structure that may be inherent in a given data matrix obtained on a collection of objects observed over a set of variables. The actual components of a factorization are now of prime importance and not just as a mechanism for solving another problem. We review some connections between NLA and AS/P and their respective concerns with matrix factorization and the subsequent rank reduction of a matrix. We note in particular that several results available for many decades in AS/P were more recently (re)discovered in the NLA literature. Two other distinctions between NLA and AS/P are also discussed briefly: how a generalized singular value decomposition might be defined, and the differing uses for the (newer) methods of optimization based on cyclic or iterative projections.

In this survey paper, we describe how strands of work that are important in two different fields, matrix theory and complex function theory, have come together in some work on fast computational algorithms for matrices with what we call displacement structure. In particular, a fast triangularization procedure can be developed for such matrices, generalizing in a striking way an algorithm presented by Schur (1917) [J. Reine Angew. Math., 147 (1917), pp. 205–232] in a paper on checking when a power series is bounded in the unit disc. This factorization algorithm has a surprisingly wide range of significant applications going far beyond numerical linear algebra. We mention, among others, inverse scattering, analytic and unconstrained rational interpolation theory, digital filter design, adaptive filtering, and state-space least-squares estimation.

This paper reviews the complexity of polynomial and matrix computations, as well as their various correlations to each other and some major techniques for the design of algebraic and numerical algorithms.

Scientific and engineering research is becoming increasingly dependent upon the development and implementation of efficient parallel algorithms on modern high-performance computers. Numerical linear algebra is an indispensable tool in such research and this paper attempts to collect and describe a selection of some of its more important parallel algorithms. The purpose is to review the current status and to provide an overall perspective of parallel algorithms for solving dense, banded, or block-structured problems arising in the major areas of direct solution of linear systems, least squares computations, eigenvalue and singular value computations, and rapid elliptic solvers. A major emphasis is given here to certain computational primitives whose efficient execution on parallel and vector computers is essential in order to obtain high performance algorithms.

This paper examines common implementations of linear algebra algorithms, such as matrix-vector multiplication, matrix-matrix multiplication and the solution of linear equations. The different versions are examined for efficiency on a computer architecture which uses vector processing and has pipelined instruction execution. By using the advanced architectural features of such machines, one can usually achieve maximum performance, and tremendous improvements in terms of execution speed can be seen over conventional computers.

The existence of parallel and pipeline computers has inspired a new approach to algorithmic analysis. Classical numerical methods are generally unable to exploit multiple processors and powerful vector-oriented hardware. Efficient parallel algorithms can be created by reformulating familiar algorithms or by discovering new ones, and the results are often surprising. A comprehensive survey of parallel techniques for problems in linear algebra is given. Specific topics include: relevant computer models and their consequences for programs, evaluation of arithmetic expressions, solution of general and special linear systems of equations, and computation of eigenvalues.

Probability one homotopy algorithms are a class of methods for solving nonlinear systems of equations that are globally convergent with probability one. These methods are theoretically powerful, and if constructed and implemented properly, are robust, numerically stable, accurate, and practical. The concomitant numerical linear algebra problems deal with rectangular matrices, and good algorithms require a delicate balance (not always achieved) of accuracy, robustness, and efficiency in both space and time. The author's experience with globally convergent homotopy algorithms is surveyed here, and some of the linear algebra difficulties for dense and sparse problems are discussed.

This is a survey of selected computational aspects of linear algebra, addressed to the nonspecialist in numerical analysis. Some current methods of solving systems of linear equations, and computing eigenvalues of symmetric and unsymmetric matrices are outlined. A bibliography containing 62 titles is included.