Search

PySINDy is a Python package for the discovery of governing dynamical systems models from data. In particular, PySINDy provides tools for applying the sparse identification of nonlinear dynamics (SINDy) (Brunton et al. 2016) approach to model discovery. In this work we provide a brief description of the mathematical underpinnings of SINDy, an overview and demonstration of the features implemented in PySINDy (with code examples), practical advice for users, and a list of potential extensions to PySINDy. Software is available at https://github.com/dynamicslab/pysindy.

We propose a multi-resolution convolutional autoencoder (MrCAE) architecture that integrates and leverages three highly successful mathematical architectures: (i) multigrid methods, (ii) convolutional autoencoders and (iii) transfer learning. The method provides an adaptive, hierarchical architecture that capitalizes on a progressive training approach for multiscale spatio-temporal data. This framework allows for inputs across multiple scales: starting from a compact (small number of weights) network architecture and low-resolution data, our network progressively deepens and widens itself in a principled manner to encode new information in the higher resolution data based on its current performance of reconstruction. Basic transfer learning techniques are applied to ensure information learned from previous training steps can be rapidly transferred to the larger network. As a result, the network can dynamically capture different scaled features at different depths of the network. The performance gains of this adaptive multiscale architecture are illustrated through a sequence of numerical experiments on synthetic examples and real-world spatial-temporal data., Comment: 20 pages, 11 figures

Accurately modeling the nonlinear dynamics of a system from measurement data is a challenging yet vital topic. The sparse identification of nonlinear dynamics (SINDy) algorithm is one approach to discover dynamical systems models from data. Although extensions have been developed to identify implicit dynamics, or dynamics described by rational functions, these extensions are extremely sensitive to noise. In this work, we develop SINDy-PI (parallel, implicit), a robust variant of the SINDy algorithm to identify implicit dynamics and rational nonlinearities. The SINDy-PI framework includes multiple optimization algorithms and a principled approach to model selection. We demonstrate the ability of this algorithm to learn implicit ordinary and partial differential equations and conservation laws from limited and noisy data. In particular, we show that the proposed approach is several orders of magnitude more noise robust than previous approaches, and may be used to identify a class of complex ODE and PDE dynamics that were previously unattainable with SINDy, including for the double pendulum dynamics and the Belousov Zhabotinsky (BZ) reaction., Comment: 25 pages, 9 figures, 5 tables

We consider cost-constrained sparse sensor selection for full-state reconstruction, applying a well-known greedy algorithm to dynamical systems for which the usual singular value decomposition (SVD) basis may not be available or preferred. We apply the cost-modified, column-pivoted QR decomposition to a physically relevant basis -- the pivots correspond to sensor locations, and these locations are penalized with a heterogeneous cost function. In considering different bases, we are able to account for the dynamics of the particular system, yielding sensor arrays that are nearly Pareto optimal in sensor cost and performance in the chosen basis. This flexibility extends our framework to include actuation and dynamic estimation, and to select sensors without training data. We provide three examples from the physical and engineering sciences and evaluate sensor selection in three dynamically relevant bases: truncated balanced modes for control systems, dynamic mode decomposition (DMD) modes, and a basis of analytic modes. We find that these bases all yield effective sensor arrays and reconstructions for their respective systems. When possible, we compare to results using an SVD basis and evaluate tradeoffs between methods., Comment: 12 pages, 12 figures

We develop a theoretical framework that predicts and fully characterizes the diverse experimental observations of the nonlinear, combustion wave propagation in a rotating detonation engine (RDE), including the nucleation and formation of combustion pulses, the soliton-like interactions between these combustion fronts, and the fundamental, underlying Hopf bifurcation to time-periodic modulation of the waves. In this framework, the mode-locked structures are classified as autosolitons, or stably-propagating nonlinear waves where the local physics of nonlinearity, dispersion, gain, and dissipation exactly balance. We find that the global dominant balance physics in the RDE combustion chamber are dissipative and multi-scale in nature, with local fast scale (nano- to microseconds) combustion balances generating the fundamental mode-locked autosoliton state, while slow scale (milliseconds) gain-loss balances determine the instabilities and structure of the total number of autosolitons. In this manner, the global multi-scale balance physics give rise to the stable structures - not exclusively the frontal dynamics prescribed by classical detonation theory. Experimental observations and numerical models of the RDE combustion chamber are in strong qualitative agreement with no parameter tuning. Moreover, numerical continuation (computational bifurcation tracking) of the RDE analog system establishes that a Hopf bifurcation of the steadily propagating pulse train leads to the fundamental instability of the RDE, or time-periodic modulation of the waves. Along branches of Hopf orbits in parameter space exist a continuum of wave-pair interactions that exhibit solitonic interactions of varying strength., Comment: 18 pages, 13 figures

Throughout the history of science, physics-based modeling has relied on judiciously approximating observed dynamics as a balance between a few dominant processes. However, this traditional approach is mathematically cumbersome and only applies in asymptotic regimes where there is a strict separation of scales in the physics. Here, we automate and generalize this approach to non-asymptotic regimes by introducing the idea of an equation space, in which different local balances appear as distinct subspace clusters. Unsupervised learning can then automatically identify regions where groups of terms may be neglected. We show that our data-driven balance models successfully delineate dominant balance physics in a much richer class of systems. In particular, this approach uncovers key mechanistic models in turbulence, combustion, nonlinear optics, geophysical fluids, and neuroscience., Comment: 30 pages, 13 figures

Recent whole-brain calcium imaging recordings of the nematode C. elegans have demonstrated that neural activity is dominated by dynamics on a low-dimensional manifold that can be clustered according to behavioral states. Despite progress in modeling the dynamics with linear or locally linear models, it remains unclear how a single network of neurons can produce the observed features. In particular, there are multiple clusters, or fixed points, observed in the data which cannot be characterized by a single linear model. We propose a nonlinear control model which is global and parameterized by only four free parameters that match the features displayed by the low-dimensional C. elegans neural activity. In addition to reproducing the average probability distribution of the data, long and short time-scale changes in transition statistics can be characterized via changes in a single parameter. Some of these macro-scale transitions have experimental correlates to single neuro-modulators that seem to act as biological controls, allowing this model to generate testable hypotheses about the effect of these neuro-modulators on the global dynamics. The theory provides an elegant characterization of the neuron population dynamics in C. elegans. Moreover, the mathematical structure of the nonlinear control framework provides a paradigm that can be generalized to more complex systems with an arbitrary number of behavioral states.

Recent whole brain imaging experiments on $\textit{C. elegans}$ has revealed that the neural population dynamics encode motor commands and stereotyped transitions between behaviors on low dimensional manifolds. Efforts to characterize the dynamics on this manifold have used piecewise linear models to describe the entire state space, but it is unknown how a single, global dynamical model can generate the observed dynamics. Here, we propose a control framework to achieve such a global model of the dynamics, whereby underlying linear dynamics is actuated by sparse control signals. This method learns the control signals in an unsupervised way from data, then uses $\textit{ Dynamic Mode Decomposition with control}$ (DMDc) to create the first global, linear dynamical system that can reconstruct whole-brain imaging data. These control signals are shown to be implicated in transitions between behaviors. In addition, we analyze the time-delay encoding of these control signals, showing that these transitions can be predicted from neurons previously implicated in behavioral transitions, but also additional neurons previously unidentified. Moreover, our decomposition method allows one to understand the observed nonlinear global dynamics instead as linear dynamics with control. The proposed mathematical framework is generic and can be generalized to other neurosensory systems, potentially revealing transitions and their encodings in a completely unsupervised way., Comment: 10 pages, 5 figures

We develop a deep autoencoder architecture that can be used to find a coordinate transformation which turns a nonlinear PDE into a linear PDE. Our architecture is motivated by the linearizing transformations provided by the Cole-Hopf transform for Burgers equation and the inverse scattering transform for completely integrable PDEs. By leveraging a residual network architecture, a near-identity transformation can be exploited to encode intrinsic coordinates in which the dynamics are linear. The resulting dynamics are given by a Koopman operator matrix $\mathbf{K}$. The decoder allows us to transform back to the original coordinates as well. Multiple time step prediction can be performed by repeated multiplication by the matrix $\mathbf{K}$ in the intrinsic coordinates. We demonstrate our method on a number of examples, including the heat equation and Burgers equation, as well as the substantially more challenging Kuramoto-Sivashinsky equation, showing that our method provides a robust architecture for discovering interpretable, linearizing transforms for nonlinear PDEs., Comment: 23 pages, 18 figures

We develop an unsupervised machine learning algorithm for the automated discovery and identification of traveling waves in spatio-temporal systems governed by partial differential equations (PDEs). Our method uses sparse regression and subspace clustering to robustly identify translational invariances that can be leveraged to build improved reduced order models (ROMs). Invariances, whether translational or rotational, are well known to compromise the ability of ROMs to produce accurate and/or low-rank representations of the spatio-temporal dynamics. However, by discovering translations in a principled way, data can be shifted into a coordinate systems where quality, low-dimensional ROMs can be constructed. This approach can be used on either numerical or experimental data with or without knowledge of the governing equations. We demonstrate our method on a variety of PDEs of increasing difficulty, taken from the field of fluid dynamics, showing the efficacy and robustness of the proposed approach., Comment: 14 pages, 8 figures

First principles modeling of physical systems has led to significant technological advances across all branches of science. For nonlinear systems, however, small modeling errors can lead to significant deviations from the true, measured behavior. Even in mechanical systems, where the equations are assumed to be well-known, there are often model discrepancies corresponding to nonlinear friction, wind resistance, etc. Discovering models for these discrepancies remains an open challenge for many complex systems. In this work, we use the sparse identification of nonlinear dynamics (SINDy) algorithm to discover a model for the discrepancy between a simplified model and measurement data. In particular, we assume that the model mismatch can be sparsely represented in a library of candidate model terms. We demonstrate the efficacy of our approach on several examples including experimental data from a double pendulum on a cart. We further design and implement a feed-forward controller in simulations, showing improvement with a discrepancy model., Comment: 8 pages, 5 figures, accepted by Conference on Decision and Control 2019

Poincar\'e maps are an integral aspect to our understanding and analysis of nonlinear dynamical systems. Despite this fact, the construction of these maps remains elusive and is primarily left to simple motivating examples. In this manuscript we propose a method of data-driven discovery of Poincar\'e maps based upon sparse regression techniques, specifically the sparse identification of nonlinear dynamics (SINDy) algorithm. This work can be used to determine the dynamics on and near invariant manifolds of a given dynamical system, as well as provide long-time forecasting of the coarse-grained dynamics of multiscale systems. Moreover, the method provides a mathematical formalism for determining nonlinear Floquet theory for the stability of nonlinear periodic orbits. The methods are applied to a range of examples including both ordinary and partial differential equations that exhibit periodic, quasi-periodic, and chaotic behavior.

Direct observation of a Rotating Detonation Engine combustion chamber has enabled the extraction of the kinematics of its detonation waves. These records exhibit a rich set of instabilities and bifurcations arising from the interaction of coherent wave fronts and global gain dynamics. We develop a model of the observed dynamics by recasting the Majda detonation analog as an autowave. The solution fronts become attractors of the engine; i.e., mode-locked rotating detonation waves. We find that detonative energy release competes with dissipation and gain recovery to produce the observed dynamics and a bifurcation structure common to driven-dissipative systems, such as mode-locked lasers., Comment: 10 pages, 11 figures

A traveling wave model for a semiconductor diode laser based on quantum wells is presented as well as a comprehensive theoretical model of the lasing dynamics produced by the intensity discrimination of the nonlinear mode-coupling in a waveguide array. By leveraging a recently developed model for the detailed semiconductor gain dynamics, the temporal shaping effects of the nonlinear mode-coupling induced by the waveguide arrays can be characterized. Specifically, the enhanced nonlinear pulse shaping provided by the waveguides are capable of generating stable frequency combs wavelength of 800 nm in a GaAs device, a parameter regime not feasible for stable combline generation using a single waveguide. Extensive numerical simulations showed that stable waveform generation could be achieved and optimized by an appropriate choice of the linear waveguide coupling coefficient, quantum well depth, and the input currents to the first and second waveguides. The model provides a first demonstration that a compact, efficient and robust on-chip comb source can be produced in GaAs., Comment: 9 pages, 10 figures. arXiv admin note: substantial text overlap with arXiv:1707.01582, arXiv:1906.11097

Machine learning (ML) is redefining what is possible in data-intensive fields of science and engineering. However, applying ML to problems in the physical sciences comes with a unique set of challenges: scientists want physically interpretable models that can (i) generalize to predict previously unobserved behaviors, (ii) provide effective forecasting predictions (extrapolation), and (iii) be certifiable. Autonomous systems will necessarily interact with changing and uncertain environments, motivating the need for models that can accurately extrapolate based on physical principles (e.g. Newton's universal second law for classical mechanics, $F=ma$). Standard ML approaches have shown impressive performance for predicting dynamics in an interpolatory regime, but the resulting models often lack interpretability and fail to generalize. We introduce a unified sparse optimization framework that learns governing dynamical systems models from data, selecting relevant terms in the dynamics from a library of possible functions. The resulting models are parsimonious, have physical interpretations, and can generalize to new parameter regimes. Our framework allows the use of non-convex sparsity promoting regularization functions and can be adapted to address key challenges in scientific problems and data sets, including outliers, parametric dependencies, and physical constraints. We show that the approach discovers parsimonious dynamical models on several example systems. This flexible approach can be tailored to the unique challenges associated with a wide range of applications and data sets, providing a powerful ML-based framework for learning governing models for physical systems from data., Comment: 22 pages, 5 figures

We develop a stable and efficient numerical scheme for modeling the optical field evolution in a nonlinear dispersive cavity with counter propagating waves and complex, semiconductor physics gain dynamics that are expensive to evaluate. Our stability analysis is characterized by a von-Neumann analysis which shows that many standard numerical schemes are unstable due to competing physical effects in the propagation equations. We show that the combination of a predictor-corrector scheme with an operator-splitting not only results in a stable scheme, but provides a highly efficient, single-stage evaluation of the gain dynamics. Given that the gain dynamics is the rate-limiting step of the algorithm, our method circumvents the numerical instability induced by the other cavity physics when evaluating the gain in an efficient manner. We demonstrate the stability and efficiency of the algorithm on a diode laser model which includes three waveguides and semiconductor gain dynamics. The laser is able to produce a repeating temporal waveform and stable optical comblines, thus demonstrating that frequency combs generation may be possible in chip scale, diode lasers., Comment: 11 pages, 5 figures. arXiv admin note: text overlap with arXiv:1707.01582

Machine learning (ML) and artificial intelligence (AI) algorithms are now being used to automate the discovery of physics principles and governing equations from measurement data alone. However, positing a universal physical law from data is challenging without simultaneously proposing an accompanying discrepancy model to account for the inevitable mismatch between theory and measurements. By revisiting the classic problem of modeling falling objects of different size and mass, we highlight a number of nuanced issues that must be addressed by modern data-driven methods for automated physics discovery. Specifically, we show that measurement noise and complex secondary physical mechanisms, like unsteady fluid drag forces, can obscure the underlying law of gravitation, leading to an erroneous model. We use the sparse identification of nonlinear dynamics (SINDy) method to identify governing equations for real-world measurement data and simulated trajectories. Incorporating into SINDy the assumption that each falling object is governed by a similar physical law is shown to improve the robustness of the learned models, but discrepancies between the predictions and observations persist due to subtleties in drag dynamics. This work highlights the fact that the naive application of ML/AI will generally be insufficient to infer universal physical laws without further modification.

Dynamic mode decomposition (DMD) is a data-driven method that models high-dimensional time series as a sum of spatiotemporal modes, where the temporal modes are constrained by linear dynamics. For nonlinear dynamical systems exhibiting strongly coherent structures, DMD can be a useful approximation to extract dominant, interpretable modes. In many domains with large spatiotemporal data---including fluid dynamics, video processing, and finance---the dynamics of interest are often perturbations about fixed points or equilibria, which motivates the application of DMD to centered (i.e. mean-subtracted) data. In this work, we show that DMD with centered data is equivalent to incorporating an affine term in the dynamic model and is not equivalent to computing a discrete Fourier transform. Importantly, DMD with centering can always be used to compute eigenvalue spectra of the dynamics. However, in many cases DMD without centering cannot model the corresponding dynamics, most notably if the dynamics have full effective rank. Additionally, we generalize the notion of centering to extracting arbitrary, but known, fixed frequencies from the data. We corroborate these theoretical results numerically on three nonlinear examples: the Lorenz system, a surveillance video, and brain recordings. Since centering the data is simple and computationally efficient, we recommend it as a preprocessing step before DMD; furthermore, we suggest that it can be readily used in conjunction with many other popular implementations of the DMD algorithm.

We introduce the dynamics mode decomposition for monitoring wide-area power grid networks from sparse measurement data. The mathematical framework fuses data from multiple sensors based on multivariate statistics, providing accurate full state estimation from limited measurements and generating data-driven forecasts for the state of the system. Our proposed data-driven strategy, which is based on energy metrics, can be used for the analysis of major disturbances in the network. The approach is tested and validated using time domain simulations in the IEEE 118 bus system under various disturbance scenarios and under different sparse observations of the system. In addition to state reconstruction, the minimal number of sensors required for monitoring disturbances can be evaluated. Visualization techniques are developed in order to aid in the analysis and characterization of the system after disturbance., Comment: 11 pages, 10 figures

The control of complex systems is of critical importance in many branches of science, engineering, and industry. Controlling an unsteady fluid flow is particularly important, as flow control is a key enabler for technologies in energy (e.g., wind, tidal, and combustion), transportation (e.g., planes, trains, and automobiles), security (e.g., tracking airborne contamination), and health (e.g., artificial hearts and artificial respiration). However, the high-dimensional, nonlinear, and multi-scale dynamics make real-time feedback control infeasible. Fortunately, these high-dimensional systems exhibit dominant, low-dimensional patterns of activity that can be exploited for effective control in the sense that knowledge of the entire state of a system is not required. Advances in machine learning have the potential to revolutionize flow control given its ability to extract principled, low-rank feature spaces characterizing such complex systems. We present a novel deep learning model predictive control (DeepMPC) framework that exploits low-rank features of the flow in order to achieve considerable improvements to control performance. Instead of predicting the entire fluid state, we use a recurrent neural network (RNN) to accurately predict the control relevant quantities of the system. The RNN is then embedded into a MPC framework to construct a feedback loop, and incoming sensor data is used to perform online updates to improve prediction accuracy. The results are validated using varying fluid flow examples of increasing complexity.

The discovery of governing equations from scientific data has the potential to transform data-rich fields that lack well-characterized quantitative descriptions. Advances in sparse regression are currently enabling the tractable identification of both the structure and parameters of a nonlinear dynamical system from data. The resulting models have the fewest terms necessary to describe the dynamics, balancing model complexity with descriptive ability, and thus promoting interpretability and generalizability. This provides an algorithmic approach to Occam's razor for model discovery. However, this approach fundamentally relies on an effective coordinate system in which the dynamics have a simple representation. In this work, we design a custom autoencoder to discover a coordinate transformation into a reduced space where the dynamics may be sparsely represented. Thus, we simultaneously learn the governing equations and the associated coordinate system. We demonstrate this approach on several example high-dimensional dynamical systems with low-dimensional behavior. The resulting modeling framework combines the strengths of deep neural networks for flexible representation and sparse identification of nonlinear dynamics (SINDy) for parsimonious models. It is the first method of its kind to place the discovery of coordinates and models on an equal footing., Comment: 25 pages, 6 figures; added acknowledgments

We present a data-driven method for separating complex, multiscale systems into their constituent time-scale components using a recursive implementation of dynamic mode decomposition (DMD). Local linear models are built from windowed subsets of the data, and dominant time scales are discovered using spectral clustering on their eigenvalues. This approach produces time series data for each identified component, which sum to a faithful reconstruction of the input signal. It differs from most other methods in the field of multiresolution analysis (MRA) in that it 1) accounts for spatial and temporal coherencies simultaneously, making it more robust to scale overlap between components, and 2) yields a closed-form expression for local dynamics at each scale, which can be used for short-term prediction of any or all components. Our technique is an extension of multi-resolution dynamic mode decomposition (mrDMD), generalized to treat a broader variety of multiscale systems and more faithfully reconstruct their isolated components. In this paper we present an overview of our algorithm and its results on two example physical systems, and briefly discuss some advantages and potential forecasting applications for the technique.

The Koopman and Perron Frobenius transport operators are fundamentally changing how we approach dynamical systems, providing linear representations for even strongly nonlinear dynamics. Although there is tremendous potential benefit of such a linear representation for estimation and control, transport operators are infinite-dimensional, making them difficult to work with numerically. Obtaining low-dimensional matrix approximations of these operators is paramount for applications, and the dynamic mode decomposition has quickly become a standard numerical algorithm to approximate the Koopman operator. Related methods have seen rapid development, due to a combination of an increasing abundance of data and the extensibility of DMD based on its simple framing in terms of linear algebra. In this chapter, we review key innovations in the data-driven characterization of transport operators for control, providing a high-level and unified perspective. We emphasize important recent developments around sparsity and control, and discuss emerging methods in big data and machine learning., Comment: 37 pages, 4 figures

In many applications, it is important to reconstruct a fluid flow field, or some other high-dimensional state, from limited measurements and limited data. In this work, we propose a shallow neural network-based learning methodology for such fluid flow reconstruction. Our approach learns an end-to-end mapping between the sensor measurements and the high-dimensional fluid flow field, without any heavy preprocessing on the raw data. No prior knowledge is assumed to be available, and the estimation method is purely data-driven. We demonstrate the performance on three examples in fluid mechanics and oceanography, showing that this modern data-driven approach outperforms traditional modal approximation techniques which are commonly used for flow reconstruction. Not only does the proposed method show superior performance characteristics, it can also produce a comparable level of performance with traditional methods in the area, using significantly fewer sensors. Thus, the mathematical architecture is ideal for emerging global monitoring technologies where measurement data are often limited.

Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an array $D$ of class response distributions, where $D_{ij}$ is the distribution of the $j^{th}$ pre-Softmax readout neuron's responses to the $i^{th}$ class. Given a test sample, Softmax implicitly uses only the row of this array $D$ that corresponds to the readout neurons' responses to the sample's true class. Leveraging more of this array $D$ can improve classifier accuracy, because the likelihoods of two competing classes can be encoded in other rows of $D$. To explore this potential resource, we develop a hybrid classifier (Softmax-Pooling Hybrid, $SPH$) that uses Softmax on high-scoring samples, but on low-scoring samples uses a log-likelihood method that pools the information from the full array $D$. We apply $SPH$ to models trained on a vectorized MNIST dataset to varying levels of accuracy. $SPH$ replaces only the final Softmax layer in the trained NN, at test time only. All training is the same as for Softmax. Because the pooling classifier performs better than Softmax on low-scoring samples, $SPH$ reduces test set error by 6% to 23%, using the exact same trained model, whatever the baseline Softmax accuracy. This reduction in error reflects hidden capacity of the trained NN that is left unused by Softmax., Comment: 9 pages, 6 figures

Optimal sensor and actuator selection is a central challenge in high-dimensional estimation and control. Nearly all subsequent control decisions are affected by these sensor/actuator locations, and optimal placement amounts to an intractable brute-force search among the combinatorial possibilities. In this work, we exploit balanced model reduction and greedy optimization to efficiently determine sensor and actuator selections that optimize observability and controllability. In particular, we determine locations that optimize scalar measures of observability and controllability via greedy matrix QR pivoting on the dominant modes of the direct and adjoint balancing transformations. Pivoting runtime scales linearly with the state dimension, making this method tractable for high-dimensional systems. The results are demonstrated on the linearized Ginzburg-Landau system, for which our algorithm approximates known optimal placements computed using costly gradient descent methods., Comment: 8 pages, 6 figures

The analysis of high-dimensional dynamical systems generally requires the integration of simulation data with experimental measurements. Experimental data often has substantial amounts of measurement noise that compromises the ability to produce accurate dimensionality reduction, parameter estimation, reduced order models, and/or balanced models for control. Data assimilation attempts to overcome the deleterious effects of noise by producing a set of algorithms for state estimation from noisy and possibly incomplete measurements. Indeed, methods such as Kalman filtering and smoothing are vital tools for scientists in fields ranging from electronics to weather forecasting. In this work we develop a novel framework for smoothing data based on known or partially known nonlinear governing equations. The method yields superior results to current techniques when applied to problems with known deterministic dynamics. By exploiting the numerical time-stepping constraints of the deterministic system, an optimization formulation can readily extract the noise from the nonlinear dynamics in a principled manner. The superior performance is due in part to the fact that it optimizes global state estimates. We demonstrate the efficiency and efficacy of the method on a number of canonical examples, thus demonstrating its viability for the wide range of potential applications stated above.

Conserved quantities, i.e. constants of motion, are critical for characterizing many dynamical systems in science and engineering. These quantities are related to underlying symmetries and they provide fundamental knowledge about physical laws, describe the evolution of the system, and enable system reduction. In this work, we formulate a data-driven architecture for discovering conserved quantities based on Koopman theory. The Koopman operator has emerged as a principled linear embedding of nonlinear dynamics, and its eigenfunctions establish intrinsic coordinates along which the dynamics behave linearly. Interestingly, eigenfunctions of the Koopman operator associated with vanishing eigenvalues correspond to conserved quantities of the underlying system. In this paper, we show that these invariants may be identified with data-driven regression and power series expansions, based on the infinitesimal generator of the Koopman operator. We further establish a connection between the Koopman framework, conserved quantities, and the Lie-Poisson bracket. This data-driven method for discovering conserved quantities is demonstrated on the three-dimensional rigid body equations, where we simultaneously discover the total energy and angular momentum and use these intrinsic coordinates to develop a model predictive controller to track a given reference value., Comment: 7 pages, 2 figures, 57th IEEE Conference on Decision and Control (CDC 2018)

Nonlinear dynamical systems are ubiquitous in science and engineering, yet analysis and prediction of these systems remains a challenge. Koopman operator theory circumvents some of these issues by considering the dynamics in the space of observable functions on the state, in which the dynamics are intrinsically linear and thus amenable to standard techniques from numerical analysis and linear algebra. However, practical issues remain with this approach, as the space of observables is infinite-dimensional and selecting a subspace of functions in which to accurately represent the system is a nontrivial task. In this work we consider time-delay observables to represent nonlinear dynamics in the Koopman operator framework. We prove the surprising result that Koopman operators for different systems admit universal (system-independent) representations in these coordinates, and give analytic expressions for these representations. In addition, we show that for certain systems a restricted class of these observables form an optimal finite-dimensional basis for representing the Koopman operator, and that the analytic representation of the Koopman operator in these coordinates coincides with results computed by the dynamic mode decomposition. We provide numerical examples to complement our results. In addition to being theoretically interesting, these results have implications for a number of linearization algorithms for dynamical systems., Comment: 28 pages, 6 figures

We develop data-driven dynamical models of the nonlinear aeroelastic effects on a long-span suspension bridge from sparse, noisy sensor measurements which monitor the bridge. Using the {\em sparse identification of nonlinear dynamics} (SINDy) algorithm, we are able to identify parsimonious, time-varying dynamical systems that capture vortex-induced vibration (VIV) events in the bridge. Thus we are able to posit new, data-driven models highlighting the aeroelastic interaction of the bridge structure with VIV events. The bridge dynamics are shown to have distinct, time-dependent modes of behavior, thus requiring parametric models to account for the diversity of dynamics. Our method generates hitherto unknown bridge-wind interaction models that go beyond current theoretical and computational descriptions. Our proposed method for real-time monitoring and model discovery allow us to move our model predictions beyond lab theory to practical engineering design, which has the potential to assess bad engineering configurations that are susceptible to deleterious bridge-wind interactions. With the rise of real-time sensor networks on major bridges, our model discovery methods can enhance an engineers ability to assess the nonlinear aeroelastic interactions of the bridge with its wind environment., Comment: 14 pages, 12 figures

Machine learning (ML) classifiers always benefit from more informative input features. We seek to auto-generate stronger feature sets in order to address the difficulty that ML methods often experience given limited training data. A wide range of biological neural nets (BNNs) excel at fast learning, implying that they are adept at extracting informative features. We can thus look to BNNs for tools to improve ML performance in this low-data regime. The insect olfactory network learns new odors very rapidly, by means of three key elements: A competitive inhibition layer; a high-dimensional sparse plastic layer; and Hebbian updates of synaptic weights. In this work, we deployed MothNet, a computational model of the insect olfactory network, as an automatic feature generator: Attached as a front-end pre-processor, its Readout Neurons provided new features, derived from the original features, for use by standard ML classifiers. We found that these "insect cyborgs", i.e. classifiers that are part-insect model and part-ML method, had significantly better performance than baseline ML methods alone on a vectorized MNIST dataset. The MothNet feature generator also substantially out-performed other feature generating methods such as PCA, PLS, and NNs, as well as pre-training to initialize NN weights. Cyborgs improved relative test set accuracy by an average of 6% to 33% depending on baseline ML accuracy, while relative reduction in test set error exceeded 50% for higher baseline accuracy ML models. These results indicate the potential value of BNN-inspired feature generators in the ML context., Comment: 14 pages, 5 figures, 6 tables

Hybrid systems are traditionally difficult to identify and analyze using classical dynamical systems theory. Moreover, recently developed model identification methodologies largely focus on identifying a single set of governing equations solely from measurement data. In this article, we develop a new methodology, Hybrid-Sparse Identification of Nonlinear Dynamics (Hybrid-SINDy), which identifies separate nonlinear dynamical regimes, employs information theory to manage uncertainty, and characterizes switching behavior. Specifically, we utilize the nonlinear geometry of data collected from a complex system to construct a set of coordinates based on measurement data and augmented variables. Clustering the data in these measurement-based coordinates enables the identification of nonlinear hybrid systems. This methodology broadly empowers nonlinear system identification without constraining the data locally in time and has direct connections to hybrid systems theory. We demonstrate the success of this method on numerical examples including a mass-spring hopping model and an infectious disease model. Characterizing complex systems that switch between dynamic behaviors is integral to overcoming modern challenges such as eradication of infectious diseases, the design of efficient legged robots, and the protection of cyber infrastructures., Comment: 22 pages, 5 figures, 1 table. Code repository: https://github.com/niallmm/Hybrid-SINDy

A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples., Comment: Updated a reference to reflect publication

Most organisms suffer neuronal damage throughout their lives, which can impair performance of core behaviors. Their neural circuits need to maintain function despite injury, which in particular requires preserving key system outputs. In this work, we explore whether and how certain structural and functional neuronal network motifs act as injury mitigation mechanisms. Specifically, we examine how (i) Hebbian learning, (ii) high levels of noise, and (iii) parallel inhibitory and excitatory connections contribute to the robustness of the olfactory system in the Manduca sexta moth. We simulate injuries on a detailed computational model of the moth olfactory network calibrated to in vivo data. The injuries are modeled on focal axonal swellings, a ubiquitous form of axonal pathology observed in traumatic brain injuries and other brain disorders. Axonal swellings effectively compromise spike train propagation along the axon, reducing the effective neural firing rate delivered to downstream neurons. All three of the network motifs examined significantly mitigate the effects of injury on readout neurons, either by reducing injury's impact on readout neuron responses or by restoring these responses to pre-injury levels. These motifs may thus be partially explained by their value as adaptive mechanisms to minimize the functional effects of neural injury. More generally, robustness to injury is a vital design principle to consider when analyzing neural systems., Comment: 17 pages, 10 figures

Regularized regression problems are ubiquitous in statistical modeling, signal processing, and machine learning. Sparse regression in particular has been instrumental in scientific model discovery, including compressed sensing applications, variable selection, and high-dimensional analysis. We propose a broad framework for sparse relaxed regularized regression, called SR3. The key idea is to solve a relaxation of the regularized problem, which has three advantages over the state-of-the-art: (1) solutions of the relaxed problem are superior with respect to errors, false positives, and conditioning, (2) relaxation allows extremely fast algorithms for both convex and nonconvex formulations, and (3) the methods apply to composite regularizers such as total variation (TV) and its nonconvex variants. We demonstrate the advantages of SR3 (computational efficiency, higher accuracy, faster convergence rates, greater flexibility) across a range of regularized regression problems with synthetic and real data, including applications in compressed sensing, LASSO, matrix completion, TV regularization, and group sparsity. To promote reproducible research, we also provide a companion MATLAB package that implements these examples., Comment: 19 pages, 14 figures

We propose a new solution to the blind source separation problem that factors mixed time-series signals into a sum of spatiotemporal modes, with the constraint that the temporal components are intrinsic mode functions (IMF's). The key motivation is that IMF's allow the computation of meaningful Hilbert transforms of non-stationary data, from which instantaneous time-frequency representations may be derived. Our spatiotemporal intrinsic mode decomposition (STIMD) method leverages spatial correlations to generalize the extraction of IMF's from one-dimensional signals, commonly performed using the empirical mode decomposition (EMD), to multi-dimensional signals. Further, this data-driven method enables future-state prediction. We demonstrate STIMD on several synthetic examples, comparing it to common matrix factorization techniques, namely singular value decomposition (SVD), independent component analysis (ICA), and dynamic mode decomposition (DMD). We show that STIMD outperforms these methods at reconstruction and extracting interpretable modes. Next, we apply STIMD to analyze two real-world datasets, gravitational wave data and neural recordings from the rodent hippocampus., Comment: 23 pages, 10 figures

In this work we present a data-driven method for the discovery of parametric partial differential equations (PDEs), thus allowing one to disambiguate between the underlying evolution equations and their parametric dependencies. Group sparsity is used to ensure parsimonious representations of observed dynamics in the form of a parametric PDE, while also allowing the coefficients to have arbitrary time series, or spatial dependence. This work builds on previous methods for the identification of constant coefficient PDEs, expanding the field to include a new class of equations which until now have eluded machine learning based identification methods. We show that group sequentially thresholded ridge regression outperforms group LASSO in identifying the fewest terms in the PDE along with their parametric dependency. The method is demonstrated on four canonical models with and without the introduction of noise.

Networked power grid systems are susceptible to a phenomenon known as Coherent Swing Instability (CSI), in which a subset of machines in the grid lose synchrony with the rest of the network. We develop network level evaluation metrics to (i) identify community substructures in the power grid network, (ii) determine weak points in the network that are particularly sensitive to CSI, and (iii) produce an engineering approach for the addition of transmission lines to reduce the incidences of CSI in existing networks, or design new power grid networks that are robust to CSI by their network design. For simulations on a reduced model for the American Northeast power grid, where a block of buses representing the New England region exhibit a strong propensity for CSI, we show that modifying the network's connectivity structure can markedly improve the grid's resilience to CSI. Our analysis provides a versatile diagnostic tool for evaluating the efficacy of adding lines to a power grid which is known to be prone to CSI. This is a particularly relevant problem in large-scale power systems, where improving stability and robustness to interruptions by increasing overall network connectivity is not feasible due to financial and infrastructural constraints., Comment: 15 pages, 10 figures

A major challenge in the study of dynamical systems is that of model discovery: turning data into models that are not just predictive, but provide insight into the nature of the underlying dynamical system that generated the data. This problem is made more difficult by the fact that many systems of interest exhibit diverse behaviors across multiple time scales. We introduce a number of data-driven strategies for discovering nonlinear multiscale dynamical systems and their embeddings from data. We consider two canonical cases: (i) systems for which we have full measurements of the governing variables, and (ii) systems for which we have incomplete measurements. For systems with full state measurements, we show that the recent sparse identification of nonlinear dynamical systems (SINDy) method can discover governing equations with relatively little data and introduce a sampling method that allows SINDy to scale efficiently to problems with multiple time scales. Specifically, we can discover distinct governing equations at slow and fast scales. For systems with incomplete observations, we show that the Hankel alternative view of Koopman (HAVOK) method, based on time-delay embedding coordinates, can be used to obtain a linear model and Koopman invariant measurement system that nearly perfectly captures the dynamics of nonlinear quasiperiodic systems. We introduce two strategies for using HAVOK on systems with multiple time scales. Together, our approaches provide a suite of mathematical strategies for reducing the data required to discover and model nonlinear multiscale systems., Comment: 21 pages, 6 figures

The problem of optimally placing sensors under a cost constraint arises naturally in the design of industrial and commercial products, as well as in scientific experiments. We consider a relaxation of the full optimization formulation of this problem and then extend a well-established QR-based greedy algorithm for the optimal sensor placement problem without cost constraints. We demonstrate the effectiveness of this algorithm on data sets related to facial recognition, climate science, and fluid mechanics. This algorithm is scalable and often identifies sparse sensors with near optimal reconstruction performance, while dramatically reducing the overall cost of the sensors. We find that the cost-error landscape varies by application, with intuitive connections to the underlying physics. Additionally, we include experiments for various pre-processing techniques and find that a popular technique based on the singular value decomposition is often sub-optimal., Comment: 13 pages, 12 figures

Sparse principal component analysis (SPCA) has emerged as a powerful technique for modern data analysis, providing improved interpretation of low-rank structures by identifying localized spatial structures in the data and disambiguating between distinct time scales. We demonstrate a robust and scalable SPCA algorithm by formulating it as a value-function optimization problem. This viewpoint leads to a flexible and computationally efficient algorithm. Further, we can leverage randomized methods from linear algebra to extend the approach to the large-scale (big data) setting. Our proposed innovation also allows for a robust SPCA formulation which obtains meaningful sparse principal components in spite of grossly corrupted input data. The proposed algorithms are demonstrated using both synthetic and real world data, and show exceptional computational efficiency and diagnostic performance.

Big data has become a critically enabling component of emerging mathematical methods aimed at the automated discovery of dynamical systems, where first principles modeling may be intractable. However, in many engineering systems, abrupt changes must be rapidly characterized based on limited, incomplete, and noisy data. Many leading automated learning techniques rely on unrealistically large data sets and it is unclear how to leverage prior knowledge effectively to re-identify a model after an abrupt change. In this work, we propose a conceptual framework to recover parsimonious models of a system in response to abrupt changes in the low-data limit. First, the abrupt change is detected by comparing the estimated Lyapunov time of the data with the model prediction. Next, we apply the sparse identification of nonlinear dynamics (SINDy) regression to update a previously identified model with the fewest changes, either by addition, deletion, or modification of existing model terms. We demonstrate this sparse model recovery on several examples for abrupt system change detection in periodic and chaotic dynamical systems. Our examples show that sparse updates to a previously identified model perform better with less data, have lower runtime complexity, and are less sensitive to noise than identifying an entirely new model. The proposed abrupt-SINDy architecture provides a new paradigm for the rapid and efficient recovery of a system model after abrupt changes.

Diffusion maps are an emerging data-driven technique for non-linear dimensionality reduction, which are especially useful for the analysis of coherent structures and nonlinear embeddings of dynamical systems. However, the computational complexity of the diffusion maps algorithm scales with the number of observations. Thus, long time-series data presents a significant challenge for fast and efficient embedding. We propose integrating the Nystr\"om method with diffusion maps in order to ease the computational demand. We achieve a speedup of roughly two to four times when approximating the dominant diffusion map components.