Search

Deep learning models offer superior performance compared to other machine learning techniques for a variety of tasks and domains, but pose their own challenges. In particular, deep learning models require larger training times as the depth of a model increases, and suffer from vanishing gradients. Several solutions address these problems independently, but there have been minimal efforts to identify an integrated solution that improves the performance of a model by addressing vanishing gradients, as well as accelerates the training process to achieve higher performance at larger learning rates. In this work, we intelligently determine which layers of a deep learning model to apply gradient amplification to, using a formulated approach that analyzes gradient fluctuations of layers during training. Detailed experiments are performed for simpler and deeper neural networks using two different intelligent measures and two different thresholds that determine the amplification layers, and a training strategy where gradients are amplified only during certain epochs. Results show that our amplification offers better performance compared to the original models, and achieves accuracy improvement of around 2.5% on CIFAR- 10 and around 4.5% on CIFAR-100 datasets, even when the models are trained with higher learning rates.

Using the technique of Poincar\'{e} return maps, we disclose an intricate order of the subsequent homoclinics near the primary homoclinic bifurcation of the Shilnikov saddle-focus in systems with reflection symmetry. We also reveal the admissible shapes of the corresponding bifurcation curves in a parameter plane of such systems. The scalability ratio of geometry and organization is proven to be universal for such homoclinic bifurcations of higher orders. Two applications with similar dynamics due to the Shilnikov saddle-foci, a smooth adaptation of the Chua circuit and a 3D normal form, are used to illustrate the theory

We study the origin of homoclinic chaos in the classical 3D model proposed by O. R\"ossler in 1976. Of our particular interest are the convoluted bifurcations of the Shilnikov saddle-foci and how their synergy determines the global unfolding of the model, along with transformations of its chaotic attractors. We apply two computational methods proposed, 1D return maps and a symbolic approach specifically tailored to this model, to scrutinize homoclinic bifurcations, as well as to detect the regions of structurally stable and chaotic dynamics in the parameter space of the R\"ossler model.

We present a case study elaborating on the multiplicity and self-similarity of homoclinic and heteroclinic bifurcation structures in the 2D and 3D parameter spaces of a nonlinear laser model with a Lorenz-like chaotic attractor. In a symbiotic approach combining the traditional parameter continuation methods using MatCont and a newly developed technique called the Deterministic Chaos Prospector (DCP) utilizing symbolic dynamics on fast parallel computing hardware with graphics processing units (GPUs), we exhibit how specific codimension-two bifurcations originate and pattern regions of chaotic and simple dynamics in this classical model. We show detailed computational reconstructions of key bifurcation structures such as Bykov T-point spirals and inclination flips in 2D parameter space, as well as the spatial organization and 3D embedding of bifurcation surfaces, parametric saddles, and isolated closed curves (isolas)., Comment: 28 pages, 23 figures

We disclose the generality of the intrinsic mechanisms underlying multistability in reciprocally inhibitory 3-cell circuits composed of simplified, low-dimensional models of oscillatory neurons, as opposed to those of a detailed Hodgkin- Huxley type . The computational reduction to return maps for the phase-lags between neurons reveals a rich multiplicity of rhythmic patterns in such circuits. We perform a detailed bifurcation analysis to show how such rhythms can emerge, disappear, and gain or lose stability, as the parameters of the individual cells and the synapses are varied.

We developed a powerful computational approach to elaborate on onset mechanisms of deterministic chaos due to complex homoclinic bifurcations in diverse systems. Its core is the reduction of phase space dynamics to symbolic binary representations that lets one detect regions of simple and complex dynamics as well as fine organization structures of the latter in parameter space. Massively parallel simulations shorten the computational time to disclose highly detailed bifurcation diagrams to a few seconds.

A suite of analytical and computational techniques based on symbolic representations of simple and complex dynamics, is further developed and employed to unravel the global organization of bi-parametric structures that underlie the emergence of chaos in a simplified resonantly coupled wave triplet system, known as the Rabinovich system. Bi-parametric scans reveal the stunning intricacy and intramural connections between homoclinic and heteroclinic connections, and codimension-2 Bykov T-points and saddle structures, which are the prime organizing centers of complexity of the bifurcation unfolding of the given system. This suite includes Deterministic Chaos Prospector (DCP) to sweep and effectively identify regions of simple (Morse-Smale) and chaotic structurally unstable dynamics in the system. Our analysis provides striking new insights into the complex behaviors exhibited by this and similar systems., Comment: Challenges in Complexity: Dynamics, Patterns, and Cognition, Springer series Nonlinear Systems and Complexity 2017

Neural network models have been used to construct energy landscapes for modeling biological phenomena, in which the minima of the landscape correspond to memory patterns stored by the network. Here, we show that dynamic properties of those landscapes, such as the sizes of the basins of attraction and the density of stable and metastable states, depend strongly on the correlations between the memory patterns and can be altered by introducing hierarchical structures. Our findings suggest dynamic features of energy landscapes can be controlled by choosing the correlations between patterns, Comment: 5 pages, 2 figures

Cellular reprogramming, the conversion of one cell type to another, has fundamentally transformed our conception of cell types. Cellular reprogramming induces global changes in gene expression involving hundreds of transcription factors and thousands of genes and understanding how cells globally alter their gene expression profile during reprogramming is an open problem. Here we reanalyze time-series data on cellular reprogramming from differentiated cell types to induced pluripotent stem cells (iPSCs) and show that gene expression dynamics during reprogramming follow a simple one-dimensional reaction coordinate. This reaction coordinate is independent of both the time it takes to reach the iPSC state as well as the details of experimental protocol used. Using Monte-Carlo simulations, we show that such a reaction coordinate emerges naturally from epigenetic landscape models of cell identity where cellular reprogramming is viewed as a "barrier-crossing" between the starting and ending cell fates. The model also provides gene-level insight into reprogramming dynamics and resolves a debate in the stem cell field about the different phases of reprogramming dynamics. Overall, our analysis and model suggest that gene expression dynamics during reprogramming follow a canonical trajectory consistent with the idea of an "optimal path" in gene expression space for reprogramming., Comment: 19 pages, 15 figures