Search

Musculoskeletal simulations are useful in biomechanics to investigate the causes of movement disorder, to estimate non-measurable physiological quantities or to study the optimality of human movement. We introduce Bioptim, an easy-to-use Python framework for biomechanical optimal control, handling musculoskeletal models. Relying on algorithmic differentiation and the multiple shooting formulation, Bioptim interfaces nonlinear solvers to quickly provide dynamically consistent optimal solutions. The software is both computationally efficient (C++ core) and easily customizable, thanks to its Python interface. It allows to quickly define a variety of biomechanical problems such as motion tracking/prediction, muscle-driven simulations, parameters optimization, multiphase problems, etc. It is also intended for real-time applications such as moving horizon estimation and model predictive control. Six contrasting examples are presented, comprising various models, dynamics, objective functions and constraints. They include data-driven simulations (i.e., a multiphase muscle driven gait cycle and an upper-limb real-time moving horizon estimation of muscle forces) and predictive simulations (i.e., a muscle-driven pointing task, a twisting somersault with a quaternion-based model, a position controller using external forces, and a multiphase torque-driven maximum-height jump motion).

Background COVID-19 has caused 3.34m deaths till 13/May/2021. It is now still causing confirmed cases and ongoing deaths every day. Method This study investigated whether fusing chest CT with chest X-ray can help improve the AI's diagnosis performance. Data harmonization is employed to make a homogeneous dataset. We create an end-to-end multiple-input deep convolutional attention network (MIDCAN) by using the convolutional block attention module (CBAM). One input of our model receives 3D chest CT image, and other input receives 2D X-ray image. Besides, multiple-way data augmentation is used to generate fake data on training set. Grad-CAM is used to give explainable heatmap. Results The proposed MIDCAN achieves a sensitivity of 98.10±1.88%, a specificity of 97.95±2.26%, and an accuracy of 98.02±1.35%. Conclusion Our MIDCAN method provides better results than 8 state-of-the-art approaches. We demonstrate the using multiple modalities can achieve better results than individual modality. Also, we demonstrate that CBAM can help improve the diagnosis performance.

With the explosive growth of computational resources and data generation, deep machine learning has been successfully employed in various applications. One important and emerging scientific application of deep learning involves solving differential equations. Here, physics-informed neural networks (PINNs) are developed to solve the differential equations associated with a specific scientific problem. As such, algorithms for solving the differential equations by embedding their initial and boundary conditions in the cost function of the artificial neural networks using algorithmic differentiation must also be developed. In this study, various PINNs are adopted to estimate the stresses in the tablets and the interphase of a single lap joint. The proposed model is represented by two fourth-order non-homogeneous coupled partial differential equations, with the axial stresses in the upper and lower tablets adopted as the dependent variables. The axial stresses are a function of the tablet length, which presents the independent variable. Therefore, the axial stresses in the tablets are estimated by solving the coupled partial differential equations when subjected to the boundary conditions, whereas the remaining stress components are expressed in terms of axial stresses. The results obtained using the developed methodology are validated using the results obtained via MAPLE software.

Neural networks with physical governing equations as constraints have recently created a new trend in machine learning research. In this context, a review of related research is first presented and discussed. The potential offered by such physics-informed deep learning models for computations in geomechanics is demonstrated by application to one-dimensional (1D) consolidation. The governing equation for 1D problems is applied as a constraint in the deep learning model. The deep learning model relies on automatic differentiation for applying the governing equation as a constraint, based on the mathematical approximations established by the neural network. The total loss is measured as a combination of the training loss (based on analytical and model predicted solutions) and the constraint loss (a requirement to satisfy the governing equation). Two classes of problems are considered: forward and inverse problems. The forward problems demonstrate the performance of a physically constrained neural network model in predicting solutions for 1D consolidation problems. Inverse problems show prediction of the coefficient of consolidation. Terzaghi’s problem, with varying boundary conditions, is used as a numerical example and the deep learning model shows a remarkable performance in both the forward and inverse problems. While the application demonstrated here is a simple 1D consolidation problem, such a deep learning model integrated with a physical law has significant implications for use in, such as, faster real-time numerical prediction for digital twins, numerical model reproducibility and constitutive model parameter optimization.

Herein, we present for the first time a general methodology for obtaining arbitrary-order nuclear coordinate derivatives of electronic energies derived from quantum chemistry methods. By leveraging modern advances in automatic differentiation software, we demonstrate that exact derivatives can be obtained for any method. This innovation completely bypasses the issues associated with the computational stability of applying numerical differentiation methods and dispenses the need to derive challenging formulae for analytic energy derivatives. We describe a freely available and open-source software implementation of our scheme and demonstrate its use in obtaining exact nuclear derivatives of energies from Hartree-Fock theory, second-order Møller-Plesset perturbation theory (MP2), and coupled cluster theory with single, double, and perturbative triple excitations [CCSD(T)]. Our sample computations include up to sextic derivatives and span a variety of test systems with up to 100 basis functions, confirming the viability of this scheme for a wide range of applications. Many of the results obtained have hitherto been unobtainable by exact means due to a lack of higher-order derivative formulae. The details of our implementation and possible further developments are discussed.

Ultrathin meta-optics offer unmatched, multifunctional control of light. Next-generation optical technologies, however, demand unprecedented performance. This will likely require design algorithms surpassing the capability of human intuition. For the adjoint method, this requires explicitly deriving gradients, which is sometimes challenging for certain photonics problems. Existing techniques also comprise a patchwork of application-specific algorithms, each focused in scope and scatterer type. Here, we leverage algorithmic differentiation as used in artificial neural networks, treating photonic design parameters as trainable weights, optical sources as inputs, and encapsulating device performance in the loss function. By solving a complex, degenerate eigenproblem and formulating rigorous coupled-wave analysis as a computational graph, we support both arbitrary, parameterized scatterers and topology optimization. With iteration times below the cost of two forward simulations typical of adjoint methods, we generate multilayer, multifunctional, and aperiodic meta-optics. As an open-source platform adaptable to other algorithms and problems, we enable fast and flexible meta-optical design. The design of meta-optics and photonics components poses a number of high-dimensional, fabrication-constrained, computationally expensive optimization problems. Here, the authors propose a framework based on algorithmic differentiation to optimize arbitrary meta-optical devices with any shape or topology, enabling rapid inverse design of photonics and complex meta-optical devices.

The classical Clarke subdifferential alone is inadequate for understanding automatic differentiation in nonsmooth contexts. Instead, we can sometimes rely on enlarged generalized gradients called “...

This paper uses gradient-based optimization to minimize the mass of a solar-regenerative high-altitude long-endurance flying-wing aircraft while accounting for nonlinear aeroelastic effects. The ai...

A current challenge for the structural topology optimization methods is the development of trustful techniques to account for different physics interactions. This paper devises a technique that considers separate physics analysis and optimization within the context of fluid-structure interaction (FSI) systems. Steady-state laminar flow and small structural displacements are assumed. We solve the compliance minimization problem subject to single or multiple volume constraints considering design-dependent FSI loads. For that, the TOBS (topology optimization of binary structures) method is applied. The TOBS approach uses binary {0,1} design variables, which can be advantageous when dealing with design-dependent physics interactions, e.g., in cases where fluid-structure boundaries are allowed to change during optimization. The COMSOL Multiphysics software is used to solve the fluid-structure equations and output the sensitivities using automatic differentiation. The TOBS optimizer provides a new set of {0,1} variables at every iteration. Numerical examples show smoothly converged solutions.

Large Bayesian vector autoregressions with the natural conjugate prior are now routinely used for forecasting and structural analysis. It has been shown that selecting the prior hyperparameters in a data‐driven manner can often substantially improve forecast performance. We propose a computationally efficient method to obtain the optimal hyperparameters based on automatic differentiation, which is an efficient way to compute derivatives. Using a large US data set, we show that using the optimal hyperparameter values leads to substantially better forecast performance. Moreover, the proposed method is much faster than the conventional grid‐search approach, and is applicable in high‐dimensional optimization problems. The new method thus provides a practical and systematic way to develop better shrinkage priors for forecasting in a data‐rich environment.

Python has become the programming language of choice for research and industry projects related to data science, machine learning, and deep learning. Since optimization is an inherent part of these research fields, more optimization related frameworks have arisen in the past few years. Only a few of them support optimization of multiple conflicting objectives at a time, but do not provide comprehensive tools for a complete multi-objective optimization task. To address this issue, we have developed pymoo, a multi-objective optimization framework in Python. We provide a guide to getting started with our framework by demonstrating the implementation of an exemplary constrained multi-objective optimization scenario. Moreover, we give a high-level overview of the architecture of pymoo to show its capabilities followed by an explanation of each module and its corresponding sub-modules. The implementations in our framework are customizable and algorithms can be modified/extended by supplying custom operators. Moreover, a variety of single, multi- and many-objective test problems are provided and gradients can be retrieved by automatic differentiation out of the box. Also, pymoo addresses practical needs, such as the parallelization of function evaluations, methods to visualize low and high-dimensional spaces, and tools for multi-criteria decision making. For more information about pymoo, readers are encouraged to visit: https://pymoo.org.

This paper presents the winning submission of the M4 forecasting competition. The submission utilizes a dynamic computational graph neural network system that enables a standard exponential smoothing model to be mixed with advanced long short term memory networks into a common framework. The result is a hybrid and hierarchical forecasting method.

Seismic full waveform inversion is a common technique that is used in the investigation of subsurface geology. Its classic implementation involves forward modeling of seismic wavefield based on a certain type of wave equation, which reflects the physics nature of subsurface seismic wavefield propagation. However, obtaining a good inversion result using traditional seismic waveform inversion methods usually comes with a high computational cost. Recently, with the emerging popularity of deep learning techniques in various computer vision tasks, deep neural network (DNN) has demonstrated an impressive ability in dealing with complex nonlinear problems, including seismic velocity inversion. Now, extensive efforts have been made in developing a DNN architecture to tackle the problem of seismic velocity inversion, and promising results have been achieved. However, due to the dependence of a labeled dataset, i.e., the barely accessible true velocity model corresponding to real seismic data, the current supervised deep learning inversion framework may suffer from limitations on generalization. One possible solution to mitigate this issue is to impose the governing physics into this kind of purely data-driven method. Thus, following the procedures of traditional seismic full waveform inversion, we propose a seismic waveform inversion network, namely SWINet, based on wave-equation-based forward modeling network cells. By treating the single-shot observation data and its corresponding shot position as training data pairs, the inverted velocity model can be obtained as the trainable network parameters. Moreover, since the proposed seismic waveform inversion method is performed in a neural-network way, its implementation and inversion effect could benefit from some built-in tools in Pytorch, such as automatic differentiation, Adam optimizer and mini-batch strategy, etc. Numerical examples indicate that the SWINet method may possess great potential in resulting a good velocity inversion effect with relatively fast convergence and lower computation cost.

Algorithmic decision-making and similar types of artificial intelligence (AI) may lead to improvements in all sectors of society but can also have discriminatory effects. While current non-discrimination law offers people some protection, algorithmic decision-making presents the lawmakers and law enforcement with several challenges. For instance, algorithms can generate new categories of people based on seemingly innocuous characteristics, such as web browser preference or apartment number, or more complicated categories combining many data points. Such new types of differentiation could evade non-discrimination law, as browser type and house number are not protected characteristics, but such differentiation could still be unfair, for instance if it reinforces social inequality. This paper attempts to determine which system of non-discrimination law would best be applied to algorithmic decision-making, considering that algorithms can differentiate on the basis of characteristics that do not correlate with protected grounds of discrimination such as ethnicity or gender. The paper analyses the current loopholes in the protection offered by non-discrimination law and explores the best way for lawmakers to approach the issue of algorithmic differentiation. While we focus on Europe, this paper’s concentration on concept and theory rather than specific application, should prove useful for scholars and policymakers from other regions as they encounter similar problems with algorithmic decision-making.

OBJECTIVES The aim of this study is to develop a Machine Learning (ML) algorithm for an automatic classification of fetal occiput position at transperineal ultrasound (TPU) during the second stage of labor. METHODS Prospective cohort study including singleton term pregnancies (> 37 weeks of gestation) in the second stage of labor, with the fetus in cephalic presentation. Transabdominal ultrasound was preliminarily performed to assess the actual fetal occiput position, which was labeled as occiput anterior (OA) or non-occiput anterior (non-OA). Subsequently, for each case, one sonographic image of the fetal head was acquired on the axial plane using TPU and archived on a cloud for remote analysis. Using the transabdominal sonographic diagnosis as the gold standard, a ML algorithm based on a pattern recognition feed-forward neural network was trained on the transperineal images to discriminate between OA and non-OA cases. In the training phase the model tuned its parameters in order to approximate correctly the training data - i.e., the training dataset - in order to correctly assess the fetal head position, by exploiting geometric, morphological and intensity-based features of the images. In the testing phase, the diagnostic performance of the algorithm was evaluated on unlabeled data, which represented the testing dataset. On this group the ability of the ML algorithm to differentiate the OA from the non-OA fetal positions was assessed in terms of diagnostic accuracy. The F1 -score and Precision-Recall Area Under the Curve (PR-AUC) were also calculated to assess the algorithm's performance. The Cohen's kappa (k) was finally added to evaluate the agreement between the algorithm and the gold standard. RESULTS Over a period of 24 months, 1219 women in the second stage of labor were enrolled. They were classified as OA (n=801 or 65.7%) or non-OA (n=418 or 34.3%) on the basis of transabdominal ultrasound. From both the sub-groups (OA and non-OA), 70% of the patients were randomly assigned to the training dataset (824 patients) while the remaining 30% (395 patients) were used as testing dataset. On the latter group the ML based algorithm yielded a correct classification of the fetal occiput position in 90.6% of cases (357 out of 395), including 224 out of 246 OA (91.0%) and of 133 out of 149 non-OA images (89.3%). Moreover, for the evaluation the algorithm's performance we found a F1 -score=88.7% and PR-AUC=85.4%. The algorithm showed a balanced performance in the recognition of both anterior and non-anterior occiput positions. Eventually, the robustness of the proposed algorithm was confirmed by a high agreement with the gold standard method (k = 0.81; p

In this paper we analyze the effects of using nonlinear least squares for parameter identification of symbolic regression models and integrate it as local search mechanism in tree-based genetic programming. We employ the Levenberg–Marquardt algorithm for parameter optimization and calculate gradients via automatic differentiation. We provide examples where the parameter identification succeeds and fails and highlight its computational overhead. Using an extensive suite of symbolic regression benchmark problems we demonstrate the increased performance when incorporating nonlinear least squares within genetic programming. Our results are compared with recently published results obtained by several genetic programming variants and state of the art machine learning algorithms. Genetic programming with nonlinear least squares performs among the best on the defined benchmark suite and the local search can be easily integrated in different genetic programming algorithms as long as only differentiable functions are used within the models.

There are several AD tools available, which all implement different strategies for the reverse mode of AD. The major strategies are primal value taping (implemented e.g. by ADOL-c) and Jacobi taping (implemented e.g. by adept and dco/c++). Especially for Jacobi taping, recent advances by using expression templates make this approach very attractive for large scale software. The current implementations are either closed source or miss essential features and flexibility. Therefore, we present the new AD tool CoDiPack (Code Differentiation Package) in this paper. It is specifically designed for a minimal memory consumption and optimal runtime, such that it can be used for the differentiation of large scale software. An essential part of the design of CoDiPack is the modular layout and the recursive data structures, which do not only allow the efficient implementation of the Jacobi taping approach, but will also enable other approaches like the primal value taping or new research ideas. We will also present the performance value of CoDiPack on a generic PDE example and on the SU2 code., Comment: 21 pages, 11 figures, 6 tables, CoDiPack: https://github.com/SciCompKL/CoDiPack

Numerical models in atmospheric sciences not only need to approximate the flow equations on a suitable computational grid, they also need to include subgrid effects of many non-resolved physical processes. Among others, the formation and evolution of cloud particles is an example of such subgrid processes. Moreover, to date there is no universal mathematical description of a cloud, hence many cloud schemes have been proposed and these schemes typically contain several uncertain parameters. In this study, we propose the use of algorithmic differentiation (AD) as a method to identify parameters within the cloud scheme, to which the output of the cloud scheme is most sensitive. We illustrate the methodology by analyzing a scheme for liquid clouds, incorporated into a parcel model framework. Since the occurrence of uncertain parameters is not limited to cloud schemes, the AD methodology may help to identify the most sensitive uncertain parameters in any subgrid scheme and therefore help limiting the application of uncertainty quantification to the most crucial parameters.

It is the tradition for the fluid community to study fluid dynamics problems via numerical simulations such as finite-element, finite-difference and finite-volume methods. These approaches use various mesh techniques to discretize a complicated geometry and eventually convert governing equations into finite-dimensional algebraic systems. To date, many attempts have been made by exploiting machine learning to solve flow problems. However, conventional data-driven machine learning algorithms require heavy inputs of large labeled data, which is computationally expensive for complex and multi-physics problems. In this paper, we proposed a data-free, physics-driven deep learning approach to solve various low-speed flow problems and demonstrated its robustness in generating reliable solutions. Instead of feeding neural networks large labeled data, we exploited the known physical laws and incorporated this physics into a neural network to relax the strict requirement of big data and improve prediction accuracy. The employed physics-informed neural networks (PINNs) provide a feasible and cheap alternative to approximate the solution of differential equations with specified initial and boundary conditions. Approximate solutions of physical equations can be obtained via the minimization of the customized objective function, which consists of residuals satisfying differential operators, the initial/boundary conditions as well as the mean-squared errors between predictions and target values. This new approach is data efficient and can greatly lower the computational cost for large and complex geometries. The capacity and generality of the proposed method have been assessed by solving various flow and transport problems, including the flow past cylinder, linear Poisson, heat conduction and the Taylor–Green vortex problem.

According to recent trend of explosive growth of computation power and accumulated data, demand for the deep learning application in various research fields is increasing. As following this trend, remarkable achievements are presented in the experimental fluid mechanics field. One of the most outstanding research is Physics Informed Neural Networks (PINN) Raissi et al. (2020). Physical knowledge, which has been accumulated by humans, is imposed on the neural networks. PINN was used the automatic differentiation for implementing the governing equations as a physical constraint. By utilizing this concept, physical constraints make neural networks finding physical meaning of phenomena instead of simply fitting to the label data.

Brain-Computer Interfaces based on the analysis of ElectroEncephaloGraphy (EEG) are composed of several elements to process and classify brain input signals. A relevant phase of these systems is the decision making module, in which often the outputs from different classifiers are fused into a single one. In this work, the use of weighted-moderate deviation based functions is proposed to improve the Enhanced-Multimodal Fusion BCI Framework (EMF) decision making phase. Moderate Deviation-based aggregation functions (MDs) allow us to choose the best value to aggregate a vector of points involving a moderate deviation function. Using a weighted MD, the relative importance of each dimension in the multi-dimensional aggregated data set can also be taken into account. By applying these functions in the EMF, each one of the different brain signals can be weighted according to their importance. Moreover, using automatic differentiation, it is possible to optimize them for the present problem.

The phase retrieval problem, where one aims to recover a complex-valued image from far-field intensity measurements, is a classic problem encountered in a range of imaging applications. Modern phase retrieval approaches usually rely on gradient descent methods in a nonlinear minimization framework. Calculating closed-form gradients for use in these methods is tedious work, and formulating second order derivatives is even more laborious. Additionally, second order techniques often require the storage and inversion of large matrices of partial derivatives, with memory requirements that can be prohibitive for data-rich imaging modalities. We use a reverse-mode automatic differentiation (AD) framework to implement an efficient matrix-free version of the Levenberg-Marquardt (LM) algorithm, a longstanding method that finds popular use in nonlinear least-square minimization problems but which has seen little use in phase retrieval. Furthermore, we extend the basic LM algorithm so that it can be applied for more general constrained optimization problems (including phase retrieval problems) beyond just the least-square applications. Since we use AD, we only need to specify the physics-based forward model for a specific imaging application; the first and second-order derivative terms are calculated automatically through matrix-vector products, without explicitly forming the large Jacobian or Gauss-Newton matrices typically required for the LM method. We demonstrate that this algorithm can be used to solve both the unconstrained ptychographic object retrieval problem and the constrained “blind” ptychographic object and probe retrieval problems, under the popular Gaussian noise model as well as the Poisson noise model. We compare this algorithm to state-of-the-art first order ptychographic reconstruction methods to demonstrate empirically that this method outperforms best-in-class first-order methods: it provides excellent convergence guarantees with (in many cases) a superlinear rate of convergence, all with a computational cost comparable to, or lower than, the tested first-order algorithms.

We have developed PND, a differential equation solver software based on a physics-informed neural network (PINN) for molecular dynamics simulators. Based on automatic differentiation technique provided by PyTorch, our software allows users to flexibly implement equation of motion for atoms, initial and boundary conditions, and conservation laws as loss function to train the network. PND comes with a parallel molecular dynamic engine in order to examine and optimize loss function design, and different conservation laws and boundary conditions, and hyperparameters, thereby accelerating PINN-based development for molecular applications.

Linear Discriminant Analysis (LDA) is an important supervised dimensionality reduction method. Traditional LDA makes use of the eigenvalue decomposition of the scatter matrices based on the entire dataset. However, in some settings the whole dataset may not be available at once. Our approach considers an incremental LDA framework where the model receives training data in the form of chunks for subsequent analysis. We propose the Grassmann-Incremental Linear Discriminant Analysis (GILDA++) using the proxy matrix optimization method (PMO). The PMO method does not directly optimize a matrix on the manifold, but uses an auxiliary or proxy matrix in ambient space which is retracted to the closest location on the manifold along the loss minimizing geodesic. PMO makes use of an LDA objective by incrementally updating the scatter matrices to handle chunks of data. It makes use of automatic differentiation and stochastic gradient descent to find the lower dimensional LDA projection matrix. GILDA++ is able to handle chunk data, where each chunk has new samples from existing classes or novel classes. Our experiments demonstrate that GILDA++ outperforms the prevailing incremental LDA methods in various datasets.

Flying wing layout has both favorable aerodynamic performance and outstanding radar scattering characteristics, which is regarded as an ideal stealth aircraft layout. In this paper, a gradient-based aerodynamic and stealth optimization design method is established by coupling the Free-Form Deformation approach (FFD), Radial Basis Function algorithm (RBF), Computational Fluid Dynamics (CFD), Physical Optics (PO) and the Sequential Quadratic Programming algorithm (SQP). The gradient of the aerodynamic characteristic parameters could be obtained by solving the discrete adjoint equations. With PO code differentiated by the automatic differentiation tool, the Radar Cross Section (RCS) and its gradient would be collected. Based on the aero-stealth optimization design method, three kinds of optimization strategies are applied to design a certain flying wing layout aircraft: 1) The drag coefficient is chosen as the objective function while the RCS is set as a constraint condition; 2) The RCS is selected as the objective function while the drag coefficient is set as a constraint condition; 3) The objective function is composed of the drag coefficient and the RCS through the Weighted Sum method. The optimization results show that when choosing the first two strategies, the optimization algorithm would push single discipline performance to the edge. Alternatively, it would obtain several trade-off configurations when selecting the last strategy. The results demonstrate that the gradient-based aero-stealth optimization design method can deal with the multidisciplinary optimization problems which require a large number of design variables. Without a large number of function evaluations, this method could rapidly and efficiently obtain the shape that meets the requirements of aerodynamic and stealth performances.

A new toolbox based on operator overloading is introduced for automatic differentiation of scientific computing codes – and in particular legacy computational fluid dynamics solvers that are developed using Fortran. The method can be readily implemented into existing iterative solvers with minimal changes to the primal code. The integrated toolbox can efficiently calculate the sensitivities of any objective function with respect to all variables (design or intermediate) that can later be used for gradient-based design optimization, uncertainty quantification, error estimation, and mesh adaptation. The underlying definition of the current automatic differentiation is directly related to the discrete adjoint sensitivity analysis. Unlike most traditional operator overloading-based adjoint approaches reported in the literature, the current technique offers huge reductions in the memory footprint. To demonstrate the advantages of the current approach, various solvers/problems are considered. It is shown that the proposed technique can be used as an efficient toolbox for automatic differentiation of scientific solvers requiring only a handful additional lines of coding.

Differential sensitivity analysis is indispensable in fitting parameters, understanding uncertainty, and forecasting the results of both thought and lab experiments. Although there are many methods currently available for performing differential sensitivity analysis of biological models, it can be difficult to determine which method is best suited for a particular model. In this paper, we explain a variety of differential sensitivity methods and assess their value in some typical biological models. First, we explain the mathematical basis for three numerical methods: adjoint sensitivity analysis, complex-perturbation sensitivity analysis, and forward-mode sensitivity analysis. We then carry out four instructive case studies. (i) The CARRGO model for tumor-immune interaction highlights the additional information that differential sensitivity analysis provides beyond traditional naive sensitivity methods, (ii) the deterministic SIR model demonstrates the value of using second-order sensitivity in refining model predictions, (iii) the stochastic SIR model shows how differential sensitivity can be attacked in stochastic modeling, and (iv) a discrete birth-death-migration model illustrates how the complex-perturbation method of differential sensitivity can be generalized to a broader range of biological models. Finally, we compare the speed, accuracy, and ease of use of these methods. We find that forward-mode automatic differentiation has the quickest computation time, while the complex-perturbation method is the simplest to implement and the most generalizable.Author SummaryOver the past few decades, mathematical modeling has become an indispensable tool in the biologist’s toolbox. From deterministic to stochastic to statistical models, computational modeling is ubiquitous in almost every field of biology. Because model parameter estimates are often noisy or depend on poorly understood interactions, it is crucial to examine how both quantitative and qualitative predictions change as parameter estimates change, especially as the number of parameters increase. Sensitivity analysis is the process of understanding how a model’s behavior depends on parameter values. Sensitivity analysis simultaneously quantifies prediction certainty and clarifies the underlying biological mechanisms that drive computational models. While sensitivity analysis is universally recognized to be an important step in modeling, it is often unclear how to best leverage the available differential sensitivity methods. In this manuscript we explain and compare various differential sensitivity methods in the hope that best practices will be widely adopted. In particular, we stress the relative advantages of existing software and their limitations. We also present a new numerical technique for computing differential sensitivity.

Background99mTc-pertechnetate thyroid scintigraphy is a valid complementary avenue for evaluating thyroid disease in the clinic, the image feature of thyroid scintigram is relatively simple but the interpretation still has a moderate consistency among physicians. Thus, we aimed to develop an artificial intelligence (AI) system to automatically classify the four patterns of thyroid scintigram.MethodsWe collected 3087 thyroid scintigrams from center 1 to construct the training dataset (n = 2468) and internal validating dataset (n = 619), and another 302 cases from center 2 as external validating datasets. Four pre-trained neural networks that included ResNet50, DenseNet169, InceptionV3, and InceptionResNetV2 were implemented to construct AI models. The models were trained separately with transfer learning. We evaluated each model’s performance with metrics as following: accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), recall, precision, and F1-score.ResultsThe overall accuracy of four pre-trained neural networks in classifying four common uptake patterns of thyroid scintigrams all exceeded 90%, and the InceptionV3 stands out from others. It reached the highest performance with an overall accuracy of 92.73% for internal validation and 87.75% for external validation, respectively. As for each category of thyroid scintigrams, the area under the receiver operator characteristic curve (AUC) was 0.986 for ‘diffusely increased,’ 0.997 for ‘diffusely decreased,’ 0.998 for ‘focal increased,’ and 0.945 for ‘heterogeneous uptake’ in internal validation, respectively. Accordingly, the corresponding performances also obtained an ideal result of 0.939, 1.000, 0.974, and 0.915 in external validation, respectively.ConclusionsDeep convolutional neural network-based AI model represented considerable performance in the classification of thyroid scintigrams, which may help physicians improve the interpretation of thyroid scintigrams more consistently and efficiently.

We present ADerrors.jl, a software for linear error propagation and analysis of Monte Carlo data. Although the focus is in data analysis in Lattice QCD, where estimates of the observables have to be computed from Monte Carlo samples, the software also deals with variables with uncertainties, either correlated or uncorrelated. Thanks to automatic differentiation techniques linear error propagation is performed exactly, even in iterative algorithms (i.e. errors in parameters of non-linear fits). In this contribution we present an overview of the capabilities of the software, including access to uncertainties in fit parameters and dealing with correlated data. The software, written in julia, is available for download and use in https://gitlab.ift.uam-csic.es/alberto/aderrors.jl, Contribution to Tools for High Energy Physics and Cosmology - TOOLS2020. Code available in https://gitlab.ift.uam-csic.es/alberto/aderrors.jl

Automatic differentiation of benign and malignant breast lesions on multiple DCE-MRI series is a challenging task. The performance of the Convolutional Neural Networks (CNNs) based methods is severely affected when the number of DCE-MRI series is inadequate or inconsistent. This paper is motivated by the need of capturing spatial-temporal features from consistent DCE-MRI series for most CNN-based classification methods, and aims at designing an interpolation network that can enlarge the DCE-MRI series. Therefore, our method achieves the objective of breast lesion classification for inconsistent DCE-MRI series with a two-stage method, i.e., DCE-MRI interpolation and classification. Inspired by the learning-based data augmentation, we propose a variable-length multiple DCE-MRI series interpolation method using learned transformations to enlarge DCE-MRI series. Specifically, the forward and backward contrast transformations are learned to estimate the kinetic and spatial variation between different DCE-MRI series. Then, an adaptive warping method is proposed to generate multiple interpolated DCE-MRI series. Finally, the spatial-temporal features are extracted by a new two-stream network from the interpolated DCE-MRI and they are further used to classify breast lesions. We justify the proposed method through extensive experiments using 1223 DCE-MRI slices. Comparing to other methods, it achieves better results on both single series interpolation and multiple series interpolation. The interpolated DCE-MRI greatly improves the classification accuracy nearly by 5% and the best accuracy is 81.9%.

Non-causal modelling is a powerful approach to modelling physical systems in a variety of domains from science and engineering. Non-causal modelling languages enable a high-level and modular approach to modelling. However, it is hard to compile non-causal languages modularly (in the sense of separate compilation). This causes difficulties when simulating large models for which code generation takes a long time, or structurally singular models in which parts of the model are allowed to change at runtime. In this work, we introduce a technique we call order-parametric differentiation to allow truly modular compilation. The idea is to generate (machine) code that can compute derivatives of any order of an expression as needed, thus allowing for ahead-of-time modular compilation of a hybrid non-causal language. We also develop a compilation scheme that enables using partial models as first-class objects in a seamless way and simulating them without the need for just-in-time compilation, even in the presence of structural dynamism. We present a performance evaluation of the scheme we used and study its shortcomings and possible improvements, demonstrating that it is a feasible complement to existing implementation techniques for cases where true modular compilation is a primary objective.

Finding optimal parameters for force fields used in molecular simulation is a challenging and time-consuming task, partly due to the difficulty of tuning multiple parameters at once. Automatic differentiation presents a general solution: run a simulation, obtain gradients of a loss function with respect to all the parameters, and use these to improve the force field. This approach takes advantage of the deep learning revolution whilst retaining the interpretability and efficiency of existing force fields. We demonstrate that this is possible by parameterising a simple coarse-grained force field for proteins, based on training simulations of up to 2,000 steps learning to keep the native structure stable. The learned potential matches chemical knowledge and PDB data, can fold and reproduce the dynamics of small proteins, and shows ability in protein design and model scoring applications. Problems in applying differentiable molecular simulation to all-atom models of proteins are discussed along with possible solutions and the variety of available loss functions. The learned potential, simulation scripts and training code are made available athttps://github.com/psipred/cgdms.

With the increased interest in machine learning, and deep learning in particular, the use of automatic differentiation has become more wide-spread in computation. There have been two recent developments to provide the theoretical support for this types of structure. One approach, due to Abadi and Plotkin, provides a simple differential programming language. Another approach is the notion of a reverse differential category. In the present paper we bring these two approaches together. In particular, we show how an extension of reverse derivative categories models Abadi and Plotkin's language, and describe how this categorical model allows one to consider potential improvements to the operational semantics of the language., In Proceedings ACT 2020, arXiv:2101.07888

In this paper, with the aid of symbolic computation system Python and based on the deep neural network (DNN), automatic differentiation (AD), and limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) optimization algorithms, we discussed the modified Korteweg-de Vries (mkdv) equation to obtain numerical solutions. From the predicted solution and the expected solution, the resulting prediction error reaches10−6. The method that we used in this paper had demonstrated the powerful mathematical and physical ability of deep learning to flexibly simulate the physical dynamic state represented by differential equations and also opens the way for us to understand more physical phenomena later.

We propose to power lens design ray-tracing engines with automatic differentiation. Along with the primal, derivatives are algorithmically obtained for lens parameters. Exploiting the gradient information, applications are shown including design optimization and sensitivity analysis.

This paper introduces the concept of automatic differentiation in the evaluation of the absorbed power density in the mmWave frequency spectrum for the new generation of mobile telecommunication technology. Automatic differentiation has been shown to be far superior over numerical differentiation by means of speed and accuracy. To demonstrate the full capacity of the proposed method, a comprehensive analysis of computing the absorbed power density on the surface of irradiated human skin in various configurations is presented.

X-ray ptychography is an advanced computational microscopy technique, which is delivering exceptionally detailed quantitative imaging of biological and nanotechnology specimens, which can be used for high-precision X-ray measurements. However, coarse parametrisation in propagation distance, position errors and partial coherence frequently threaten the experimental viability. In this work, we formally introduce these actors, solving the whole reconstruction as an optimisation problem. A modern deep learning framework was used to autonomously correct the setup incoherences, thus improving the quality of a ptychography reconstruction. Automatic procedures are indeed crucial to reduce the time for a reliable analysis, which has a significant impact on all the fields that use this kind of microscopy. We implemented our algorithm in our software framework, SciComPty, releasing it as open-source. We tested our system on both synthetic datasets, as well as on real data acquired at the TwinMic beamline of the Elettra synchrotron facility.

While exploring stochastic gradient descent in Chapter 3, we treated the computation of gradients of the loss function ∇xL(x) as a black box. In this chapter, we open the black box and cover the theory and practice of automatic differentiation, as well as explore PyTorch’s Autograd module that implements the same. Automatic differentiation is a mature method that allows for the effortless and efficient computation of gradients of arbitrarily complicated loss functions. This is critical when it comes to minimizing loss functions of interest; at the heart of building any deep learning model lies an optimization problem that is invariably solved using stochastic gradient descent, which, in turn, requires one to compute gradients.

The lattice Boltzmann method (LBM) is an efficient simulation technique for computational fluid mechanics and beyond. It is based on a simple stream-and-collide algorithm on Cartesian grids, which is easily compatible with modern machine learning architectures. While it is becoming increasingly clear that deep learning can provide a decisive stimulus for classical simulation techniques, recent studies have not addressed possible connections between machine learning and LBM. Here, we introduce Lettuce, a PyTorch-based LBM code with a threefold aim. Lettuce enables GPU accelerated calculations with minimal source code, facilitates rapid prototyping of LBM models, and enables integrating LBM simulations with PyTorch’s deep learning and automatic differentiation facility. As a proof of concept for combining machine learning with the LBM, a neural collision model is developed, trained on a doubly periodic shear layer and then transferred to a different flow, a decaying turbulence. We also exemplify the added benefit of PyTorch’s automatic differentiation framework in flow control and optimization. To this end, the spectrum of a forced isotropic turbulence is maintained without further constraining the velocity field. The source code is freely available from https://github.com/lettucecfd/lettuce.

During the process of modelling an existing dynamic physical system, it may be hard to capture some of the phenomena exactly on the basis of only textbook-equations. With measurement data from the real system, approximators like artificial neural networks can help improve the models. However, simulation and machine learning are usually done in different software applications. A unified environment for modeling, simulation and optimization would be highly valuable. We here present a framework within the Julia programming language that encompasses tools for acausal modeling, automatic differentiation rsp. sensitivity analysis involving solvers for differential equations. We use it to build and evaluate an easily interpretable model based on both physics and data.

We describe and demonstrate an optimization-based X-ray image reconstruction framework called Adorym. Our framework provides a generic forward model, allowing one code framework to be used for a wide range of imaging methods ranging from near-field holography to fly-scan ptychographic tomography. By using automatic differentiation for optimization, Adorym has the flexibility to refine experimental parameters including probe positions, multiple hologram alignment, and object tilts. It is written with strong support for parallel processing, allowing large datasets to be processed on high-performance computing systems. We demonstrate its use on several experimental datasets to show improved image quality through parameter refinement. (C) 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

We employ a class of deep learning, known as Physics Informed Neural Networks (PINNs), to solve rectangular waveguide problems. In particular, we successfully apply PINNs to the task of solving electric and magnetic fields, which can be described by partial differential equations (PDEs). PINNs are neural networks that can naturally encode PDEs into the loss function, while partial derivatives with respect to input variables can be calculated through automatic differentiation (AD) built in modern deep learning libraries. We also show the applicability of the framework for predicting the unknown parameters such as wavenumber. We validate our model in terms of the convergence speed and accuracy. In addition, we suggest an approach to scale the input vector in order to accelerate training.

Conjugate Heat Transfer (CHT) simulations allow the prediction of complex interactions between fluid and solid mediums. Our application is the optimization of heat transfer between heat sinks and a cooling fluid, used to extract the heat from server infrastructure. Adjoint methods allow the optimization of high dimensional parameter settings, using sensitivity information. Compared to classical approaches to sensitivity generation, e.g. finite differences, a significant improvement in run time can be achieved, as the complexity of deriving the sensitivity scales with the output dimension, instead of the input (parameter) dimension. As an initial prove of concept, our discrete adjoint OpenFOAM framework has been extended to facilitate the differentiation of the chtMultiRegionSimpleFoam solver. To combat prohibitive memory loads a traditional and a novel checkpointing approach are used. We will present results of the heat transfer of a copper heat sink immersed in water.

Ongoing efforts to develop a fully open-source framework for the aero-structural design and optimization of wings, including aerodynamic and structural geometric nonlinearities, are presented. The framework is self-contained and relies on the well-established SU2 suite for the computation of the aerodynamic part of the problem. SU2 is a python-wrapped C++ suite for multiphysics problems, able to compute accurate adjoint sensitivities by means of Algorithmic Differentiation techniques. For the structural problem, a C++ library featuring a nonlinear FE beam is employed. The library is fully wrapped in python and coupled to SU2 by means of a python orchestrator and a splining module for force and displacement transferring. The applicability of this approach is demonstrated using a known aeroelastic test case based on the ONERA M6 wing geometry. The structural solver is differentiated by means of Algorithmic Differentiation and structural and coupled adjoint-based sensitivities are evaluated and validated by comparison to Finite Differences for a variety of cases. The final goal of this research is to provide an integrated infrastructure for aeroelastic design and optimization of wings by means of coupled adjoint sensitivities, including challenging cases in which wings are operating in non-linear aerodynamic regimes, e.g., transonic flows, and subject to large displacements.

We present a modular differentiable renderer design that yields performance superior to previous methods by leveraging existing, highly optimized hardware graphics pipelines. Our design supports all crucial operations in a modern graphics pipeline: rasterizing large numbers of triangles, attribute interpolation, filtered texture lookups, as well as user-programmable shading and geometry processing, all in high resolutions. Our modular primitives allow custom, high-performance graphics pipelines to be built directly within automatic differentiation frameworks such as PyTorch or TensorFlow. As a motivating application, we formulate facial performance capture as an inverse rendering problem and show that it can be solved efficiently using our tools. Our results indicate that this simple and straightforward approach achieves excellent geometric correspondence between rendered results and reference imagery.

As our knowledge and understanding of atmospheric aerosol particle evolution and impact grows, designing community mechanistic models requires an ability to capture increasing chemical, physical and therefore numerical complexity. As the landscape of computing software and hardware evolves, it is important to profile the usefulness of emerging platforms in tackling this complexity. Julia is a relatively new programming language that promises computational performance close to that of Fortran, for example, without sacrificing flexibility offered by languages such as Python. With this in mind, in this paper we present and demonstrate the initial development of a high-performance community mixed phase atmospheric 0D box-model, JlBox, written in Julia. In JlBox v1.0 we provide the option to simulate the chemical kinetics of a gas phase whilst also providing a fully coupled gasparticle model with dynamic partitioning to a fully moving sectional size distribution, in the first instance. JlBox is built around chemical mechanism files, using existing informatics software to provide parameters required for mixed phase simulations. In this study we use mechanisms from a subset and the complete Master Chemical Mechanism (MCM). Exploiting the ability to perform automatic differentiation of Jacobian matrices within Julia, we profile the use of sparse linear solvers and preconditioners, whilst also using a range of stiff solvers included within the expanding ODE solver suite the Julia environment provides, including the development of an adjoint model. Case studies range from a single volatile organic compound [VOC] with 305 equations to a full complexity MCM mixed phase simulation with 47544 variables. Comparison with an existing mixed phase model shows significant improvements in performance and potential for developments in a number of areas.

Computing derivatives of tensor expressions, also known as tensor calculus, is a fundamental task in machine learning. A key concern is the efficiency of evaluating the expressions and their derivatives that hinges on the representation of these expressions. Recently, an algorithm for computing higher order derivatives of tensor expressions like Jacobians or Hessians has been introduced that is a few orders of magnitude faster than previous state-of-the-art approaches. Unfortunately, the approach is based on Ricci notation and hence cannot be incorporated into automatic differentiation frameworks from deep learning like TensorFlow, PyTorch, autograd, or JAX that use the simpler Einstein notation. This leaves two options, to either change the underlying tensor representation in these frameworks or to develop a new, provably correct algorithm based on Einstein notation. Obviously, the first option is impractical. Hence, we pursue the second option. Here, we show that using Ricci notation is not necessary for an efficient tensor calculus and develop an equally efficient method for the simpler Einstein notation. It turns out that turning to Einstein notation enables further improvements that lead to even better efficiency. The methods that are described in this paper have been implemented in the online tool www.MatrixCalculus.org for computing derivatives of matrix and tensor expressions. An extended abstract of this paper appeared as "A Simple and Efficient Tensor Calculus", AAAI 2020.

Full waveform inversion (FWI) has been widely used for recovering a high-resolution subsurface structure from the entire content of measured seismic data. However, FWI requires a good starting model which needs to be close to the true model. If it is not, FWI may converge to a local minimum and will yield an incorrect subsurface image. To mitigate this problem, we propose a hybrid machine learning (HML) inversion method that uses a low dimensional representation of the measured seismic data for inverting subsurface velocity distribution. This low dimensional representation, also denoted as latent space (LS) feature, is automatically extracted from seismic data by using an autoencoder neural network. A small dimensional LS feature mainly contains the kinematic information of seismic data, such as traveltime. However, a large dimensional LS feature can also preserve the dynamic information of seismic data, such as waveform variations. Therefore, HML inversion can recover both the low- and high-wavenumber velocity information by inverting the LS feature with different dimensions. Meanwhile, HML inversion does not require a good starting model and less prone to local minima compared to FWI. The calculation of the HML gradient requires computing the derivative between the LS feature and velocity. However, there are no existing equations used to describe the relationship between these two terms. To address this problem, we use automatic differentiation (AD) to connect the two terms. Following this step, we use the wave equation inversion to invert the LS features for the subsurface velocity distribution. The numerical tests show that HML inversion can efficiently recover both the low and high-wavenumber velocity information from a poor starting model., Open-Access Online Publication: March 03, 2023

Arbitrary linear transformations are of crucial importance in a plethora of photonic applications spanning classical signal processing, communication systems, quantum information processing and machine learning. Here, we present a new photonic architecture to achieve arbitrary linear transformations by harnessing the synthetic frequency dimension of photons. Our structure consists of dynamically modulated micro-ring resonators that implement tunable couplings between multiple frequency modes carried by a single waveguide. By inverse design of these short- and long-range couplings using automatic differentiation, we realize arbitrary scattering matrices in synthetic space between the input and output frequency modes with near-unity fidelity and favorable scaling. We show that the same physical structure can be reconfigured to implement a wide variety of manipulations including single-frequency conversion, nonreciprocal frequency translations, and unitary as well as non-unitary transformations. Our approach enables compact, scalable and reconfigurable integrated photonic architectures to achieve arbitrary linear transformations in both the classical and quantum domains using current state-of-the-art technology., 12 pages, 7 figures