Search

Implicit networks are a class of neural networks whose outputs are defined by the fixed point of a parameterized operator. They have enjoyed success in many applications including natural language processing, image processing, and numerous other applications. While they have found abundant empirical success, theoretical work on its generalization is still under-explored. In this work, we consider a large family of implicit networks defined parameterized contractive fixed point operators. We show a generalization bound for this class based on a covering number argument for the Rademacher complexity of these architectures.

Atom probe tomography (APT) is ideally suited to characterize and understand the interplay of chemical segregation and microstructure in modern multicomponent materials. Yet, the quantitative analysis typically relies on human expertise to define regions of interest. We introduce a computationally efficient, multistage machine learning strategy to identify chemically distinct domains in a semi automated way, and subsequently quantify their geometric and compositional characteristics. In our algorithmic pipeline, we first coarse grain the APT data into voxels, collect the composition statistics, and decompose it via clustering in composition space. The composition classification then enables the real space segmentation via a density based clustering algorithm, thus revealing the microstructure at voxel resolution. Our approach is demonstrated for a Sm(Co,Fe)ZrCu alloy. The alloy exhibits two precipitate phases with a plate-like, but intertwined morphology. The primary segmentation is further refined to disentangle these geometrically complex precipitates into individual plate like parts by an unsupervised approach based on principle component analysis, or a U-Net based semantic segmentation trained on the former. Following the chemical and geometric analysis, detailed chemical distribution and segregation effects relative to the predominant plate-like geometry can be readily mapped without resorting to the initial voxelization.

Phase transformations and crystallographic defects are two essential tools to drive innovations in materials. Bulk materials design via tuning chemical compositions has been systematized using phase diagrams. We show here that the same thermodynamic concept can be applied to understand the chemistry at defects. We present a combined experimental and modelling approach to scope and build phase diagrams for defects. The discovery was enabled by triggering phase transformations of individual defects through local alloying, and sequentially imaging the structural and chemical changes using atomic-resolution scanning transmission electron microscopy. By observing atomic-scale phase transformations of a Mg grain boundary through Ga alloying, we exemplified the method to construct a grain boundary phase diagram using ab initio simulations and thermodynamic principles. The methodology enables a systematic development of defect phase diagrams to propel a new paradigm for materials design utilizing chemical complexity and phase transformations at defects.

During the last decade, automatic data analysis methods concerning different aspects of crystal analysis have been developed, e.g., unsupervised primitive unit cell extraction and automated crystal distortion and defects detection. However, an automatic, unsupervised motif extraction method is still not widely available yet. Here, we propose and demonstrate a novel method for the automatic motif extraction in real space from crystalline images based on a variational approach involving the unit cell projection operator. Due to the non-convex nature of the resulting minimization problem, a multi-stage algorithm is used. First, we determine the primitive unit cell in form of two lattice vectors. Second, a motif image is estimated using the unit cell information. Finally, the motif is determined in terms of atom positions inside the unit cell. The method was tested on various synthetic and experimental HAADF STEM images. The results are a representation of the motif in form of an image, atomic positions, primitive unit cell vectors, and a denoised and a modeled reconstruction of the input image. The method was applied to extract the primitive cells of complex $\mu$-phase structures Nb$_\text{6.4}$Co$_\text{6.6}$ and Nb$_\text{7}$Co$_\text{6}$, where subtle differences between their interplanar spacings were determined.

High performance sheet metals with a multi-phase microstructure suffer from deformation induced damage formation during forming in the constituent phases but importantly also where these intersect. To capture damage in terms of the physical processes in three dimensions (3D) and its stochastic nature during deformation, two challenges remain to be tackled: First, bridging high resolution analysis towards large scales to consider statistical data and, second, characterising in 3D with a resolution appropriate for sub-micron sized voids at a large scale. Here, we present how this can be achieved using panoramic scanning electron microscopy (SEM), metallographic serial sectioning, and deep-learning assisted automatic image analysis. This brings together the 3D evolution of active damage mechanisms with volumetric and environmental information for thousands of individual damage sites. We also assess potential surface preparation artefacts in 2D analyses. Overall, we find that for the material considered here, a dual phase (DP800) steel, martensite cracking is the dominant but not sole origin of deformation induced damage and that for a quantitative comparison of damage density, metallographic preparation can induce additional surface damage density far exceeding what is commonly induced between uniaxial straining steps.

Brittle topologically close-packed precipitates form in many advanced alloys. Due to their complex structures little is known about their plasticity. Here, we present a strategy to understand and tailor the deformability of these complex phases by considering the Nb-Co {\mu}-phase as an archetypal material. The plasticity of the Nb-Co {\mu}-phase is controlled by the Laves phase building block that forms parts of its unit cell. We find that between the bulk C15-NbCo$_2$ Laves and Nb-Co {\mu}-phase, the interplanar spacing and local elastic modulus of the Laves phase building block change, leading to a strong reduction in hardness and elastic modulus, as well as a transition from synchroshear to crystallographic slip. Furthermore, as the composition changes from Nb$_6$Co$_7$ to Nb$_7$Co$_6$, the Co atoms in the triple layer are substituted such that the triple layer of the Laves phase building block becomes a slab of pure Nb, resulting in inhomogeneous changes in elasticity and a transition from crystallographic slip to a glide-and-shuffle mechanism. These findings open opportunities to purposefully tailor the plasticity of these topologically close-packed phases in bulk, but at the atomic scale of interplanar spacing and local shear modulus of the fundamental crystal building blocks in their large unit cells.

Cell tracking is an essential tool in live-cell imaging to determine single-cell features, such as division patterns or elongation rates. Unlike in common multiple object tracking, in microbial live-cell experiments cells are growing, moving, and dividing over time, to form cell colonies that are densely packed in mono-layer structures. With increasing cell numbers, following the precise cell-cell associations correctly over many generations becomes more and more challenging, due to the massively increasing number of possible associations. To tackle this challenge, we propose a fast parameter-free cell tracking approach, which consists of activity-prioritized nearest neighbor assignment of growing cells and a combinatorial solver that assigns splitting mother cells to their daughters. As input for the tracking, Omnipose is utilized for instance segmentation. Unlike conventional nearest-neighbor-based tracking approaches, the assignment steps of our proposed method are based on a Gaussian activity-based metric, predicting the cell-specific migration probability, thereby limiting the number of erroneous assignments. In addition to being a building block for cell tracking, the proposed activity map is a standalone tracking-free metric for indicating cell activity. Finally, we perform a quantitative analysis of the tracking accuracy for different frame rates, to inform life scientists about a suitable (in terms of tracking performance) choice of the frame rate for their cultivation experiments, when cell tracks are the desired key outcome., Comment: Accepted for publication at the 5th IEEE International Conference on Image Processing Applications and Systems 2022 (IPAS)

We here propose an automated pipeline for the microscopy image-based characterization of catalytically active inclusion bodies (CatIBs), which includes a fully automatic experimental high-throughput workflow combined with a hybrid approach for multi-object microbial cell segmentation. For automated microscopy, a CatIB producer strain was cultivated in a microbioreactor from which samples were injected into a flow chamber. The flow chamber was fixed under a microscope and an integrated camera took a series of images per sample. To explore heterogeneity of CatIB development during the cultivation and track the size and quantity of CatIBs over time, a hybrid image processing pipeline approach was developed, which combines an ML-based detection of in-focus cells with model-based segmentation. The experimental setup in combination with an automated image analysis unlocks high-throughput screening of CatIB production, saving time and resources. Biotechnological relevance - CatIBs have wide application in synthetic chemistry and biocatalysis, but also could have future biomedical applications such as therapeutics. The proposed hybrid automatic image processing pipeline can be adjusted to treat comparable biological microorganisms, where fully data-driven ML-based segmentation approaches are not feasible due to the lack of training data. Our work is the first step towards image-based bioprocess control.

In this paper, we propose a hybrid approach for multi-object microbial cell segmentation. The approach combines an ML-based detection with a geometry-aware variational-based segmentation using B-splines that are parametrized based on a geometric model of the cell shape. The detection is done first using YOLOv5. In a second step, each detected cell is segmented individually. Thus, the segmentation only needs to be done on a per-cell basis, which makes it amenable to a variational approach that incorporates prior knowledge on the geometry. Here, the contour of the segmentation is modelled as closed uniform cubic B-spline, whose control points are parametrized using the known cell geometry. Compared to purely ML-based segmentation approaches, which need accurate segmentation maps as training data that are very laborious to produce, our method just needs bounding boxes as training data. Still, the proposed method performs on par with ML-based segmentation approaches usually used in this context. We study the performance of the proposed method on time-lapse microscopy data of Corynebacterium glutamicum.

Hyperspectral images provide a rich representation of the underlying spectrum for each pixel, allowing for a pixel-wise classification/segmentation into different classes. As the acquisition of labeled training data is very time-consuming, unsupervised methods become crucial in hyperspectral image analysis. The spectral variability and noise in hyperspectral data make this task very challenging and define special requirements for such methods. Here, we present a novel unsupervised hyperspectral segmentation framework. It starts with a denoising and dimensionality reduction step by the well-established Minimum Noise Fraction (MNF) transform. Then, the Mumford-Shah (MS) segmentation functional is applied to segment the data. We equipped the MS functional with a novel robust distribution-dependent indicator function designed to handle the characteristic challenges of hyperspectral data. To optimize our objective function with respect to the parameters for which no closed form solution is available, we propose an efficient fixed point iteration scheme. Numerical experiments on four public benchmark datasets show that our method produces competitive results, which outperform three state-of-the-art methods substantially on three of these datasets.

High thermoelectric energy conversion efficiency requires a large figure-of-merit, zT, over a broad temperature range. To achieve this, we optimize the carrier concentrations of n-type PbTe from room up to hot-end temperatures by co-doping Bi and Ag. Bi is an efficient n-type dopant in PbTe, often leading to excessive carrier concentration at room temperature. As revealed by density functional theory calculations, the formation of Bi and Ag defect complexes is exploited to optimize the room temperature carrier concentration. At elevated temperatures, we demonstrate the dynamic dissolution of Ag2Te precipitates in PbTe in situ by heating in a scanning transmission electron microscope. The release of n-type Ag interstitials with increasing temperature fulfills the requirement of higher carrier concentrations at the hot end. Moreover, as characterized by atom probe tomography, Ag atoms aggregate along parallel dislocation arrays to form Cottrell atmospheres. This results in enhanced phonon scattering and leads to a low lattice thermal conductivity. As a result of the synergy of dynamic doping and phonon scattering at decorated dislocations, an average zT of 1.0 is achieved in n-type Bi/Ag-codoped PbTe between 400 and 825 K. Introducing dopants with temperature-dependent solubility and strong interaction with dislocation cores enables simultaneous optimization of the average power factor and thermal conductivity, providing a new concept to exploit in the field of thermoelectrics.

Parallelism patterns (e.g., map or reduce) have proven to be effective tools for parallelizing high-performance applications. In this paper, we study the recursive registration of a series of electron microscopy images - a time consuming and imbalanced computation necessary for nano-scale microscopy analysis. We show that by translating the image registration into a specific instance of the prefix scan, we can convert this seemingly sequential problem into a parallel computation that scales to over thousand of cores. We analyze a variety of scan algorithms that behave similarly for common low-compute operators and propose a novel work-stealing procedure for a hierarchical prefix scan. Our evaluation shows that by identifying a suitable and well-optimized prefix scan algorithm, we reduce time-to-solution on a series of 4,096 images spanning ten seconds of microscopy acquisition from over 10 hours to less than 3 minutes (using 1024 Intel Haswell cores), enabling derivation of material properties at nanoscale for long microscopy image series.

The hemocompatibility of blood-contacting medical devices remains one of the major challenges in biomedical engineering and makes research in the field of new and improved materials inevitable. However, current in-vitro test and analysis methods are still lacking standardization and comparability, which impedes advances in material design. For example, the optical platelet analysis of material in-vitro hemocompatibility tests is carried out manually or semi-manually by each research group individually. As a step towards standardization, this paper proposes an automation approach for the optical platelet count and analysis. To this end, fluorescence images are segmented using Zach's convexification of the multiphase-phase piecewise constant Mumford--Shah model. The resulting connected components of the non-background segments then need to be classified as platelet or no platelet. Therefore, a supervised random forest is applied to feature vectors derived from the components using features like area, perimeter and circularity. With an overall high accuracy and low error rates, the random forest achieves reliable results. This is supported by high areas under the receiver-operator and the prediction-recall curve, respectively. We developed a new method for a fast, user-independent and reproducible analysis of material hemocompatibility tests, which is therefore a unique and powerful tool for advances in biomaterial research.

We propose a novel method for multi-phase segmentation of images based on high-dimensional local feature vectors. While the method was developed for the segmentation of extremely noisy crystal images based on localized Fourier transforms, the resulting framework is not tied to specific feature descriptors. For instance, using local spectral histograms as features, it allows for robust texture segmentation. The segmentation itself is based on the multi-phase Mumford-Shah model. Initializing the high-dimensional mean features directly is computationally too demanding and ill-posed in practice. This is resolved by projecting the features onto a low-dimensional space using principle component analysis. The resulting objective functional is minimized using a convexification and the Chambolle-Pock algorithm. Numerical results are presented, illustrating that the algorithm is very competitive in texture segmentation with state-of-the-art performance on the Prague benchmark and provides new possibilities in crystal segmentation, being robust to extreme noise and requiring no prior knowledge of the crystal structure.

Topologically close packed phases, among them the $\mu$-phase studied here, are commonly considered as being hard and brittle due to their close packed and complex structure. Nanoindentation enables plastic deformation and therefore investigation of the structure of mobile defects in the $\mu$-phase, which, in contrast to grown-in defects, has not been examined yet. High resolution transmission electron microscopy (HR-TEM) performed on samples deformed by nanoindentation revealed stacking faults which are likely induced by plastic deformation. These defects were compared to theoretically possible stacking faults within the $\mu$-phase building blocks, and in particular Laves phase layers. The experimentally observed stacking faults were found resulting from synchroshear assumed to be associated with deformation in the Laves-phase building blocks.

Aberration corrected scanning transmission electron microscopes (STEM) enable to determine local strain fields, composition and bonding states at atomic resolution. The precision to locate atomic columns is often obstructed by scan artifacts limiting the quantitative interpretation of STEM datasets. Here, a novel bias-corrected non-rigid registration approach is presented that compensates for fast and slow scan artifacts in STEM image series. The bias-correction is responsible for the correction of the slow scan artifacts and based on a explicit coupling of the deformations of the individual images in a series via a minimization of the average deformation. This allows to reduce fast scan noise in an image series and slow scan distortions simultaneously. The novel approach is tested on synthetic and experimental images and its implication on atomic resolution strain and elemental mapping is discussed.

Images generated by a transmission electron microscope (TEM) are blurred by aberrations from the objective lens and can be difficult to interpret correctly. One possible solution to this problem is to reconstruct the so-called exit wave, i.e. the electron wave in the microscope right before it passes the objective lens, from a series of TEM images acquired with varying focus. While the forward model of simulating a TEM image from a given exit wave is known and easy to evaluate, it is in general not possible to reconstruct the exit wave from a series of images analytically. The corresponding inverse problem can be formulated as a minimization problem, which is done in the well known MAL and MIMAP methods. We propose a generalization of these methods by performing the exit wave reconstruction and the registration of the image series simultaneously. We show that our objective functional is not convex with respect to the exit wave, which also carries over to the MAL and MIMAP functionals. The main result is the existence of minimizers of our objective functional. These results are based on the properties of a generalization of the cross-correlation. Finally, the applicability of our method is verified with a numerical experiment on simulated input data.

Region of interest (ROI) alignment in medical images plays a crucial role in diagnostics, procedure planning, treatment, and follow-up. Frequently, a model is represented as triangulated mesh while the patient data is provided from CAT scanners as pixel or voxel data. Previously, we presented a 2D method for curve-to-pixel registration. This paper contributes (i) a general mesh-to-raster (M2R) framework to register ROIs in multi-modal images; (ii) a 3D surface-to-voxel application, and (iii) a comprehensive quantitative evaluation in 2D using ground truth provided by the simultaneous truth and performance level estimation (STAPLE) method. The registration is formulated as a minimization problem where the objective consists of a data term, which involves the signed distance function of the ROI from the reference image, and a higher order elastic regularizer for the deformation. The evaluation is based on quantitative light-induced fluoroscopy (QLF) and digital photography (DP) of decalcified teeth. STAPLE is computed on 150 image pairs from 32 subjects, each showing one corresponding tooth in both modalities. The ROI in each image is manually marked by three experts (900 curves in total). In the QLF-DP setting, our approach significantly outperforms the mutual information-based registration algorithm implemented with the Insight Segmentation and Registration Toolkit (ITK) and Elastix.

Nowadays, modern electron microscopes deliver images at atomic scale. The precise atomic structure encodes information about material properties. Thus, an important ingredient in the image analysis is to locate the centers of the atoms shown in micrographs as precisely as possible. Here, we consider scanning transmission electron microscopy (STEM), which acquires data in a rastering pattern, pixel by pixel. Due to this rastering combined with the magnification to atomic scale, movements of the specimen even at the nanometer scale lead to random image distortions that make precise atom localization difficult. Given a series of STEM images, we derive a Bayesian method that jointly estimates the distortion in each image and reconstructs the underlying atomic grid of the material by fitting the atom bumps with suitable bump functions. The resulting highly non-convex minimization problems are solved numerically with a trust region approach. Well-posedness of the reconstruction method and the model behavior for faster and faster rastering are investigated using variational techniques. The performance of the method is finally evaluated on both synthetic and real experimental data.

The binary Mumford-Shah model is a widespread tool for image segmentation and can be considered as a basic model in shape optimization with a broad range of applications in computer vision, ranging from basic segmentation and labeling to object reconstruction. This paper presents robust a posteriori error estimates for a natural error quantity, namely the area of the non properly segmented region. To this end, a suitable strictly convex and non-constrained relaxation of the originally non-convex functional is investigated and Repin's functional approach for a posteriori error estimation is used to control the numerical error for the relaxed problem in the $L^2$-norm. In combination with a suitable cut out argument, a fully practical estimate for the area mismatch is derived. This estimate is incorporated in an adaptive meshing strategy. Two different adaptive primal-dual finite element schemes, and the most frequently used finite difference discretization are investigated and compared. Numerical experiments show qualitative and quantitative properties of the estimates and demonstrate their usefulness in practical applications., Comment: 18 pages, 7 figures, 1 table

In this paper the space of images is considered as a Riemannian manifold using the metamorphosis approach, where the underlying Riemannian metric simultaneously measures the cost of image transport and intensity variation. A robust and effective variational time discretization of geodesics paths is proposed. This requires to minimize a discrete path energy consisting of a sum of consecutive image matching functionals over a set of image intensity maps and pairwise matching deformations. For square-integrable input images the existence of discrete, connecting geodesic paths defined as minimizers of this variational problem is shown. Furthermore, $\Gamma$-convergence of the underlying discrete path energy to the continuous path energy is proved. This includes a diffeomorphism property for the induced transport and the existence of a square-integrable weak material derivative in space and time. A spatial discretization via finite elements combined with an alternating descent scheme in the set of image intensity maps and the set of matching deformations is presented to approximate discrete geodesic paths numerically. Computational results underline the efficiency of the proposed approach and demonstrate important qualitative properties., Comment: 27 pages, 7 figures

The extraordinary improvements of modern imaging devices offer access to data with unprecedented information content. However, widely used image processing methodologies fall far short of exploiting the full breadth of information offered by numerous types of scanning probe, optical, and electron microscopies. In many applications, it is necessary to keep measurement intensities below a desired threshold. We propose a methodology for extracting an increased level of information by processing a series of data sets suffering, in particular, from high degree of spatial uncertainty caused by complex multiscale motion during the acquisition process. An important role is played by a nonrigid pixel-wise registration method that can cope with low signal-to-noise ratios. This is accompanied by formulating objective quality measures which replace human intervention and visual inspection in the processing chain. Scanning transmission electron microscopy of siliceous zeolite material exhibits the above-mentioned obstructions and therefore serves as orientation and a test of our procedures.

Brain shift, i.e. the change in configuration of the brain after opening the dura mater, is a key problem in neuronavigation. We present an approach to co-register intra-operative microscope images with pre-operative MRI to adapt and optimize intra-operative neuronavigation. The tools are a robust classification of sulci on MRI extracted cortical surfaces, guided user marking of most prominent sulci on a microscope image, and the actual variational registration method with a fidelity energy for 3D deformations of the cortical surface combined with a higher order, linear elastica type prior energy. Furthermore, the actual registration is validated on an artificial testbed with known ground truth deformation and on real data of a neuro clinical patient.