Search

Exploring the structural, chemical, and physical properties of matter on the nano- and atomic scales has become possible with the recent advances in aberration-corrected electron energy-loss spectroscopy (EELS) in scanning transmission electron microscopy (STEM). However, the current paradigm of STEM-EELS relies on the classical rectangular grid sampling, in which all surface regions are assumed to be of equal a priori interest. This is typically not the case for real-world scenarios, where phenomena of interest are concentrated in a small number of spatial locations. One of foundational problems is the discovery of nanometer- or atomic scale structures having specific signatures in EELS spectra. Here we systematically explore the hyperparameters controlling deep kernel learning (DKL) discovery workflows for STEM-EELS and identify the role of the local structural descriptors and acquisition functions on the experiment progression. In agreement with actual experiment, we observe that for certain parameter combinations the experiment path can be trapped in the local minima. We demonstrate the approaches for monitoring automated experiment in the real and feature space of the system and monitor knowledge acquisition of the DKL model. Based on these, we construct intervention strategies, thus defining human-in the loop automated experiment (hAE). This approach can be further extended to other techniques including 4D STEM and other forms of spectroscopic imaging.

Microscopy, in particular scanning probe and electron microscopy, has been pivotal in improving our understanding of structure-function relationships at the nanoscale and is by now ubiquitous in most research characterization labs and facilities. However, traditional microscopy operations are still limited largely by a human-centric click-and-go paradigm utilizing vendor-provided software, which necessarily limits the scope, utility, efficiency, effectiveness, and at times reproducibility of microscopy experiments. Here, we develop a coupled hardware-software platform that consists of a field-programmable gate array (FPGA) device, with LabView-built customized acquisition scripts, along with a software package termed AEcroscoPy (short for Automated Experiments in Microscopy driven by Python) that overcome these limitations and provide the necessary abstractions towards full automation of microscopy platforms. The platform works across multiple vendor devices on scanning probe microscopes and scanning transmission electron microscopes. It enables customized scan trajectories, processing functions that can be triggered locally or remotely on processing servers, user-defined excitation waveforms, standardization of data models, and completely seamless operation through simple Python commands to enable a plethora of microscopy experiments to be performed in a reproducible, automated manner. This platform can be readily coupled with existing machine learning libraries as well as simulations, to provide automated decision-making and active theory-experiment optimization loops to turn microscopes from characterization tools to instruments capable of autonomous model refinement and physics discovery., Comment: 19 pages, 9 figures

On- and off-axis electron energy loss spectroscopy (EELS) is a powerful method for probing local electronic structure on single atom level. However, many materials undergo electron-beam induced transformation during the scanning transmission electron microscopy (STEM) and spectroscopy, the problem particularly acute for off-axis EELS signals. Here, we propose and operationalize the rapid object detection and action system (RODAS) for dynamic exploration of the structure-property relationships in STEM-EELS. In this approach, the electron beam is used to induce dynamic transformations creating new defect types at sufficiently small rates and avoiding complete material destruction. The deep convolutional neural networks trained via the ensemble learning iterative training (ELIT) approach are used to identify the defects as they form and perform EELS measurements only at specific defect types. Overall, in this case the EEL spectra are collected only at predefined objects of interest, avoiding measurements on the ideal regions or holes. We note that this approach can be extended to identify new defect classes as they appear, allowing for efficient collection of structure-property relationship data via balanced sampling over defect types.

Directed atomic fabrication using an aberration-corrected scanning transmission electron microscope (STEM) opens new pathways for atomic engineering of functional materials. In this approach, the electron beam is used to actively alter the atomic structure through electron beam induced irradiation processes. One of the impediments that has limited widespread use thus far has been the ability to understand the fundamental mechanisms of atomic transformation pathways at high spatiotemporal resolution. Here, we develop a workflow for obtaining and analyzing high-speed spiral scan STEM data, up to 100 fps, to track the atomic fabrication process during nanopore milling in monolayer MoS2. An automated feedback-controlled electron beam positioning system combined with deep convolution neural network (DCNN) was used to decipher fast but low signal-to-noise datasets and classify time-resolved atom positions and nature of their evolving atomic defect configurations. Through this automated decoding, the initial atomic disordering and reordering processes leading to nanopore formation was able to be studied across various timescales. Using these experimental workflows a greater degree of speed and information can be extracted from small datasets without compromising spatial resolution. This approach can be adapted to other 2D materials systems to gain further insights into the defect formation necessary to inform future automated fabrication techniques utilizing the STEM electron beam.

Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.

Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centered experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for the edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows and the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation., Comment: Review Article

Experimental science is enabled by the combination of synthesis, imaging, and functional characterization. Synthesis of a new material is typically followed by a set of characterization methods aiming to provide feedback for optimization or discover fundamental mechanisms. However, the sequence of synthesis and characterization methods and their interpretation, or research workflow, has traditionally been driven by human intuition and is highly domain specific. Here we explore concepts of scientific workflows that emerge at the interface between theory, characterization, and imaging. We discuss the criteria by which these workflows can be constructed for special cases of multi-resolution structural imaging and structural and functional characterization. Some considerations for theory-experiment workflows are provided. We further pose that the emergence of user facilities and cloud labs disrupt the classical progression from ideation, orchestration, and execution stages of workflow development and necessitate development of universal frameworks for workflow design, including universal hyper-languages describing laboratory operation, reward functions and their integration between domains, and policy development for workflow optimization. These tools will enable knowledge-based workflow optimization, enable lateral instrumental networks, sequential and parallel orchestration of characterization between dissimilar facilities, and empower distributed research., Comment: 33 pages; 8 figures

The impulsive excitation of ensembles of excitons or color centers by a high-energy electron beam results in the observation of photon bunching in the second-order correlation function of the cathodoluminescence generated by those emitters. Photon bunching in cathodoluminescence microscopy can be used to resolve the excited-state dynamics and the excitation and emission efficiency of nanoscale materials, and it can be used to probe interactions between emitters and nanophotonic cavities. Here, we report substantial changes in the measured bunching induced by indirect electron interactions (with indirect electron excitation inducing $g^{2}(0)$ values approaching $10^4$). This result is critical to the interpretation of $g^{2}(\tau)$ in cathodoluminescence microscopies, and, more importantly, it provides a foundation for the nanoscale characterization of optical properties in beam-sensitive materials.

Advanced electron microscopy workflows require an ecosystem of microscope instruments and computing systems possibly located at different sites to conduct remotely steered and automated experiments. Current workflow executions involve manual operations for steering and measurement tasks, which are typically performed from control workstations co-located with microscopes; consequently, their operational tempo and effectiveness are limited. We propose an approach based on separate data and control channels for such an ecosystem of Scanning Transmission Electron Microscopes (STEM) and computing systems, for which no general solutions presently exist, unlike the neutron and light source instruments. We demonstrate automated measurement transfers and remote steering of Nion STEM physical instruments over site networks. We propose a Virtual Infrastructure Twin (VIT) of this ecosystem, which is used to develop and test our steering software modules without requiring access to the physical instrument infrastructure. Additionally, we develop a VIT for a multiple laboratory scenario, which illustrates the applicability of this approach to ecosystems connected over wide-area networks, for the development and testing of software modules and their later field deployment., Comment: 11 pages, 16 figures, accepted at IEEE eScience 2022 conference

We pose that microscopy offers an ideal real-world experimental environment for the development and deployment of active Bayesian and reinforcement learning methods. Indeed, the tremendous progress achieved by machine learning (ML) and artificial intelligence over the last decade has been largely achieved via the utilization of static data sets, from the paradigmatic MNIST to the bespoke corpora of text and image data used to train large models such as GPT3, DALLE and others. However, it is now recognized that continuous, minute improvements to state-of-the-art do not necessarily translate to advances in real-world applications. We argue that a promising pathway for the development of ML methods is via the route of domain-specific deployable algorithms in areas such as electron and scanning probe microscopy and chemical imaging. This will benefit both fundamental physical studies and serve as a test bed for more complex autonomous systems such as robotics and manufacturing. Favorable environment characteristics of scanning and electron microscopy include low risk, extensive availability of domain-specific priors and rewards, relatively small effects of exogeneous variables, and often the presence of both upstream first principles as well as downstream learnable physical models for both statics and dynamics. Recent developments in programmable interfaces, edge computing, and access to APIs facilitating microscope control, all render the deployment of ML codes on operational microscopes straightforward. We discuss these considerations and hope that these arguments will lead to creating a novel set of development targets for the ML community by accelerating both real-world ML applications and scientific progress.

The advent of modern, high-speed electron detectors has made the collection of multidimensional hyperspectral transmission electron microscopy datasets, such as 4D-STEM, a routine. However, many microscopists find such experiments daunting since such datasets' analysis, collection, long-term storage, and networking remain challenging. Some common issues are the large and unwieldy size of the said datasets, often running into several gigabytes, non-standardized data analysis routines, and a lack of clarity about the computing and network resources needed to utilize the electron microscope fully. However, the existing computing and networking bottlenecks introduce significant penalties in each step of these experiments, and thus, real-time analysis-driven automated experimentation for multidimensional TEM is exceptionally challenging. One solution is integrating microscopy with edge computing, where moderately powerful computational hardware performs the preliminary analysis before handing off the heavier computation to HPC systems. In this perspective, we trace the roots of computation in modern electron microscopy, demonstrate deep learning experiments running on an edge system, and discuss the networking requirements for tying together microscopes, edge computers, and HPC systems., Comment: Perspective on automated microscopy. 3 figures

The robust approach for real-time analysis of the scanning transmission electron microscopy (STEM) data streams, based on the ensemble learning and iterative training (ELIT) of deep convolutional neural networks, is implemented on an operational microscope, enabling the exploration of the dynamics of specific atomic configurations under electron beam irradiation via an automated experiment in STEM. Combined with beam control, this approach allows studying beam effects on selected atomic groups and chemical bonds in a fully automated mode. Here, we demonstrate atomically precise engineering of single vacancy lines in transition metal dichalcogenides and the creation and identification of topological defects graphene. The ELIT-based approach opens the pathway toward the direct on-the-fly analysis of the STEM data and engendering real-time feedback schemes for probing electron beam chemistry, atomic manipulation, and atom by atom assembly.

Over the last two decades, Electron Energy Loss Spectroscopy (EELS) imaging with a scanning transmission electron microscope (STEM) has emerged as a technique of choice for visualizing complex chemical, electronic, plasmonic, and phononic phenomena in complex materials and structures. The availability of the EELS data necessitates the development of methods to analyze multidimensional datasets with complex spatial and energy structures. Traditionally, the analysis of these data sets has been based on analysis of individual spectra, one at a time, whereas the spatial structure and correlations between individual spatial pixels containing the relevant information of the physics of underpinning processes have generally been ignored and analyzed only via the visualization as 2D maps. Here we develop a machine learning-based approach and workflows for the analysis of spatial structures in 3D EELS data sets using a combination of dimensionality reduction and multichannel rotationally-invariant variational autoencoders. This approach is illustrated for the analysis of both the plasmonic phenomena in a system of nanowires and in the core excitations in functional oxides using low loss and core loss EELS, respectively. The code developed in this manuscript is open sourced and freely available and provided as a Jupyter notebook for the interested reader here., Comment: Associated Jupyter Notebook: https://github.com/kevinroccapriore/Multichannel-rVAE

Automated experiments in 4D Scanning Transmission Electron Microscopy are implemented for rapid discovery of local structures, symmetry-breaking distortions, and internal electric and magnetic fields in complex materials. Deep kernel learning enables active learning of the relationship between local structure and a 4D-STEM based descriptors. With this, efficient and "intelligent" probing of dissimilar structural elements to discover desired physical functionality is made possible. This approach allows effective navigation of the sample in an automated fashion guided by either a pre-determined physical phenomenon, such as strongest electric field magnitude, or in an exploratory fashion. We verify the approach first on pre-acquired 4D-STEM data, and further implement it experimentally on an operational STEM. The experimental discovery workflow is demonstrated using graphene, and subsequently extended towards a lesser-known layered 2D van der Waal material, MnPS3. This approach establishes a paradigm for physics-driven automated 4D-STEM experiments that enable probing the physics of strongly correlated systems and quantum materials and devices, as well as exploration of beam sensitive materials., Comment: The data used for analysis as well as additional materials are available through the Jupyter notebook located at: https://github.com/kevinroccapriore/AE-DKL-4DSTEM

Ordered mesoscale structures in 2D materials induced by small misorientations have opened pathways for a wide variety of novel electronic, ferroelectric, and quantum phenomena. Until now, the only mechanism to induce this periodic ordering was via mechanical rotations between the layers, with the periodicity of the resulting moir\'e pattern being directly related to twist angle. Here we report a fundamentally new mechanism for emergence of mesoscopic periodic patterns in multilayer sulfur-containing metal phosphorous trichalcogenide, MnPS$_{3}$, induced by the electron beam. The formation under the beam of periodic hexagonal patterns with several characteristic length scales, nucleation and transitions between the phases, and local dynamics are demonstrated. The associated mechanisms are attributed to the relative contraction of the layers caused by beam-induced sulphur vacancy formation with subsequent ordering and lattice parameter change. As a result, the plasmonic response of the system is locally altered, suggesting an element of control over plasmon resonances by electron beam patterning. We pose that harnessing this phenomenon provides both insight into fundamental physics of quantum materials and opens a pathway towards device applications by enabling controlled periodic potentials on the atomic scale., Comment: Electron microscopy data and analysis codes are freely available here: https://github.com/kevinroccapriore/MnPS3

Physics-driven discovery in an autonomous experiment has emerged as a dream application of machine learning in physical sciences. Here we develop and experimentally implement a deep kernel learning workflow combining the correlative prediction of the target functional response and its uncertainty from the structure, and physics-based selection of acquisition function, which autonomously guides the navigation of the image space. Compared to classical Bayesian optimization methods, this approach allows to capture the complex spatial features present in the images of realistic materials, and dynamically learn structure-property relationships. In combination with the flexible scalarizer function that allows to ascribe the degree of physical interest to predicted spectra, this enables physical discovery in automated experiment. Here, this approach is illustrated for nanoplasmonic studies of nanoparticles and experimentally implemented in a truly autonomous fashion for bulk- and edge plasmon discovery in MnPS3, a lesser-known beam-sensitive layered 2D material. This approach is universal, can be directly used as-is with any specimen, and is expected to be applicable to any probe-based microscopic techniques including other STEM modalities, Scanning Probe Microscopies, chemical, and optical imaging.

Spatial confinement of matter in functional nanostructures has propelled these systems to the forefront of nanoscience, both as a playground for exotic physics and quantum phenomena and in multiple applications including plasmonics, optoelectronics, and sensing. In parallel, the emergence of monochromated electron energy loss spectroscopy (EELS) has enabled exploration of local nanoplasmonic functionalities within single nanoparticles and the collective response of nanoparticle assemblies, providing deep insight into the associated mechanisms. However, modern synthesis processes for plasmonic nanostructures are often limited in the types of accessible geometry and materials, and even then, limited to spatial precisions on the order of tens of nm, precluding the direct exploration of critical aspects of the structure-property relationships. Here, we use the atomic-sized probe of the scanning transmission electron microscope (STEM) to perform precise sculpting and design of nanoparticle configurations. Furthermore, using low-loss (EELS), we provide dynamic analyses of evolution of the plasmonic response during the sculpting process. We show that within self-assembled systems of nanoparticles, individual nanoparticles can be selectively removed, reshaped, or arbitrarily patterned with nanometer-level resolution, effectively modifying the plasmonic response in both space and energy domains. This process significantly increases the scope for design possibilities and presents opportunities for arbitrary structure development, which are ultimately key for nanophotonic design. Nanosculpting introduces yet another capability to the electron microscope.

The adatom arrays on surfaces offer an ideal playground to explore the mechanisms of chemical bonding via changes in the local electronic tunneling spectra. While this information is readily available in hyperspectral scanning tunneling spectroscopy data, its analysis has been considerably impeded by a lack of suitable analytical tools. Here we develop a machine learning based workflow combining supervised feature identification in the spatial domain and un-supervised clustering in the energy domain to reveal the details of structure-dependent changes of the electronic structure in adatom arrays on the Co3Sn2S2 cleaved surface. This approach, in combination with first-principles calculations, provides insight for using artificial neural networks to detect adatoms and classifies each based on their local neighborhood comprised of other adatoms. These structurally classified adatoms are further spectrally deconvolved. The unexpected inhomogeneity of electronic structures among adatoms in similar configurations is unveiled using this method, suggesting there is not a single atomic species of adatoms, but rather multiple types of adatoms on the Co3Sn2S2 surface. This is further supported by a slight contrast difference in the images (or slight size variation) of the topography of the adatoms.

Design of nanoscale structures with desired nanophotonic properties are key tasks for nanooptics and nanophotonics. Here, the correlative relationship between local nanoparticle geometries and their plasmonic responses is established using encoder-decoder neural networks. In the im2spec network, the correlative relationship between local particle geometries and local spectra is established via encoding the observed geometries to a small number of latent variables and subsequently decoding into plasmonic spectra; in the spec2im network, the relationship is reversed. Surprisingly, these reduced descriptions allow high-veracity predictions of the local responses based on geometries for fixed compositions and chemical states of the surface. The analysis of the latent space distributions and the corresponding decoded and closest (in latent space) encoded images yields insight into the generative mechanisms of plasmonic interactions in the nanoparticle arrays. Ultimately, this approach creates a path toward determining configurations that can yield the spectrum closest to the desired one, paving the way for stochastic design of nanoplasmonic structures.

Low-loss electron energy loss spectroscopy (EELS) has emerged as a technique of choice for exploring the localization of plasmonic phenomena at the nanometer level, necessitating analysis of physical behaviors from 3D spectral data sets. For systems with high localization, linear unmixing methods provide an excellent basis for exploratory analysis, while in more complex systems large numbers of components are needed to accurately capture the true plasmonic response and the physical interpretability of the components becomes uncertain. Here, we explore machine learning based analysis of low-loss EELS data on heterogeneous self-assembled monolayer films of doped-semiconductor nanoparticles, which support infrared resonances. We propose a pathway for supervised analysis of EELS datasets that separate and classify regions of the films with physically distinct spectral responses. The classifications are shown to be robust, to accurately capture the common spatiospectral tropes of the complex nanostructures, and to be transferable between different datasets to allow high-throughput analysis of large areas of the sample. As such, it can be used as a basis for automated experiment workflows based on Bayesian optimization, as demonstrated on the ex situ data. We further demonstrate the use of non-linear autoencoders (AE) combined with clustering in the latent space of the AE yields highly reduced representations of the system response that yield insight into the relevant physics that do not depend on operator input and bias. The combination of these supervised and unsupervised tools provides complementary insight into the nanoscale plasmonic phenomena.

Scanning transmission electron microscopy (STEM) has become the technique of choice for quantitative characterization of atomic structure of materials, where the minute displacements of atomic columns from high-symmetry positions can be used to map strain, polarization, octahedra tilts, and other physical and chemical order parameter fields. The latter can be used as inputs into mesoscopic and atomistic models, providing insight into the correlative relationships and generative physics of materials on the atomic level. However, these quantitative applications of STEM necessitate understanding the microscope induced image distortions and developing the pathways to compensate them both as part of a rapid calibration procedure for in situ imaging, and the post-experimental data analysis stage. Here, we explore the spatiotemporal structure of the microscopic distortions in STEM using multivariate analysis of the atomic trajectories in the image stacks. Based on the behavior of principal component analysis (PCA), we develop the Gaussian process (GP)-based regression method for quantification of the distortion function. The limitations of such an approach and possible strategies for implementation as a part of in-line data acquisition in STEM are discussed. The analysis workflow is summarized in a Jupyter notebook that can be used to retrace the analysis and analyze the reader's data.

Alternate stacking of a highly conducting metallic layer with a magnetic triangular layer found in delafossite PdCrO2 provides an excellent platform for discovering intriguing correlated quantum phenomena. Thin film growth of the material may enable not only tuning the basic physical properties beyond what bulk materials can exhibit, but also developing novel hybrid materials by interfacing with dissimilar materials, yet this has proven to be extremely challenging. Here, we report the epitaxial growth of metallic delafossite PdCrO2 films by pulsed laser epitaxy (PLE). The fundamental role of the PLE growth conditions, epitaxial strain, and chemical and structural characteristics of the substrate is investigated by growing under various growth conditions and on various types of substrates. While strain plays a large role in improving the crystallinities, the direct growth of epitaxial PdCrO2 films without impurity phases was not successful. We attribute this difficulty to both the chemical and structural dissimilarities between the substrates and volatile nature of PdO layer, which make nucleation of the right phase difficult. This difficulty was overcome by growing CuCrO2 buffer layers before PdCrO2 were grown. Unlike PdCrO2, CuCrO2 films were rather readily grown with a relatively wide growth window. Only monolayer thick buffer layer was sufficient to grow the correct PdCrO2 phase. This result indicates that the epitaxy of Pd-based delafossites is extremely sensitive to the chemistry and structure of the interface, necessitating near perfect substrate materials. The resulting films are commensurately strained and show an antiferromagnetic transition at 40 K that persists down to as thin as 3.6 nm in thickness.