Search

Cathodoluminescence microscopy is now a well-established and powerful tool for probing the photonic properties of nanoscale materials, but in many cases, nanophotonic materials are easily damaged by the electron-beam doses necessary to achieve reasonable cathodoluminescence signal-to-noise ratios. Two-dimensional materials have proven particularly susceptible to beam-induced modifications, yielding both obstacles to high spatial-resolution measurement and opportunities for beam-induced patterning of quantum photonic systems. Here pan-sharpening techniques are applied to cathodoluminescence microscopy in order to address these challenges and experimentally demonstrate the promise of pan-sharpening for minimally-perturbative high-spatial-resolution spectrum imaging of beam-sensitive materials.

The rapid validation of newly predicted materials through autonomous synthesis requires real-time adaptive control methods that exploit physics knowledge, a capability that is lacking in most systems. Here, we demonstrate an approach to enable the real-time control of thin film synthesis by combining in situ optical diagnostics with a Bayesian state estimation method. We developed a physical model for film growth and applied the Direct Filter (DF) method for real-time estimation of nucleation and growth rates during pulsed laser deposition (PLD) of transition metal dichalcogenides. We validated the approach on simulated and previously acquired reflectivity data for WSe$_2$ growth and ultimately deployed the algorithm on an autonomous PLD system during growth of 1T$^\prime$-MoTe$_2$ under various synthesis conditions. We found that the DF robustly estimates growth parameters in real-time at early stages of growth, down to 15% percent monolayer area coverage. This approach opens new opportunities for adaptive film growth control based on a fusion of in situ diagnostics, modern data assimilation methods, and physical models which promises to enable control of synthesis trajectories towards desired material states.

Both computational and experimental material discovery bring forth the challenge of exploring multidimensional and multimodal parameter spaces, such as phase diagrams of Hamiltonians with multiple interactions, composition spaces of combinatorial libraries, material structure image spaces, and molecular embedding spaces. Often these systems are black-box and time-consuming to evaluate, which resulted in strong interest towards active learning methods such as Bayesian optimization (BO). However, these systems are often noisy which make the black box function severely multi-modal and non-differentiable, where a vanilla BO can get overly focused near a single or faux optimum, deviating from the broader goal of scientific discovery. To address these limitations, here we developed Strategic Autonomous Non-Smooth Exploration (SANE) to facilitate an intelligent Bayesian optimized navigation with a proposed cost-driven probabilistic acquisition function to find multiple global and local optimal regions, avoiding the tendency to becoming trapped in a single optimum. To distinguish between a true and false optimal region due to noisy experimental measurements, a human (domain) knowledge driven dynamic surrogate gate is integrated with SANE. We implemented the gate-SANE into a pre-acquired Piezoresponse spectroscopy data of a ferroelectric combinatorial library with high noise levels in specific regions, and a piezoresponse force microscopy (PFM) hyperspectral data. SANE demonstrated better performance than classical BO to facilitate the exploration of multiple optimal regions and thereby prioritized learning with higher coverage of scientific values in autonomous experiments. Our work showcases the potential application of this method to real-world experiment, where such combined strategic and human intervening approaches can be critical to unlocking new discoveries in autonomous research., Comment: 25 pages, 7 figures in main text, 2 figures in Supp Mat

EuM2As2 (M = Zn, Cd, In, Sn etc.) is an excellent material system for studying topological properties, which can be easily tuned by magnetism involved. Theoretical calculations predict gapped and flat bands in EuZn2As2 but gapless structure in EuCd2As2. In this work, low-temperature (77 K) cleaved EuZn2As2 crystals are studied using scanning tunneling microscopy/spectroscopy (STM/S) and density functional theory (DFT) calculations. Defects-induced local density of states (LDOS) modification with a triangular shape helps identify the surface terminations: Eu versus AsZn surface. While large bandgaps (~1.5 eV at 77 K) are observed on both pristine surfaces, the bandgap width is found to be very sensitive to local heterogeneity, such as defects and step edges, with the tendency of reduction. Combining experimental data with DFT simulations, we conclude that the modified bandgap in the heterogeneous area arises from Zn vacancies and/or substitution by As atoms. Our investigation offers important information for reevaluating the electron topology of the EuM2As2 family., Comment: 17 pages, 4 figures

Noncollinear ferroic materials are sought after as testbeds to explore the intimate connections between topology and symmetry, which result in electronic, optical and magnetic functionalities not observed in collinear ferroic materials. For example, ferroaxial materials have ordered rotational structural distortions that break mirror symmetry and induce chirality. When ferroaxial order is coupled with ferroelectricity arising from a broken inversion symmetry, it offers the prospect of electric-field-control of the ferroaxial distortions and opens up new tunable functionalities. However, chiral multiferroics, especially ones stable at room temperature, are rare. We report the discovery of a strain-stabilized, room-temperature chiral multiferroic phase in single crystals of BaTiS$_3$, a quasi-one-dimensional (1D) hexagonal chalcogenide. Using first-principles calculations, we predict the stabilization of this multiferroic phase having $P6_3$ space group for biaxial tensile strains exceeding 1.5% applied on the basal ab-plane of the room temperature $P6_3cm$ phase of BaTiS$_3$. The chiral multiferroic phase is characterized by rotational distortions of select TiS$_6$ octahedra around the long $c$-axis and polar displacement of Ti atoms along the $c$-axis. We used an innovative approach using focused ion beam milling to make appropriately strained samples of BaTiS$_3$. The ferroaxial and ferroelectric distortions, and their domains in $P6_3$-BaTiS$_3$ were directly resolved using atomic resolution scanning transmission electron microscopy. Landau-based phenomenological modeling predicts a strong coupling between the ferroelectric and the ferroaxial order making $P6_3$-BaTiS$_3$ an attractive test bed for achieving electric-field control of chirality-related phenomena such as circular photo-galvanic current and the Rashba effect., Comment: 20 pages, 4 figures

Microscopy techniques have played vital roles in materials science, biology, and nanotechnology, offering high-resolution imaging and detailed insights into properties at nanoscale and atomic level. The automation of microscopy experiments, in combination with machine learning approaches, is a transformative advancement, offering increased efficiency, reproducibility, and the capability to perform complex experiments. Our previous work on autonomous experimentation with scanning probe microscopy (SPM) demonstrated an active learning framework using deep kernel learning (DKL) for structure-property relationship discovery. This approach has demonstrated broad applications in various microscopy techniques. Here, to address limitations of workflows based on DKL, we developed methods to incorporate prior knowledge and human interest into DKL-based workflows and implemented these workflows in SPM. By integrating expected rewards from structure libraries or spectroscopic features, we enhanced the exploration efficiency of autonomous microscopy, demonstrating more efficient and targeted exploration in autonomous microscopy. We demonstrated the application of these methods in SPM, but we suggest that these methods can be seamlessly applied to other microscopy and imaging techniques. Furthermore, the concept can be adapted for general Bayesian optimization in material discovery across a broad range of autonomous experimental fields.

The fusion of experimental automation and machine learning has catalyzed a new era in materials research, prominently featuring Gaussian Process Bayesian Optimization (GPBO) driven autonomous experiments navigating complex experimental conditions for accelerated scientific discovery. In traditional GPBO-driven experiments, a predefined scalarizer function is often required to preprocess the experimental data, transforming non-scalar raw data into scalar descriptors for GP training. However, such predefined scalarizer functions have limitations, which likely fail to accommodate the diversity and complexity of real-world experimental data, potentially skewing experimental outcomes. Thus, oversight and quality control are necessitated over the process to avoid GPBO from being misled by low quality scalarizers. To address the limitation, we introduce a Dual-GP approach that enhances traditional GPBO by adding a secondary surrogate model to dynamically constrain the experimental space based on real-time assessments of the raw experimental data. This Dual-GP approach enhances the optimization efficiency of traditional GPBO by isolating more promising space for BO sampling and more valuable experimental data for primary GP training. We also incorporate a flexible, human-in-the-loop intervention method in the Dual-GP workflow to adjust for unanticipated results. We demonstrate the effectiveness of the Dual-GP model with synthetic model data and implement this approach in autonomous pulsed laser deposition experimental data. This Dual-GP approach has broad applicability in diverse GPBO-driven experimental settings, providing a more adaptable and precise framework for refining autonomous experimentation for more efficient optimization., Comment: 18 pages; 5 figures

Scientific advancement is universally based on the dynamic interplay between theoretical insights, modelling, and experimental discoveries. However, this feedback loop is often slow, including delayed community interactions and the gradual integration of experimental data into theoretical frameworks. This challenge is particularly exacerbated in domains dealing with high-dimensional object spaces, such as molecules and complex microstructures. Hence, the integration of theory within automated and autonomous experimental setups, or theory in the loop automated experiment, is emerging as a crucial objective for accelerating scientific research. The critical aspect is not only to use theory but also on-the-fly theory updates during the experiment. Here, we introduce a method for integrating theory into the loop through Bayesian co-navigation of theoretical model space and experimentation. Our approach leverages the concurrent development of surrogate models for both simulation and experimental domains at the rates determined by latencies and costs of experiments and computation, alongside the adjustment of control parameters within theoretical models to minimize epistemic uncertainty over the experimental object spaces. This methodology facilitates the creation of digital twins of material structures, encompassing both the surrogate model of behavior that includes the correlative part and the theoretical model itself. While demonstrated here within the context of functional responses in ferroelectric materials, our approach holds promise for broader applications, the exploration of optical properties in nanoclusters, microstructure-dependent properties in complex materials, and properties of molecular systems. The analysis code that supports the funding is publicly available at https://github.com/Slautin/2024_Co-navigation/tree/main, Comment: 23 pages, 10 figures

Atomic arrangements and local sub-structures fundamentally influence emergent material functionalities. The local structures are conventionally probed using spatially resolved studies and the property correlations are usually deciphered by a researcher based on sequential explorations and auxiliary information, thus limiting the throughput efficiency. Here we demonstrate a Bayesian deep learning based framework that automatically correlates material structure with its electronic properties using scanning tunneling microscopy (STM) measurements in real-time. Its predictions are used to autonomously direct exploration toward regions of the sample that optimize a given material property. This autonomous method is deployed on the low-temperature ultra-high vacuum STM to understand the structure-property relationship in a europium-based semimetal, EuZn2As2, one of the promising candidates for studying the magnetism-driven topological properties. The framework employs a sparse sampling approach to efficiently construct the scalar-property space using a minimal number of measurements, about 1 - 10 % of the data required in standard hyperspectral imaging methods. We further demonstrate a target-property-guided active learning of structures within a multiscale framework. This is implemented across length scales in a hierarchical fashion for the autonomous discovery of structural origins for an observed material property. This framework offers the choice to select and derive a suitable scalar property from the spectroscopic data to steer exploration across the sample space. Our findings reveal correlations of the electronic properties unique to surface terminations, local defect density, and point defects.

Both computational and experimental material discovery bring forth the challenge of exploring multidimensional and often non-differentiable parameter spaces, such as phase diagrams of Hamiltonians with multiple interactions, composition spaces of combinatorial libraries, processing spaces, and molecular embedding spaces. Often these systems are expensive or time-consuming to evaluate a single instance, and hence classical approaches based on exhaustive grid or random search are too data intensive. This resulted in strong interest towards active learning methods such as Bayesian optimization (BO) where the adaptive exploration occurs based on human learning (discovery) objective. However, classical BO is based on a predefined optimization target, and policies balancing exploration and exploitation are purely data driven. In practical settings, the domain expert can pose prior knowledge on the system in form of partially known physics laws and often varies exploration policies during the experiment. Here, we explore interactive workflows building on multi-fidelity BO (MFBO), starting with classical (data-driven) MFBO, then structured (physics-driven) sMFBO, and extending it to allow human in the loop interactive iMFBO workflows for adaptive and domain expert aligned exploration. These approaches are demonstrated over highly non-smooth multi-fidelity simulation data generated from an Ising model, considering spin-spin interaction as parameter space, lattice sizes as fidelity spaces, and the objective as maximizing heat capacity. Detailed analysis and comparison show the impact of physics knowledge injection and on-the-fly human decisions for improved exploration, current challenges, and potential opportunities for algorithm development with combining data, physics and real time human decisions., Comment: Main text includes 29 pages and 10 figures, Supplementary mat. includes 4 pages and 4 figures

With the advent of large language models (LLMs), in both the open source and proprietary domains, attention is turning to how to exploit such artificial intelligence (AI) systems in assisting complex scientific tasks, such as material synthesis, characterization, analysis and discovery. Here, we explore the utility of LLM, particularly ChatGPT4, in combination with application program interfaces (APIs) in tasks of experimental design, programming workflows, and data analysis in scanning probe microscopy, using both in-house developed API and API given by a commercial vendor for instrument control. We find that the LLM can be especially useful in converting ideations of experimental workflows to executable code on microscope APIs. Beyond code generation, we find that the GPT4 is capable of analyzing microscopy images in a generic sense. At the same time, we find that GPT4 suffers from inability to extend beyond basic analyses or more in-depth technical experimental design. We argue that a LLM specifically fine-tuned for individual scientific domains can potentially be a better language interface for converting scientific ideations from human experts to executable workflows, such a synergy between human expertise and LLM efficiency in experimentation can open new door for accelerating scientific research, enabling effective experimental protocols archive and sharing in scientific community., Comment: 16 pages; 7 figures

Microscopy, in particular scanning probe and electron microscopy, has been pivotal in improving our understanding of structure-function relationships at the nanoscale and is by now ubiquitous in most research characterization labs and facilities. However, traditional microscopy operations are still limited largely by a human-centric click-and-go paradigm utilizing vendor-provided software, which necessarily limits the scope, utility, efficiency, effectiveness, and at times reproducibility of microscopy experiments. Here, we develop a coupled hardware-software platform that consists of a field-programmable gate array (FPGA) device, with LabView-built customized acquisition scripts, along with a software package termed AEcroscoPy (short for Automated Experiments in Microscopy driven by Python) that overcome these limitations and provide the necessary abstractions towards full automation of microscopy platforms. The platform works across multiple vendor devices on scanning probe microscopes and scanning transmission electron microscopes. It enables customized scan trajectories, processing functions that can be triggered locally or remotely on processing servers, user-defined excitation waveforms, standardization of data models, and completely seamless operation through simple Python commands to enable a plethora of microscopy experiments to be performed in a reproducible, automated manner. This platform can be readily coupled with existing machine learning libraries as well as simulations, to provide automated decision-making and active theory-experiment optimization loops to turn microscopes from characterization tools to instruments capable of autonomous model refinement and physics discovery., Comment: 19 pages, 9 figures

Materials synthesis platforms that are designed for autonomous experimentation are capable of collecting multimodal diagnostic data that can be utilized for feedback to optimize material properties. Pulsed laser deposition (PLD) is emerging as a viable autonomous synthesis tool, and so the need arises to develop machine learning (ML) techniques that are capable of extracting information from in situ diagnostics. Here, we demonstrate that intensified-CCD image sequences of the plasma plume generated during PLD can be used for anomaly detection and the prediction of thin film growth kinetics. We developed a multi-output (2$+$1)D convolutional neural network regression model that extracts deep features from plume dynamics that not only correlate with the measured chamber pressure and incident laser energy, but more importantly, predict parameters of an auto-catalytic film growth model derived from in situ laser reflectivity experiments. Our results are the first demonstration of how ML with in situ plume diagnostics data in PLD can be utilized to maintain deposition conditions in an optimal regime. Further, the predictive capabilities of plume dynamics on the kinetics of film growth or other film properties prior to deposition provides a means for rapid pre-screening of growth conditions for the non-expert, which promises to accelerate materials optimization with PLD.

Upgrades to advanced scientific user facilities such as next-generation x-ray light sources, nanoscience centers, and neutron facilities are revolutionizing our understanding of materials across the spectrum of the physical sciences, from life sciences to microelectronics. However, these facility and instrument upgrades come with a significant increase in complexity. Driven by more exacting scientific needs, instruments and experiments become more intricate each year. This increased operational complexity makes it ever more challenging for domain scientists to design experiments that effectively leverage the capabilities of and operate on these advanced instruments. Large language models (LLMs) can perform complex information retrieval, assist in knowledge-intensive tasks across applications, and provide guidance on tool usage. Using x-ray light sources, leadership computing, and nanoscience centers as representative examples, we describe preliminary experiments with a Context-Aware Language Model for Science (CALMS) to assist scientists with instrument operations and complex experimentation. With the ability to retrieve relevant information from facility documentation, CALMS can answer simple questions on scientific capabilities and other operational procedures. With the ability to interface with software tools and experimental hardware, CALMS can conversationally operate scientific instruments. By making information more accessible and acting on user needs, LLMs could expand and diversify scientific facilities' users and accelerate scientific output.

Scanning Tunneling microscopy (STM) is a widely used tool for atomic imaging of novel materials and its surface energetics. However, the optimization of the imaging conditions is a tedious process due to the extremely sensitive tip-surface interaction, and thus limits the throughput efficiency. Here we deploy a machine learning (ML) based framework to achieve optimal-atomically resolved imaging conditions in real time. The experimental workflow leverages Bayesian optimization (BO) method to rapidly improve the image quality, defined by the peak intensity in the Fourier space. The outcome of the BO prediction is incorporated into the microscope controls, i.e., the current setpoint and the tip bias, to dynamically improve the STM scan conditions. We present strategies to either selectively explore or exploit across the parameter space. As a result, suitable policies are developed for autonomous convergence of the control-parameters. The ML-based framework serves as a general workflow methodology across a wide range of materials., Comment: 31 pages, 5 figures and Supplementary Information

Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.

Synthesis of thin films has traditionally relied upon slow, sequential processes carried out with substantial human intervention, frequently utilizing a mix of experience and serendipity to optimize material structure and properties. With recent advances in autonomous systems which combine synthesis, characterization, and decision making with artificial intelligence (AI), large parameter spaces can be explored autonomously at rates beyond what is possible by human experimentalists, greatly accelerating discovery, optimization, and understanding in materials synthesis which directly address the grand challenges in synthesis science. Here, we demonstrate autonomous synthesis of a contemporary 2D material by combining the highly versatile pulsed laser deposition (PLD) technique with automation and machine learning (ML). We incorporated in situ and real-time spectroscopy, a high-throughput methodology, and cloud connectivity to enable autonomous synthesis workflows with PLD. Ultrathin WSe2 films were grown using co-ablation of two targets and showed a 10x increase in throughput over traditional PLD workflows. Gaussian process regression and Bayesian optimization were used with in situ Raman spectroscopy to autonomously discover two distinct growth windows and the process-property relationship after sampling only 0.25% of a large 4D parameter space. Any material that can be grown with PLD could be autonomously synthesized with our platform and workflows, enabling accelerated discovery and optimization of a vast number of materials.

Developing methods to understand and control defect formation in nanomaterials offers a promising route for materials discovery. Monolayer MX2 phases represent a particularly compelling case for defect engineering of nanomaterials due to the large variability in their physical properties as different defects are introduced into their structure. However, effective identification and quantification of defects remains a challenge even as high-throughput scanning tunneling electron microscopy (STEM) methods improve. This study highlights the benefits of employing first principles calculations to produce digital twins for training deep learning segmentation models for defect identification in monolayer MX2 phases. Around 600 defect structures were obtained using density functional theory calculations, with each monolayer MX2 structure being subjected to multislice simulations for the purpose of generating the digital twins. Several deep learning segmentation architectures were trained on this dataset, and their performances evaluated under a variety of conditions such as recognizing defects in the presence of unidentified impurities, beam damage, grain boundaries, and with reduced image quality from low electron doses. This digital twin approach allows benchmarking different deep learning architectures on a theory dataset, which enables the study of defect classification under a broad array of finely controlled conditions. It thus opens the door to resolving the underpinning physical reasons for model shortcomings, and potentially chart paths forward for automated discovery of materials defect phases in experiments., Comment: 17 pages, 7 figures

Optimization of experimental materials synthesis and characterization through active learning methods has been growing over the last decade, with examples ranging from measurements of diffraction on combinatorial alloys at synchrotrons, to searches through chemical space with automated synthesis robots for perovskites. In virtually all cases, the target property of interest for optimization is defined apriori with limited human feedback during operation. In contrast, here we present the development of a new type of human in the loop experimental workflow, via a Bayesian optimized active recommender system (BOARS), to shape targets on the fly, employing human feedback. We showcase examples of this framework applied to pre-acquired piezoresponse force spectroscopy of a ferroelectric thin film, and then implement this in real time on an atomic force microscope, where the optimization proceeds to find symmetric piezoresponse amplitude hysteresis loops. It is found that such features appear more affected by subsurface defects than the local domain structure. This work shows the utility of human-augmented machine learning approaches for curiosity-driven exploration of systems across experimental domains. The analysis reported here is summarized in Colab Notebook for the purpose of tutorial and application to other data: https://github.com/arpanbiswas52/varTBO, Comment: 7 figures in main text, 3 figures in Supp Material

The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards data set and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The jupyter notebooks that encapsulate the complete analysis accompany the article., Comment: 8 Figures, 26 pages

Reducing the dimensions of ferroelectric materials down to the nanoscale has strong implications on the ferroelectric polarization pattern and on the ability to switch the polarization. As the size of ferroelectric domains shrinks to nanometer scale, the heterogeneity of the polarization pattern becomes increasingly pronounced, enabling a large variety of possible polar textures in nanocrystalline and nanocomposite materials. Critical to the understanding of fundamental physics of such materials and hence their applications in electronic nanodevices, is the ability to investigate their ferroelectric polarization at the nanoscale in a non-destructive way. We show that contact Kelvin probe force microscopy (cKPFM) combined with a k-means response clustering algorithm enables to measure the ferroelectric response at a mapping resolution of 8 nm. In a BaTiO3 thin film on silicon composed of tetragonal and hexagonal nanocrystals, we determine a nanoscale lateral distribution of discrete ferroelectric response clusters, fully consistent with the nanostructure determined by transmission electron microscopy. Moreover, we apply this data clustering method to the cKPFM responses measured at different temperatures, which allows us to follow the corresponding change in polarization pattern as the Curie temperature is approached and across the phase transition. This work opens up perspectives for mapping complex ferroelectric polarization textures such as curled/swirled polar textures that can be stabilized in epitaxial heterostructures and more generally mapping the polar domain distribution of any spatially-highly-heterogeneous ferroelectric materials.

Major advancements in fields as diverse as biology and quantum computing have relied on a multitude of microscopic techniques. All optical, electron and scanning probe microscopy advanced with new detector technologies and integration of spectroscopy, imaging, and diffraction. Despite the considerable proliferation of these instruments, significant bottlenecks remain in terms of processing, analysis, storage, and retrieval of acquired datasets. Aside from the lack of file standards, individual domain-specific analysis packages are often disjoint from the underlying datasets. Thus, keeping track of analysis and processing steps remains tedious for the end-user, hampering reproducibility. Here, we introduce the pycroscopy ecosystem of packages, an open-source python-based ecosystem underpinned by a common data model. Our data model, termed the N-dimensional spectral imaging data format, is realized in pycroscopy's sidpy package. This package is built on top of dask arrays, thus leveraging dask array attributes but expanding them to accelerate microscopy-relevant analysis and visualization. Several examples of the use of the pycroscopy ecosystem to create workflows for data ingestion and analysis are shown. Adoption of such standardized routines will be critical to usher in the next generation of autonomous instruments where processing, computation, and meta-data storage will be critical to overall experimental operations., Comment: 14 pages, 6 figures

The broad adoption of machine learning (ML)-based automated and autonomous experiments (AE) in physical characterization and synthesis requires development of strategies for understanding and intervention in the experimental workflow. Here, we introduce and realize strategies for post-acquisition forensic analysis applied to the deep kernel learning based AE scanning probe microscopy. This approach yields real-time and post-acquisition indicators of the progression of an active learning process interacting with an experimental system. We further illustrate that this approach can be extended towards human-in-the-loop autonomous experiments, where human operators make high-level decisions at high latencies setting the policies for AE, and the ML algorithm performs low-level fast decisions. The proposed approach is universal and can be extended to other physical and chemical imaging techniques and applications such as combinatorial library analysis. The full forensic analysis notebook is publicly available on GitHub at https://github.com/yongtaoliu/Forensics-DKL-BEPS., Comment: 24 pages, 8 figures

Experimental science is enabled by the combination of synthesis, imaging, and functional characterization. Synthesis of a new material is typically followed by a set of characterization methods aiming to provide feedback for optimization or discover fundamental mechanisms. However, the sequence of synthesis and characterization methods and their interpretation, or research workflow, has traditionally been driven by human intuition and is highly domain specific. Here we explore concepts of scientific workflows that emerge at the interface between theory, characterization, and imaging. We discuss the criteria by which these workflows can be constructed for special cases of multi-resolution structural imaging and structural and functional characterization. Some considerations for theory-experiment workflows are provided. We further pose that the emergence of user facilities and cloud labs disrupt the classical progression from ideation, orchestration, and execution stages of workflow development and necessitate development of universal frameworks for workflow design, including universal hyper-languages describing laboratory operation, reward functions and their integration between domains, and policy development for workflow optimization. These tools will enable knowledge-based workflow optimization, enable lateral instrumental networks, sequential and parallel orchestration of characterization between dissimilar facilities, and empower distributed research., Comment: 33 pages; 8 figures

Electronic transport and hysteresis in metal halide perovskites (MHPs) are key to the applications in photovoltaics, light emitting devices, and light and chemical sensors. These phenomena are strongly affected by the materials microstructure including grain boundaries, ferroic domain walls, and secondary phase inclusions. Here, we demonstrate an active machine learning framework for 'driving' an automated scanning probe microscope (SPM) to discover the microstructures responsible for specific aspects of transport behavior in MHPs. In our setup, the microscope can discover the microstructural elements that maximize the onset of conduction, hysteresis, or any other characteristic that can be derived from a set of current-voltage spectra. This approach opens new opportunities for exploring the origins of materials functionality in complex materials by SPM and can be integrated with other characterization techniques either before (prior knowledge) or after (identification of locations of interest for detail studies) functional probing., Comment: 19 pages; 7 figures

We pose that microscopy offers an ideal real-world experimental environment for the development and deployment of active Bayesian and reinforcement learning methods. Indeed, the tremendous progress achieved by machine learning (ML) and artificial intelligence over the last decade has been largely achieved via the utilization of static data sets, from the paradigmatic MNIST to the bespoke corpora of text and image data used to train large models such as GPT3, DALLE and others. However, it is now recognized that continuous, minute improvements to state-of-the-art do not necessarily translate to advances in real-world applications. We argue that a promising pathway for the development of ML methods is via the route of domain-specific deployable algorithms in areas such as electron and scanning probe microscopy and chemical imaging. This will benefit both fundamental physical studies and serve as a test bed for more complex autonomous systems such as robotics and manufacturing. Favorable environment characteristics of scanning and electron microscopy include low risk, extensive availability of domain-specific priors and rewards, relatively small effects of exogeneous variables, and often the presence of both upstream first principles as well as downstream learnable physical models for both statics and dynamics. Recent developments in programmable interfaces, edge computing, and access to APIs facilitating microscope control, all render the deployment of ML codes on operational microscopes straightforward. We discuss these considerations and hope that these arguments will lead to creating a novel set of development targets for the ML community by accelerating both real-world ML applications and scientific progress.

The advent of modern, high-speed electron detectors has made the collection of multidimensional hyperspectral transmission electron microscopy datasets, such as 4D-STEM, a routine. However, many microscopists find such experiments daunting since such datasets' analysis, collection, long-term storage, and networking remain challenging. Some common issues are the large and unwieldy size of the said datasets, often running into several gigabytes, non-standardized data analysis routines, and a lack of clarity about the computing and network resources needed to utilize the electron microscope fully. However, the existing computing and networking bottlenecks introduce significant penalties in each step of these experiments, and thus, real-time analysis-driven automated experimentation for multidimensional TEM is exceptionally challenging. One solution is integrating microscopy with edge computing, where moderately powerful computational hardware performs the preliminary analysis before handing off the heavier computation to HPC systems. In this perspective, we trace the roots of computation in modern electron microscopy, demonstrate deep learning experiments running on an edge system, and discuss the networking requirements for tying together microscopes, edge computers, and HPC systems., Comment: Perspective on automated microscopy. 3 figures

We report the development and experimental implementation of the automated experiment workflows for the identification of the best predictive channel for a phenomenon of interest in spectroscopic measurements. The approach is based on the combination of ensembled deep kernel learning for probabilistic predictions and a basic reinforcement learning policy for channel selection. It allows the identification of which of the available observational channels, sampled sequentially, are most predictive of selected behaviors, and hence have the strongest correlations. We implement this approach for multimodal imaging in Piezoresponse Force Microscopy (PFM), with the behaviors of interest manifesting in piezoresponse spectroscopy. We illustrate the best predictive channel for polarization-voltage hysteresis loop and frequency-voltage hysteresis loop areas is amplitude in the model samples. The same workflow and code are universal and applicable for any multimodal imaging and local characterization methods., Comment: 17 pages, 5 figures

We develop and implement an automated experiment in multimodal imaging to probe structural, chemical, and functional behaviors in complex materials and elucidate the dominant physical mechanisms that control device function. Here the emergence of non-linear electromechanical responses in piezoresponse force microscopy (PFM) is explored. Non-linear responses in PFM can originate from multiple mechanisms, including intrinsic material responses often controlled by domain structure, surface topography that affects the mechanical phenomena at the tip-surface junction, and, potentially, the presence of surface contaminants. Using an automated experiment to probe the origins of non-linear behavior in model ferroelectric lead titanate (PTO) and ferroelectric Al0.93B0.07N films, it was found that PTO showed asymmetric nonlinear behavior across a/c domain walls and a broadened high nonlinear response region around c/c domain walls. In contrast, for Al0.93B0.07N, well-poled regions showed high linear piezoelectric responses paired with low non-linear responses and regions that were multidomain indicated low linear responses and high nonlinear responses. We show that formulating dissimilar exploration strategies in deep kernel learning as alternative hypotheses allows for establishing the preponderant physical mechanisms behind the non-linear behaviors, suggesting that this approach automated experiments can potentially discern between competing physical mechanisms. This technique can also be extended to electron, probe, and chemical imaging., Comment: 18 pages, 5 figures

Unsupervised and semi-supervised ML methods such as variational autoencoders (VAE) have become widely adopted across multiple areas of physics, chemistry, and materials sciences due to their capability in disentangling representations and ability to find latent manifolds for classification and regression of complex experimental data. Like other ML problems, VAEs require hyperparameter tuning, e.g., balancing the Kullback Leibler (KL) and reconstruction terms. However, the training process and resulting manifold topology and connectivity depend not only on hyperparameters, but also their evolution during training. Because of the inefficiency of exhaustive search in a high-dimensional hyperparameter space for the expensive to train models, here we explored a latent Bayesian optimization (zBO) approach for the hyperparameter trajectory optimization for the unsupervised and semi-supervised ML and demonstrate for joint-VAE with rotational invariances. We demonstrate an application of this method for finding joint discrete and continuous rotationally invariant representations for MNIST and experimental data of a plasmonic nanoparticles material system. The performance of the proposed approach has been discussed extensively, where it allows for any high dimensional hyperparameter tuning or trajectory optimization of other ML models., Comment: 32 pages, including 11 figures in the main text and Appendixes with 2 figures. arXiv admin note: text overlap with arXiv:2108.12889

Automated chemical synthesis, materials fabrication, and spectroscopic physical measurements often bring forth the challenge of process trajectory optimization, i.e., discovering the time dependence of temperature, electric field, or pressure that gives rise to optimal properties. Due to the high dimensionality of the corresponding vectors, these problems are not directly amenable to Bayesian Optimization (BO). Here we propose an approach based on the combination of the generative statistical models, specifically variational autoencoders, and Bayesian optimization. Here, the set of potential trajectories is formed based on best practices in the field, domain intuition, or human expertise. The variational autoencoder is used to encode the thus generated trajectories as a latent vector, and also allows for the generation of trajectories via sampling from latent space. In this manner, Bayesian Optimization of the process is realized in the latent space of the system, reducing the problem to a low-dimensional one. Here we apply this approach to a ferroelectric lattice model and demonstrate that this approach allows discovering the field trajectories that maximize curl in the system. The analysis of the corresponding polarization and curl distributions allows the relevant physical mechanisms to be decoded., Comment: 22 pages and 9 figures

Recent progress in machine learning methods, and the emerging availability of programmable interfaces for scanning probe microscopes (SPMs), have propelled automated and autonomous microscopies to the forefront of attention of the scientific community. However, enabling automated microscopy requires the development of task-specific machine learning methods, understanding the interplay between physics discovery and machine learning, and fully defined discovery workflows. This, in turn, requires balancing the physical intuition and prior knowledge of the domain scientist with rewards that define experimental goals and machine learning algorithms that can translate these to specific experimental protocols. Here, we discuss the basic principles of Bayesian active learning and illustrate its applications for SPM. We progress from the Gaussian Process as a simple data-driven method and Bayesian inference for physical models as an extension of physics-based functional fits to more complex deep kernel learning methods, structured Gaussian Processes, and hypothesis learning. These frameworks allow for the use of prior data, the discovery of specific functionalities as encoded in spectral data, and exploration of physical laws manifesting during the experiment. The discussed framework can be universally applied to all techniques combining imaging and spectroscopy, SPM methods, nanoindentation, electron microscopy and spectroscopy, and chemical imaging methods, and can be particularly impactful for destructive or irreversible measurements., Comment: 39 pages, 10 figures

The design of materials structure for optimizing functional properties and potentially, the discovery of novel behaviors is a keystone problem in materials science. In many cases microstructural models underpinning materials functionality are available and well understood. However, optimization of average properties via microstructural engineering often leads to combinatorically intractable problems. Here, we explore the use of the reinforcement learning (RL) for microstructure optimization targeting the discovery of the physical mechanisms behind enhanced functionalities. We illustrate that RL can provide insights into the mechanisms driving properties of interest in a 2D discrete Landau ferroelectrics simulator. Intriguingly, we find that non-trivial phenomena emerge if the rewards are assigned to favor physically impossible tasks, which we illustrate through rewarding RL agents to rotate polarization vectors to energetically unfavorable positions. We further find that strategies to induce polarization curl can be non-intuitive, based on analysis of learned agent policies. This study suggests that RL is a promising machine learning method for material design optimization tasks, and for better understanding the dynamics of microstructural simulations., Comment: 5 figures

We report the development and implementation of a hypothesis learning based automated experiment, in which the microscope operating in the autonomous mode identifies the physical laws behind the material's response. Specifically, we explore the bias induced transformations that underpin the functionality of broad classes of devices and functional materials from batteries and memristors to ferroelectrics and antiferroelectrics. Optimization and design of these materials require probing the mechanisms of these transformations on the nanometer scale as a function of the broad range of control parameters such as applied potential and time, often leading to experimentally intractable scenarios. At the same time, often the behaviors of these systems are understood within potentially competing theoretical models, or hypotheses. Here, we develop a hypothesis list that covers the possible limiting scenarios for the domain growth, including thermodynamic, domain wall pinning, and screening limited. We further develop and experimentally implement the hypothesis driven automated experiment in Piezoresponse Force Microscopy, autonomously identifying the mechanisms of the bias induced domain switching. This approach can be applied for a broad range of physical and chemical experiments with relatively low dimensional control parameter space and for which the possible competing models of the system behavior that ideally cover the full range of physical eventualities are known or can be created. These include other scanning probe microscopy modalities such as force distance curve measurements and nanoindentation, as well as materials synthesis and optimization., Comment: 25 pages, 6 figures