-
Unsupervised segmentation of irradiation$\unicode{x2010}$induced order$\unicode{x2010}$disorder phase transitions in electron microscopy
Authors:
Arman H Ter-Petrosyan,
Jenna A Bilbrey,
Christina M Doty,
Bethany E Matthews,
Le Wang,
Yingge Du,
Eric Lang,
Khalid Hattar,
Steven R Spurgeon
Abstract:
We present a method for the unsupervised segmentation of electron microscopy images, which are powerful descriptors of materials and chemical systems. Images are oversegmented into overlapping chips, and similarity graphs are generated from embeddings extracted from a domain$\unicode{x2010}$pretrained convolutional neural network (CNN). The Louvain method for community detection is then applied to…
▽ More
We present a method for the unsupervised segmentation of electron microscopy images, which are powerful descriptors of materials and chemical systems. Images are oversegmented into overlapping chips, and similarity graphs are generated from embeddings extracted from a domain$\unicode{x2010}$pretrained convolutional neural network (CNN). The Louvain method for community detection is then applied to perform segmentation. The graph representation provides an intuitive way of presenting the relationship between chips and communities. We demonstrate our method to track irradiation$\unicode{x2010}$induced amorphous fronts in thin films used for catalysis and electronics. This method has potential for "on$\unicode{x2010}$the$\unicode{x2010}$fly" segmentation to guide emerging automated electron microscopes.
△ Less
Submitted 14 November, 2023;
originally announced November 2023.
-
Deep Learning for Automated Experimentation in Scanning Transmission Electron Microscopy
Authors:
Sergei V. Kalinin,
Debangshu Mukherjee,
Kevin M. Roccapriore,
Ben Blaiszik,
Ayana Ghosh,
Maxim A. Ziatdinov,
A. Al-Najjar,
Christina Doty,
Sarah Akers,
Nageswara S. Rao,
Joshua C. Agar,
Steven R. Spurgeon
Abstract:
Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centered experiment workflow design and…
▽ More
Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centered experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for the edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows and the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.
△ Less
Submitted 4 April, 2023;
originally announced April 2023.
-
Deep-learning-based prediction of nanoparticle phase transitions during in situ transmission electron microscopy
Authors:
Wenkai Fu,
Steven R. Spurgeon,
Chongmin Wang,
Yuyan Shao,
Wei Wang,
Amra Peles
Abstract:
We develop the machine learning capability to predict a time sequence of in-situ transmission electron microscopy (TEM) video frames based on the combined long-short-term-memory (LSTM) algorithm and the features de-entanglement method. We train deep learning models to predict a sequence of future video frames based on the input of a sequence of previous frames. This unique capability provides insi…
▽ More
We develop the machine learning capability to predict a time sequence of in-situ transmission electron microscopy (TEM) video frames based on the combined long-short-term-memory (LSTM) algorithm and the features de-entanglement method. We train deep learning models to predict a sequence of future video frames based on the input of a sequence of previous frames. This unique capability provides insight into size dependent structural changes in Au nanoparticles under dynamic reaction condition using in-situ environmental TEM data, informing models of morphological evolution and catalytic properties. The model performance and achieved accuracy of predictions are desirable based on, for scientific data characteristic, based on limited size of training data sets. The model convergence and values for the loss function mean square error show dependence on the training strategy, and structural similarity measure between predicted structure images and ground truth reaches the value of about 0.7. This computed structural similarity is smaller than values obtained when the deep learning architecture is trained using much larger benchmark data sets, it is sufficient to show the structural transition of Au nanoparticles. While performance parameters of our model applied to scientific data fall short of those achieved for the non-scientific big data sets, we demonstrate model ability to predict the evolution, even including the particle structural phase transformation, of Au nano particles as catalyst for CO oxidation under the chemical reaction conditions. Using this approach, it may be possible to anticipate the next steps of a chemical reaction for emerging automated experimentation platforms.
△ Less
Submitted 23 May, 2022;
originally announced May 2022.
-
An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics
Authors:
Matthew Olszta,
Derek Hopkins,
Kevin R. Fiedler,
Marjolein Oostrom,
Sarah Akers,
Steven R. Spurgeon
Abstract:
Artificial intelligence (AI) promises to reshape scientific inquiry and enable breakthrough discoveries in areas such as energy storage, quantum computing, and biomedicine. Scanning transmission electron microscopy (STEM), a cornerstone of the study of chemical and materials systems, stands to benefit greatly from AI-driven automation. However, present barriers to low-level instrument control, as…
▽ More
Artificial intelligence (AI) promises to reshape scientific inquiry and enable breakthrough discoveries in areas such as energy storage, quantum computing, and biomedicine. Scanning transmission electron microscopy (STEM), a cornerstone of the study of chemical and materials systems, stands to benefit greatly from AI-driven automation. However, present barriers to low-level instrument control, as well as generalizable and interpretable feature detection, make truly automated microscopy impractical. Here, we discuss the design of a closed-loop instrument control platform guided by emerging sparse data analytics. We demonstrate how a centralized controller, informed by machine learning combining limited $a$ $priori$ knowledge and task-based discrimination, can drive on-the-fly experimental decision-making. This platform unlocks practical, automated analysis of a variety of material features, enabling new high-throughput and statistical studies.
△ Less
Submitted 29 September, 2021;
originally announced September 2021.
-
Design of a Graphical User Interface for Few-Shot Machine Learning Classification of Electron Microscopy Data
Authors:
Christina Doty,
Shaun Gallagher,
Wenqi Cui,
Wenya Chen,
Shweta Bhushan,
Marjolein Oostrom,
Sarah Akers,
Steven R. Spurgeon
Abstract:
The recent growth in data volumes produced by modern electron microscopes requires rapid, scalable, and flexible approaches to image segmentation and analysis. Few-shot machine learning, which can richly classify images from a handful of user-provided examples, is a promising route to high-throughput analysis. However, current command-line implementations of such approaches can be slow and unintui…
▽ More
The recent growth in data volumes produced by modern electron microscopes requires rapid, scalable, and flexible approaches to image segmentation and analysis. Few-shot machine learning, which can richly classify images from a handful of user-provided examples, is a promising route to high-throughput analysis. However, current command-line implementations of such approaches can be slow and unintuitive to use, lacking the real-time feedback necessary to perform effective classification. Here we report on the development of a Python-based graphical user interface that enables end users to easily conduct and visualize the output of few-shot learning models. This interface is lightweight and can be hosted locally or on the web, providing the opportunity to reproducibly conduct, share, and crowd-source few-shot analyses.
△ Less
Submitted 21 July, 2021;
originally announced July 2021.
