Computer Vision for Recognition of Materials and Vessels in Chemistry Lab Settings and the Vector-LabPics Data Set

Self-Driving Laboratories for Chemistry and Materials Science

Self-driving laboratories (SDLs) promise an accelerated application of the scientific method. Through the automation of experimental workflows, along with autonomous experimental planning, SDLs hold the potential to greatly accelerate research in chemistry and materials discovery. This review provides an in-depth analysis of the state-of-the-art in SDL technology, its applications across various scientific disciplines, and the potential implications for research and industry. This review additionally provides an overview of the enabling technologies for SDLs, including their hardware, software, and integration with laboratory infrastructure. Most importantly, this review explores the diverse range of scientific domains where SDLs have made significant contributions, from drug discovery and materials science to genomics and chemistry. We provide a comprehensive review of existing real-world examples of SDLs, their different levels of automation, and the challenges and limitations associated with each domain.

Continuous and autonomous-flow separation of laccase enzyme utilizing functionalized aqueous two-phase system with computer vision control

Downstream processing of biomolecules, particularly therapeutic proteins and enzymes, presents a formidable challenge due to intricate unit operations and high costs. This study introduces a novel cysteine (cys) functionalized aqueous two-phase system (ATPS) utilizing polyethylene glycol (PEG) and potassium phosphate, referred as PEG-K3PO4/cys, for selective extraction of laccase from complex protein mixtures. A 3D-baffle micro-mixer and phase separator was meticulously designed and equipped with computer vision controller, to enable precise mixing and continuous phase separation under automated-flow. Microfluidic-assisted ATPS exhibits substantial increase in partition coefficient (Kflow = 16.3) and extraction efficiency (EEflow = 88 %) for laccase compared to conventional batch process. Integrated and continuous-flow process efficiently partitioned laccase, even in low concentrations and complex crude extracts. Circular dichroism spectra of laccase confirm structural stability of enzyme throughout the purification process. Eventually, continuous-flow microfluidic bioseparation is highly useful for seamless downstream processing of target biopharmaceuticals in integrated and autonomous manner.

Comprehensive image dataset for enhancing object detection in chemical experiments

The application of image recognition in chemical experiments has the potential to enhance experiment recording and risk management. However, the current scarcity of suitable benchmarking datasets restricts the applications of machine vision techniques in chemical experiments. This data article presents an image dataset featuring common chemical apparatuses and experimenter's hands. The images have been meticulously annotated, providing detailed information for precise object detection through deep learning methods. The images were captured from videos filmed in organic chemistry laboratories. This dataset comprises a total of 5078 images including diverse backgrounds and situations surrounding the objects. Detailed annotations are provided in accompanying text files. The dataset is organized into training, validation, and test subsets. Each subset is stored within independent folders for easy access and utilization.

A focus on computer vision for non-contact monitoring of catalyst degradation and product formation kinetics

Chemists know the value of looking at a reaction for clues about reaction progress and success, but what-it-looks-like has never been quantified. Reid and co-workers (C. Yan, M. Cowie, C. Howcutt, K. M. P. Wheelhouse, N. S. Hodnett, M. Kollie, M. Gildea, M. H. Goodfellow and M. Reid, Chem. Sci., 2023, 14, 5323-5331, https://doi.org/10.1039/d2sc05702f) have developed an approach that uses camera footage of reactions to obtain quantitative descriptors of changes in reaction mixtures to support kinetic analysis.

Go with the flow: deep learning methods for autonomous viscosity estimations

Closed-loop experiments can accelerate material discovery by automating both experimental manipulations and decisions that have traditionally been made by researchers. Fast and non-invasive measurements are particularly attractive for closed-loop strategies. Viscosity is a physical property for fluids that is important in many applications. It is fundamental in application areas such as coatings; also, even if viscosity is not the key property of interest, it can impact our ability to do closed-loop experimentation. For example, unexpected increases in viscosity can cause liquid-handling robots to fail. Traditional viscosity measurements are manual, invasive, and slow. Here we use convolutional neural networks (CNNs) as an alternative to traditional viscometry by non-invasively extracting the spatiotemporal features of fluid motion under flow. To do this, we built a workflow using a dual-armed collaborative robot that collects video data of fluid motion autonomously. This dataset was then used to train a 3-dimensional convolutional neural network (3D-CNN) for viscosity estimation, either by classification or by regression. We also used these models to identify unknown laboratory solvents, again based on differences in fluid motion. The 3D-CNN model performance was compared with the performance of a panel of human participants for the same classification tasks. Our models strongly outperformed human classification in both cases. For example, even with training on fewer than 50 videos for each liquid, the 3D-CNN model gave an average accuracy of 88% for predicting the identity of five different laboratory solvents, compared to an average accuracy of 32% for human observation. For comparison, random category selection would give an average accuracy of 20%. Our method offers an alternative to traditional viscosity measurements for autonomous chemistry workflows that might be used both for process control (e.g., choosing not to pipette liquids that are too viscous) or for materials discovery (e.g., identifying new polymerization catalysts on the basis of viscosification).

AtomVision: A Machine Vision Library for Atomistic Images

Computer vision techniques have immense potential for materials design applications. In this work, we introduce an integrated and general-purpose AtomVision library that can be used to generate and curate microscopy image (such as scanning tunneling microscopy and scanning transmission electron microscopy) data sets and apply a variety of machine learning techniques. To demonstrate the applicability of this library, we (1) establish an atomistic image data set of about 10 000 materials with large structural and chemical diversity, (2) develop and compare convolutional and atomistic line graph neural network models to classify the Bravais lattices, (3) demonstrate the application of fully convolutional neural networks using U-Net architecture to pixelwise classify atom versus background, (4) use a generative adversarial network for super resolution, (5) curate an image data set on the basis of natural language processing using an open-access arXiv data set, and (6) integrate the computational framework with experimental microscopy images for Rh, Fe₃O₄, and SnS systems. The AtomVision library is available at https://github.com/usnistgov/atomvision.

Machine Learning in Unmanned Systems for Chemical Synthesis

Chemical synthesis is state-of-the-art, and, therefore, it is generally based on chemical intuition or experience of researchers. The upgraded paradigm that incorporates automation technology and machine learning (ML) algorithms has recently been merged into almost every subdiscipline of chemical science, from material discovery to catalyst/reaction design to synthetic route planning, which often takes the form of unmanned systems. The ML algorithms and their application scenarios in unmanned systems for chemical synthesis were presented. The prospects for strengthening the connection between reaction pathway exploration and the existing automatic reaction platform and solutions for improving autonomation through information extraction, robots, computer vision, and intelligent scheduling were proposed.

An Optical Approach for Cell Pellet Detection

Cell-based screening methods are increasingly used in diagnostics and drug development. As a result, various research groups from around the world have been working on this topic to develop methods and algorithms that increase the degree of automation of various measurement techniques. The field of computer vision is becoming increasingly important and has therefore a significant influence on the development of various processes in modern laboratories. In this work we describe an approach for detecting two height information, the phase boundary of a cell pellet and the bottom edge of the tube, and thereby a method for determining the highest point of the topology. The starting point for the development of the method described are cells obtained by various procedures and stabilized by a fixative. Centrifugation of the tube causes the cells to settle to the bottom of the tube, resulting in a cell pellet with a clear phase boundary between the cells and the fixative. For further studies, the supernatant fixative has to be removed without reducing the number of cells. The fixative is to be extracted automatically by a liquid robot, which is only possible by accurately determining the cell pellet height. Due to centrifugation, an uneven topology is formed, which is why the entire phase boundary must be examined to detect the highest point of the cell pellet. For this approach, the tube to be examined, which contains the cells and the fixative, is rotated 360° in defined small steps after centrifugation. During rotation, an image is captured in each step, after which a defined image area is separated from the center of the image and merged into a panoramic image. This produces a panoramic image of the cell topology which represents the complete phase boundary, the boundary located on the outside of the tube. This panoramic image is modified through various image processing steps to extract and detect the phase boundary. Various image processing algorithms from the OpenCV library are used. In the first step, the panoramic image is convolved with a Gaussian blur filter to reduce noise. In the following step, a black and white image is generated by a thresholding process. This black and white image, or binary image, is convolved with a Sobel operator in the x and y directions and the results are superimposed. This overlaid image shows the top edge of the cell pellet and other edges located in the image. A logical exclusion method of the obtained boundaries is used for the detection of the phase boundary. To detect the tube bottom, a multilevel model was trained in advance with an appropriate data set. This model can detect and localize in near real time the tube bottom in an image. By using the two-height information of the different boundaries, phase boundary and tube bottom, the highest point of the cell pellet can be detected. This information is then passed on to a higher-level process so that the liquid robot can approach this point with the pipette tip to remove the excess fixative. By determining the highest point, the probability of being able to remove a larger amount of fixative without reducing the number of cells is highest. This ensures that post-processing studies have the largest possible number of cells available with complete automation.

Autonomous Chemical Experiments: Challenges and Perspectives on Establishing a Self-Driving Lab

We must accelerate the pace at which we make technological advancements to address climate change and disease risks worldwide. This swifter pace of discovery requires faster research and development cycles enabled by better integration between hypothesis generation, design, experimentation, and data analysis. Typical research cycles take months to years. However, data-driven automated laboratories, or self-driving laboratories, can significantly accelerate molecular and materials discovery. Recently, substantial advancements have been made in the areas of machine learning and optimization algorithms that have allowed researchers to extract valuable knowledge from multidimensional data sets. Machine learning models can be trained on large data sets from the literature or databases, but their performance can often be hampered by a lack of negative results or metadata. In contrast, data generated by self-driving laboratories can be information-rich, containing precise details of the experimental conditions and metadata. Consequently, much larger amounts of high-quality data are gathered in self-driving laboratories. When placed in open repositories, this data can be used by the research community to reproduce experiments, for more in-depth analysis, or as the basis for further investigation. Accordingly, high-quality open data sets will increase the accessibility and reproducibility of science, which is sorely needed.In this Account, we describe our efforts to build a self-driving lab for the development of a new class of materials: organic semiconductor lasers (OSLs). Since they have only recently been demonstrated, little is known about the molecular and material design rules for thin-film, electrically-pumped OSL devices as compared to other technologies such as organic light-emitting diodes or organic photovoltaics. To realize high-performing OSL materials, we are developing a flexible system for automated synthesis via iterative Suzuki-Miyaura cross-coupling reactions. This automated synthesis platform is directly coupled to the analysis and purification capabilities. Subsequently, the molecules of interest can be transferred to an optical characterization setup. We are currently limited to optical measurements of the OSL molecules in solution. However, material properties are ultimately most important in the solid state (e.g., as a thin-film device). To that end and for a different scientific goal, we are developing a self-driving lab for inorganic thin-film materials focused on the oxygen evolution reaction.While the future of self-driving laboratories is very promising, numerous challenges still need to be overcome. These challenges can be split into cognition and motor function. Generally, the cognitive challenges are related to optimization with constraints or unexpected outcomes for which general algorithmic solutions have yet to be developed. A more practical challenge that could be resolved in the near future is that of software control and integration because few instrument manufacturers design their products with self-driving laboratories in mind. Challenges in motor function are largely related to handling heterogeneous systems, such as dispensing solids or performing extractions. As a result, it is critical to understand that adapting experimental procedures that were designed for human experimenters is not as simple as transferring those same actions to an automated system, and there may be more efficient ways to achieve the same goal in an automated fashion. Accordingly, for self-driving laboratories, we need to carefully rethink the translation of manual experimental protocols.

Deep neural network analysis of nanoparticle ordering to identify defects in layered carbon materials

Smoothness/defectiveness of the carbon material surface is a key issue for many applications, spanning from electronics to reinforced materials, adsorbents and catalysis. Several surface defects cannot be observed with conventional analytic techniques, thus requiring the development of a new imaging approach. Here, we evaluate a convenient method for mapping such "hidden" defects on the surface of carbon materials using 1-5 nm metal nanoparticles as markers. A direct relationship between the presence of defects and the ordering of nanoparticles was studied experimentally and modeled using quantum chemistry calculations and Monte Carlo simulations. An automated pipeline for analyzing microscopic images is described: the degree of smoothness of experimental images was determined by a classification neural network, and then the images were searched for specific types of defects using a segmentation neural network. An informative set of features was generated from both networks: high-dimensional embeddings of image patches and statics of defect distribution.

Automated solubility screening platform using computer vision

Solubility screening is an essential, routine process that is often labor intensive. Robotic platforms have been developed to automate some aspects of the manual labor involved. However, many of the existing systems rely on traditional analytic techniques such as high-performance liquid chromatography, which require pre-calibration for each compound and can be resource consuming. In addition, automation is not typically end-to-end, requiring user intervention to move vials, establish analytical methods for each compound and interpret the raw data. We developed a closed-loop, flexible robotic system with integrated solid and liquid dosing capabilities that relies on computer vision and iterative feedback to successfully measure caffeine solubility in multiple solvents. After initial researcher input (<2 min), the system ran autonomously, screening five different solvent systems (20-80 min each). The resulting solubility values matched those obtained using traditional manual techniques.

•

We demonstrate a modular, closed-loop robotic platform for solubility screening

•

Automated solvent titration is informed by computer vision and turbidity monitoring

•

No human intervention or HPLC analysis is required during the experimental loop

•

Solubility values obtained by the system match those obtained via traditional methods