Inference in artificial intelligence with deep optics and photonics

Electronic imaging of photoisomerisation process in photochromic crystals with scanning tunnelling spectroscopy

Nanoscale photoisomerisation in photochromic crystals is promising for nanoscale optoelectronic applications. To further understand this phenomenon and address fundamental difficulties in measuring photoisomerisation at the nanoscale, we used scanning tunnelling spectroscopy (STS) to measure the changes in the electronic properties of the microcrystals that correspond to the photoisomerisation state. The results showed reversible changes in the electronic properties of the crystals in response to photoisomerisation between the coloured (closed ring) and decoloured (open ring) states. Furthermore, the measured tunnel current fluctuation increased as the crystal was uniformly decolourised, indicating that the measurement was sensitive to the position of the molecular-scale tunnel junction. STS measurements on the crystals visualised the relationship between the thickness of the crystals and the degree of photoisomerisation in a two-dimensional image without the use of light. Isomerisation-dependent electronic properties provide a method to visualise the photoisomerisation progress on the crystal at the nanometre scale without extra photoisomerisations and show potential as a fundamental technology for nano-optoelectronics integration.

Experimental benchmarking of quantum state overlap estimation strategies with photonic systems

Accurately estimating the overlap between quantum states is a fundamental task in quantum information processing. While various strategies using distinct quantum measurements have been proposed for overlap estimation, the lack of experimental benchmarks on estimation precision limits strategy selection in different situations. Here we compare the performance of four practical strategies for overlap estimation, including tomography-tomography, tomography-projection, Schur collective measurement and optical swap test using photonic quantum systems. We encode the quantum states on the polarization and path degrees of freedom of single photons. The corresponding measurements are performed by photon detection on certain modes following single-photon mode transformation or two-photon interference. We further propose an adaptive strategy with optimized precision in full-range overlap estimation. Our results shed new light on extracting the parameter of interest from quantum systems, prompting the design of efficient quantum protocols.

Real-time holographic camera for obtaining real 3D scene hologram

As a frontier technology, holography has important research values in fields such as bio-micrographic imaging, light field modulation and data storage. However, the real-time acquisition of 3D scenes and high-fidelity reconstruction technology has not yet made a breakthrough, which has seriously hindered the development of holography. Here, a novel holographic camera is proposed to solve the above inherent problems completely. The proposed holographic camera consists of the acquisition end and the calculation end. At the acquisition end of the holographic camera, specially configured liquid materials and liquid lens structure based on voice-coil motor-driving are used to produce the liquid camera, so that the liquid camera can quickly capture the focus stack of the real 3D scene within 15 ms. At the calculation end, a new structured focus stack network (FS-Net) is designed for hologram calculation. After training the FS-Net with the focus stack renderer and learnable Zernike phase, it enables hologram calculation within 13 ms. As the first device to achieve real-time incoherent acquisition and high-fidelity holographic reconstruction of a real 3D scene, our proposed holographic camera breaks technical bottlenecks of difficulty in acquiring the real 3D scene, low quality of the holographic reconstructed image, and incorrect defocus blur. The experimental results demonstrate the effectiveness of our holographic camera in the acquisition of focal plane information and hologram calculation of the real 3D scene. The proposed holographic camera opens up a new way for the application of holography in fields such as 3D display, light field modulation, and 3D measurement.

Deep empirical neural network for optical phase retrieval over a scattering medium

Supervised learning, a popular tool in modern science and technology, thrives on huge amounts of labeled data. Physics-enhanced deep neural networks offer an effective solution to alleviate the data burden by incorporating an analytical model that interprets the underlying physical processes. However, it completely fails in tackling systems without analytical solution, where wave scattering systems with multiple input multiple output are typical examples. Herein, we propose a concept of deep empirical neural network (DENN) that is a hybridization of a deep neural network and an empirical model, which enables seeing through an opaque scattering medium in an untrained manner. The DENN does not rely on labeled data, all while delivering as high as 58% improvement in fidelity compared with the supervised learning using 30000 data pairs for achieving the same goal of optical phase retrieval. The DENN might shed new light on the applications of deep learning in physics, information science, biology, chemistry and beyond.

Polarimetric Compressed Sensing with Hollow, Self-Assembled Diffractive Films

Sensing light's polarization and wavefront direction enables surface curvature assessment, material identification, shadow differentiation, and improved image quality in turbid environments. Traditional polarization cameras utilize multiple sensor measurements per pixel and polarization-filtering optics, which result in reduced image resolution. We propose a nanophotonic pipeline that enables compressive sensing and reduces the sampling requirements with a low-refractive-index, self-assembled optical encoder. These nanostructures scatter light into lattice modes, which encode the wavefront direction and the polarization ellipticity in the linearly polarized components of the diffracted, interference patterns. Combining optical encoders with a neural network, the system predicts pointing and polarization when the interference patterns are adequately sampled. A comparison of "ordered" and "random" optical encoders shows that the latter both blurs the interference patterns and achieves higher resolution. Our work centers on the unexpected modulation and spatial multiplexing of incident light polarization by self-assembled hollow nanocavity arrays as a class of materials distinct from traditional metasurfaces that will not only enable encoding for polarization and optical computing but also for compressed sensing and imaging.

Empowering nanophotonic applications via artificial intelligence: pathways, progress, and prospects

Empowering nanophotonic devices via artificial intelligence (AI) has revolutionized both scientific research methodologies and engineering practices, addressing critical challenges in the design and optimization of complex systems. Traditional methods for developing nanophotonic devices are often constrained by the high dimensionality of design spaces and computational inefficiencies. This review highlights how AI-driven techniques provide transformative solutions by enabling the efficient exploration of vast design spaces, optimizing intricate parameter systems, and predicting the performance of advanced nanophotonic materials and devices with high accuracy. By bridging the gap between computational complexity and practical implementation, AI accelerates the discovery of novel nanophotonic functionalities. Furthermore, we delve into emerging domains, such as diffractive neural networks and quantum machine learning, emphasizing their potential to exploit photonic properties for innovative strategies. The review also examines AI's applications in advanced engineering areas, e.g., optical image recognition, showcasing its role in addressing complex challenges in device integration. By facilitating the development of highly efficient, compact optical devices, these AI-powered methodologies are paving the way for next-generation nanophotonic systems with enhanced functionalities and broader applications.

Femtojoule optical nonlinearity for deep learning with incoherent illumination

Optical neural networks (ONNs) are a promising computational alternative for deep learning due to their inherent massive parallelism for linear operations. However, the development of energy-efficient and highly parallel optical nonlinearities, a critical component in ONNs, remains an outstanding challenge. Here, we introduce a nonlinear optical microdevice array (NOMA) compatible with incoherent illumination by integrating the liquid crystal cell with silicon photodiodes at the single-pixel level. We fabricate NOMA with more than half a million pixels, each functioning as an optical analog of the rectified linear unit at ultralow switching energy down to 100 femtojoules per pixel. With NOMA, we demonstrate an optical multilayer neural network. Our work holds promise for large-scale and low-power deep ONNs, computer vision, and real-time optical image processing.

A novel spectroscopy-deep learning approach for aqueous multi-heavy metal detection

Addressing heavy metal contamination in water bodies is a critical concern for environmental scientists. Traditional detection methods are often complex and costly. Recent advancements in artificial intelligence (AI), particularly machine learning (ML) and deep learning (DL), have shown significant potential in analytical chemistry. However, these AI models require extensive spectral data, which traditional methods struggle to provide quickly. To overcome this challenge, we developed a new digital spectral imaging system and rapidly collected 3000 digital spectra from mixed heavy metal samples. We then created an end-to-end regression model for predicting heavy metal concentrations in mixed water samples using deep convolutional neural networks (ResNet-50, Inception V1, and SqueezeNet V1.1). The results indicated that the trained ResNet-50 model can effectively detect arsenic, chromium, and copper simultaneously, with a linear fitting coefficient exceeding 0.99 between true and predicted values. This study offers an efficient approach for rapid heavy metal detection in complex water environments and serves as a reference for developing intelligent analytical techniques.

Programmable wave-based analog computing machine: a metastructure that designs metastructures

The ability to perform mathematical computations using metastructures is an emergent paradigm that carries the potential of wave-based analog computing to the realm of near-speed-of-light, low-loss, compact devices. We theoretically introduce and experimentally verify the concept of a reconfigurable metastructure that performs analog complex mathematical computations using electromagnetic waves. Reconfigurable, RF-based components endow our device with the ability to perform stationary and non-stationary iterative algorithms. After demonstrating matrix inversion (stationary problem), we use the machine to tackle two major non-stationary problems: root finding with Newton's method and inverse design (constrained optimization) via the Lagrange multiplier method. The platform enables possible avenues for wave-based, analog computations for general linear algebraic problems and beyond in compact, ultrafast, and parallelized ways.

High-integrated photonic tensor core utilizing high-dimensional lightwave and microwave multidomain multiplexing

The burgeoning volume of parameters in artificial neural network models has posed substantial challenges to conventional tensor computing hardware. Benefiting from the available optical multidimensional information entropy, optical intelligent computing is used as an alternative solution to address the emerging challenges of electrical computing. These limitations, in terms of device size and photonic integration scale, have hindered the performance of optical chips. Herein, an ultrahigh computing density optical tensor processing unit (OTPU), which is grounded in an individual microring resonator (MRR), is introduced to respond to these challenges. Through the independent tuning of multiwavelength lasers, the operational capabilities of an MRR are orchestrated, culminating in the formation of an optical tensor core. This design facilitates the execution of tensor convolution operations via the lightwave and microwave multidomain hybrid multiplexing in terms of the time, wavelength, and frequency of microwaves. The experimental results for the MRR-based OTPU show an extraordinary computing density of 34.04 TOPS/mm2. Additionally, the achieved accuracy rate in recognizing MNIST handwritten digits was 96.41%. These outcomes signify a significant advancement toward the realization of high-performance optical tensor processing chips.

Quantum-limited stochastic optical neural networks operating at a few quanta per activation

Energy efficiency in computation is ultimately limited by noise, with quantum limits setting the fundamental noise floor. Analog physical neural networks hold promise for improved energy efficiency compared to digital electronic neural networks. However, they are typically operated in a relatively high-power regime so that the signal-to-noise ratio (SNR) is large (>10), and the noise can be treated as a perturbation. We study optical neural networks where all layers except the last are operated in the limit that each neuron can be activated by just a single photon, and as a result the noise on neuron activations is no longer merely perturbative. We show that by using a physics-based probabilistic model of the neuron activations in training, it is possible to perform accurate machine-learning inference in spite of the extremely high shot noise (SNR  ~ 1). We experimentally demonstrated MNIST handwritten-digit classification with a test accuracy of 98% using an optical neural network with a hidden layer operating in the single-photon regime; the optical energy used to perform the classification corresponds to just 0.038 photons per multiply-accumulate (MAC) operation. Our physics-aware stochastic training approach might also prove useful with non-optical ultra-low-power hardware.

TOPS-speed complex-valued convolutional accelerator for feature extraction and inference

Complex-valued neural networks process both amplitude and phase information, in contrast to conventional artificial neural networks, achieving additive capabilities in recognizing phase-sensitive data inherent in wave-related phenomena. The ever-increasing data capacity and network scale place substantial demands on underlying computing hardware. In parallel with the successes and extensive efforts made in electronics, optical neuromorphic hardware is promising to achieve ultra-high computing performances due to its inherent analog architecture and wide bandwidth. Here, we report a complex-valued optical convolution accelerator operating at over 2 Tera operations per second (TOPS). With appropriately designed phasors we demonstrate its performance in the recognition of synthetic aperture radar (SAR) images captured by the Sentinel-1 satellite, which are inherently complex-valued and more intricate than what optical neural networks have previously processed. Experimental tests with 500 images yield an 83.8% accuracy, close to in-silico results. This approach facilitates feature extraction of phase-sensitive information, and represents a pivotal advance in artificial intelligence towards real-time, high-dimensional data analysis of complex and dynamic environments.

Silicon photonic MEMS switches based on split waveguide crossings

The continuous push for high-performance photonic switches is one of the most crucial premises for the sustainable scaling of programmable and reconfigurable photonic circuits for a wide spectrum of applications. Conventional optical switches rely on the perturbative mechanisms of mode coupling or mode interference, resulting in inherent bottlenecks in their switching performance concerning size, power consumption and bandwidth. Here we propose and realize a silicon photonic 2×2 elementary switch based on a split waveguide crossing (SWX) consisting of two halves. The propagation direction of the incident light is manipulated to implement the OFF/ON states by splitting/combining the two halves of the SWX, showing excellent performance with low excess loss and low crosstalk over an ultrawide bandwidth. Both elementary switch and a 64×64 switch array based on Benes topology are fabricated and characterized, demonstrating great potential for practical scenarios such as photonic interconnect/routing, Lidar and spectroscopy, photonic computing, as well as microwave photonics.

Robust pattern retrieval in an optical Hopfield neural network

Hopfield neural networks (HNNs) promise broad applications in areas such as combinatorial optimization, memory storage, and pattern recognition. Among various implementations, optical HNNs are particularly interesting because they can take advantage of fast optical matrix-vector multiplications. Yet their studies so far have mostly been on the theoretical side, and the effects of optical imperfections and robustness against memory errors remain to be quantified. Here we demonstrate an optical HNN in a simple experimental setup using a spatial light modulator with 100 neurons. It successfully stores and retrieves 13 patterns, which approaches the critical capacity limit of α c  = 0.138. It is robust against random phase flipping errors of the stored patterns, achieving high fidelity in recognizing and storing patterns even when 30% pixels are randomly flipped. Our results highlight the potential of optical HNNs in practical applications such as real-time image processing for autonomous driving, enhanced AI with fast memory retrieval, and other scenarios requiring efficient data processing.

Selective learning for sensing using shift-invariant spectrally stable undersampled networks

The amount of data collected for sensing tasks in scientific computing is based on the Shannon-Nyquist sampling theorem proposed in the 1940s. Sensor data generation will surpass 73 trillion GB by 2025 as we increase the high-fidelity digitization of the physical world. Skyrocketing data infrastructure costs and time to maintain and compute on all this data are increasingly common. To address this, we introduce a selective learning approach, where the amount of data collected is problem dependent. We develop novel shift-invariant and spectrally stable neural networks to solve real-time sensing problems formulated as classification or regression problems. We demonstrate that (i) less data can be collected while preserving information, and (ii) test accuracy improves with data augmentation (size of training data), rather than by collecting more than a certain fraction of raw data, unlike information theoretic approaches. While sampling at Nyquist rates, every data point does not have to be resolved at Nyquist and the network learns the amount of data to be collected. This has significant implications (orders of magnitude reduction) on the amount of data collected, computation, power, time, bandwidth, and latency required for several embedded applications ranging from low earth orbit economy to unmanned underwater vehicles.

Neural networks with quantum states of light

Quantum optical networks are instrumental in addressing the fundamental questions and enable applications ranging from communication to computation and, more recently, machine learning (ML). In particular, photonic artificial neural networks (ANNs) offer the opportunity to exploit the advantages of both classical and quantum optics. Photonic neuro-inspired computation and ML have been successfully demonstrated in classical settings, while quantum optical networks have triggered breakthrough applications such as teleportation, quantum key distribution and quantum computing. We present a perspective on the state of the art in quantum optical ML and the potential advantages of ANNs in circuit designs and beyond, in more general, analogue settings characterized by recurrent and coherent complex interactions. We consider two analogue neuro-inspired applications, namely quantum reservoir computing and quantum associative memories, and discuss the enhanced capabilities offered by quantum substrates, highlighting the specific role of light squeezing in this context.This article is part of the theme issue 'The quantum theory of light'.

Pattern memory cannot be completely and truly realized in deep neural networks

The unknown boundary issue, between superior computational capability of deep neural networks (DNNs) and human cognitive ability, has becoming crucial and foundational theoretical problem in AI evolution. Undoubtedly, DNN-empowered AI capability is increasingly surpassing human intelligence in handling general intelligent tasks. However, the absence of DNN's interpretability and recurrent erratic behavior remain incontrovertible facts. Inspired by perceptual characteristics of human vision on optical illusions, we propose a novel working capability analysis framework for DNNs through innovative cognitive response characteristics on visual illusion images, accompanied with fine adjustable sample image construction strategy. Our findings indicate that, although DNNs can infinitely approximate human-provided empirical standards in pattern classification, object detection and semantic segmentation, they are still unable to truly realize independent pattern memorization. All super cognitive abilities of DNNs purely come from their powerful sample classification performance on similar known scenes. Above discovery establishes a new foundation for advancing artificial general intelligence.

Low-power scalable multilayer optoelectronic neural networks enabled with incoherent light

Optical approaches have made great strides towards the goal of high-speed, energy-efficient computing necessary for modern deep learning and AI applications. Read-in and read-out of data, however, limit the overall performance of existing approaches. This study introduces a multilayer optoelectronic computing framework that alternates between optical and optoelectronic layers to implement matrix-vector multiplications and rectified linear functions, respectively. Our framework is designed for real-time, parallelized operations, leveraging 2D arrays of LEDs and photodetectors connected via independent analog electronics. We experimentally demonstrate this approach using a system with a three-layer network with two hidden layers and operate it to recognize images from the MNIST database with a recognition accuracy of 92% and classify classes from a nonlinear spiral data with 86% accuracy. By implementing multiple layers of a deep neural network simultaneously, our approach significantly reduces the number of read-ins and read-outs required and paves the way for scalable optical accelerators requiring ultra low energy.

Training all-mechanical neural networks for task learning through in situ backpropagation

Recent advances unveiled physical neural networks as promising machine learning platforms, offering faster and more energy-efficient information processing. Compared with extensively-studied optical neural networks, the development of mechanical neural networks remains nascent and faces significant challenges, including heavy computational demands and learning with approximate gradients. Here, we introduce the mechanical analogue of in situ backpropagation to enable highly efficient training of mechanical neural networks. We theoretically prove that the exact gradient can be obtained locally, enabling learning through the immediate vicinity, and we experimentally demonstrate this backpropagation to obtain gradient with high precision. With the gradient information, we showcase the successful training of networks in simulations for behavior learning and machine learning tasks, achieving high accuracy in experiments of regression and classification. Furthermore, we present the retrainability of networks involving task-switching and damage, demonstrating the resilience. Our findings, which integrate the theory for training mechanical neural networks and experimental and numerical validations, pave the way for mechanical machine learning hardware and autonomous self-learning material systems.

Photonic diffractive generators through sampling noises from scattering media

Photonic computing, with potentials of high parallelism, low latency and high energy efficiency, have gained progressive interest at the forefront of neural network (NN) accelerators. However, most existing photonic computing accelerators concentrate on discriminative NNs. Large-scale generative photonic computing machines remain largely unexplored, partly due to poor data accessibility, accuracy and hardware feasibility. Here, we harness random light scattering in disordered media as a native noise source and leverage large-scale diffractive optical computing to generate images from above noise, thereby achieving hardware consistency by solely pursuing the spatial parallelism of light. To realize experimental data accessibility, we design two encoding strategies between images and optical noise latent space that effectively solves the training problem. Furthermore, we utilize advanced photonic NN architectures including cascaded and parallel configurations of diffraction layers to enhance the image generation performance. Our results show that the photonic generator is capable of producing clear and meaningful synthesized images across several standard public datasets. As a photonic generative machine, this work makes an important contribution to photonic computing and paves the way for more sophisticated applications such as real world data augmentation and multi modal generation.

Spatial Optical Simulator for Classical Statistical Models

Optical simulators for the Ising model have demonstrated great promise for solving challenging problems in physics and beyond. Here, we develop a spatial optical simulator for a variety of classical statistical systems, including the clock, XY, Potts, and Heisenberg models, utilizing a digital micromirror device composed of a large number of tiny mirrors. Spins, with desired amplitudes or phases of the statistical models, are precisely encoded by a patch of mirrors with a superpixel approach. Then, by modulating the light field in a sequence of designed patterns, the spin-spin interaction is realized in such a way that the Hamiltonian symmetries are preserved. We successfully simulate statistical systems on a fully connected network, with ferromagnetic or Mattis-type random interactions, and observe the corresponding phase transitions between the paramagnetic and the ferromagnetic or spin-glass phases. Our results largely extend the research scope of spatial optical simulators and their versatile applications.

Optoelectronic metadevices

Metasurfaces have introduced new opportunities in photonic design by offering unprecedented, nanoscale control over optical wavefronts. These artificially structured layers have largely been used to passively manipulate the flow of light by controlling its phase, amplitude, and polarization. However, they can also dynamically modulate these quantities and manipulate fundamental light absorption and emission processes. These valuable traits can extend their application domain to chipscale optoelectronics and conceptually new optical sources, displays, spatial light modulators, photodetectors, solar cells, and imaging systems. New opportunities and challenges have also emerged in the materials and device integration with existing technologies. This Review aims to consolidate the current research landscape and provide perspectives on metasurface capabilities specific to optoelectronic devices, giving new direction to future research and development efforts in academia and industry.

Programmable nonlinear optical neuromorphic computing with bare 2D material MoS2

Nonlinear optical responses in two-dimensional (2D) materials can build free-space optical neuromorphic computing systems. Ensuring the high performance and the tunability of the system is essential to encode diverse functions. However, common strategies, including the integration of external electrode arrays or photonic structures with 2D materials, and barely patterned 2D materials, exhibit a contradiction between performance and tunability. Because the unique band dispersions of 2D materials can provide hidden paths to boost nonlinear responses independently, here we introduced a new free-space optical computing concept within a bare molybdenum disulfide array. This system can preserve high modulation performance with fast speed, low energy consumption, and high signal-to-noise ratio. Due to the freedom from the restrictions of fixed photonic structures, the tunability is also enhanced through the synergistic encodings of the 2D cells and the excitation pulses. The computing mechanism of transition from two-photon absorption to synergistic excited states absorption intrinsically improved the modulation capability of nonlinear optical responses, revealed from the relative transmittance modulated by a pump-probe-control strategy. Optical artificial neural network (ANN) and digital processing were demonstrated, revealing the feasibility of the free-space optical computing based on bare 2D materials toward neuromorphic applications.

Miniaturized on-chip optical differentiator based on 2F-structured metasurfaces

Analog optical computing based on Fourier optics has attracted ever-growing attention, offering unprecedented low power consumption and high parallelism computation at the speed of light. Typically, classical optical 4F systems have been widely employed as one of the most common approaches for analog optical computing. However, most existing schemes replicate the original architecture relying on two Fourier transforms and one spatial-frequency filtering, resulting in bulky size and complex structure. Here, we propose a novel, to the best of our knowledge, on-chip 2F structure that achieves ultra-miniaturized optical analog computing. Taking advantage of the exceptional design flexibility of metasurfaces, we reduce the optical path length through a combination of phase compensation and complex amplitude modulation, thereby significantly simplifying the system structure without sacrificing accuracy compared to the traditional 4F system. As a proof-of-concept demonstration, we design and fabricate an on-chip optical differentiator on a silicon-on-insulator platform, achieving 84.01% and 79.81% differentiation accuracy in simulation and experiment, respectively.

Large-scale scattering-augmented optical encryption

Data proliferation in the digital age necessitates robust encryption techniques to protect information privacy. Optical encryption leverages the multiple degrees of freedom inherent in light waves to encode information with parallel processing and enhanced security features. However, implementations of large-scale, high-security optical encryption have largely remained theoretical or limited to digital simulations due to hardware constraints, signal-to-noise ratio challenges, and precision fabrication of encoding elements. Here, we present an optical encryption platform utilizing scattering multiplexing ptychography, simultaneously enhancing security and throughput. Unlike optical encoders which rely on computer-generated randomness, our approach leverages the inherent complexity of light scattering as a natural unclonable function. This enables multi-dimensional encoding with superior randomness. Furthermore, the ptychographic configuration expands encryption throughput beyond hardware limitations through spatial multiplexing of different scatterer regions. We propose a hybrid decryption algorithm integrating model- and data-driven strategies, ensuring robust decryption against various sources of measurement noise and communication interference. We achieved optical encryption at a scale of ten-megapixel pixels with 1.23 µm resolution. Communication experiments validate the resilience of our decryption algorithm, yielding high-fidelity results even under extreme transmission conditions characterized by a 20% bit error rate. Our encryption platform offers a holistic solution for large-scale, high-security, and cost-effective cryptography.

Vaporchromic Domino Transformation and Polarized Photonic Heterojunctions of Organoplatinum Microrods

Photonic heterostructures with codable properties have shown great values as versatile information carriers at the micro- and nanoscale. These heterostructures are typically prepared by a step-by-step growth or post-functionalization method to achieve varied emission colors with different building blocks. In order to realize high-throughput and multivariate information loading, we report here a strategy to integrate polarization signals into photonic heterojunctions. A U-shaped di-Pt(II) complex has been assembled into highly polarized yellow-phosphorescent crystalline microrods (Y-rod) by strong intermolecular Pt⋅⋅⋅Pt interaction. Upon end-initiated desorption of the incorporated CH2Cl2 solvents, the Y-rod is transformed in a domino fashion into tri-block polarized photonic heterojunctions (PPHs) with alternate red-yellow-red emissions or red-phosphorescent microrods (R-rods). The red emissions of these structures are also highly polarized; however, their polarization directions are just orthogonal to those of the yellow phosphorescence of the Y-rod. With the aid of a patterned mask, the R-rod can be further programmed into multi-block PPHs with precisely controlled block sizes by side-allowed adsorption of CH2Cl2 vapor. X-ray diffraction analysis and theoretical calculations suggest that the solvent-regulated modulation of the crystal packing and excited-state property is critical for the construction of these PPHs.

Spatially varying nanophotonic neural networks

The explosive growth in computation and energy cost of artificial intelligence has spurred interest in alternative computing modalities to conventional electronic processors. Photonic processors, which use photons instead of electrons, promise optical neural networks with ultralow latency and power consumption. However, existing optical neural networks, limited by their designs, have not achieved the recognition accuracy of modern electronic neural networks. In this work, we bridge this gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture, before recording on the sensor. We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization. We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era.

Advancements and Challenges in the Integration of Indium Arsenide and Van der Waals Heterostructures

The strategic integration of low-dimensional InAs-based materials and emerging van der Waals systems is advancing in various scientific fields, including electronics, optics, and magnetics. With their unique properties, these InAs-based van der Waals materials and devices promise further miniaturization of semiconductor devices in line with Moore's Law. However, progress in this area lags behind other 2D materials like graphene and boron nitride. Challenges include synthesizing pure crystalline phase InAs nanostructures and single-atomic-layer 2D InAs films, both vital for advanced van der Waals heterostructures. Also, diverse surface state effects on InAs-based van der Waals devices complicate their performance evaluation. This review discusses the experimental advances in the van der Waals epitaxy of InAs-based materials and the working principles of InAs-based van der Waals devices. Theoretical achievements in understanding and guiding the design of InAs-based van der Waals systems are highlighted. Focusing on advancing novel selective area growth and remote epitaxy, exploring multi-functional applications, and incorporating deep learning into first-principles calculations are proposed. These initiatives aim to overcome existing bottlenecks and accelerate transformative advancements in integrating InAs and van der Waals heterostructures.

Ultra-broadband all-optical nonlinear activation function enabled by MoTe2/optical waveguide integrated devices

All-optical nonlinear activation functions (NAFs) are crucial for enabling rapid optical neural networks (ONNs). As linear matrix computation advances in integrated ONNs, on-chip all-optical NAFs face challenges such as limited integration, high latency, substantial power consumption, and a high activation threshold. In this work, we develop an integrated nonlinear optical activator based on the butt-coupling integration of two-dimensional (2D) MoTe2 and optical waveguides (OWGs). The activator exhibits an ultra-broadband response from visible to near-infrared wavelength, a low activation threshold of 0.94 μW, a small device size (~50 µm2), an ultra-fast response rate (2.08 THz), and high-density integration. The excellent nonlinear effects and broadband response of 2D materials have been utilized to create all-optical NAFs. These activators were applied to simulate MNIST handwritten digit recognition, achieving an accuracy of 97.6%. The results underscore the potential application of this approach in ONNs. Moreover, the classification of more intricate CIFAR-10 images demonstrated a generalizable accuracy of 94.6%. The present nonlinear activator promises a general platform for three-dimensional (3D) ultra-broadband ONNs with dense integration and low activation thresholds by integrating a variety of strong nonlinear optical (NLO) materials (e.g., 2D materials) and OWGs in glass.

Deep learning with photonic neural cellular automata

Rapid advancements in deep learning over the past decade have fueled an insatiable demand for efficient and scalable hardware. Photonics offers a promising solution by leveraging the unique properties of light. However, conventional neural network architectures, which typically require dense programmable connections, pose several practical challenges for photonic realizations. To overcome these limitations, we propose and experimentally demonstrate Photonic Neural Cellular Automata (PNCA) for photonic deep learning with sparse connectivity. PNCA harnesses the speed and interconnectivity of photonics, as well as the self-organizing nature of cellular automata through local interactions to achieve robust, reliable, and efficient processing. We utilize linear light interference and parametric nonlinear optics for all-optical computations in a time-multiplexed photonic network to experimentally perform self-organized image classification. We demonstrate binary (two-class) classification of images using as few as 3 programmable photonic parameters, achieving high experimental accuracy with the ability to also recognize out-of-distribution data. The proposed PNCA approach can be adapted to a wide range of existing photonic hardware and provides a compelling alternative to conventional photonic neural networks by maximizing the advantages of light-based computing whilst mitigating their practical challenges. Our results showcase the potential of PNCA in advancing photonic deep learning and highlights a path for next-generation photonic computers.

Single-Shot Multispectral Encoding: Advancing Optical Lithography for Encryption and Spectroscopy

Most modern optical display and sensing devices utilize a limited number of spectral units within the visible range, based on human color perception. In contrast, the rapid advancement of machine-based pattern recognition and spectral analysis could facilitate the use of multispectral functional units, yet the challenge of creating complex, high-definition, and reproducible patterns with an increasing number of spectral units limits their widespread application. Here, we report a technique for optical lithography that employs a single-shot exposure to reproduce perovskite films with spatially controlled optical band gaps through light-induced compositional modulations. Luminescent patterns are designed to program correlations between spatial and spectral information, covering the entire visible spectral range. Using this platform, we demonstrate multispectral encoding patterns for encryption and multivariate optical converters for dispersive optics-free spectroscopy with high spectral resolution. The fabrication process is conducted at room temperature and can be extended to other material and device platforms.

Lanthanide ion-doped upconversion nanoparticles for low-energy super-resolution applications

Energy-intensive technologies and high-precision research require energy-efficient techniques and materials. Lens-based optical microscopy technology is useful for low-energy applications in the life sciences and other fields of technology, but standard techniques cannot achieve applications at the nanoscale because of light diffraction. Far-field super-resolution techniques have broken beyond the light diffraction limit, enabling 3D applications down to the molecular scale and striving to reduce energy use. Typically targeted super-resolution techniques have achieved high resolution, but the high light intensity needed to outperform competing optical transitions in nanomaterials may result in photo-damage and high energy consumption. Great efforts have been made in the development of nanomaterials to improve the resolution and efficiency of these techniques toward low-energy super-resolution applications. Lanthanide ion-doped upconversion nanoparticles that exhibit multiple long-lived excited energy states and emit upconversion luminescence have enabled the development of targeted super-resolution techniques that need low-intensity light. The use of lanthanide ion-doped upconversion nanoparticles in these techniques for emerging low-energy super-resolution applications will have a significant impact on life sciences and other areas of technology. In this review, we describe the dynamics of lanthanide ion-doped upconversion nanoparticles for super-resolution under low-intensity light and their use in targeted super-resolution techniques. We highlight low-energy super-resolution applications of lanthanide ion-doped upconversion nanoparticles, as well as the related research directions and challenges. Our aim is to analyze targeted super-resolution techniques using lanthanide ion-doped upconversion nanoparticles, emphasizing fundamental mechanisms governing transitions in lanthanide ions to surpass the diffraction limit with low-intensity light, and exploring their implications for low-energy nanoscale applications.

Electrically tunable planar liquid-crystal singlets for simultaneous spectrometry and imaging

Conventional hyperspectral cameras cascade lenses and spectrometers to acquire the spectral datacube, which forms the fundamental framework for hyperspectral imaging. However, this cascading framework involves tradeoffs among spectral and imaging performances when the system is driven toward miniaturization. Here, we propose a spectral singlet lens that unifies optical imaging and computational spectrometry functions, enabling the creation of minimalist, miniaturized and high-performance hyperspectral cameras. As a paradigm, we capitalize on planar liquid crystal optics to implement the proposed framework, with each liquid-crystal unit cell acting as both phase modulator and electrically tunable spectral filter. Experiments with various targets show that the resulting millimeter-scale hyperspectral camera exhibits both high spectral fidelity ( > 95%) and high spatial resolutions ( ~1.7 times the diffraction limit). The proposed "two-in-one" framework can resolve the conflicts between spectral and imaging resolutions, which paves a practical pathway for advancing hyperspectral imaging systems toward miniaturization and portable applications.

Photonic probabilistic machine learning using quantum vacuum noise

Probabilistic machine learning utilizes controllable sources of randomness to encode uncertainty and enable statistical modeling. Harnessing the pure randomness of quantum vacuum noise, which stems from fluctuating electromagnetic fields, has shown promise for high speed and energy-efficient stochastic photonic elements. Nevertheless, photonic computing hardware which can control these stochastic elements to program probabilistic machine learning algorithms has been limited. Here, we implement a photonic probabilistic computer consisting of a controllable stochastic photonic element - a photonic probabilistic neuron (PPN). Our PPN is implemented in a bistable optical parametric oscillator (OPO) with vacuum-level injected bias fields. We then program a measurement-and-feedback loop for time-multiplexed PPNs with electronic processors (FPGA or GPU) to solve certain probabilistic machine learning tasks. We showcase probabilistic inference and image generation of MNIST-handwritten digits, which are representative examples of discriminative and generative models. In both implementations, quantum vacuum noise is used as a random seed to encode classification uncertainty or probabilistic generation of samples. In addition, we propose a path towards an all-optical probabilistic computing platform, with an estimated sampling rate of  ~1 Gbps and energy consumption of  ~5 fJ/MAC. Our work paves the way for scalable, ultrafast, and energy-efficient probabilistic machine learning hardware.

Self-healing spiral phase contrast imaging

Spiral phase contrast imaging alleviates the information load by extracting the geometric features of objects and is one of the most representative branches of instant imaging processing. The self-healing capacity of edge detectors can enhance their robustness to obstacles in practical applications. Here, a self-healing spiral phase contrast imaging scheme is proposed and experimentally demonstrated by a liquid crystal edge detector combining a spiral phase, an axicon phase, and a lens phase. The spiral phase is encoded into a liquid crystal by photopatterning. Self-healing contrast imaging is characterized by a series of edge images of both high-contrast amplitude-type and low-contrast phase-type objects. This work extends the self-healing capacity of these detectors to instant imaging processing and paves the way for optical applications with self-healing features.

Thin film ferroelectric photonic-electronic memory

To reduce system complexity and bridge the interface between electronic and photonic circuits, there is a high demand for a non-volatile memory that can be accessed both electrically and optically. However, practical solutions are still lacking when considering the potential for large-scale complementary metal-oxide semiconductor compatible integration. Here, we present an experimental demonstration of a non-volatile photonic-electronic memory based on a 3-dimensional monolithic integrated ferroelectric-silicon ring resonator. We successfully demonstrate programming and erasing the memory using both electrical and optical methods, assisted by optical-to-electrical-to-optical conversion. The memory cell exhibits a high optical extinction ratio of 6.6 dB at a low working voltage of 5 V and an endurance of 4 × 104 cycles. Furthermore, the multi-level storage capability is analyzed in detail, revealing stable performance with a raw bit-error-rate smaller than 5.9 × 10-2. This ground-breaking work could be a key technology enabler for future hybrid electronic-photonic systems, targeting a wide range of applications such as photonic interconnect, high-speed data communication, and neuromorphic computing.

Terahertz spoof plasmonic neural network for diffractive information recognition and processing

All-optical diffractive neural networks, as analog artificial intelligence accelerators, leverage parallelism and analog computation for complex data processing. However, their low space transmission efficiency or large spatial dimensions hinder miniaturization and broader application. Here, we propose a terahertz spoof plasmonic neural network on a planar diffractive platform for direct multi-target recognition. Our approach employs a spoof surface plasmon polariton coupler array to construct a diffractive network layer, resulting in a compact, efficient, and easily integrable architecture. We designed three schemes: basis vector classification, multi-user recognition, and MNIST handwritten digit classification. Experimental results reveal that the terahertz spoof plasmonic neural network successfully classifies basis vectors, recognizes multi-user orientation information, and directly processes handwritten digits using a designed input framework comprising a metal grating array, transmitters, and receivers. This work broadens the application of terahertz plasmonic metamaterials, paving the way for terahertz on-chip integration, intelligent communication, and advanced computing systems.

Curriculum learning for ab initio deep learned refractive optics

Deep optical optimization has recently emerged as a new paradigm for designing computational imaging systems using only the output image as the objective. However, it has been limited to either simple optical systems consisting of a single element such as a diffractive optical element or metalens, or the fine-tuning of compound lenses from good initial designs. Here we present a DeepLens design method based on curriculum learning, which is able to learn optical designs of compound lenses ab initio from randomly initialized surfaces without human intervention, therefore overcoming the need for a good initial design. We demonstrate the effectiveness of our approach by fully automatically designing both classical imaging lenses and a large field-of-view extended depth-of-field computational lens in a cellphone-style form factor, with highly aspheric surfaces and a short back focal length.

Fully forward mode training for optical neural networks

Optical computing promises to improve the speed and energy efficiency of machine learning applications1-6. However, current approaches to efficiently train these models are limited by in silico emulation on digital computers. Here we develop a method called fully forward mode (FFM) learning, which implements the compute-intensive training process on the physical system. The majority of the machine learning operations are thus efficiently conducted in parallel on site, alleviating numerical modelling constraints. In free-space and integrated photonics, we experimentally demonstrate optical systems with state-of-the-art performances for a given network size. FFM learning shows training the deepest optical neural networks with millions of parameters achieves accuracy equivalent to the ideal model. It supports all-optical focusing through scattering media with a resolution of the diffraction limit; it can also image in parallel the objects hidden outside the direct line of sight at over a kilohertz frame rate and can conduct all-optical processing with light intensity as weak as subphoton per pixel (5.40 × 1018- operations-per-second-per-watt energy efficiency) at room temperature. Furthermore, we prove that FFM learning can automatically search non-Hermitian exceptional points without an analytical model. FFM learning not only facilitates orders-of-magnitude-faster learning processes, but can also advance applied and theoretical fields such as deep neural networks, ultrasensitive perception and topological photonics.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1-5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6-8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9-11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson's disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Nonlinear optical encoding enabled by recurrent linear scattering

Optical information processing and computing can potentially offer enhanced performance, scalability and energy efficiency. However, achieving nonlinearity-a critical component of computation-remains challenging in the optical domain. Here we introduce a design that leverages a multiple-scattering cavity to passively induce optical nonlinear random mapping with a continuous-wave laser at a low power. Each scattering event effectively mixes information from different areas of a spatial light modulator, resulting in a highly nonlinear mapping between the input data and output pattern. We demonstrate that our design retains vital information even when the readout dimensionality is reduced, thereby enabling optical data compression. This capability allows our optical platforms to offer efficient optical information processing solutions across applications. We demonstrate our design's efficacy across tasks, including classification, image reconstruction, keypoint detection and object detection, all of which are achieved through optical data compression combined with a digital decoder. In particular, high performance at extreme compression ratios is observed in real-time pedestrian detection. Our findings open pathways for novel algorithms and unconventional architectural designs for optical computing.

Nonlinear processing with linear optics

Deep neural networks have achieved remarkable breakthroughs by leveraging multiple layers of data processing to extract hidden representations, albeit at the cost of large electronic computing power. To enhance energy efficiency and speed, the optical implementation of neural networks aims to harness the advantages of optical bandwidth and the energy efficiency of optical interconnections. In the absence of low-power optical nonlinearities, the challenge in the implementation of multilayer optical networks lies in realizing multiple optical layers without resorting to electronic components. Here we present a novel framework that uses multiple scattering, and which is capable of synthesizing programmable linear and nonlinear transformations concurrently at low optical power by leveraging the nonlinear relationship between the scattering potential, represented by data, and the scattered field. Theoretical and experimental investigations show that repeating the data by multiple scattering enables nonlinear optical computing with low-power continuous-wave light. Moreover, we empirically find that scaling of this optical framework follows a power law.

Pyramid diffractive optical networks for unidirectional image magnification and demagnification

Diffractive deep neural networks (D2NNs) are composed of successive transmissive layers optimized using supervised deep learning to all-optically implement various computational tasks between an input and output field-of-view. Here, we present a pyramid-structured diffractive optical network design (which we term P-D2NN), optimized specifically for unidirectional image magnification and demagnification. In this design, the diffractive layers are pyramidally scaled in alignment with the direction of the image magnification or demagnification. This P-D2NN design creates high-fidelity magnified or demagnified images in only one direction, while inhibiting the image formation in the opposite direction-achieving the desired unidirectional imaging operation using a much smaller number of diffractive degrees of freedom within the optical processor volume. Furthermore, the P-D2NN design maintains its unidirectional image magnification/demagnification functionality across a large band of illumination wavelengths despite being trained with a single wavelength. We also designed a wavelength-multiplexed P-D2NN, where a unidirectional magnifier and a unidirectional demagnifier operate simultaneously in opposite directions, at two distinct illumination wavelengths. Furthermore, we demonstrate that by cascading multiple unidirectional P-D2NN modules, we can achieve higher magnification factors. The efficacy of the P-D2NN architecture was also validated experimentally using terahertz illumination, successfully matching our numerical simulations. P-D2NN offers a physics-inspired strategy for designing task-specific visual processors.

Super-resolution diffractive neural network for all-optical direction of arrival estimation beyond diffraction limits

Wireless sensing of the wave propagation direction from radio sources lays the foundation for communication, radar, navigation, etc. However, the existing signal processing paradigm for the direction of arrival estimation requires the radio frequency electronic circuit to demodulate and sample the multichannel baseband signals followed by a complicated computing process, which places the fundamental limit on its sensing speed and energy efficiency. Here, we propose the super-resolution diffractive neural networks (S-DNN) to process electromagnetic (EM) waves directly for the DOA estimation at the speed of light. The multilayer meta-structures of S-DNN generate super-oscillatory angular responses in local angular regions that can perform the all-optical DOA estimation with angular resolutions beyond the diffraction limit. The spatial-temporal multiplexing of passive and reconfigurable S-DNNs is utilized to achieve high-resolution DOA estimation over a wide field of view. The S-DNN is validated for the DOA estimation of multiple radio sources over 5 GHz frequency bandwidth with estimation latency over two to four orders of magnitude lower than the state-of-the-art commercial devices in principle. The results achieve the angular resolution over an order of magnitude, experimentally demonstrated with four times, higher than diffraction-limited resolution. We also apply S-DNN's edge computing capability, assisted by reconfigurable intelligent surfaces, for extremely low-latency integrated sensing and communication with low power consumption. Our work is a significant step towards utilizing photonic computing processors to facilitate various wireless sensing and communication tasks with advantages in both computing paradigms and performance over electronic computing.

Non-convex optimization for inverse problem solving in computer-generated holography

Computer-generated holography is a promising technique that modulates user-defined wavefronts with digital holograms. Computing appropriate holograms with faithful reconstructions is not only a problem closely related to the fundamental basis of holography but also a long-standing challenge for researchers in general fields of optics. Finding the exact solution of a desired hologram to reconstruct an accurate target object constitutes an ill-posed inverse problem. The general practice of single-diffraction computation for synthesizing holograms can only provide an approximate answer, which is subject to limitations in numerical implementation. Various non-convex optimization algorithms are thus designed to seek an optimal solution by introducing different constraints, frameworks, and initializations. Herein, we overview the optimization algorithms applied to computer-generated holography, incorporating principles of hologram synthesis based on alternative projections and gradient descent methods. This is aimed to provide an underlying basis for optimized hologram generation, as well as insights into the cutting-edge developments of this rapidly evolving field for potential applications in virtual reality, augmented reality, head-up display, data encryption, laser fabrication, and metasurface design.

Nanowatt all-optical 3D perception for mobile robotics

Three-dimensional (3D) perception is vital to drive mobile robotics' progress toward intelligence. However, state-of-the-art 3D perception solutions require complicated postprocessing or point-by-point scanning, suffering computational burden, latency of tens of milliseconds, and additional power consumption. Here, we propose a parallel all-optical computational chipset 3D perception architecture (Aop3D) with nanowatt power and light speed. The 3D perception is executed during the light propagation over the passive chipset, and the captured light intensity distribution provides a direct reflection of the depth map, eliminating the need for extensive postprocessing. The prototype system of Aop3D is tested in various scenarios and deployed to a mobile robot, demonstrating unprecedented performance in distance detection and obstacle avoidance. Moreover, Aop3D works at a frame rate of 600 hertz and a power consumption of 33.3 nanowatts per meta-pixel experimentally. Our work is promising toward next-generation direct 3D perception techniques with light speed and high energy efficiency.

Role of spatial coherence in diffractive optical neural networks

Diffractive optical neural networks (DONNs) have emerged as a promising optical hardware platform for ultra-fast and energy-efficient signal processing for machine learning tasks, particularly in computer vision. Previous experimental demonstrations of DONNs have only been performed using coherent light. However, many real-world DONN applications require consideration of the spatial coherence properties of the optical signals. Here, we study the role of spatial coherence in DONN operation and performance. We propose a numerical approach to efficiently simulate DONNs under incoherent and partially coherent input illumination and discuss the corresponding computational complexity. As a demonstration, we train and evaluate simulated DONNs on the MNIST dataset of handwritten digits to process light with varying spatial coherence.

The Application Potential of Artificial Intelligence and Numerical Simulation in the Research and Formulation Design of Drilling Fluid Gel Performance

Drilling fluid is pivotal for efficient drilling. However, the gelation performance of drilling fluids is influenced by various complex factors, and traditional methods are inefficient and costly. Artificial intelligence and numerical simulation technologies have become transformative tools in various disciplines. This work reviews the application of four artificial intelligence techniques-expert systems, artificial neural networks (ANNs), support vector machines (SVMs), and genetic algorithms-and three numerical simulation techniques-computational fluid dynamics (CFD) simulations, molecular dynamics (MD) simulations, and Monte Carlo simulations-in drilling fluid design and performance optimization. It analyzes the current issues in these studies, pointing out that challenges in applying these two technologies to drilling fluid gelation performance research include difficulties in obtaining field data and overly idealized model assumptions. From the literature review, it can be estimated that 52.0% of the papers are related to ANNs. Leakage issues are the primary concern for practitioners studying drilling fluid gelation performance, accounting for over 17% of research in this area. Based on this, and in conjunction with the technical requirements of drilling fluids and the development needs of drilling intelligence theory, three development directions are proposed: (1) Emphasize feature engineering and data preprocessing to explore the application potential of interpretable artificial intelligence. (2) Establish channels for open access to data or large-scale oil and gas field databases. (3) Conduct in-depth numerical simulation research focusing on the microscopic details of the spatial network structure of drilling fluids, reducing or even eliminating data dependence.

Advancements in fluorescence lifetime imaging microscopy Instrumentation: Towards high speed and 3D

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging tool offering molecular specific insights into samples through the measurement of fluorescence decay time, with promising applications in diverse research fields. However, to acquire two-dimensional lifetime images, conventional FLIM relies on extensive scanning in both the spatial and temporal domain, resulting in much slower acquisition rates compared to intensity-based approaches. This problem is further magnified in three-dimensional imaging, as it necessitates additional scanning along the depth axis. Recent advancements have aimed to enhance the speed and three-dimensional imaging capabilities of FLIM. This review explores the progress made in addressing these challenges and discusses potential directions for future developments in FLIM instrumentation.