1
|
Song P, Jadan HV, Howe CL, Foust AJ, Dragotti PL. Model-Based Explainable Deep Learning for Light-Field Microscopy Imaging. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:3059-3074. [PMID: 38656840 PMCID: PMC11100862 DOI: 10.1109/tip.2024.3387297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Revised: 01/27/2024] [Accepted: 03/12/2024] [Indexed: 04/26/2024]
Abstract
In modern neuroscience, observing the dynamics of large populations of neurons is a critical step of understanding how networks of neurons process information. Light-field microscopy (LFM) has emerged as a type of scanless, high-speed, three-dimensional (3D) imaging tool, particularly attractive for this purpose. Imaging neuronal activity using LFM calls for the development of novel computational approaches that fully exploit domain knowledge embedded in physics and optics models, as well as enabling high interpretability and transparency. To this end, we propose a model-based explainable deep learning approach for LFM. Different from purely data-driven methods, the proposed approach integrates wave-optics theory, sparse representation and non-linear optimization with the artificial neural network. In particular, the architecture of the proposed neural network is designed following precise signal and optimization models. Moreover, the network's parameters are learned from a training dataset using a novel training strategy that integrates layer-wise training with tailored knowledge distillation. Such design allows the network to take advantage of domain knowledge and learned new features. It combines the benefit of both model-based and learning-based methods, thereby contributing to superior interpretability, transparency and performance. By evaluating on both structural and functional LFM data obtained from scattering mammalian brain tissues, we demonstrate the capabilities of the proposed approach to achieve fast, robust 3D localization of neuron sources and accurate neural activity identification.
Collapse
Affiliation(s)
- Pingfan Song
- Department of EngineeringUniversity of CambridgeCB2 1PZCambridgeU.K
| | - Herman Verinaz Jadan
- Faculty of Electrical and Computer EngineeringEscuela Superior Politécnica del Litoral (ESPOL)GuayaquilEC090903Ecuador
| | - Carmel L. Howe
- Department of Chemical Physiology and BiochemistryOregon Health and Science UniversityPortlandOR97239USA
| | - Amanda J. Foust
- Center for NeurotechnologyDepartment of BioengineeringImperial College LondonSW7 2AZLondonU.K
| | - Pier Luigi Dragotti
- Department of Electronic and Electrical EngineeringImperial College LondonSW7 2AZLondonUK
| |
Collapse
|
2
|
Song P, Jadan HV, Howe CL, Quicke P, Foust AJ, Dragotti PL. 3D Localization for Light-Field Microscopy via Convolutional Sparse Coding on Epipolar Images. IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING 2020; 6:1017-1032. [PMID: 32851121 PMCID: PMC7442043 DOI: 10.1109/tci.2020.2997301] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/09/2020] [Revised: 04/16/2020] [Accepted: 05/18/2020] [Indexed: 05/27/2023]
Abstract
Light-field microscopy (LFM) is a type of all-optical imaging system that is able to capture 4D geometric information of light rays and can reconstruct a 3D model from a single snapshot. In this paper, we propose a new 3D localization approach to effectively detect 3D positions of neuronal cells from a single light-field image with high accuracy and outstanding robustness to light scattering. This is achieved by constructing a depth-aware dictionary and by combining it with convolutional sparse coding. Specifically, our approach includes 3 key parts: light-field calibration, depth-aware dictionary construction, and localization based on convolutional sparse coding (CSC). In the first part, an observed raw light-field image is calibrated and then decoded into a two-plane parameterized 4D format which leads to the epi-polar plane image (EPI). The second part involves simulating a set of light-fields using a wave-optics forward model for a ball-shaped volume that is located at different depths. Then, a depth-aware dictionary is constructed where each element is a synthetic EPI associated to a specific depth. Finally, by taking full advantage of the sparsity prior and shift-invariance property of EPI, 3D localization is achieved via convolutional sparse coding on an observed EPI with respect to the depth-aware EPI dictionary. We evaluate our approach on both non-scattering specimen (fluorescent beads suspended in agarose gel) and scattering media (brain tissues of genetically encoded mice). Extensive experiments demonstrate that our approach can reliably detect the 3D positions of granular targets with small Root Mean Square Error (RMSE), high robustness to optical aberration and light scattering in mammalian brain tissues.
Collapse
Affiliation(s)
- Pingfan Song
- Department of Electronic & Electrical
EngineeringImperial College LondonLondonSW7 2AZU.K.
| | - Herman Verinaz Jadan
- Department of Electronic & Electrical
EngineeringImperial College LondonLondonSW7 2AZU.K.
| | - Carmel L. Howe
- Department of Bioengineering, and Center
for NeurotechnologyImperial College LondonLondonSW7 2AZU.K.
| | - Peter Quicke
- Department of Bioengineering, and Center
for NeurotechnologyImperial College LondonLondonSW7 2AZU.K.
| | - Amanda J. Foust
- Department of Bioengineering, and Center
for NeurotechnologyImperial College LondonLondonSW7 2AZU.K.
| | - Pier Luigi Dragotti
- Department of Electronic & Electrical
EngineeringImperial College LondonLondonSW7 2AZU.K.
| |
Collapse
|
3
|
Gil G, Savino G, Piantini S, Pierini M. Motorcycle That See: Multifocal Stereo Vision Sensor for Advanced Safety Systems in Tilting Vehicles. SENSORS 2018; 18:s18010295. [PMID: 29351267 PMCID: PMC5795592 DOI: 10.3390/s18010295] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/31/2017] [Revised: 01/17/2018] [Accepted: 01/17/2018] [Indexed: 11/16/2022]
Abstract
Advanced driver assistance systems, ADAS, have shown the possibility to anticipate crash accidents and effectively assist road users in critical traffic situations. This is not the case for motorcyclists, in fact ADAS for motorcycles are still barely developed. Our aim was to study a camera-based sensor for the application of preventive safety in tilting vehicles. We identified two road conflict situations for which automotive remote sensors installed in a tilting vehicle are likely to fail in the identification of critical obstacles. Accordingly, we set two experiments conducted in real traffic conditions to test our stereo vision sensor. Our promising results support the application of this type of sensors for advanced motorcycle safety applications.
Collapse
Affiliation(s)
- Gustavo Gil
- Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze, Santa Marta 3, 50139 Firenze, Italy.
| | - Giovanni Savino
- Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze, Santa Marta 3, 50139 Firenze, Italy.
- Accident Research Centre, Monash University, Melbourne, 21 Alliance Lane, Clayton, VIC 3800, Australia.
| | - Simone Piantini
- Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze, Santa Marta 3, 50139 Firenze, Italy.
| | - Marco Pierini
- Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze, Santa Marta 3, 50139 Firenze, Italy.
| |
Collapse
|
4
|
Terabayashi K, Morita T, Okamoto H, Umeda K. 3D Measurement Using a Fish-Eye Camera Based on EPI Analysis. JOURNAL OF ROBOTICS AND MECHATRONICS 2012. [DOI: 10.20965/jrm.2012.p0677] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
In car driving support systems and mobile robots, it is important to understand three-dimensional environment widely at once. In this paper, we use a fish-eye camera as a sensor to measure three-dimensional (3D) environments. This camera can take a wide-range and distortional image and can be easily mounted on cars. We propose a method for reconstructing 3D environment using fish-eye images based on Epipolar-Plane Image (EPI) analysis. This method enables easy and stable matching of feature points. The effectiveness of the proposed method is verified by experiments.
Collapse
|
5
|
Yang Ran, Qinfen Zheng, Chellappa R, Strat TM. Applications of a Simple Characterization of Human Gait in Surveillance. ACTA ACUST UNITED AC 2010; 40:1009-20. [DOI: 10.1109/tsmcb.2010.2044173] [Citation(s) in RCA: 39] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
6
|
Min C, Medioni G. Inferring segmented dense motion layers using 5D tensor voting. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2008; 30:1589-1602. [PMID: 18617717 DOI: 10.1109/tpami.2007.70802] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
We present a novel local spatiotemporal approach to produce motion segmentation and dense temporal trajectories from an image sequence. A common representation of image sequences is a 3D spatiotemporal volume, (x,y,t), and its corresponding mathematical formalism is the fiber bundle. However, directly enforcing the spatiotemporal smoothness constraint is difficult in the fiber bundle representation. Thus, we convert the representation into a new 5D space (x,y,t,vx,vy) with an additional velocity domain, where each moving object produces a separate 3D smooth layer. The smoothness constraint is now enforced by extracting 3D layers using the tensor voting framework in a single step that solves both correspondence and segmentation simultaneously. Motion segmentation is achieved by identifying those layers, and the dense temporal trajectories are obtained by converting the layers back into the fiber bundle representation. We proceed to address three applications (tracking, mosaic, and 3D reconstruction) that are hard to solve from the video stream directly because of the segmentation and dense matching steps, but become straightforward with our framework. The approach does not make restrictive assumptions about the observed scene or camera motion and is therefore generally applicable. We present results on a number of data sets.
Collapse
|
7
|
Liu S, Kang K, Tarel JP, Cooper DB. Free-form object reconstruction from silhouettes, occluding edges and texture edges: a unified and robust operator based on duality. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2008; 30:131-146. [PMID: 18000330 DOI: 10.1109/tpami.2007.1143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
In this paper, the duality in differential form is developed between a 3D primal surface and its dual manifold formed by the surface's tangent planes, i.e., each tangent plane of the primal surface is represented as a four-dimensional vector which constitutes a point on the dual manifold. The iterated dual theorem shows that each tangent plane of the dual manifold corresponds to a point on the original 3D surface, i.e., the dual of the dual goes back to the primal. This theorem can be directly used to reconstruct 3D surface from image edges by estimating the dual manifold from these edges. In this paper we further develop the work in our original conference papers resulting in the robust differential dual operator. We argue that the operator makes good use of the information available in the image data, by using both points of intensity discontinuity and their edge directions; we provide a simple physical interpretation of what the abstract algorithm is actually estimating and why it makes sense in terms of estimation accuracy; our algorithm operates on all edges in the images, including silhouette edges, self occlusion edges, and texture edges, without distinguishing their types (thus resulting in improved accuracy and handling locally concave surface estimation if texture edges are present); the algorithm automatically handles various degeneracies; and the algorithm incorporates new methodologies for implementing the required operations such as appropriately relating edges in pairs of images, evaluating and using the algorithm's sensitivity to noise to determine the accuracy of an estimated 3D point. Experiments with both synthetic and real images demonstrate that the operator is accurate, robust to degeneracies and noise, and general for reconstructing free-form objects from occluding edges and texture edges detected in calibrated images or video sequences.
Collapse
Affiliation(s)
- Shubao Liu
- Division of Engineering, Brown University, Providence, RI 02912, USA.
| | | | | | | |
Collapse
|
8
|
On the motion of 3D curves and its relationship to optical flow. ACTA ACUST UNITED AC 2005. [DOI: 10.1007/bfb0014856] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register]
|
9
|
On the Motion and Appearance of Specularities in Image Sequences. COMPUTER VISION — ECCV 2002 2002. [DOI: 10.1007/3-540-47969-4_34] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/02/2023]
|
10
|
|
11
|
Computation and analysis of image motion: A synopsis of current problems and methods. Int J Comput Vis 1996. [DOI: 10.1007/bf00131147] [Citation(s) in RCA: 71] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
12
|
|
13
|
Horridge GA. Ratios of template responses as the basis of semivision. Philos Trans R Soc Lond B Biol Sci 1991; 331:189-97. [PMID: 1674153 DOI: 10.1098/rstb.1991.0007] [Citation(s) in RCA: 17] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022] Open
Abstract
The template model starts with a layer of receptors that in the case of vision are leaky detectors or counters of photons. In many animals, the ratio of the responses of a few spectral types is the basis of colour vision irrespective of intensity. Ratios of template responses are now introduced as the basis of form discrimination. In insects, the second-order neurons on the visual pathway appear to detect temporal contrast at the spatial resolution of the retina. At the next level, in the optic medulla, we find a large number of small local neurons in a column on each visual axis. The template theory is a hypothesis about how the above system functions. All possible combinations of positive, indeterminate or negative temporal contrast are considered, at two adjacent visual axes at two successive instants, giving 81 possible local templates. These templates are therefore phasic detectors of all the possible spatiotemporal contrast combinations. Some of the template responses indicate polarity of edge, flicker, or direction of motion and other abstracted features of the stimulus pattern with the maximum spatial and temporal resolution. The ratios of numbers of template responses, in higher fields at a higher level, yield quantitative measures of the qualities of edges independently of the number of edges, but taking ratios causes a corresponding loss of the spatiotemporal resolution and the pattern within each field. Templates respond to transients without computation, are readily modified or selected in evolution and can be simulated in artificial vision.
Collapse
Affiliation(s)
- G A Horridge
- Centre for Visual Sciences, Research School of Biological Sciences, Australian National University, Canberra
| |
Collapse
|
14
|
|
15
|
|