Interactive 3D reconstruction of the spine from radiographs using a statistical shape model and second-order cone programming

Automatic 3D reconstruction of vertebrae from orthogonal bi-planar radiographs

When conducting spine-related diagnosis and surgery, the three-dimensional (3D) upright posture of the spine under natural weight bearing is of significant clinical value for physicians to analyze the force on the spine. However, existing medical imaging technologies cannot meet current requirements of medical service. On the one hand, the mainstream 3D volumetric imaging modalities (e.g. CT and MRI) require patients to lie down during the imaging process. On the other hand, the imaging modalities conducted in an upright posture (e.g. radiograph) can only realize 2D projections, which lose the valid information of spinal anatomy and curvature. Developments of deep learning-based 3D reconstruction methods bring potential to overcome the limitations of the existing medical imaging technologies. To deal with the limitations of current medical imaging technologies as is described above, in this paper, we propose a novel deep learning framework, ReVerteR, which can realize automatic 3D Reconstruction of Vertebrae from orthogonal bi-planar Radiographs. With the utilization of self-attention mechanism and specially designed loss function combining Dice, Hausdorff, Focal, and MSE, ReVerteR can alleviate the sample-imbalance problem during the reconstruction process and realize the fusion of the centroid annotation and the focused vertebra. Furthermore, aiming at automatic and customized 3D spinal reconstruction in real-world scenarios, we extend ReVerteR to a clinical deployment-oriented framework, and develop an interactive interface with all functions in the framework integrated so as to enhance human-computer interaction during clinical decision-making. Extensive experiments and visualization conducted on our constructed datasets based on two benchmark datasets of spinal CT, VerSe 2019 and VerSe 2020, demonstrate the effectiveness of our proposed ReVerteR. In this paper, we propose an automatic 3D reconstruction method of vertebrae based on orthogonal bi-planar radiographs. With the 3D upright posture of the spine under natural weight bearing effectively constructed, our proposed method is expected to better support doctors make clinical decision during spine-related diagnosis and surgery.

Survey of methods and principles in three-dimensional reconstruction from two-dimensional medical images

Three-dimensional (3D) reconstruction of human organs has gained attention in recent years due to advances in the Internet and graphics processing units. In the coming years, most patient care will shift toward this new paradigm. However, development of fast and accurate 3D models from medical images or a set of medical scans remains a daunting task due to the number of pre-processing steps involved, most of which are dependent on human expertise. In this review, a survey of pre-processing steps was conducted, and reconstruction techniques for several organs in medical diagnosis were studied. Various methods and principles related to 3D reconstruction were highlighted. The usefulness of 3D reconstruction of organs in medical diagnosis was also highlighted.

Region-Based Convolutional Neural Network-Based Spine Model Positioning of X-Ray Images

Background

Idiopathic scoliosis accounts for over 80% of all cases of scoliosis but has an unclear pathogenic mechanism. Many studies have introduced conventional image processing methods, but the results often fail to meet expectations. With the improvement and evolution of research in neural networks in the field of deep learning, many research efforts related to spinal reconstruction using the convolutional neural network (CNN) architecture of deep learning have shown promise.

Purpose

To investigate the use of CNN for spine modeling.

Methods

The primary technique used in this study involves Mask Region-based CNN (R-CNN) image segmentation and object detection methods as applied to spine model positioning of radiographs. The methods were evaluated based on common evaluation criteria for vertebral segmentation and object detection. Evaluations were performed using the loss function, mask loss function, classification loss function, target box loss function, average accuracy, and average recall.

Results

Many bony structures were directly identified in one step, including the lumbar spine (L1-L5) and thoracic spine (T1-T12) in frontal and lateral radiographs, thereby achieving initial positioning of the statistical spine model to provide spine model positioning for future reconstruction and classification prediction. An average detection box accuracy of 97.4% and an average segmentation accuracy of 96.8% were achieved for the prediction efficacy of frontal images, with good image visualization. Moreover, the results for lateral images were satisfactory considering the evaluation parameters and image visualization.

Conclusion

Mask R-CNN can be used for effective positioning in spine model studies for future reconstruction and classification prediction.

Toward Automated 3D Spine Reconstruction from Biplanar Radiographs Using CNN for Statistical Spine Model Fitting

To date, 3D spine reconstruction from biplanar radiographs involves intensive user supervision and semi-automated methods that are time-consuming and not effective in clinical routine. This paper proposes a new, fast, and automated 3D spine reconstruction method through which a realistic statistical shape model of the spine is fitted to images using convolutional neural networks (CNN). The CNNs automatically detect the anatomical landmarks controlling the spine model deformation through a hierarchical and gradual iterative process. The performance assessment used a set of 68 biplanar radiographs, composed of both asymptomatic subjects and adolescent idiopathic scoliosis patients, in order to compare automated reconstructions with ground truths build using multiple experts-supervised reconstructions. The mean (SD) errors of landmark locations (3D Euclidean distances) were 1.6 (1.3) mm, 1.8 (1.3) mm, and 2.3 (1.4) mm for the vertebral body center, endplate centers, and pedicle centers, respectively. The clinical parameters extracted from the automated 3D reconstruction (reconstruction time is less than one minute) presented an absolute mean error between 2.8° and 4.7° for the main spinal parameters and between 1° and 2.1° for pelvic parameters. Automated and expert's agreement analysis reported that, on average, 89% of automated measurements were inside the expert's confidence intervals. The proposed automated 3D spine reconstruction method provides an important step that should help the dissemination and adoption of 3D measurements in clinical routine.

Postoperative 3D spine reconstruction by navigating partitioning manifolds

PURPOSE

The postoperative evaluation of scoliosis patients undergoing corrective treatment is an important task to assess the strategy of the spinal surgery. Using accurate 3D geometric models of the patient's spine is essential to measure longitudinal changes in the patient's anatomy. On the other hand, reconstructing the spine in 3D from postoperative radiographs is a challenging problem due to the presence of instrumentation (metallic rods and screws) occluding vertebrae on the spine.

METHODS

This paper describes the reconstruction problem by searching for the optimal model within a manifold space of articulated spines learned from a training dataset of pathological cases who underwent surgery. The manifold structure is implemented based on a multilevel manifold ensemble to structure the data, incorporating connections between nodes within a single manifold, in addition to connections between different multilevel manifolds, representing subregions with similar characteristics.

RESULTS

The reconstruction pipeline was evaluated on x-ray datasets from both preoperative patients and patients with spinal surgery. By comparing the method to ground-truth models, a 3D reconstruction accuracy of 2.24 ± 0.90 mm was obtained from 30 postoperative scoliotic patients, while handling patients with highly deformed spines.

CONCLUSIONS

This paper illustrates how this manifold model can accurately identify similar spine models by navigating in the low-dimensional space, as well as computing nonlinear charts within local neighborhoods of the embedded space during the testing phase. This technique allows postoperative follow-ups of spinal surgery using personalized 3D spine models and assess surgical strategies for spinal deformities.

3D spine reconstruction of postoperative patients from multi-level manifold ensembles

The quantitative assessment of surgical outcomes using personalized anatomical models is an essential task for the treatment of spinal deformities such as adolescent idiopathic scoliosis. However an accurate 3D reconstruction of the spine from postoperative X-ray images remains challenging due to presence of instrumentation (metallic rods and screws) occluding vertebrae on the spine. In this paper, we formulate the reconstruction problem as an optimization over a manifold of articulated spine shapes learned from pathological training data. The manifold itself is represented using a novel data structure, a multi-level manifold ensemble, which contains links between nodes in a single hierarchical structure, as well as links between different hierarchies, representing overlapping partitions. We show that this data structure allows both efficient localization and navigation on the manifold, for on-the-fly building of local nonlinear models (manifold charting). Our reconstruction framework was tested on pre- and postoperative X-ray datasets from patients who underwent spinal surgery. Compared to manual ground-truth, our method achieves a 3D reconstruction accuracy of 2.37 +/- 0.85 mm for postoperative spine models and can deal with severe cases of scoliosis.

Three-dimensional spine model reconstruction using one-class SVM regularization

Statistical shape models have become essential for medical image registration or segmentation and are used in many biomedical applications. These models are often based on Gaussian distributions learned from a training set. We propose in this paper a shape model which does not rely on the estimation of a Gaussian distribution, but on similarities computed with a kernel function. Our model takes advantage of the one-class support vector machine (OCSVM) to do so. In this context, we propose in this paper a method for reconstructing the spine of scoliotic patients using OCSVM regularization. Current state-of-the-art methods use conventional statistical shape models, and the reconstruction is commonly processed by minimizing a Mahalanobis distance. Nevertheless, when a shape differs significantly from the statistical model, the associated Mahalanobis distance often overstates the need for statistical regularization. We show that OCSVM regularization is more robust and is less sensitive to weak landmarks definition and is hardly influenced by the presence of outliers in the training data. The proposed OCSVM model applied to 3-D spine reconstruction was evaluated on real patient data, and results showed that our approach allows precise reconstruction.