Repurposing traditional U-Net predictions for sparse SAM prompting in medical image segmentation

Objective:Automated medical image segmentation (MIS) using deep learning has traditionally relied on models built and trained from scratch, or at least fine-tuned on a target dataset. The Segment Anything Model (SAM) by Meta challenges this paradigm by providing zero-shot generalisation capabilities. This study aims to develop and compare methods for refining traditional U-Net segmentations by repurposing them for automated SAM prompting.Approach:A 2D U-Net with EfficientNet-B4 encoder was trained using 4-fold cross-validation on an in-house brain metastases dataset. Segmentation predictions from each validation set were used for automatic sparse prompt generation via a bounding box prompting method (BBPM) and novel implementations of the point prompting method (PPM). The PPMs frequently produced poor slice predictions (PSPs) that required identification and substitution. A slice was identified as a PSP if it (1) contained multiple predicted regions per lesion or (2) possessed outlier foreground pixel counts relative to the patient's other slices. Each PSP was substituted with a corresponding initial U-Net or SAM BBPM prediction. The patients' mean volumetric dice similarity coefficient (DSC) was used to evaluate and compare the methods' performances.Main results:Relative to the initial U-Net segmentations, the BBPM improved mean patient DSC by 3.93 ± 1.48% to 0.847 ± 0.008 DSC. PSPs constituted 20.01-21.63% of PPMs' predictions and without substitution performance dropped by 82.94 ± 3.17% to 0.139 ± 0.023 DSC. Pairing the two PSP identification techniques yielded a sensitivity to PSPs of 92.95 ± 1.20%. By combining this approach with BBPM prediction substitution, the PPMs achieved segmentation accuracies on par with the BBPM, improving mean patient DSC by up to 4.17 ± 1.40% and reaching 0.849 ± 0.007 DSC.Significance:The proposed PSP identification and substitution techniques bridge the gap between PPM and BBPM performance for MIS. Additionally, the uniformity observed in our experiments' results demonstrates the robustness of SAM to variations in prompting style. These findings can assist in the design of both automatically and manually prompted pipelines.

Application of the concept of biologically effective dose (BED) to patients with Vestibular Schwannomas treated by radiosurgery

In the application of stereotactic radiosurgery, using the Gamma Knife, there are large variations in the overall treatment time for the same prescription dose, given in a single treatment session, for different patients. This is due to not only changes in the activity of the Cobolt-60 sources, but also to variations in the number of iso-centers used, the collimator size for a particular iso-center, and the time gap between the different iso-centers. Although frequently viewed as a single dose treatment the concept of biologically effective dose (BED), incorporating concurrent fast and a slow components of repair of sublethal damage, would imply potential variations in BED because of the influence of these different variables associated with treatment. This was investigated in 26 patients, treated for Vestibular Schwannomas, using the Series B Gamma-Knife, between 1999 and 2005. The iso-center number varied between 2 and 13, and the overall treatment time from 25.4-129.58 min. The prescription doses varied from 10-14 Gy. To obtain physical dose and dose-rates from each iso-center, in a number of locations in the region of interest, a prototype version of the Leksell GammaPlan^® was used. For an individual patient, BED values varied by up to 15% for a given physical iso-dose. This was due to variation in the dose prescription at different locations on that iso-dose. Between patients there was a decline in the range of BED values as the overall treatment time increased. This increased treatment time was partly a function of the slow decline in the activity of the sources with time but predominantly due to changes in the number of iso-centers used. Thus, variations in BED values did not correlate with prescription dose but was modified by the overall treatment time.

An orientation anisotropy in the effects of scaling vertical disparities

Gårding et al. (Vis Res 1995;35:703-722) proposed a two-stage theory of stereopsis. The first uses horizontal disparities for relief computations after they have been subjected to a process called disparity correction that utilises vertical disparities. The second stage, termed disparity normalisation, is concerned with computing metric representations from the output of stage one. It uses vertical disparities to a much lesser extent, if at all, for small field stimuli. We report two psychophysical experiments that tested whether human vision implements this two-stage theory. They tested the prediction that scaling vertical disparities to simulate different viewing distances to the fixation point should affect the perceived amplitudes of vertically but not horizontally oriented ridges. The first used elliptical half-cylinders and the 'apparently circular cylinder' judgement task of Johnston (Vis Res 1991;31:1351-1360). The second experiment used parabolic ridges and the amplitude judgement task of Buckley and Frisby (Vis Res 1993;33:919-934). Both studies broadly confirmed the anisotropy prediction by finding that large scalings of vertical disparities simulating near distances had a strong effect on the perceived amplitudes of the vertically oriented stimuli but little effect on the horizontal ones. When distances > 25 cm were simulated there were no significant differential effects and various methodological reasons are offered for this departure from expectations.

Pooling of vertical disparities by the human visual system

Two experiments are described in which the effects of scaling vertical disparities on the perceived amplitudes of dome-shaped surfaces depicted with horizontal disparities were examined. The Mayhew and Longuet-Higgins's theory and the regional-disparity-correction theory of Garding et al predict that scaling should generate a change in perceived depth appropriate to the viewing distance simulated by the scaled vertical disparities. Significant depth changes were observed, by means of a nulling task in which the vertical-disparity-scaling effect was cancelled by the observer choosing a pattern of horizontal disparities that made the dome-shaped surface appear flat. The sizes of the scaling effects were less than those predicted by either theory, suggesting that other cues to fixation distance such as oculomotor information played an appreciable role. In conditions in which 50% of the texture elements were given one value of vertical-disparity scaling and the remaining 50% were left unscaled, the size of the scaling effect on perceived depth could be accounted for by equally weighted pooling of the vertical-disparity information unless the two scalings were very dissimilar, in which case the lower scaling factor tended to dominate. These findings are discussed in terms of a Hough parameter estimation model of the vertical-disparity-pooling process.

Interaction of stereo and texture cues in the perception of three-dimensional steps

A computational method for calibrating stereo using shape-from-texture is described together with five experiments that tested whether the human visual system implements the method. The experiments all tested the prediction that the perceived size of a step between two planar and slanted real surfaces should be affected by texture slant cues projected on to them that are inconsistent with the disparity cues. The predicted effect was observed but the results could be accounted for by a new phenomenon revealed in control conditions: the perceived size of a step between two slanted planes is in part determined by the size of the slants even when texture and stereo cues are held consistent. We conclude that the hypothesis that human stereo is calibrated by texture is not confirmed.

Stereopsis, vertical disparity and relief transformations

The pattern of retinal binocular disparities acquired by a fixating visual system depends on both the depth structure of the scene and the viewing geometry. This paper treats the problem of interpreting the disparity pattern in terms of scene structure without relying on estimates of fixation position from eye movement control and proprioception mechanisms. We propose a sequential decomposition of this interpretation process into disparity correction, which is used to compute three-dimensional structure up to a relief transformation, and disparity normalization, which is used to resolve the relief ambiguity to obtain metric structure. We point out that the disparity normalization stage can often be omitted, since relief transformations preserve important properties such as depth ordering and coplanarity. Based on this framework we analyse three previously proposed computational models of disparity processing; the Mayhew and Longuet-Higgins model, the deformation model and the polar angle disparity model. We show how these models are related, and argue that none of them can account satisfactorily for available psychophysical data. We therefore propose an alternative model, regional disparity correction. Using this model we derive predictions for a number of experiments based on vertical disparity manipulations, and compare them to available experimental data. The paper is concluded with a summary and a discussion of the possible architectures and mechanisms underling stereopsis in the human visual system.