Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology

Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.

On the generalizability of diffusion MRI signal representations across acquisition parameters, sequences and tissue types: Chronicles of the MEMENTO challenge

Diffusion MRI (dMRI) has become an invaluable tool to assess the microstructural organization of brain tissue. Depending on the specific acquisition settings, the dMRI signal encodes specific properties of the underlying diffusion process. In the last two decades, several signal representations have been proposed to fit the dMRI signal and decode such properties. Most methods, however, are tested and developed on a limited amount of data, and their applicability to other acquisition schemes remains unknown. With this work, we aimed to shed light on the generalizability of existing dMRI signal representations to different diffusion encoding parameters and brain tissue types. To this end, we organized a community challenge - named MEMENTO, making available the same datasets for fair comparisons across algorithms and techniques. We considered two state-of-the-art diffusion datasets, including single-diffusion-encoding (SDE) spin-echo data from a human brain with over 3820 unique diffusion weightings (the MASSIVE dataset), and double (oscillating) diffusion encoding data (DDE/DODE) of a mouse brain including over 2520 unique data points. A subset of the data sampled in 5 different voxels was openly distributed, and the challenge participants were asked to predict the remaining part of the data. After one year, eight participant teams submitted a total of 80 signal fits. For each submission, we evaluated the mean squared error, the variance of the prediction error and the Bayesian information criteria. The received submissions predicted either multi-shell SDE data (37%) or DODE data (22%), followed by cartesian SDE data (19%) and DDE (18%). Most submissions predicted the signals measured with SDE remarkably well, with the exception of low and very strong diffusion weightings. The prediction of DDE and DODE data seemed more challenging, likely because none of the submissions explicitly accounted for diffusion time and frequency. Next to the choice of the model, decisions on fit procedure and hyperparameters play a major role in the prediction performance, highlighting the importance of optimizing and reporting such choices. This work is a community effort to highlight strength and limitations of the field at representing dMRI acquired with trending encoding schemes, gaining insights into how different models generalize to different tissue types and fiber configurations over a large range of diffusion encodings.

Sparse Reconstruction Challenge for diffusion MRI: Validation on a physical phantom to determine which acquisition scheme and analysis method to use?

Diffusion magnetic resonance imaging (dMRI) is the modality of choice for investigating in-vivo white matter connectivity and neural tissue architecture of the brain. The diffusion-weighted signal in dMRI reflects the diffusivity of water molecules in brain tissue and can be utilized to produce image-based biomarkers for clinical research. Due to the constraints on scanning time, a limited number of measurements can be acquired within a clinically feasible scan time. In order to reconstruct the dMRI signal from a discrete set of measurements, a large number of algorithms have been proposed in recent years in conjunction with varying sampling schemes, i.e., with varying b-values and gradient directions. Thus, it is imperative to compare the performance of these reconstruction methods on a single data set to provide appropriate guidelines to neuroscientists on making an informed decision while designing their acquisition protocols. For this purpose, the SPArse Reconstruction Challenge (SPARC) was held along with the workshop on Computational Diffusion MRI (at MICCAI 2014) to validate the performance of multiple reconstruction methods using data acquired from a physical phantom. A total of 16 reconstruction algorithms (9 teams) participated in this community challenge. The goal was to reconstruct single b-value and/or multiple b-value data from a sparse set of measurements. In particular, the aim was to determine an appropriate acquisition protocol (in terms of the number of measurements, b-values) and the analysis method to use for a neuroimaging study. The challenge did not delve on the accuracy of these methods in estimating model specific measures such as fractional anisotropy (FA) or mean diffusivity, but on the accuracy of these methods to fit the data. This paper presents several quantitative results pertaining to each reconstruction algorithm. The conclusions in this paper provide a valuable guideline for choosing a suitable algorithm and the corresponding data-sampling scheme for clinical neuroscience applications.

A Unifying Framework for Spatial and Temporal Diffusion in Diffusion MRI

We propose a novel framework to simultaneously represent the diffusion-weighted MRI (dMRI) signal over diffusion times, gradient strengths and gradient directions. Current frameworks such as the 3D Simple Harmonic Oscillator Reconstruction and Estimation basis (3D-SHORE) only represent the signal over the spatial domain, leaving the temporal dependency as a fixed parameter. However, microstructure-focused techniques such as Axcaliber and ActiveAx provide evidence of the importance of sampling the dMRI space over .diffusion time. Up to now there exists no generalized framework that simultaneously models the dependence of the dMRI signal in space and time. We use a functional basis to fit the 3D+t spatio-temporal dMRI signal, similarly to the 3D-SHORE basis in three dimensional 'q-space'. The lowest order term in this expansion contains an isotropic diffusion tensor that characterizes the Gaussian displacement distribution, multiplied by a negative exponential. We regularize the signal fitting by minimizing the norm of the analytic Laplacian of the basis, and validate our technique on synthetic data generated using the theoretical model proposed by Callaghan et al. We show that our method is robust to noise, and can accurately describe the restricted spatio-temporal signal decay originating from tissue models such as cylindrical pores. From the fitting we can then estimate the axon radius distribution parameters along any direction using approaches similar to AxCaliber. We also apply our method on real data from an ActiveAx acquisition. Overall, our approach allows one to represent the complete 3D+t dMRI signal, which should prove helpful in understanding normal and pathologic nervous tissue.

The aging population and Natural Occurring Retirement Communities (NORCs); local government, healthy aging, and healthy-NORCs

BACKGROUND

The term Naturally Occurring Retirement Communities (NORCs) has been used since the 1980s. NORCs are defined as communities where people remain or move to when they retire. NORCs develop 'naturally', meaning that seniors tend to remain or move there when they retire, although the residences and physical environment were not constructed for a senior population. The term, Healthy-NORC, has been introduced and is associated with healthy aging.

OBJECTIVES/METHODS

We describe how demographic trends will facilitate a dramatic growth in NORCs. Acknowledging the 'Determinants of Health' model, we suggest that some determinants impact people differently at different ages. We also suggest that more attention be focused on the impact of physical/social environments on health, and that some determinants of health are particularly relevant for seniors. We argue that NORCs exist on a spectrum, from NORC to H-NORC, and that health benefits for seniors increase as NORCs adopt additional characteristics associated with improved senior health. We also illustrate H-NORC research methods and policy options for local governments.

RESULTS/CONCLUSION

Compared to the provision of additional medical and social services, H-NORCs represent a low-cost approach to facilitating healthy aging. Municipal governments can promote healthy aging and should pursue policies that will stimulate H-NORC development.

Effect of omalizumab on symptoms of seasonal allergic rhinitis: a randomized controlled trial

CONTEXT

Seasonal allergic rhinitis is a common IgE-mediated disorder that produces troublesome symptoms. A recombinant humanized monoclonal anti-IgE antibody (omalizumab) forms complexes with free IgE, blocking its interaction with mast cells and basophils and lowering free IgE levels in the circulation.

OBJECTIVE

To assess the efficacy and safety of omalizumab for prophylaxis of symptoms in patients with seasonal allergic rhinitis.

DESIGN

Randomized, double-blind, dose-ranging, placebo-controlled trial conducted from July 25 through November 21, 1997.

SETTING

Twenty-five outpatient centers throughout the United States.

PATIENTS

Five hundred thirty-six patients aged 12 to 75 years with at least a 2-year history of moderate to severe ragweed-induced seasonal allergic rhinitis and a baseline IgE level between 30 and 700 IU/mL.

INTERVENTIONS

Patients were randomly assigned to receive omalizumab, 50 mg (n = 137), 150 mg (n = 134), or 300 mg (n = 129), or placebo (n = 136) subcutaneously just prior to ragweed season and repeated during the pollen season every 3 weeks in patients with baseline IgE levels of 151 to 700 IU/mL (4 total treatments) and every 4 weeks in patients with baseline IgE levels of 30 to 150 IU/mL (3 total treatments).

MAIN OUTCOME MEASURES

Self-assessed daily nasal symptom severity scores (range, 0-3), rescue antihistamine use, and rhinitis-specific quality of life during the 12 weeks from the start of treatment.

RESULTS

Nasal symptom severity scores were significantly lower in patients who received 300 mg of omalizumab than in those who received placebo (least squares means, 0.75 vs 0.98, respectively; P =.002). A significant association was observed between IgE reduction and nasal symptoms and rescue antihistamine use. Rhinitis-specific quality of life scores were consistently better in patients who received 300 mg of omalizumab than in those who received lower dosages or placebo and did not decline during peak season. The frequency of adverse events was not significantly different among the omalizumab and placebo groups.

CONCLUSION

Omalizumab decreased serum free IgE levels and provided clinical benefit in a dose-dependent fashion in patients with seasonal allergic rhinitis.

Subtle overlapping deletions in the terminal region of chromosome 6q24.2-q26: three cases studied using FISH

Interstitial deletions in the terminal region of chromosome 6 are rare. We describe three new cases with subtle interstitial deletions in the q24-q26 region of the long arm of chromosome 6. The karyotypes were analyzed at a 550 band level. Patient1 is a 9-month-old boy with an interstitial deletion, del(6)(q24.2q25.1), developmental delay, low birth weight, hypotonia, heart murmur, respiratory distress, craniofacial and genital anomalies. This is the first report of a case with deletion del(6)(q24.2q25.1). Patient 2 is a 17-year-old young man with an interstitial deletion del(6)(q25.1q25.3), developmental delay, short stature, mental retardation, autism, head, face, chest, hand and feet anomalies and a history of seizures. For the first time autism was described as a manifestation in 6q deletions. Patient 3 is baby boy with a de novo interstitial deletion, del(6)(q25.1q26), anomalies of the brain, genital organs, limbs and feet. This is the first report of a case with deletion, del(6)(q25.1q26). In all three patients, fluorescence in situ hybridization (FISH) using chromosome 6 painting probe ruled out an insertion. The ESR (6q25.1) and TBP (6q27) probes were used to confirm the breakpoints. Since TBP signal is present in all cases, it confirmed an interstitial deletion proximal to this probe. Patient 1 has a deletion of the ESR locus; Patient 2 and 3 have signals for the ESR locus on both chromosomes 6. Therefore the deletion in Patients 2 and 3 are between ESR and TBP loci distal to that of Patient 1. FISH validated the deletion breakpoints assessed by conventional cytogenetics.

Correlation of CT findings with clinical evaluations in 261 patients with symptomatic bronchiectasis

OBJECTIVE

In a multicenter study, we evaluated the relationships between the extent and severity of bronchiectasis on CT and clinical symptoms, spirometric abnormality, and sputum characteristics.

SUBJECTS AND METHODS

The study population included 261 patients with symptomatic, physiologically significant bronchiectasis, who were enrolled in another study evaluating the clinical efficacy of deoxyribonudease in treatment of bronchiectasis. Patients with cystic fibrosis, allergic bronchopulmonary aspergillosis, and fungal or mycobacterial infection were excluded. In addition to high-resolution CT scanning, all patients underwent clinical evaluation, spirometry, and sputum culture. CT features scored by consensus of two observers included the extent of bronchiectasis, type of bronchiectasis (cylindric, varicose, or cystic), extent of mucoid impaction, and degree of bronchial wall thickening.

RESULTS

Scores for the severity and extent of bronchiectasis correlated with the forced expiratory volume in 1 sec (FEV1) (r = -.362, p < .0001) and with the forced vital capacity (FVC) (r = -.362, p < .0001). Scores for bronchial wall thickening correlated with the FEV1 (r = -.367, p < .0001) and FVC (r = -.239, p < .001). Patients with cystic bronchiectasis were significantly more likely to grow Pseudomonas from their sputa and to have purulent sputa than were patients with cylindric or varicose bronchiectasis. Patients with cystic bronchiectasis had significantly lower FEV1 and FVC values than did patients with cylindric or varicose bronchiectasis.

CONCLUSION

In this patient population, we found weak but significant correlations between the degree of morphologic abnormality on CT and the extent of physiologic impairment. Cystic bronchiectasis was associated with sputum purulence and with the growth of Pseudomonas. CT classification of the type of bronchiectasis may be useful as an index of severity of disease.

The relationship between infection and inflammation in the early stages of lung disease from cystic fibrosis

We examined the relationship of pulmonary infection and inflammation in cystic fibrosis (CF) by performing 31 bronchoalveolar lavages (BAL) in 14 young children with minimal lung disease from CF. While 10 of the 14 patients had elevated polymorphonuclear leukocyte (PMN) counts initially, only 4 had bacteria generally regarded as pathogenic in the recovered BAL fluid. Three of these 4 and 6 of the others had follow-up bronchoscopies at 6 months intervals. PMN counts remained normal for only one patient. However, pathogenic bacteria were recovered during the repeat BALs only in those patients who were colonized initially. Proinflammatory cytokines and proteinases were generally elevated, and interleukin-8 (IL-8) concentration correlated inversely with oxygen saturation (SaO2). No complications of the procedure occurred. We conclude that BAL identifies inflammation and the presence of bacteria in the lower airway at an early stage of the disease. This information may be used to guide therapy in patients too young or otherwise unable to produce sputum. These data also suggest that inflammation is present early in the course of CF lung disease before colonization and infection of the lungs with potentially pathogenic bacteria occurs. Since inflammation appears to be the earliest detectable evidence of lung disease in CF, monitoring of inflammation with BAL may serve as a useful marker of clinical benefits from new treatments in patients with minimal lung disease.

[Cytologic follow-up examination of patients with a suspicious Papanicoaou type 3D smear (author's transl)]

The cytological differential diagnosis "dysplasia" was made in 176 cases between January 1968 and June 1975. An immediate microscopic diagnosis was possible in 18 cases (10.22%). 14 cases eluded further follow-up. In 144 cases, long term observation was carried out. In some cases up to 6 years. Cytological regression to a permanently negative smear occurred in 57 patients (39.58%). A microscopic confirmation of the diagnosis was not obtained in these patients. In cytologically persistent cases microscopic confirmation was obtained after varying times of observation. The cytological differential diagnosis was correct in comparison to the histologic findings in 82.6% of the cases. 62 cases of the total (43.05%) showed cytological persistence of the suspicious smear. A cytological progression became apparent in 25 cases (17.36%) and was always subjected to microscopic confirmation by cone biopsy or primary hysterectomy. In 71 cases with microscopic confirmation persistent dysplasia was found in 64.78% of the cases and a progression occurred in 30.98% of the cases. 21.12% showed carcinoma in situ, 7.04% (5) cases showed a microinvasive carcinoma and 2 cases (2.82%) showed an invasive carcinoma. Cervical dysplasias are apparently capable of regression in a large number of cases. However about 10% of the cases will show progression to a micro-invasive or invasive carcinoma after varying lengths of time. In order to avoid unnecessary operations and to improve our knowledge on the biology of dysplasias, observation with cytological diagnosis dysplasia (Papanicolaou 3D) is justified.