Evaluation of medical technologies: a generalized ROC analysis

Trajectories of Treatment Response and Nonresponse in Youth at High Risk for Suicide

OBJECTIVE

To examine trajectories of treatment response in suicidal youth who participated in a randomized controlled trial comparing dialectical behavior therapy (DBT) and individual and group supportive therapy.

METHOD

Using latent class analysis across both treatment conditions, secondary analyses were conducted of data from a multisite randomized controlled trial comprising 173 youths ages 12-18 with repetitive self-harm (SH) (including ≥1 lifetime suicide attempts) and elevated suicidal ideation (SI). The sample was 95% female, 56.4% White, and 27.49% Latina. Participants received 6 months of DBT or individual and group supportive therapy and 6 months of follow-up. Primary outcomes were SH and SI.

RESULTS

Of the sample, 63% and 74% were members of latent classes that showed improvement in SI and SH, respectively; 13% were total nonresponders, with no improvement in SI or SH. SH nonresponse emerged at the midpoint of treatment (3 months), with nonresponders showing a sharp increase in SH over the remainder of treatment and follow-up. Youth receiving DBT were significantly more likely to be an SH responder vs nonresponder than youths in individual and group supportive therapy (ꭓ21 = 6.53, p = .01). An optimal threshold cut point using multivariate predictors of total nonresponse (White, externalizing symptoms, total SH, and SI) predicted total nonresponders to DBT with 100% accuracy.

CONCLUSION

This is the first study to identify trajectories of both SI and SH response to treatment in a sample of adolescents at risk of suicide. Results may inform personalized treatment approaches.

CLINICAL TRIAL REGISTRATION INFORMATION

Collaborative Adolescent Research on Emotions and Suicide (CARES); https://www.

CLINICALTRIALS

gov/; NCT01528020.

Value of information methods for assessing a new diagnostic test

Value-of-information methods are applied to assess the evidence in support of a new diagnostic test and, where the evidence is insufficient for decision making, to determine the optimal sample size for future studies. Net benefit formulations are derived under various diagnostic and treatment scenarios. The expressions for the expected opportunity loss of adopting strategies that include the new test are given. Expressions for the expected value of information from future studies are derived. One-sample and two-sample designs, with or without known prevalence, are considered. An example is given.

Developing and validating a model to predict the success of an IHCS implementation: the Readiness for Implementation Model

OBJECTIVE

To develop and validate the Readiness for Implementation Model (RIM). This model predicts a healthcare organization's potential for success in implementing an interactive health communication system (IHCS). The model consists of seven weighted factors, with each factor containing five to seven elements.

DESIGN

Two decision-analytic approaches, self-explicated and conjoint analysis, were used to measure the weights of the RIM with a sample of 410 experts. The RIM model with weights was then validated in a prospective study of 25 IHCS implementation cases.

MEASUREMENTS

Orthogonal main effects design was used to develop 700 conjoint-analysis profiles, which varied on seven factors. Each of the 410 experts rated the importance and desirability of the factors and their levels, as well as a set of 10 different profiles. For the prospective 25-case validation, three time-repeated measures of the RIM scores were collected for comparison with the implementation outcomes.

RESULTS

Two of the seven factors, 'organizational motivation' and 'meeting user needs,' were found to be most important in predicting implementation readiness. No statistically significant difference was found in the predictive validity of the two approaches (self-explicated and conjoint analysis). The RIM was a better predictor for the 1-year implementation outcome than the half-year outcome.

LIMITATIONS

The expert sample, the order of the survey tasks, the additive model, and basing the RIM cut-off score on experience are possible limitations of the study.

CONCLUSION

The RIM needs to be empirically evaluated in institutions adopting IHCS and sustaining the system in the long term.

The psychometric properties of the MASC in a pediatric psychiatric sample

The goals of this study were twofold: to examine the psychometric properties of the Multidimensional Anxiety Scale for Children (MASC) in a clinical sample of 193 children and adolescents who had received a diagnosis of major depressive or anxiety disorder, and to discriminate between these two groups of patients. Participants had volunteered in randomized psychopharmacological clinical trials. The MASC four-factor structure was confirmed and its subscales were found to be reliable. The MASC correlated well with other self-report measures of anxiety, and less so with measures of depressive symptoms. The MASC subscales and two MASC items as well as age differentiated between anxious and depressed pediatric patients. If these results are replicated in an independent study, those items could be used by clinicians to discriminate between these two disorders. The MASC is a clinically useful measure to discriminate between anxious and depressed pediatric patients. Limitations due to the highly selective sample are noted.

Three validation metrics for automated probabilistic image segmentation of brain tumours

The validity of brain tumour segmentation is an important issue in image processing because it has a direct impact on surgical planning. We examined the segmentation accuracy based on three two-sample validation metrics against the estimated composite latent gold standard, which was derived from several experts' manual segmentations by an EM algorithm. The distribution functions of the tumour and control pixel data were parametrically assumed to be a mixture of two beta distributions with different shape parameters. We estimated the corresponding receiver operating characteristic curve, Dice similarity coefficient, and mutual information, over all possible decision thresholds. Based on each validation metric, an optimal threshold was then computed via maximization. We illustrated these methods on MR imaging data from nine brain tumour cases of three different tumour types, each consisting of a large number of pixels. The automated segmentation yielded satisfactory accuracy with varied optimal thresholds. The performances of these validation metrics were also investigated via Monte Carlo simulation. Extensions of incorporating spatial correlation structures using a Markov random field model were considered.

Developing and testing a model to predict outcomes of organizational change

OBJECTIVE

To test the effectiveness of a Bayesian model employing subjective probability estimates for predicting success and failure of health care improvement projects.

DATA SOURCES

Experts' subjective assessment data for model development and independent retrospective data on 221 healthcare improvement projects in the United States, Canada, and The Netherlands collected between 1996 and 2000 for validation.

METHODS

A panel of theoretical and practical experts and literature in organizational change were used to identify factors predicting the outcome of improvement efforts. A Bayesian model was developed to estimate probability of successful change using subjective estimates of likelihood ratios and prior odds elicited from the panel of experts. A subsequent retrospective empirical analysis of change efforts in 198 health care organizations was performed to validate the model. Logistic regression and ROC analysis were used to evaluate the model's performance using three alternative definitions of success.

DATA COLLECTION

For the model development, experts' subjective assessments were elicited using an integrative group process. For the validation study, a staff person intimately involved in each improvement project responded to a written survey asking questions about model factors and project outcomes.

RESULTS

Logistic regression chi-square statistics and areas under the ROC curve demonstrated a high level of model performance in predicting success. Chi-square statistics were significant at the 0.001 level and areas under the ROC curve were greater than 0.84.

CONCLUSIONS

A subjective Bayesian model was effective in predicting the outcome of actual improvement projects. Additional prospective evaluations as well as testing the impact of this model as an intervention are warranted.

Measurement of image quality in diagnostic radiology

The aim in radiology is to obtain images which are adequate for the clinical purpose with the minimum radiation dose to the patient. If optimum performance is to be achieved, assessments of image quality must be made to balance against patient dose. The subjective nature of image interpretation makes an objective approach to such assessment difficult. Methods widely applied involve the use of test objects, which although providing a measure of imaging performance may be difficult to link to clinical image formation. The ideal method for evaluation of imaging techniques is through clinical trials and this should be used to address major questions. Scoring of quality criteria, relating to features observed in a normal clinical radiograph, provides a simple method through which image quality can be assessed in every hospital department.

A critical evaluation of several alcohol screening instruments using the CIDI-SAM as a criterion measure

Four alcohol screening instruments (the AUDIT, CAGE, MAST, and Svanum's scale) were administered to a sample of 306 undergraduate students at a Midwestern university and were compared with regard to several test characteristics, using the alcohol section of the CIDI-SAM (DSM-IV version) as the criterion measure. The performance of these instruments was evaluated using two subsets of subjects: (1) students who currently met diagnostic criteria for alcohol dependence (n = 35); and (2) students who met diagnostic criteria for alcohol dependence in the past and/or at present (i.e., lifetime diagnosis; n = 50). The AUDIT performed significantly better than the other three instruments in identifying students who were currently alcohol dependent, providing a moderate degree of clinical utility with this group. The four instruments did not differ significantly in their ability to identify students with a lifetime diagnosis; each measure provided only a modest degree of clinical utility with this group.

Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves

Diagnostic tests are often evaluated by comparison of the areas under receiver operating characteristic (ROC) curves. In this study the authors compared this approach with a more direct method that takes into account consequences of a diagnosis. Data from a prospective study of diagnosis of pulmonary embolism were used for a motivating example. Using multivariable logistic regression analysis, three diagnostic models were built and compared based on their ROC curves. Although model 1 (0.706) and model 2 (0.702) had the same ROC-curve area, they performed differently when risks and benefits of subsequent decisions were considered by applying the treatment probability threshold. Models 1 and 3 (0.611) had substantially different ROC-curve areas but performed similarly taking into account the therapeutic consequences. This demonstrates that comparison of diagnostic tests using the areas under the ROC curves may lead to erroneous conclusions about therapeutic usefulness. To correspond to daily practice, it would be more appropriate to also consider the clinical implications in evaluating diagnostic tests. This is made feasible by explicit definition and application of a treatment threshold.

Optimal cut-points when screening for more than one disease state: an example from the Canadian Study of Health and Aging

A method is described for choosing optimal cut-points in screening test with a continuous score in order to divide people into any number of disease categories. The cut-points are chosen to minimize the overall expected loss, given a matrix of losses incurred by misclassifications. It is shown that the optimal cut-points depend on the column differences of the loss matrix. The effect of changing the loss matrix is examined. For the error counting matrix, where misclassifications are given equal weight, it is shown that the relationship between the minimal loss cut-point and the receiver operating characteristic (ROC) curve for two disease categories extends to the general case. The results of a study in which elderly people were screened for cognitive impairment and dementia are used to illustrate the method.

Removing the assumption of conditional independence from Bayesian decision models by using artificial neural networks: some practical techniques and a case study

The article describes how artificial neural networks with special designs can be applied to approximate a subjective Bayesian decision model without the assumption of conditional independence. New techniques are proposed to resolve some of the practical difficulties during the processes of problem structuring, knowledge elicitation, quantitative modeling, and model interpretation. A Bayesian model considering the conditional dependencies to predict a teenager's marijuana use was constructed by experts using these techniques, and compared to another conventional Bayesian model which assumed conditional independence. The new approach without the assumption of conditional independence had predictive power (r = 0.7) in the test of linearity compared to the conventional approach (r = 0.58) on a data set (n = 129). Its receiver operating characteristic curve dominated the alternative approach within the range (true positive fraction > 0.7) that we were interested in. The interpretations of the possible conditional dependencies provided by the artificial neural network after the training process were consistent with the expert's descriptions.