Individual risk of cutaneous melanoma in New Zealand: developing a clinical prediction aid

Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC₁ was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC₁ ranging from 0.071 to 0.535). After training, the multi-rater AC₁ ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC₁: 0.405, 95%-CI 0.149-0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.

Using the Prediction Model Risk of Bias Assessment Tool (PROBAST) to Evaluate Melanoma Prediction Studies

Simple Summary

The rising incidence of cutaneous melanoma over recent decades, combined with a general interest in cancer risk prediction, has led to a high number of published melanoma risk prediction models. The aim of our work was to assess the validity of these models in order to discuss the current state of knowledge about how to predict incident cutaneous melanoma. To assess the risk of bias, we used a standardized procedure based on PROBAST (Prediction model Risk Of Bias ASsessment Tool). Only one of the 42 studies identified was rated as having a low risk of bias. However, it was encouraging to observe a recent reduction of problematic statistical methods used in the analyses. Nevertheless, the evidence base of high-quality studies that can be used to draw conclusions on the prediction of incident cutaneous melanoma is currently much weaker than the high number of studies on this topic would suggest.

Abstract

Rising incidences of cutaneous melanoma have fueled the development of statistical models that predict individual melanoma risk. Our aim was to assess the validity of published prediction models for incident cutaneous melanoma using a standardized procedure based on PROBAST (Prediction model Risk Of Bias ASsessment Tool). We included studies that were identified by a recent systematic review and updated the literature search to ensure that our PROBAST rating included all relevant studies. Six reviewers assessed the risk of bias (ROB) for each study using the published “PROBAST Assessment Form” that consists of four domains and an overall ROB rating. We further examined a temporal effect regarding changes in overall and domain-specific ROB rating distributions. Altogether, 42 studies were assessed, of which the vast majority (n = 34; 81%) was rated as having high ROB. Only one study was judged as having low ROB. The main reasons for high ROB ratings were the use of hospital controls in case-control studies and the omission of any validation of prediction models. However, our temporal analysis results showed a significant reduction in the number of studies with high ROB for the domain “analysis”. Nevertheless, the evidence base of high-quality studies that can be used to draw conclusions on the prediction of incident cutaneous melanoma is currently much weaker than the high number of studies on this topic would suggest.

Reporting Quality of Studies Developing and Validating Melanoma Prediction Models: An Assessment Based on the TRIPOD Statement

Transparent and accurate reporting is essential to evaluate the validity and applicability of risk prediction models. Our aim was to evaluate the reporting quality of studies developing and validating risk prediction models for melanoma according to the TRIPOD (Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis) checklist. We included studies that were identified by a recent systematic review and updated the literature search to ensure that our TRIPOD rating included all relevant studies. Six reviewers assessed compliance with all 37 TRIPOD components for each study using the published “TRIPOD Adherence Assessment Form”. We further examined a potential temporal effect of the reporting quality. Altogether 42 studies were assessed including 35 studies reporting the development of a prediction model and seven studies reporting both development and validation. The median adherence to TRIPOD was 57% (range 29% to 78%). Study components that were least likely to be fully reported were related to model specification, title and abstract. Although the reporting quality has slightly increased over the past 35 years, there is still much room for improvement. Adherence to reporting guidelines such as TRIPOD in the publication of study results must be adopted as a matter of course to achieve a sufficient level of reporting quality necessary to foster the use of the prediction models in applications.

Risk Prediction Models for Melanoma: A Systematic Review on the Heterogeneity in Model Development and Validation

The rising incidence of cutaneous melanoma over the past few decades has prompted substantial efforts to develop risk prediction models identifying people at high risk of developing melanoma to facilitate targeted screening programs. We review these models, regarding study characteristics, differences in risk factor selection and assessment, evaluation, and validation methods. Our systematic literature search revealed 40 studies comprising 46 different risk prediction models eligible for the review. Altogether, 35 different risk factors were part of the models with nevi being the most common one (n = 35, 78%); little consistency in other risk factors was observed. Results of an internal validation were reported for less than half of the studies (n = 18, 45%), and only 6 performed external validation. In terms of model performance, 29 studies assessed the discriminative ability of their models; other performance measures, e.g., regarding calibration or clinical usefulness, were rarely reported. Due to the substantial heterogeneity in risk factor selection and assessment as well as methodologic aspects of model development, direct comparisons between models are hardly possible. Uniform methodologic standards for the development and validation of risk prediction models for melanoma and reporting standards for the accompanying publications are necessary and need to be obligatory for that reason.

Real-world outcomes of melanoma surveillance using the MoleMap NZ telemedicine platform

BACKGROUND

MoleMap NZ is a novel New Zealand-based store-and-forward telemedicine service to detect melanoma. It uses expert review of total body photography and close-up and dermoscopic images of skin lesions that are suspicious for malignancy.

OBJECTIVE

The purpose of this study was to assess the effectiveness of MoleMap NZ as a melanoma early detection program.

METHODS

We conducted a review of 2108 melanocytic lesions recommended for biopsy/excision by MoleMap NZ dermoscopists between January 2015 and December 2016.

RESULTS

Pathologic diagnoses were available for 1571 lesions. Of these, 1303 (83%) lesions were benign and 260 (17%) lesions were diagnosed as melanoma, for a melanoma-specific benign:malignant ratio of 5.0:1. The number needed to biopsy to obtain 1 melanoma was 6. Among melanomas with available tumor thickness data (n = 137), 92% were <0.8 mm (range in situ to 3.1 mm), with in situ melanomas comprising 74%.

LIMITATIONS

Only lesions recommended for excision were analyzed. Pathology results were available for 75% of these cases. Tumor thickness data were available for 53% of melanomas diagnosed.

CONCLUSIONS

This real-world study of MoleMap NZ, a community-based teledermoscopy program, suggests that it has the potential to increase patients' access to specialist expertise via telemedicine. Additional studies are needed to more accurately define its efficacy.

Combining common genetic variants and non-genetic risk factors to predict risk of cutaneous melanoma

Melanoma heritability is among the highest for cancer and single nucleotide polymorphisms (SNPs) contribute to it. To date, only SNPs that reached statistical significance in genome-wide association studies or few candidate SNPs have been included in melanoma risk prediction models. We compared four approaches for building polygenic risk scores (PRS) using 12 874 melanoma cases and 23 203 controls from Melanoma Meta-Analysis Consortium as a training set, and newly genotyped 3102 cases and 2301 controls from the MelaNostrum consortium for validation. We estimated adjusted odds ratios (ORs) for melanoma risk using traditional melanoma risk factors and the PRS with the largest area under the receiver operator characteristics curve (AUC). We estimated absolute risks combining the PRS and other risk factors, with age- and sex-specific melanoma incidence and competing mortality rates from Italy as an example. The best PRS, including 204 SNPs (AUC = 64.4%; 95% confidence interval (CI) = 63-65.8%), developed using winner's curse estimate corrections, had a per-quintile OR = 1.35 (95% CI = 1.30-1.41), corresponding to a 3.33-fold increase comparing the 5th to the 1st PRS quintile. The AUC improvement by adding the PRS was up to 7%, depending on adjusted factors and country. The 20-year absolute risk estimates based on the PRS, nevus count and pigmentation characteristics for a 60-year-old Italian man ranged from 0.5 to 11.8% (relative risk  = 26.34), indicating good separation.

Development and validation of a melanoma risk score based on pooled data from 16 case-control studies

BACKGROUND

We report the development of a cutaneous melanoma risk algorithm based upon seven factors; hair color, skin type, family history, freckling, nevus count, number of large nevi, and history of sunburn, intended to form the basis of a self-assessment Web tool for the general public.

METHODS

Predicted odds of melanoma were estimated by analyzing a pooled dataset from 16 case-control studies using logistic random coefficients models. Risk categories were defined based on the distribution of the predicted odds in the controls from these studies. Imputation was used to estimate missing data in the pooled datasets. The 30th, 60th, and 90th centiles were used to distribute individuals into four risk groups for their age, sex, and geographic location. Cross-validation was used to test the robustness of the thresholds for each group by leaving out each study one by one. Performance of the model was assessed in an independent UK case-control study dataset.

RESULTS

Cross-validation confirmed the robustness of the threshold estimates. Cases and controls were well discriminated in the independent dataset [area under the curve, 0.75; 95% confidence interval (CI), 0.73-0.78]. Twenty-nine percent of cases were in the highest risk group compared with 7% of controls, and 43% of controls were in the lowest risk group compared with 13% of cases.

CONCLUSION

We have identified a composite score representing an estimate of relative risk and successfully validated this score in an independent dataset.

IMPACT

This score may be a useful tool to inform members of the public about their melanoma risk.