Brief critical review: Statistical assessment of biomarker performance

A benefit risk approach in cutoff determination for diagnostic tests

A crucial step in the design of a diagnostic test is determining the cutoff point, the threshold which separates a negative measurement from a positive one. The results of a diagnostic test have clinical consequences: only when disease is accurately detected, proper treatments be administered, and vice versa. Benefit-Risk (BR) analysis should be used to determine the optimal cutoff point that optimizes the consequence. Quantitative BR analysis requires measurable benefit and risk and a function, e.g., linear or ratio, to combine all the components. When BR corresponding to the four possible diagnostic test outcomes are all scaled in units of risk resulting from an untreated disease, we propose a net BR (linear BR) equation as a function of diagnostic parameters, disease prevalence, benefit of correct diagnosis and risk of false diagnostic results. Optimal cutoff of a diagnostic test can be obtained using this function. Comparison of diagnostic tests based on their benefit and risk of tests is also discussed. Use of this function is illustrated with a biosensor rapid antigen test for SARS-CoV-2.

Early Detection of Ovarian Cancer

The risk of death from ovarian cancer is highly associated with the clinical stage at diagnosis. Efforts to implement screening for ovarian cancer have been largely unsuccessful, due to the low prevalence of the disease in the general population and the heterogeneity of the various cancer types that fall under the ovarian cancer designation. A practical test for early detection will require both high sensitivity and high specificity to balance reducing the number of cancer deaths with minimizing surgical interventions for false positive screens. The technology must be cost-effective to deliver at scale, widely accessible, and relatively noninvasive. Most importantly, a successful early detection test must be effective not only at diagnosing ovarian cancer but also in reducing ovarian cancer deaths. Stepwise or multimodal approaches among the various areas under investigation will likely be required to make early detection a reality.

Practical guide to the typical analysis of prognostic factors and biomarkers without the use of P-values

OBJECTIVES

We have seen an increasing number of studies evaluating biomarkers and prognostic factors. Biomedical researchers like to draw conclusions based on P-values. However, P-values are often not needed for this type of study. In this article, we show how most biomedical research problems in this area could be organized into three main analyses, each avoiding the use of P-values.

METHODS

The three main analyses follow the framework of prediction modeling when the outcome of interest is binary or time-to-event. The analyses make use of figures such as boxplots, nonparametric smoothing line, and nomogram, and also incorporate prediction performance measures such as the Area under the receiver operating characteristic curve, and index of predictive accuracy.

RESULTS

Our proposed framework is easy to follow. It is also in line with most of the research in the field of biomarkers and prognostic factors evaluation, such as reclassification table, net reclassification index, Akaike information criterion/Bayesian information criterion, receiver operating characteristic curve, and decision curve analysis.

CONCLUSION

Overall, we provide a step-by-step guideline that biomedical researchers could easily follow to conduct statistical analysis without using P-values, especially with the goal of evaluating biomarkers and prognostic factors.

A critique and systematic review of the clinical utility of hepatitis B core-related antigen

BACKGROUND & AIMS

Hepatitis B core-related antigen (HBcrAg) is a new biomarker for chronic hepatitis B (CHB) whose performance has not been critically or systematically appraised. Herein, we performed a systematic review to determine its clinical utility.

METHODS

We evaluated the biological pathway of HBcrAg and performed a systematic review of PubMed for clinical trials, cohort studies, and case-control studies that evaluated the clinical utility of HBcrAg. The effectiveness of HBcrAg in predicting HBV-specific clinical events (e.g. HBeAg seroconversion, phases of CHB, HBsAg loss, treatment response, and relapse after stopping therapy) was examined using receiver-operating characteristic curves. The correlation coefficients of HBcrAg with HBV DNA, quantitative HBsAg (qHBsAg), HBV RNA, and cccDNA were summarised from published studies. Median values were used as estimates.

RESULTS

HBcrAg consists of three precore/core protein products: HBcAg, HBeAg, and a 22 kDa precore protein. HBcrAg assays have been associated with false-positive rates of 9.3% and false-negative rates of between 12-35% for CHB. The new iTACT-HBcrAg is more sensitive but does not reduce the false-positive rate. A PubMed search found 248 papers on HBcrAg, of which 59 were suitable for analysis. The clinical performance of HBcrAg was evaluated using AUROC analyses, with median AUROCs of 0.860 for HBeAg seroconversion, 0.867 for predicting HBeAg(-) hepatitis, 0.645 for HBsAg loss, 0.757 for treatment response, and 0.688 for relapse after stopping therapy. The median correlation coefficient (r) was 0.630 with HBV DNA, 0.414 with qHBsAg, 0.619 with HBV RNA and 0.550 with cccDNA. Correlation decreased during antiviral therapy, but combined biomarkers improved performance.

CONCLUSIONS

HBcrAg has a mixed performance and has a poor correlation with HBsAg loss and antiviral therapy, hence HBcrAg results should be interpreted with caution.

IMPACT AND IMPLICATIONS

Hepatitis B core-related antigen (HBcrAg) has been used to assess management of patients with chronic hepatitis B (CHB) without a systematic and critical Sreview of its performance. Our finding that HBcrAg had a false-positive rate of 9% and a false-negative rate of 12-35% raises concerns, although larger studies are needed for validation. A systematic review showed that the performance of HBcrAg was variable depending on the CHB endpoint; it was excellent at predicting HBeAg seroconversion and HBeAg-negative chronic hepatitis (vs. chronic infection), which should be its main use, but it was poor for relapse after stopping antiviral therapy and for HBsAg loss. HBcrAg results should be interpreted with considerable caution, particularly by physicians, researchers, guideline committees and agencies that approve diagnostic tests.

Relying on pulse oximetry to avoid hypoxaemia and hyperoxia: A multicentre prospective cohort study in patients with circulatory failure

PURPOSE

The purpose of this study was to assess the predictive performance of pulse oximetry (SpO₂) to rule out hypoxaemia and hyperoxia in critically ill patients.

METHODS

SpO₂, arterial oxygenation (SaO₂), and arterial partial pressure of oxygen (PaO₂) were prospectively and simultaneously measured every 6 h during the first 24 h of intensive care unit admission in a multicentre cohort of critically ill patients suffering acute circulatory failure. Likelihood ratios associated with different cutoff values of SpO₂ to rule out hypoxaemia (SaO₂ < 90% or PaO₂ < 60 mmHg) or hyperoxia (SaO₂ > 95% or PaO₂ > 100 mmHg) and post-test probabilities were calculated. Mean bias between SpO₂ and SaO₂ and agreement interval were calculated. Area under the receiver operating characteristics associated with SpO₂ to predict different threshold values of SaO₂ and PaO₂ were calculated.

RESULTS

Five hundred seventy-one patients (mean [standard deviation] Simplified Acute Physiology Score II: 58.7 [20.1]; mechanically ventilated 75.6%) with 2643 available SaO₂ and PaO₂ samples and corresponding 2643 SpO₂ values were analysed. Mean bias between SpO₂ and SaO₂ was 1.1%, and its agreement interval ranged from -8.2 to +11.1%. SpO₂ cutoff values of 88%, 90%, and 92% left the possibility that 8%-13% of patients had hypoxaemia. SpO₂ < 95% left the possibility that 31% of patients had hyperoxia. All calculated areas under the receiver operating characteristics showed a lower limit of their 95% confidence interval below 0.85 CONCLUSION: In this cohort of patients with circulatory failure, SpO₂ had poor discriminative ability to rule out hypoxaemia and hyperoxia. Overconfidence upon SpO₂ monitoring may be dangerous.

Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example

The stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluated as predictors for metastasis in patients with colorectal cancer, was investigated. Data belonging to patients admitted with the diagnosis of colorectal cancer from January 2014 until September 2019 in a single hospital were used. There were 1688 patients eligible for the study, 418 in the metastatic stage. All investigated inflammatory ratios proved to be significant classification models on both the full models and on cross-validations (AUCs > 0.05). High variability of the cut-off values was observed in the unrestricted and restricted split (full models: 4.255 for NLR, 2.745 for dNLR and 255.56 for PLR; random splits: cut-off from 3.215 to 5.905 for NLR, from 2.625 to 3.575 for dNLR and from 134.67 to 335.9 for PLR), but with no effect on the models characteristics or performances. The investigated biomarkes proved limited value as predictors for metastasis (AUCs < 0.8), with largely sensitivity and specificity (from 33.3% to 79.2% for the full model and 29.1% to 82.7% in the restricted splits). Our results showed that a simple random split of observations, weighting or not the patients with and whithout metastasis, in a ROC analysis assures the performances similar to the full model, if at least 70% of the available population is included in the study.

Predicting arterial blood gas and lactate from central venous blood analysis in critically ill patients: a multicentre, prospective, diagnostic accuracy study

BACKGROUND

The estimation of arterial blood gas and lactate from central venous blood analysis and pulse oximetry [Formula: see text] readings has not yet been extensively validated.

METHODS

In this multicentre, prospective study performed in 590 patients with acute circulatory failure, we measured blood gases and lactate in simultaneous central venous and arterial blood samples at 6 h intervals during the first 24 h after insertion of central venous and arterial catheters. The study population was randomly divided in a 2:1 ratio into model derivation and validation sets. We derived predictive models of arterial pH, carbon dioxide partial pressure, oxygen saturation, and lactate, using clinical characteristics, [Formula: see text], and central venous blood gas values as predictors, and then tested their performance in the validation set.

RESULTS

In the validation set, the agreement intervals between predicted and actual values were -0.078/+0.084 units for arterial pH, -1.32/+1.36 kPa for arterial carbon dioxide partial pressure, -5.15/+4.47% for arterial oxygen saturation, and -1.07/+1.05 mmol litre(-1) for arterial lactate (i.e. around two times our predefined clinically tolerable intervals for all variables). This led to ∼5% (or less) of extreme-to-extreme misclassifications, thus giving our predictive models only marginal agreement. Thresholds of predicted variables (as determined from the derivation set) showed high predictive values (consistently >94%), to exclude abnormal arterial values in the validation set.

CONCLUSIONS

Using clinical characteristics, [Formula: see text], and central venous blood analysis, we predicted arterial blood gas and lactate values with marginal accuracy in patients with circulatory failure. Further studies are required to establish whether the developed models can be used with acceptable safety.

Site-specific Hypermethylation of RUNX3 Predicts Poor Prognosis in Gastric Cancer

BACKGROUND AND AIMS

Methylation status of RUNX3 remains largely unknown in gastric cancer (GC). The aim of this study was to prognostically evaluate the methylation level of CpG sites within RUNX3 promoter region in GC.

METHODS

Using pyrosequencing, we quantitatively explored the methylation status of 8 CpG sites within RUNX3 promoter region for 76 gastric cancer and 24 normal gastric tissues. We then analyzed the association between methylation level of each CpG site and clinicopathological characteristics and outcomes in the cohort.

RESULTS

Methylation of RUNX3 promoter was significantly higher in GC than normal subjects. Overall methylation level was closely associated with tumor invasion and TNM stage. Positive associations were found between hypermethylation of the following concerned sites and variables: site -1392, -1397, -1403, -1415 and tumor invasion, as well as TNM stage; site -1392 and lymph node metastasis along with number of lymph node metastases; site -1415 and cancer recurrence; site -1403, -1415 and cancer-related deaths. In multivariate analysis, tumor invasion was correlated with sites -1392 and -1397. Lymph node metastasis was associated with site -1392. Most importantly, methylation of site -1415 was associated with poor survival by using Cox survival regression.

CONCLUSION

Analysis of RUNX3 gene promoter by quantitative pyrosequencing suggested methylation status of RUNX3 is different in normal and tumor tissues. RUNX3 methylation level is associated with GC, especially the methylation at site -1415 contributes to the poor prognosis in GC. Thus, RUNX3 methylation may serve as a valuable diagnostic and prognostic biomarker in GC.

Methylation analysis of plasma cell-free DNA for breast cancer early detection using bisulfite next-generation sequencing

Circulating cell-free DNA (cfDNA) has been considered as a potential biomarker for non-invasive cancer detection. To evaluate the methylation levels of six candidate genes (EGFR, GREM1, PDGFRB, PPM1E, SOX17, and WRN) in plasma cfDNA as biomarkers for breast cancer early detection, quantitative analysis of the promoter methylation of these genes from 86 breast cancer patients and 67 healthy controls was performed by using microfluidic-PCR-based target enrichment and next-generation bisulfite sequencing technology. The predictive performance of different logistic models based on methylation status of candidate genes was investigated by means of the area under the ROC curve (AUC) and odds ratio (OR) analysis. Results revealed that EGFR, PPM1E, and 8 gene-specific CpG sites showed significantly hypermethylation in cancer patients' plasma and significantly associated with breast cancer (OR ranging from 2.51 to 9.88). The AUC values for these biomarkers were ranging from 0.66 to 0.75. Combinations of multiple hypermethylated genes or CpG sites substantially improved the predictive performance for breast cancer detection. Our study demonstrated the feasibility of quantitative measurement of candidate gene methylation in cfDNA by using microfluidic-PCR-based target enrichment and bisulfite next-generation sequencing, which is worthy of further validation and potentially benefits a broad range of applications in clinical oncology practice. Quantitative analysis of methylation pattern of plasma cfDNA by next-generation sequencing might be a valuable non-invasive tool for early detection of breast cancer.

Secreted frizzled-related protein promotors are hypermethylated in cutaneous squamous carcinoma compared with normal epidermis

Background

The Wnt signaling pathway is abnormally activated in many human cancers. Secreted frizzled-related proteins (SFRPs) function as negative regulators of Wnt signaling and play an important role in carcinogenesis. SFRP promoter hypermethylation has often been identified in human cancers; however, the precise role of SFRPs in cutaneous squamous cell carcinoma (SCC) is unclear.

Methods

The methylation status of the SFRP family was analyzed in an age-and sex-matched case-control study, including 40 cutaneous SCC cases and 40 normal controls, using the MassARRAY EpiTYPER system.

Results

The methylation rate of SFRP1, SFRP2, SFRP4, and SFRP5 promoters was significantly higher in cutaneous SCC tissues than in adjacent tissue and normal skin samples.

Discussion

Our manuscript mainly discussed the average methylation rate of SFRPs (SFRP1, SFRP2, SFRP4, and SFRP5) promoters are significantly high in tumor tissue samples and the average CpG island methylation rate among different pathological levels of cutaneous SCC between these genes are different.

Conclusions

Our findings suggest that promoter hypermethylation of SFRPs is associated with the development of carcinoma, and could be a useful tumor marker for cutaneous SCC and other types of cancers.

Biomarker tests for risk assessment in coronary artery disease: will they change clinical practice?

The clinical significance and economic impact of coronary artery disease has triggered major research efforts into the discovery of novel biomarkers for risk stratification in primary and secondary prevention and then the development of assays suitable for routine measurement. Nevertheless, the clinical impact of these novel biomarkers for risk stratification is still limited because they do not add substantially to traditional risk factors and they only modestly-even with a multimarker approach-improve risk stratification and patient reclassification. The most useful markers appear to be high-sensitivity C-reactive protein, natriuretic peptides, and, eventually, high-sensitivity cardiac troponins. Further research is clearly needed.

Current concepts and future directions for the assessment of autoantibodies to cellular antigens referred to as anti-nuclear antibodies

The detection of autoantibodies that target intracellular antigens, commonly termed anti-nuclear antibodies (ANA), is a serological hallmark in the diagnosis of systemic autoimmune rheumatic diseases (SARD). Different methods are available for detection of ANA and all bearing their own advantages and limitations. Most laboratories use the indirect immunofluorescence (IIF) assay based on HEp-2 cell substrates. Due to the subjectivity of this diagnostic platform, automated digital reading systems have been developed during the last decade. In addition, solid phase immunoassays using well characterized antigens have gained widespread adoption in high throughput laboratories due to their ease of use and open automation. Despite all the advances in the field of ANA detection and its contribution to the diagnosis of SARD, significant challenges persist. This review provides a comprehensive overview of the current status on ANA testing including automated IIF reading systems and solid phase assays and suggests an approach to interpretation of results and discusses meeting the problems of assay standardization and other persistent challenges.