Systematic review and meta-analysis of within-subject and between-subject biological variation estimates of 20 haematological parameters

A standard to report biological variation data studies - based on an expert opinion

There is a need for standards for generation and reporting of Biological Variation (BV) reference data. The absence of standards affects the quality and transportability of BV data, compromising important clinical applications. To address this issue, international expert groups under the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) have developed an online resource (https://tinyurl.com/bvmindmap) in the form of an interactive mind map that serves as a guideline for researchers planning, performing and reporting BV studies. The mind map addresses study design, data analysis, and reporting criteria, providing embedded links to relevant references and resources. It also incorporates a checklist approach, identifying a Minimum Data Set (MDS) to enable the transportability of BV data and incorporates the Biological Variation Data Critical Appraisal Checklist (BIVAC) to assess study quality. The mind map is open to access and is disseminated through the EFLM BV Database website, promoting accessibility and compliance to a reporting standard, thereby providing a tool to be used to ensure data quality, consistency, and comparability of BV data. Thus, comparable to the STARD initiative for diagnostic accuracy studies, the mind map introduces a Standard for Reporting Biological Variation Data Studies (STARBIV), which can enhance the reporting quality of BV studies, foster user confidence, provide better decision support, and be used as a tool for critical appraisal. Ongoing refinement is expected to adapt to emerging methodologies, ensuring a positive trajectory toward improving the validity and applicability of BV data in clinical practice.

Prediction interval: A powerful statistical tool for monitoring patients and analytical systems

Monitoring is indispensable for assessing disease prognosis and evaluating the effectiveness of treatment strategies, both of which rely on serial measurements of patients' data. It also plays a critical role in maintaining the stability of analytical systems, which is achieved through serial measurements of quality control samples. Accurate monitoring can be achieved through data collection, following a strict preanalytical and analytical protocol, and the application of a suitable statistical method. In a stable process, future observations can be predicted based on historical data collected during periods when the process was deemed reliable. This can be evaluated using the statistical prediction interval. Statistically, prediction interval gives an "interval" based on historical data where future measurement results can be located with a specified probability such as 95%. Prediction interval consists of two primary components: (i) the set point and (ii) the total variation around the set point which determines the upper and lower limits of the interval. Both can be calculated using the repeated measurement results obtained from the process during its steady-state. In this paper, (i) the theoretical bases of prediction intervals were outlined, and (ii) its practical application was explained through examples, aiming to facilitate the implementation of prediction intervals in laboratory medicine routine practice, as a robust tool for monitoring patients' data and analytical systems.

Biological variation in the estimated glomerular filtration rate of healthy individuals within 24 h calculated using 2021CKD-EPI equations

BACKGROUND AND AIMS

Use the MDRD (Modification of Diet in Renal Disease) and 2021 CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration) equation void of race coefficients (CKD-EPICrea, CKD-EPICys-C, and CKD-EPICrea+Cys-C) to estimate the BV (Biological variation) of eGFR (estimated glomerular filtration rate) within 24 h in a healthy population to help explain future studies using eGFR in the context of a known BV.

METHODS

Blood samples were collected from 30 healthy subjects at six time points within 24 h. Serum creatinine (S-Crea) and serum cystatin C (S-Cys-C) were measured, and the BV of eGFR was calculated. Outlier and variance homogeneity analyses were performed, followed by CV-ANOVA on trend-corrected data.

RESULTS

The eGFR CVI for the four equations (MDRD, CKD-EPICrea, CKD-EPICys-C, and CKD-EPICrea+Cys-C) were 8.39% (7.50-9.51%), 3.90% (3.49-4.42%), 6.58% (5.88-7.46%), and 5.03% (4.50-5.71%), respectively. The corresponding II and RCVpos/neg values were 0.69, 0.48, 0.51, and 0.31, and (29.30%, - 22.66%), (12.69%, - 11.2 6%), (20.97%, - 17.33%), and (15.88%, - 13.70%), respectively; RCVpos /neg of eGFR was highest in the MDRD equation and lowest in the CKD-EPI Crea equation. Additionally, the RCVpos/neg values of the individual was highest in the MDRD equation and lowest in the CKD-EPICrea+Cys-C equation; they are (56.51%, - 36.11%) and (5.01%, - 4.77%), respectively.

CONCLUSIONS

We present data on the 24 h BV eGFR of the 2021 CKD-EPI equations. The presence of BV has impact on the interpretation of GFR results, affecting CKD disease grading. The RCVpos/neg differences were large among the individuals. When using eGFRs based on the MDRD and CKD-EPI equations, it is necessary to combine RCVpos/neg values before interpreting the results.

Bias in Laboratory Medicine: The Dark Side of the Moon

Physicians increasingly use laboratory-produced information for disease diagnosis, patient monitoring, treatment planning, and evaluations of treatment effectiveness. Bias is the systematic deviation of laboratory test results from the actual value, which can cause misdiagnosis or misestimation of disease prognosis and increase healthcare costs. Properly estimating and treating bias can help to reduce laboratory errors, improve patient safety, and considerably reduce healthcare costs. A bias that is statistically and medically significant should be eliminated or corrected. In this review, the theoretical aspects of bias based on metrological, statistical, laboratory, and biological variation principles are discussed. These principles are then applied to laboratory and diagnostic medicine for practical use from clinical perspectives.

Biological variation estimates for serum neurofilament light chain in healthy subjects

OBJECTIVES

Neurofilament light chain (NfL) is an emerging biomarker of neurodegeneration disorders. Knowledge of the biological variation (BV) can facilitate proper interpretation between serial measurements. Here BV estimates for serum NfL (sNfL) are provided.

METHODS

Serum samples were collected weekly from 24 apparently healthy subjects for 10 consecutive weeks and analyzed in duplicate using the Siemens Healthineers sNfL assay on the Atellica® IM Analyzer. Outlier detection, variance homogeneity analyses, and trend analysis were performed followed by CV-ANOVA to determine BV and analytical variation (CVA) estimates with 95%CI and the associated reference change values (RCV) and analytical performance specifications (APS).

RESULTS

Despite observed differences in sNfL concentrations between males and females, BV estimates remained consistent across genders. Both within-subject BV (CVI) for males (10.7%, 95%CI; 9.2-12.6) and females (9.1%, 95%CI; 7.8-10.9) and between-subject BV (CVG) for males (26.1%, 95%CI; 18.0-45.6) and females (30.2%, 95%CI; 20.9-53.5) were comparable. An index of individuality value of 0.33 highlights significant individuality, indicating the potential efficacy of personalized reference intervals in patient monitoring.

CONCLUSIONS

The established BV estimates for sNfL underscore its potential as a valuable biomarker for monitoring neurodegenerative diseases, offering a foundation for improved decision-making in clinical settings.

Recommendations for Setting a Criterion for Assessing Commutability of Secondary Calibrator Certified Reference Materials

A secondary higher-order calibrator is required to be commutable with clinical samples to be suitable for use in the calibration hierarchy of an end-user clinical laboratory in vitro diagnostic medical device (IVD-MD). Commutability is a property of a reference material that means results for a reference material and for clinical samples have the same numeric relationship, within specified limits, across the measurement procedures for which the reference material is intended to be used. Procedures for assessing commutability have been described in the literature. This report provides recommendations for establishing a quantitative criterion to assess the commutability of a certified reference material (CRM). The criterion is the maximum allowable noncommutability bias (MANCB) that allows a CRM to be used as a calibrator in a calibration hierarchy for an IVD-MD without exceeding the maximum allowable combined standard uncertainty for a clinical sample result (umaxCS). Consequently, the MANCB is derived as a fraction of the umaxCS for the measurand. The suitability of an MANCB for practical use in a commutability assessment is determined by estimating the number of measurements of clinical samples and CRMs required based on the precision performance and nonselectivity for the measurand of the measurement procedures in the assessment. Guidance is also provided for evaluating indeterminate commutability conclusions and how to report results of a commutability assessment.

Clinical Validity of a Machine Learning Decision Support System for Early Detection of Hepatitis B Virus: A Binational External Validation Study

HepB LiveTest is a machine learning decision support system developed for the early detection of hepatitis B virus (HBV). However, there is a lack of evidence on its generalisability. In this study, we aimed to externally assess the clinical validity and portability of HepB LiveTest in predicting HBV infection among independent patient cohorts from Nigeria and Australia. The performance of HepB LiveTest was evaluated by constructing receiver operating characteristic curves and estimating the area under the curve. Delong's method was used to estimate the 95% confidence interval (CI) of the area under the receiver-operating characteristic curve (AUROC). Compared to the Australian cohort, patients in the derivation cohort of HepB LiveTest and the hospital-based Nigerian cohort were younger (mean age, 45.5 years vs. 38.8 years vs. 40.8 years, respectively; p < 0.001) and had a higher incidence of HBV infection (1.9% vs. 69.4% vs. 57.3%). In the hospital-based Nigerian cohort, HepB LiveTest performed optimally with an AUROC of 0.94 (95% CI, 0.91-0.97). The model provided tailored predictions that ensured most cases of HBV infection did not go undetected. However, its discriminatory measure dropped to 0.60 (95% CI, 0.56-0.64) in the Australian cohort. These findings indicate that HepB LiveTest exhibits adequate cross-site transportability and clinical validity in the hospital-based Nigerian patient cohort but shows limited performance in the Australian cohort. Whilst HepB LiveTest holds promise for reducing HBV prevalence in underserved populations, caution is warranted when implementing the model in older populations, particularly in regions with low incidence of HBV infection.

Relative Blood Volume Profiles Hours After Loop Diuretic Administration: A Systematic Review and Meta-analysis

Background

Plasma refill rates can be estimated by combining measurements of urine output with relative blood volume profiles. Change in plasma refill rates could guide decongestive loop diuretic therapy in acute heart failure. The objective of the study was to assess average relative blood volume profiles generated from 2 or 3 follow-up measurements obtained hours after loop diuretic administration in subjects with vs without baseline congestion.

Methods

A systematic review was conducted of articles written in English, French, Spanish, and German, using MEDLINE (1964 to 2019), Cochrane Reviews (1996 to 2019), and Embase (1974 to 2019). Search terms included the following: diuretics, hemoconcentration, plasma volume, and blood volume. We included studies of adults given a loop diuretic with at least one baseline and one follow-up measurement. A single author extracted subject- or group-level blood volume measurements, aggregated them when needed, and converted them to relative changes.

Results

Across all 16 studies that met the prespecified inclusion criteria, relative blood volume maximally decreased 9.2% (6.6% to 12.0%) and returned to baseline after 3 or more hours. Compared to subjects without congestion, those with congestion experienced smaller decreases in relative blood volume across all follow-up periods (P = 0.001) and returned to baseline within the final follow-up period.

Conclusions

Single doses of loop diuretics produce measurable changes in relative blood volume that follow distinct profiles for subjects with vs without congestion. Measured alongside urine output, these profiles may be used to estimate plasma refill rates-potential patient-specific targets for decongestive therapy across serial diuretic doses.

Systematic review and meta-analysis of within-subject and between-subject biological variation data of coagulation and fibrinolytic measurands

OBJECTIVES

The diagnosis and monitoring of bleeding and thrombotic disorders depend on correct haemostatic measurements. The availability of high-quality biological variation (BV) data is important in this context. Many studies have reported BV data for these measurands, but results are varied. The present study aims to deliver global within-subject (CVI) and between-subject (CVG) BV estimates for haemostasis measurands by meta-analyses of eligible studies, by assessment with the Biological Variation Data Critical Appraisal Checklist (BIVAC).

METHODS

Relevant BV studies were graded by the BIVAC. Weighted estimates for CVI and CVG were obtained via meta-analysis of the BV data derived from BIVAC-compliant studies (graded A-C; whereby A represents optimal study design) performed in healthy adults.

RESULTS

In 26 studies BV data were reported for 35 haemostasis measurands. For 9 measurands, only one eligible publication was identified and meta-analysis could not be performed. 74% of the publications were graded as BIVAC C. The CVI and CVG varied extensively between the haemostasis measurands. The highest estimates were observed for PAI-1 antigen (CVI 48.6%; CVG 59.8%) and activity (CVI 34.9%; CVG 90.2%), while the lowest were observed for activated protein C resistance ratio (CVI 1.5%; CVG 4.5%).

CONCLUSIONS

This study provides updated BV estimates of CVI and CVG with 95% confidence intervals for a wide range of haemostasis measurands. These estimates can be used to form the basis for analytical performance specifications for haemostasis tests used in the diagnostic work-up required in bleeding- and thrombosis events and for risk assessment.

Long-Term Within- and Between-Subject Biological Variation Data of Hematological Parameters in Recreational Endurance Athletes

BACKGROUND

Hematological parameters have many applications in athletes, from monitoring health to uncovering blood doping. This study aimed to deliver biological variation (BV) estimates for 9 hematological parameters by a Biological Variation Data Critical Appraisal Checklist (BIVAC) design in a population of recreational endurance athletes and to assess the effect of self-reported exercise and health-related variables on BV.

METHODS

Samples were drawn from 30 triathletes monthly for 11 months and measured in duplicate for hematological measurands on an Advia 2120 analyzer (Siemens Healthineers). After outlier and homogeneity analysis, within-subject (CVI) and between-subject (CVG) BV estimates were delivered (CV-ANOVA and log-ANOVA, respectively) and a linear mixed model was applied to analyze the effect of exercise and other related variables on the BV estimates.

RESULTS

CVI estimates ranged from 1.3% (95%CI, 1.2-1.4) for mean corpuscular volume to 23.8% (95%CI, 21.6-26.3) for reticulocytes. Sex differences were observed for platelets and OFF-score. The CVI estimates were higher than those reported for the general population based on meta-analysis of eligible studies in the European Biological Variation Database, but 95%CI overlapped, except for reticulocytes, 23.9% (95%CI, 21.6-26.5) and 9.7% (95%CI, 6.4-11.0), respectively. Factors related to exercise and athletes' state of health did not appear to influence the BV estimates.

CONCLUSIONS

This is the first BIVAC-compliant study delivering BV estimates that can be applied to athlete populations performing high-level aerobic exercise. CVI estimates of most parameters were similar to the general population and were not influenced by exercise or athletes' state of health.

Biological variation: recent development and future challenges

Biological variation (BV) data have many applications in laboratory medicine. However, these depend on the availability of relevant and robust BV data fit for purpose. BV data can be obtained through different study designs, both by experimental studies and studies utilizing previously analysed routine results derived from laboratory databases. The different BV applications include using BV data for setting analytical performance specifications, to calculate reference change values, to define the index of individuality and to establish personalized reference intervals. In this review, major achievements in the area of BV from last decade will be presented and discussed. These range from new models and approaches to derive BV data, the delivery of high-quality BV data by the highly powered European Biological Variation Study (EuBIVAS), the Biological Variation Data Critical Appraisal Checklist (BIVAC) and other standards for deriving and reporting BV data, the EFLM Biological Variation Database and new applications of BV data including personalized reference intervals and measurement uncertainty.

A Review of Routine Laboratory Biomarkers for the Detection of Severe COVID-19 Disease

As the COVID-19 pandemic continues, there is an urgent need to identify clinical and laboratory predictors of disease severity and prognosis. Once the coronavirus enters the cell, it triggers additional events via different signaling pathways. Cellular and molecular deregulation evoked by coronavirus infection can manifest as changes in laboratory findings. Understanding the relationship between laboratory biomarkers and COVID-19 outcomes would help in developing a risk-stratified approach to the treatment of patients with this disease. The purpose of this review is to investigate the role of hematological (white blood cell (WBC), lymphocyte, and neutrophil count, neutrophil-to-lymphocyte ratio (NLR), platelet, and red blood cell (RBC) count), inflammatory (C-reactive protein (CRP), erythrocyte sedimentation rate (ESR), and lactate dehydrogenase (LDH)), and biochemical (Albumin, aspartate aminotransferase (AST) and alanine aminotransferase (ALT), blood urea nitrogen (BUN), creatinine, D-dimer, total Cholesterol, low-density lipoprotein (LDL), and high-density lipoprotein (HDL)) biomarkers in the pathogenesis of COVID-19 disease and how their levels vary according to disease severity.

How is test laboratory data used and characterised by machine learning models? A systematic review of diagnostic and prognostic models developed for COVID-19 patients using only laboratory data

The current gold standard for COVID-19 diagnosis, the rRT-PCR test, is hampered by long turnaround times, probable reagent shortages, high false-negative rates and high prices. As a result, machine learning (ML) methods have recently piqued interest, particularly when applied to digital imagery (X-rays and CT scans). In this review, the literature on ML-based diagnostic and prognostic studies grounded on hematochemical parameters has been considered. By doing so, a gap in the current literature was addressed concerning the application of machine learning to laboratory medicine. Sixty-eight articles have been included that were extracted from the Scopus and PubMed indexes. These studies were marked by a great deal of heterogeneity in terms of the examined laboratory test and clinical parameters, sample size, reference populations, ML algorithms, and validation approaches. The majority of research was found to be hampered by reporting and replicability issues: only four of the surveyed studies provided complete information on analytic procedures (units of measure, analyzing equipment), while 29 provided no information at all. Only 16 studies included independent external validation. In light of these findings, we discuss the importance of closer collaboration between data scientists and medical laboratory professionals in order to correctly characterise the relevant population, select the most appropriate statistical and analytical methods, ensure reproducibility, enable the proper interpretation of the results, and gain actual utility by using machine learning methods in clinical practice.

Critical review and meta-analysis of biological variation estimates for tumor markers

OBJECTIVES

Biological variation data (BV) can be used for different applications, but this depends on the availability of robust and relevant BV data. In this study, we aimed to summarize and appraise BV studies for tumor markers, to examine the influence of study population characteristics and concentrations on BV estimates and to discuss the applicability of BV data for tumor markers in clinical practice.

METHODS

Studies reporting BV data for tumor markers related to gastrointestinal, prostate, breast, ovarian, haematological, lung, and dermatological cancers were identified by a systematic literature search. Relevant studies were evaluated by the Biological Variation Data Critical Appraisal Checklist (BIVAC) and meta-analyses were performed for BIVAC compliant studies to deliver global estimates of within-subject (CV_I) and between-subject (CV_G) BV with 95% CI.

RESULTS

The systematic review identified 49 studies delivering results for 22 tumor markers; four papers received BIVAC grade A, 3 B, 27 C and 15 D. Out of these, 29 CV_I and 29 CV_G estimates met the criteria to be included in the meta-analysis. Robust data are lacking to conclude on the relationship between BV and different disease states and tumor marker concentrations.

CONCLUSIONS

This review identifies a lack of high-quality BV studies for many tumor markers and a need for delivery of BIVAC compliant studies, including in different, disease states and tumor marker concentrations. As of yet, the state-of-the-art may still be the most appropriate model to establish analytical performance specifications for the majority of tumor markers.

Performance of Platelet Counting in Thrombocytopenic Samples: Comparison between Mindray BC-6800Plus and Sysmex XN-9000

The performance of platelet (PLT) counting in thrombocytopenic samples is crucial for transfusion decisions. We compared PLT counting and its reproducibility between Mindray BC-6800Plus (BC-6800P, Mindray, Shenzhen, China) and Sysmex XN-9000 (XN, Sysmex, Kobe, Japan), especially focused on thrombocytopenic samples. We analyzed the correlation and agreement of PLT-I channels in both analyzers and BC-6800P PLT-O mode and XN PLT-F channel in 516 samples regarding PLT counts. Ten thrombocytopenic samples (≤2.0 × 10⁹/L by XN PLT-F) were measured 10 times to investigate the reproducibility with the desirable precision criterion, 7.6%. The correlation of BC-6800P PLT-I and XN PLT-I was arranged moderate to very high; but the correlation of BC-6800P PLT-O and XN PLT-F was arranged high to very high. Both BC-6800P PLT-I vs. XN PLT-I and BC-6800P PLT-O vs. XN PLT-F showed very good agreement (κ = 0.93 and κ = 0.94). In 41 discordant samples between BC-6800P PLT-O and XN PLT-F at transfusion thresholds, BC-6800P PLT-O showed higher PLT counts than XN-PLT-F, except the one case. BC-6800P PLT-O exceeded the precision criterion in one of 10 samples; but XN PLT-F exceeded it in six of 10 samples. BC-6800P would be a reliable option for PLT counting in thrombocytopenic samples with good reproducibility.

Annual biological variation and personalized reference intervals of clinical chemistry and hematology analytes

OBJECTIVES

A large number of people undergo annual health checkup but accurate laboratory criterion for evaluating their health status is limited. The present study determined annual biological variation (BV) and derived parameters of common laboratory analytes in order to accurately evaluate the test results of the annual healthcare population.

METHODS

A total of 43 healthy individuals who had regular healthcare once a year for six consecutive years, were enrolled using physical, electrocardiogram, ultrasonography and laboratory. The annual BV data and derived parameters, such as reference change value (RCV) and index of individuality (II) were calculated and compared with weekly data. We used annual BV and homeostatic set point to calculate personalized reference intervals (RI_per) which were compared with population-based reference intervals (RI_pop).

RESULTS

We have established the annual within-subject BV (CV_I), RCV, II, RI_per of 24 commonly used clinical chemistry and hematology analytes for healthy individuals. Among the 18 comparable measurands, CV_I estimates of annual data for 11 measurands were significantly higher than the weekly data. Approximately 50% measurands of II were <0.6, the utility of their RI_pop were limited. The distribution range of RI_per for most measurands only copied small part of RI_pop with reference range index for 8 measurands <0.5.

CONCLUSIONS

Compared with weekly BV, for annual healthcare individuals, annual BV and related parameters can provide more accurate evaluation of laboratory results. RI_per based on long-term BV data is very valuable for "personalized" diagnosis on annual health assessments.

Interindividual Biological Variability of Reticulocytes and Their Maturation Fractions in the Pediatric Population

OBJECTIVES

Because published data about the variability of reticulocyte counts in children are scarce, the interindividual biological variability of the automated reticulocyte count and its maturation fractions according to age and sex were analyzed.

METHODS

A retrospective, observational, analytical study was designed to establish and compare normal values of the automated reticulocyte count and its maturation fractions in different age and sex groups. The sample was drawn from results of CBC counts performed in children aged between 2 months and 18 years using an indirect sampling methodology.

RESULTS

A total of 9,362 CBC counts were analyzed. Automated reticulocyte count decreased between 2 months and 3 years of age and slowly increased thereafter, showing higher values in girls up to the age of 9 years, and equalized by sex thereafter. Immature reticulocyte fraction increased until 7 months of age; decreased progressively until 4 years of age; and then showed a discreet but constant rise, with significantly higher values in boys older than 1 year. The low-fluorescence fraction was relatively steady, with significantly higher values in girls aged 8 months and older.

CONCLUSIONS

The automated reticulocyte count and its maturation fractions show significant variations related to age and sex in pediatric patients.

Employing multiple synchronous outcome samples per subject to improve study efficiency

Background

Accuracy can be improved by taking multiple synchronous samples from each subject in a study to estimate the endpoint of interest if sample values are not highly correlated. If feasible, it is useful to assess the value of this cluster approach when planning studies. Multiple assessments may be the only method to increase power to an acceptable level if the number of subjects is limited.

Methods

The main aim is to estimate the difference in outcome between groups of subjects by taking one or more synchronous primary outcome samples or measurements. A summary statistic from multiple samples per subject will typically have a lower sampling error. The number of subjects can be balanced against the number of synchronous samples to minimize the sampling error, subject to design constraints. This approach can include estimating the optimum number of samples given the cost per subject and the cost per sample.

Results

The accuracy improvement achieved by taking multiple samples depends on the intra-class correlation (ICC). The lower the ICC, the greater the benefit that can accrue. If the ICC is high, then a second sample will provide little additional information about the subject’s true value. If the ICC is very low, adding a sample can be equivalent to adding an extra subject. Benefits of multiple samples include the ability to reduce the number of subjects in a study and increase both the power and the available alpha. If, for example, the ICC is 35%, adding a second measurement can be equivalent to adding 48% more subjects to a single measurement study.

Conclusion

A study’s design can sometimes be improved by taking multiple synchronous samples. It is useful to evaluate this strategy as an extension of a single sample design. An Excel workbook is provided to allow researchers to explore the most appropriate number of samples to take in a given setting.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01414-7.

Problems with estimating reference change values (critical differences)

The concept of reference change values (RCVs) for diagnosis and monitoring of diseases has become well established. Several models habe been developed, e. g. one assuming a normal distribution and another one for a log-normal distribution. RCV values calculated for some measurands with both models are compared with each other and led to similar results. A few examples led to RCV values which are not plausible for diagnostic purposes. Although statistical concepts of RCV values are well established, their clinical relevance remains questionable at least for some measurands. Studies with clinicians are required whether RCVs are of practical usefulness.

The importance of being external. methodological insights for the external validation of machine learning models in medicine

Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models.

METHODS

We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations.

RESULTS

We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results.

CONCLUSIONS

In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.

The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies

This editorial aims to contribute to the current debate about the quality of studies that apply machine learning (ML) methodologies to medical data to extract value from them and provide clinicians with viable and useful tools supporting everyday care practices. We propose a practical checklist to help authors to self assess the quality of their contribution and to help reviewers to recognize and appreciate high-quality medical ML studies by distinguishing them from the mere application of ML techniques to medical data.

Biological variation of serum neurofilament light chain

OBJECTIVES

The neurofilament light chain (NfL) has emerged as a versatile biomarker for CNS-diseases and is approaching clinical use. The observed changes in NfL levels are frequently of limited magnitude and in order to make clinical decisions based on NfL measurements, it is essential that biological variation is not confused with clinically relevant changes. The present study was designed to evaluate the biological variation of serum NfL.

METHODS

Apparently healthy individuals (n=33) were submitted to blood draws for three days in a row. On the second day, blood draws were performed every third hour for 12 h. NfL was quantified in serum using the Simoa™ HD-1 platform. The within-subject variation (CV_I) and between-subject variation (CV_G) were calculated using linear mixed-effects models.

RESULTS

The overall median value of NfL was 6.3 pg/mL (range 2.1-19.1). The CV_I was 3.1% and the CV_G was 35.6%. An increase in two serial measurements had to exceed 24.3% to be classified as significant at the 95% confidence level. Serum NfL levels remained stable during the day (p=0.40), whereas a minute variation (6.0-6.6 pg/mL) was observed from day-to-day (p=0.02).

CONCLUSIONS

Serum NfL is subject to tight homeostatic regulation with none or neglectable semidiurnal and day-to-day variation, but considerable between-subject variation exists. This emphasizes serum NfL as a well-suited biomarker for disease monitoring, but warrants caution when interpreting NfL levels in relation to reference intervals in a diagnosis setting. Furthermore, NfL's tight regulation requires that the analytical variation is kept at a minimum.

Biological variation: Understanding why it is so important?

This Review will describe the increasing importance of the concepts of biological variation to clinical chemists. The idea of comparison to 'reference' is fundamental in measurement. For the biological measurands, that reference is the relevant patient population, a clinical decision point based on a trial or an individual patient's previous results. The idea of using biological variation to set quality goals was then realised for setting Quality Control (QC) and External Quality Assurance (EQA) limits. The current phase of BV integration into practice is using Patient-Based Real-Time Quality Control (PBRTQC) and Patient Based Quality Assurance (PBQA) to detect a change in assay performance. The challenge of personalised medicine is to determine an individual reference interval. The Athletes Biological Passport may provide the solution.

Stability of hematological analytes during 48 hours storage at three temperatures using Cell-Dyn hematology analyzer

Background

The complete blood count (CBC) with differential leukocyte count (DLC) is one of the most common tests requested by physicians. The results of this test are affected by storage temperature and time of incubation. This study was designed to evaluate the stability of hematologic parameters in blood specimens stored for 48 h at three temperatures.

Methods

K2-EDTA - blood was collected from 22 healthy adults. The CBC was performed using a hematology analyser immediately; 0 time point and at 4, 8, 12, 16, 20, 24, and 48 h after storage at 4 °C, 10 °C or 23 °C. Changes in values of CBC parameters from the 0 time point were determined and reported as % of the initial value.

Results

Red blood cells, platelet, hemoglobin, and mean corpuscular hemoglobin were found stable during 48 h storage at 4 °C, 10 °C or 23 °C. Hematocrite and mean corpuscular volume increased, while white blood cells decreased at 48 h when stored at 23 °C. Lymphocytes, neutrophils, eosinophils, and basophils showed significant differences after 12 h of storage at 23 °C.

Conclusions

Red blood cells, platelet, hemoglobin, and mean corpuscular hemoglobin are the only suitable parameters without refrigeration during 24 h storage. When CBC and DLC are performed, 4 °C can be recommended as the most suitable storage temperature for 12 h storage.

Basophil counting in hematology analyzers: time to discontinue?

Basophils (basophilic granulocytes) are the least abundant cells in blood. Nowadays, basophils are included in the complete blood count performed by hematology analyzers and therefore reported in practically all patients in whom hematologic investigations are requested. However, hematology analyzers are not reliable enough to report clinically useful results. This is due to a combination of very high analytical imprecision and poor specificity, because the chemical and physical methods used for basophil counting in hematology analyzers are ill-defined and thus basophils are not well recognized by the analyzers. As a result, false basophil counts are quite common. In view of increasing analytical performance demands, hematology laboratories should stop reporting basophil counts produced by hematology analyzers. Suggestions for alternative pathways are presented for those situations where basophils are of clinical relevance.

Personalized Reference Intervals in Laboratory Medicine: A New Model Based on Within-Subject Biological Variation

BACKGROUND

The concept of personalized medicine has received widespread attention in the last decade. However, personalized medicine depends on correct diagnosis and monitoring of patients, for which personalized reference intervals for laboratory tests may be beneficial. In this study, we propose a simple model to generate personalized reference intervals based on historical, previously analyzed results, and data on analytical and within-subject biological variation.

METHODS

A model using estimates of analytical and within-subject biological variation and previous test results was developed. We modeled the effect of adding an increasing number of measurement results on the estimation of the personal reference interval. We then used laboratory test results from 784 adult patients (>18 years) considered to be in a steady-state condition to calculate personalized reference intervals for 27 commonly requested clinical chemistry and hematology measurands.

RESULTS

Increasing the number of measurements had little impact on the total variation around the true homeostatic set point and using ≥3 previous measurement results delivered robust personalized reference intervals. The personalized reference intervals of the study participants were different from one another and, as expected, located within the common reference interval. However, in general they made up only a small proportion of the population-based reference interval.

CONCLUSIONS

Our study shows that, if using results from patients in steady state, only a few previous test results and reliable estimates of within-subject biological variation are required to calculate personalized reference intervals. This may be highly valuable for diagnosing patients as well as for follow-up and treatment.

Evaluación crítica y meta-análisis de estudios de variación biológica para albúmina glicosilada, glucosa y HbA1c

Objetivos

A lo largo de los años se han publicado numerosos artículos sobre variación biológica (VB) de diferente calidad. Los objetivos de este trabajo fueron realizar una revisión sistemática y una evaluación crítica de los estudios de VB para albúmina glicosilada y proporcionar datos actualizados de VB para glucosa y HbA_1c, incluyendo prestigiosos estudios recientemente publicados como el Estudio de Variación Biológica Europea (EuBIVAS).

Métodos

Se hizo una búsqueda bibliográfica sistemática para identificar estudios sobre VB, encontrándose 9 estudios no incluidos en la primera revisión: 4 para albúmina glicosilada, 3 para glucosa y 3 para HbA_1c. Se realizó una evaluación crítica de los estudios relevantes, utilizando la herramienta Biological Variation Data Critical Appraisal Checklist (BIVAC). Se obtuvieron los estimados globales de VB mediante meta-análisis de los estudios que cumplían los requisitos BIVAC, realizados en individuos sanos con estudios de diseño similar.

Resultados

Un estudio recibió el grado A, dos el B y 6 el C. en la mayoría de los casos el grado C se asoció a deficiencias en el análisis estadístico de los datos. Los estimados de VB para albúmina glicosilada fueron: CV_I = 1,4%(1,2–2,1) y CV_G = 5,7%(4,7–10,6); para HbA_1c, CV_I = 1,2%(0,3–2,5), CV_G = 5,4%(3,3–7,3) y para glucosa, CV_I = 5,0%(4,1–12,0), CV_G = 8,1%(2,7–10,8) no difirieron de los estimados globales previamente descritos.

Conclusiones

La evaluación crítica y clasificación de los estudios de VB a tenor de su calidad metodológica, seguido de un meta-análisis, genera estimados de VB robustos y fiables. Este estudio proporciona datos de VB para albúmina glicolisada, glucosa y HbA_1c actualizados y basados en la evidencia científica.

Critical appraisal and meta-analysis of biological variation studies on glycosylated albumin, glucose and HbA1c

OBJECTIVES

Numerous biological variation (BV) studies have been performed over the years, but the quality of these studies vary. The objectives of this study were to perform a systematic review and critical appraisal of BV studies on glycosylated albumin and to deliver updated BV estimates for glucose and HbA_1c, including recently published high-quality studies such as the European Biological Variation study (EuBIVAS).

METHODS

Systematic literature searches were performed to identify BV studies. Nine publications not included in a previous review were identified; four for glycosylated albumin, three for glucose, and three for HbA_1c. Relevant studies were appraised by the Biological Variation Data Critical Appraisal Checklist (BIVAC). Global BV estimates were derived by meta-analysis of BIVAC-compliant studies in healthy subjects with similar study design.

RESULTS

One study received BIVAC grade A, 2B, and 6C. In most cases, the C-grade was associated with deficiencies in statistical analysis. BV estimates for glycosylated albumin were: CV_I=1.4% (1.2-2.1) and CV_G=5.7% (4.7-10.6), whereas estimates for HbA_1c, CV_I=1.2% (0.3-2.5), CV_G=5.4% (3.3-7.3), and glucose, CV_I=5.0% (4.1-12.0), CV_G=8.1% (2.7-10.8) did not differ from previously published global estimates.

CONCLUSIONS

The critical appraisal and rating of BV studies according to their methodological quality, followed by a meta-analysis, generate robust, and reliable BV estimates. This study delivers updated and evidence-based BV estimates for glycosylated albumin, glucose and HbA_1c.

The HemoScreen hematology point-of-care device is suitable for rapid evaluation of acute leukemia patients

BACKGROUND

Hematological patients, receiving intensive chemotherapy (predominantly acute leukemia patients), have repeated postchemotherapy periods with severe bone marrow suppression. As a result, these patients require regular monitoring of the complete blood counts (CBC) for optimal patient care. To reduce the strain on the patient, there is a need for a point-of-care (POC) hematology device that provides rapid and reliable results both in general and in cytopenic samples and is suitable for outpatient clinics. We evaluated the HemoScreen device for the most used CBC parameter both overall and at the lower range.

METHODS

The HemoScreen was compared with the Sysmex XN-9000 in 206 routine venous samples and 79 capillary bedside samples focusing on white blood cells (WBC), absolute neutrophil count (ANC), red blood cells (RBC), PLT and HGB.

RESULTS

The HemoScreen was less precise compared to the acceptance criteria set for larger and more advanced hematology instrument with a CV% 3.0-3.7 for WBCs, 3.6-8.4 for ANCs, 1.1-1.5 for RBCs, 2.5-4.4 for PLTs, and 1.7-2.3 for HGB. Correlation coefficient for all five parameters for the entire range was r >.95 and r >.90 at lower range for venous and capillary samples. Bias limits were within the CTCAE acceptance limits.

CONCLUSIONS

The HemoScreen provides rapid and accurate test results, for evaluation of WBC, PLT, and HGB, as well as at low concentrations for guiding transfusions and postchemotherapy treatment. The device is easy to operate and can measure both venous and capillary samples. Therefore, the HemoScreen is well suited for smaller outpatient clinics and potentially home use.

Breakfast can Affect Routine Hematology and Coagulation Laboratory Testing: An Evaluation on Behalf of COLABIOCLI WG-PRE-LATAM

Laboratories worldwide perform both hematological and coagulation testing on patients avoiding fasting time. In 2017, the Latin America Confederation of Clinical Biochemistry (COLABIOCLI) commissioned the Latin American Working Group for Preanalytical Phase (WG-PRE-LATAM) to study preanalytical variability and establish guidelines for preanalytical procedures to be applied by clinical laboratories and health care professionals. This study, on behalf of COLABIOCLI WG-PRE-LATAM, aims to evaluate the effect of the breakfast on routine hematology and coagulation laboratory testing. We studied 20 healthy volunteers who consumed a breakfast containing a standardized amount of carbohydrates, proteins, and lipids. We collected blood specimens for routine hematology and coagulation laboratory testing before breakfast and 1, 2, and 4 hours thereafter. Significant differences between samples were assessed by the Wilcoxon ranked-pairs test. Statistically significant differences ( p  < 0.05) between basal and 4 hours after the breakfast were observed for red blood cells, hemoglobin, hematocrit, mean corpuscular volume, white blood cells, neutrophils, lymphocytes, monocytes, mean platelet volume, and activated partial thromboplastin time. In conclusion, the significant variations observed in several hematological parameters, and activated partial thromboplastin time due to breakfast feeding demonstrate that the fasting time needs to be carefully considered prior to performing routine hematological and coagulation testing to avoid interpretive mistakes of test results, and to guarantee patient safety. Therefore, COLABIOCLI WG-PRE-LATAM encourages laboratory quality managers to standardize the fasting requirements in their laboratory, i.e., 12 hours.