Validation of an approach using only patient big data from clinical laboratories to establish reference intervals for thyroid hormones based on data mining

Should we depend on reference intervals from manufacturer package inserts? Comparing TSH and FT4 reference intervals from four manufacturers with results from modern indirect methods and the direct method

OBJECTIVES

Correct interpretation of thyroid function tests relies on correct reference intervals (RIs) for thyroid-stimulating hormone (TSH) and free thyroxine (FT4). ISO15189 mandates periodic verification of RIs, but laboratories struggle with cost-effective approaches. We investigated whether indirect methods (utilizing historical laboratory data) could replace the direct approach (utilizing healthy reference individuals) and compared results with manufacturer-provided RIs for TSH and FT4.

METHODS

We collected historical data (2008-2022) from 13 Dutch laboratories to re-establish RIs by employing indirect methods, TMC (for TSH) and refineR (for FT4). Laboratories used common automated platforms (Roche, Abbott, Beckman or Siemens). Indirect RIs (IRIs) were determined per laboratory per year and clustered per manufacturer (>1.000.000 data points per manufacturer). Direct RIs (DRIs) were established in 125 healthy individuals per platform.

RESULTS

TSH IRIs remained robust over the years for all manufacturers. FT4 IRIs proved robust for three manufacturers (Roche, Beckman and Siemens), but the IRI upper reference limit (URL) of Abbott showed a decrease of 2 pmol/L from 2015. Comparison of the IRIs and DRIs for TSH and FT4 showed close agreement using adequate age-stratification. Manufacturer-provided RIs, notably Abbott, Roche and Beckman exhibited inappropriate URLs (overall difference of 0.5-1.0 µIU/mL) for TSH. For FT4, the URLs provided by Roche, Abbott and Siemens were overestimated by 1.5-3.5 pmol/L.

CONCLUSIONS

These results underscore the importance of RI verification as manufacturer-provided RIs are often incorrect and RIs may not be robust. Indirect methods offer cost-effective alternatives for laboratory-specific or platform-specific verification of RIs.

Development of next-generation reference interval models to establish reference intervals based on medical data: current status, algorithms and future consideration

Evidence derived from laboratory medicine plays a pivotal role in the diagnosis, treatment monitoring, and prognosis of various diseases. Reference intervals (RIs) are indispensable tools for assessing test results. The accuracy of clinical decision-making relies directly on the appropriateness of RIs. With the increase in real-world studies and advances in computational power, there has been increased interest in establishing RIs using big data. This approach has demonstrated cost-effectiveness and applicability across diverse scenarios, thereby enhancing the overall suitability of the RI to a certain extent. However, challenges persist when tests results are influenced by age and sex. Reliance on a single RI or a grouping of RIs based on age and sex can lead to erroneous interpretation of results with significant implications for clinical decision-making. To address this issue, the development of next generation of reference interval models has arisen at an historic moment. Such models establish a curve relationship to derive continuously changing reference intervals for test results across different age and sex categories. By automatically selecting appropriate RIs based on the age and sex of patients during result interpretation, this approach facilitates clinical decision-making and enhances disease diagnosis/treatment as well as health management practices. Development of next-generation reference interval models use direct or indirect sampling techniques to select reference individuals and then employed curve fitting methods such as splines, polynomial regression and others to establish continuous models. In light of these studies, several observations can be made: Firstly, to date, limited interest has been shown in developing next-generation reference interval models, with only a few models currently available. Secondly, there are a wide range of methods and algorithms for constructing such models, and their diversity may lead to confusion. Thirdly, the process of constructing next-generation reference interval models can be complex, particularly when employing indirect sampling techniques. At present, normative documents pertaining to the development of next-generation reference interval models are lacking. In summary, this review aims to provide an overview of the current state of development of next-generation reference interval models by defining them, highlighting inherent advantages, and addressing existing challenges. It also describes the process, advanced algorithms for model building, the tools required and the diagnosis and validation of models. Additionally, a discussion on the prospects of utilizing big data for developing next-generation reference interval models is presented. The ultimate objective is to equip clinical laboratories with the theoretical framework and practical tools necessary for developing and optimizing next-generation reference interval models to establish next-generation reference intervals while enhancing the use of medical data resources to facilitate precision medicine.

Effects of Using Different Indirect Techniques on the Calculation of Reference Intervals: Observational Study

BACKGROUND

Reference intervals (RIs) play an important role in clinical decision-making. However, due to the time, labor, and financial costs involved in establishing RIs using direct means, the use of indirect methods, based on big data previously obtained from clinical laboratories, is getting increasing attention. Different indirect techniques combined with different data transformation methods and outlier removal might cause differences in the calculation of RIs. However, there are few systematic evaluations of this.

OBJECTIVE

This study used data derived from direct methods as reference standards and evaluated the accuracy of combinations of different data transformation, outlier removal, and indirect techniques in establishing complete blood count (CBC) RIs for large-scale data.

METHODS

The CBC data of populations aged ≥18 years undergoing physical examination from January 2010 to December 2011 were retrieved from the First Affiliated Hospital of China Medical University in northern China. After exclusion of repeated individuals, we performed parametric, nonparametric, Hoffmann, Bhattacharya, and truncation points and Kolmogorov-Smirnov distance (kosmic) indirect methods, combined with log or BoxCox transformation, and Reed-Dixon, Tukey, and iterative mean (3SD) outlier removal methods in order to derive the RIs of 8 CBC parameters and compared the results with those directly and previously established. Furthermore, bias ratios (BRs) were calculated to assess which combination of indirect technique, data transformation pattern, and outlier removal method is preferrable.

RESULTS

Raw data showed that the degrees of skewness of the white blood cell (WBC) count, platelet (PLT) count, mean corpuscular hemoglobin (MCH), mean corpuscular hemoglobin concentration (MCHC), and mean corpuscular volume (MCV) were much more obvious than those of other CBC parameters. After log or BoxCox transformation combined with Tukey or iterative mean (3SD) processing, the distribution types of these data were close to Gaussian distribution. Tukey-based outlier removal yielded the maximum number of outliers. The lower-limit bias of WBC (male), PLT (male), hemoglobin (HGB; male), MCH (male/female), and MCV (female) was greater than that of the corresponding upper limit for more than half of 30 indirect methods. Computational indirect choices of CBC parameters for males and females were inconsistent. The RIs of MCHC established by the direct method for females were narrow. For this, the kosmic method was markedly superior, which contrasted with the RI calculation of CBC parameters with high |BR| qualification rates for males. Among the top 10 methodologies for the WBC count, PLT count, HGB, MCV, and MCHC with a high-BR qualification rate among males, the Bhattacharya, Hoffmann, and parametric methods were superior to the other 2 indirect methods.

CONCLUSIONS

Compared to results derived by the direct method, outlier removal methods and indirect techniques markedly influence the final RIs, whereas data transformation has negligible effects, except for obviously skewed data. Specifically, the outlier removal efficiency of Tukey and iterative mean (3SD) methods is almost equivalent. Furthermore, the choice of indirect techniques depends more on the characteristics of the studied analyte itself. This study provides scientific evidence for clinical laboratories to use their previous data sets to establish RIs.

Utilization of five data mining algorithms combined with simplified preprocessing to establish reference intervals of thyroid-related hormones for non-elderly adults

BACKGROUND

Despite the extensive research on data mining algorithms, there is still a lack of a standard protocol to evaluate the performance of the existing algorithms. Therefore, the study aims to provide a novel procedure that combines data mining algorithms and simplified preprocessing to establish reference intervals (RIs), with the performance of five algorithms assessed objectively as well.

METHODS

Two data sets were derived from the population undergoing a physical examination. Hoffmann, Bhattacharya, Expectation Maximum (EM), kosmic, and refineR algorithms combined with two-step data preprocessing respectively were implemented in the Test data set to establish RIs for thyroid-related hormones. Algorithm-calculated RIs were compared with the standard RIs calculated from the Reference data set in which reference individuals were selected following strict inclusion and exclusion criteria. Objective assessment of the methods is implemented by the bias ratio (BR) matrix.

RESULTS

RIs of thyroid-related hormones are established. There is a high consistency between TSH RIs established by the EM algorithm and the standard TSH RIs (BR = 0.063), although EM algorithms seems to perform poor on other hormones. RIs calculated by Hoffmann, Bhattacharya, and refineR methods for free and total triiodo-thyronine, free and total thyroxine respectively are close and match the standard RIs.

CONCLUSION

An effective approach for objectively evaluating the performance of the algorithm based on the BR matrix is established. EM algorithm combined with simplified preprocessing can handle data with significant skewness, but its performance is limited in other scenarios. The other four algorithms perform well for data with Gaussian or near-Gaussian distribution. Using the appropriate algorithm based on the data distribution characteristics is recommended.

Personalized reference intervals: from theory to practice

Using laboratory test results for diagnosis and monitoring requires a reliable reference to which the results can be compared. Currently, most reference data is derived from the population, and patients in this context are considered members of a population group rather than individuals. However, such reference data has limitations when used as the reference for an individual. A patient's test results preferably should be compared with their own, individualized reference intervals (RI), i.e. a personalized RI (prRI).The prRI is based on the homeostatic model and can be calculated using an individual's previous test results obtained in a steady-state situation and estimates of analytical (CV_A) and biological variation (BV). BV used to calculate the prRI can be obtained from the population (within-subject biological variation, CV_I) or an individual's own data (within-person biological variation, CV_P). Statistically, the prediction interval provides a useful tool to calculate the interval (i.e. prRI) for future observation based on previous measurements. With the development of information technology, the data of millions of patients is stored and processed in medical laboratories, allowing the implementation of personalized laboratory medicine. PrRI for each individual should be made available as part of the laboratory information system and should be continually updated as new test results become available.In this review, we summarize the limitations of population-based RI for the diagnosis and monitoring of disease, provide an outline of the prRI concept and different approaches to its determination, including statistical considerations for deriving prRI, and discuss aspects which must be further investigated prior to implementation of prRI in clinical practice.

An innovative approach based on real-world big data mining for calculating the sample size of the reference interval established using transformed parametric and non-parametric methods

Background

Currently, the direct method is the main approach for establishment of reference interval (RI). However, only a handful of studies have described the effects of sample size on establishment of RI and estimation of sample size. We describe a novel approach for estimation of the sample size when establishing RIs using the transformed parametric and non-parametric methods.

Methods

A total of 3,697 healthy participants were enrolled in this study. We adopted a two-layer nested loop sample size estimation method to determine the effects of sample size on RI, using thyroid-related hormone as an example. The sample size was selected as the calculation result when the width of the confidence interval (CI) of the upper and lower limit of the RI were both stably < 0.2 times the width of RI. Then, we calculated the sample size for establishing RIs via transformed parametric and non-parametric methods for thyroid-related hormones.

Results

Sample sizes for thyroid stimulating hormone (TSH), as required by parametric and non-parametric methods to establish RIs were 239 and 850, respectively. Sample sizes required by the transformed parametric method for free triiodothyronine (FT3), free thyroxine (FT4), total triiodothyronine (TT3) and total thyroxine (TT4) were all less than 120, while those required by the non-parametric method were more than 120.

Conclusion

We describe a novel approach for estimating sample sizes for establishment of RI. A corresponding open-source code has been developed and is available for applications. The established method is suitable for most analytes, with evidence based on thyroid-related hormones indicating that different sample sizes are required to establish RIs using different methods for analytes with different variations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01751-1.

Artificial intelligence and thyroid disease management

Artificial intelligence (AI) is transforming healthcare and offers new tools in clinical research, personalized medicine, and medical diagnostics. Thyroid function tests represent an important asset for physicians in the diagnosis and monitoring of pathologies. Artificial intelligence tools can clearly assist physicians and specialists in laboratory medicine to optimize test prescription, tests interpretation, decision making, process optimization, and assay design. Our article is reviewing several of these aspects. As thyroid AI models rely on large data sets, which often requires distributed learning from multi-center contributions, this article also briefly discusses this issue.

Establishment of Reference Intervals for Thyroid-Associated Hormones Using refineR Algorithm in Chinese Population at High-Altitude Areas

Objectives

Diagnosis of thyroid disease among individuals dwelling at high altitude remains a challenge. Reference intervals (RIs) for thyroid-associated hormones among Tibetans living at various high altitudes were established to improve diagnosis.

Methods

One thousand two hundred eighty-one subjects were randomly recruited from Nyingchi, Shigatse/Lhasa, and Ali of Tibet. Thyroid-stimulating hormone (TSH), free triiodothyronine (FT3), and free thyroxine (FT4) were measured by the Cobas e601 electrochemiluminescence analyzer. We used multiple linear regression and variance component analysis to assess the effect of sex, age, and altitude on hormones. RIs were established by refineR algorithm and compared with those provided by the manufacturer.

Results

Serum TSH was significantly lower in males than in females, while FT3 and FT4 were higher in males. Both FT3 and FT4 decreased with increasing age. FT3 increased with altitude, while TSH and FT4 were less influenced by altitude. The RI for TSH was 0.764–5.784 μIU/ml, while for FT4, the RIs were 12.36–19.38 pmol/L in females and 14.84–20.18 pmol/L in males. The RIs for FT3 at Nyingchi, Shigatse/Lhasa, and Ali in females were 4.09–4.98, 4.31–5.45, and 4.82–5.58 pmol/L, while in males, the values were 4.82–5.41, 4.88–5.95, and 5.26–6.06 pmol/L, respectively. The obtained RIs for TSH and FT4 were generally higher, while that for FT3 was narrower than the RIs provided by Cobas.

Conclusions

Specific RIs were established for thyroid-associated hormones among Tibetans, which were significantly different from those provided by the manufacturer.

Big data and reference intervals

Although reference intervals (RIs) play an important role in clinical diagnosis, there remain significant differences with respect to race, gender, age and geographic location. Accordingly, the Clinical Laboratory Standards Institute (CLSI) EP28-A3c has recommended that clinical laboratories establish RIs appropriate to their subject population. Unfortunately, the traditional and direct approach to establish RIs relies on the recruitment of a sufficient number of healthy individuals of various age groups, collection and testing of large numbers of specimens and accurate data interpretation. The advent of the big data era has, however, created a unique opportunity to "mine" laboratory information. Unfortunately, this indirect method lacks standardization, consensus support and CLSI guidance. In this review we provide a historical perspective, comprehensively assess data processing and statistical methods, and post-verification analysis to validate this big data approach in establishing laboratory specific RIs.

Disease evolution and risk-based disease trajectories in congestive heart failure patients

Congestive Heart Failure (CHF) is among the most prevalent chronic diseases worldwide, and is commonly associated with comorbidities and complex health conditions. Consequently, CHF patients are typically hospitalized frequently, and are at a high risk of premature death. Early detection of an envisaged patient disease trajectory is crucial for precision medicine. However, despite the abundance of patient-level data, cardiologists currently struggle to identify disease trajectories and track the evolution patterns of the disease over time, especially in small groups of patients with specific disease subtypes. The present study proposed a five-step method that allows clustering CHF patients, detecting cluster similarity, and identifying disease trajectories, and promises to overcome the existing difficulties. This work is based on a rich dataset of patients' records spanning ten years of hospital visits. The dataset contains all the health information documented in the hospital during each visit, including diagnoses, lab results, clinical data, and demographics. It utilizes an innovative Cluster Evolution Analysis (CEA) method to analyze the complex CHF population where each subject is potentially associated with numerous variables. We have defined sub-groups for mortality risk levels, which we used to characterize patients' disease evolution by refined data clustering in three points in time over ten years, and generating patients' migration patterns across periods. The results elicited 18, 23, and 25 clusters respective to the first, second, and third visits, uncovering clinically interesting small sub-groups of patients. In the following post-processing stage, we identified meaningful patterns. The analysis yielded fine-grained patient clusters divided into several finite risk levels, including several small-sized groups of high-risk patients. Significantly, the analysis also yielded longitudinal patterns where patients' risk levels changed over time. Four types of disease trajectories were identified: decline, preserved state, improvement, and mixed-progress. This stage is a unique contribution of the work. The resulting fine partitioning and longitudinal insights promise to significantly assist cardiologists in tailoring personalized interventions to improve care quality. Cardiologists could utilize these results to glean previously undetected relationships between symptoms and disease evolution that would allow a more informed clinical decision-making and effective interventions.

Development of nation-wide reference intervals using an indirect method and harmonized assays

OBJECTIVES

For many years, clinical laboratories have either verified or estimated reference intervals (RI) for laboratory tests. Those calculations have largely been performed by direct sampling analysis of ostensibly healthy individuals or by post-analysis biochemical screening. Recently however, indirect calculations have come to the forefront as an IFCC endorsed method by using normal and abnormal patient data.

DESIGN AND METHODS

Using a large database of patient test results from Laboratory Corporation of America, age and gender based RIs, inclusive of neonatal, pediatric, and geriatric populations, were determined using a modified indirect method of Hoffmann, and represent a diverse population distributed across the United States from a nation-wide system of laboratories and is unbiased with respect to age, gender, race or geography.

RESULTS

The tabulation of RIs using big data by an indirect method represent 72 M patient test results. The table includes 266 individual analytes consisting of approximately 2,700 age categories, including tests across multiple medical disciplines.

CONCLUSIONS

To our knowledge, this is the largest collection of RIs that were calculated by an indirect method representing clinical chemistry, endocrinology, coagulation, and hematology analytes that have been derived with very powerful "Ns" for each age bracket. This process provides more robust RIs and allows for the determination of pediatric and geriatric RIs that would otherwise be difficult to obtain using traditional direct RI determinations.

Estimation of change limits (deltacheck) in clinical laboratory

Objectives

Change limits, more commonly called delta check, are those in which a change in a patient's measured result in relation to their corresponding preceding measurement is suspected of being erroneous and should be considered as a doubtful result. The aim of this study was to provide change limits for some biochemical and haematological quantities to detect doubtful measured results and to assess its effectiveness to detect erroneous results for their application in and the standardization of the plausibility control.

Methods

Change limits have been estimated for 13 biochemical and 6 haematological quantities. For each quantity, relative differences (D), expressed as a percentage between the two consecutive measured results from the same patient (from scheduled laboratory requests), were calculated. From these differences (D), the p5 and p95 percentiles of the data distribution were calculated. To assess the effectiveness of the change limits to detect laboratory errors, 43 erroneous laboratory reports, containing different biochemical and haematological quantities, were obtained from the standard laboratory plausibility control procedure.

Results

From the 43 erroneous laboratory reports, 31 (72%) were due to endovenous administration errors and 12 (28%) were due to mislabeling errors. All erroneous laboratory reports were detected when the change limits of the quantities were combined and applied together.

Conclusions

The best combination of quantities, which detect all the erroneous reports in the same specimen were: potassium, albumin, creatinine, glucose and haemoglobin.

Effect of sample size and the traditional parametric, nonparametric, and robust methods on the establishment of reference intervals: Evidence from real world data

Sample size and statistical methods are critical for establishing reference intervals (RIs) but they tend to be overlooked. In this study, we used R (3.6.3) to stratify the reference individuals by sex, and then stratified them using the random sampling method. Fourteen sub-data sets with a sample size of 40, 80, 120, 160, 200, 500, 800, 1000, 1500, 2000, 2500, 3000, 3500, and 4000 were extracted, respectively. The sex ratios of all sub-data sets were 1:1. Transformed parametric (using log transformation), nonparametric, and robust approaches as described in the Clinical and Laboratory Standards Institute guidelines were adopted to establish the RIs and the 90% confidence interval of the thyroid-stimulating hormone (TSH) using data from the sub-data sets. The Bland-Altman plot was used to evaluate the consistency of the upper and lower limits of the RIs established using the three methods. The upper and lower limits of TSH RI tended to be stable starting from the data set with a sample size of 1500. The RIs established using the three methods were more consistent when using a sample size greater than or equal to 2000.

Potential value and impact of data mining and machine learning in clinical diagnostics

Data mining involves the use of mathematical sciences, statistics, artificial intelligence, and machine learning to determine the relationships between variables from a large sample of data. It has previously been shown that data mining can improve the prediction and diagnostic precision of type 2 diabetes mellitus. A few studies have applied machine learning to assess hypertension and metabolic syndrome-related biomarkers, as well as refine the assessment of cardiovascular disease risk. Machine learning methods have also been applied to assess new biomarkers and survival outcomes in patients with renal diseases to predict the development of chronic kidney disease, disease progression, and renal graft survival. In the latter, random forest methods were found to be the best for the prediction of chronic kidney disease. Some studies have investigated the prognosis of nonalcoholic fatty liver disease and acute liver failure, as well as therapy response prediction in patients with viral disorders, using decision tree models. Machine learning techniques, such as Sparse High-Order Interaction Model with Rejection Option, have been used for diagnosing Alzheimer's disease. Data mining techniques have also been applied to identify the risk factors for serious mental illness, such as depression and dementia, and help to diagnose and predict the quality of life of such patients. In relation to child health, some studies have determined the best algorithms for predicting obesity and malnutrition. Machine learning has determined the important risk factors for preterm birth and low birth weight. Published studies of patients with cancer and bacterial diseases are limited and should perhaps be addressed more comprehensively in future studies. Herein, we provide an in-depth review of studies in which biochemical biomarker data were analyzed using machine learning methods to assess the risk of several common diseases, in order to summarize the potential applications of data mining methods in clinical diagnosis. Data mining techniques have now been increasingly applied to clinical diagnostics, and they have the potential to support this field.