Simulation study to demonstrate biases created by diagnostic criteria of mental illnesses: major depressive episodes, dysthymia, and manic episodes

Identifying Depression Through Machine Learning Analysis of Omics Data: Scoping Review

BACKGROUND

Depression is one of the most common mental disorders that affects >300 million people worldwide. There is a shortage of providers trained in the provision of mental health care, and the nursing workforce is essential in filling this gap. The diagnosis of depression relies heavily on self-reported symptoms and clinical interviews, which are subject to implicit biases. The omics methods, including genomics, transcriptomics, epigenomics, and microbiomics, are novel methods for identifying the biological underpinnings of depression. Machine learning is used to analyze genomic data that includes large, heterogeneous, and multidimensional data sets.

OBJECTIVE

This scoping review aims to review the existing literature on machine learning methods for omics data analysis to identify individuals with depression, with the goal of providing insight into alternative objective and driven insights into the diagnostic process for depression.

METHODS

This scoping review was reported following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. Searches were conducted in 3 databases to identify relevant publications. A total of 3 independent researchers performed screening, and discrepancies were resolved by consensus. Critical appraisal was performed using the Joanna Briggs Institute Critical Appraisal Checklist for Analytical Cross-Sectional Studies.

RESULTS

The screening process identified 15 relevant papers. The omics methods included genomics, transcriptomics, epigenomics, multiomics, and microbiomics, and machine learning methods included random forest, support vector machine, k-nearest neighbor, and artificial neural network.

CONCLUSIONS

The findings of this scoping review indicate that the omics methods had similar performance in identifying omics variants associated with depression. All machine learning methods performed well based on their performance metrics. When variants in omics data are associated with an increased risk of depression, the important next step is for clinicians, especially nurses, to assess individuals for symptoms of depression and provide a diagnosis and any necessary treatment.

Translating Risk Ratios, Baseline Incidence, and Proportions Diseased to Correlations and Chi-Squared Statistics: Simulation Epidemiology

Background In a population, when a disease is causing a symptom, the overall symptom incidence can be determined by proportions diseased, baseline symptom incidence, and risk ratios of developing the symptom due to the disease. There are various measures of association, including risk ratios. How risk ratios are linked to other measures of association, such as correlation coefficients and chi-squared statistics, has not been explicitly discussed. This study aims to demonstrate their connection via equations and simulations, assuming one disease causes symptoms. Methods The equations for correlation coefficients and chi-square statistics were rewritten using epidemiological measures: proportions diseased, baseline symptom incidence, and risk ratios. Simulations were conducted to test the accuracy of the equations. The baseline symptom incidence and the proportions diseased were assumed to be 0.05, 0.1, 0.2, 0.4, or 0.8. The risk ratios were assumed to be 0.5, 1, 2, 5, 10, and 25. Another disease that correlates with this disease was created (correlation = 0, 0.3, or 0.7). For each combination of symptom incidence, proportions diseased, risk ratios, and between-disease correlations, 10,000 subjects were simulated. The correlation coefficients and chi-squared statistics were approximated with epidemiologic measures and their interaction terms. R-squared was used to assess the importance of the epidemiologic measures. Results In the simulations, the overall symptom incidence, correlation coefficients, and chi-squared statistics between the disease and symptoms could be fully explained by the epidemiologic measures in the equations (R-squared = 1). When approximating correlation coefficients and chi-squared statistics with individual measures or their interaction terms, the importance of these measures depended on whether the at-risk incidence reached 1 or not. The numbers in the four cells in the contingency table predicted correlation coefficients, or chi-squared statistics, with different R-squared. Conclusion To our knowledge, this is the first study to translate the three epidemiologic measures (risk ratios, baseline symptom incidence, and proportions diseased) into correlation coefficients and chi-squared statistics. However, chi-squared statistics also depend on sample sizes. This study also provides a platform for developing teaching cases for students to investigate the causal relationship between diseases and symptoms or exposure and outcomes.

Mental Illness Diagnoses May Not Cause All Mental Symptoms: A Simulation Study for Major Depressive Episodes, Dysthymic Disorder, and Manic Episodes

Objectives This study aims to understand the statistical significance of the associations between diagnoses and symptoms based on simulations that have been used to understand the interpretability of mental illness diagnoses. Methods The symptoms for the diagnosis of major depressive episodes, dysthymic disorder, and manic episodes were extracted from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR, American Psychiatric Association, Philadelphia, Pennsylvania). Without real-world symptom data, we simulated populations using various combinations of symptom prevalence and correlations. Assuming symptoms occurred with similar prevalence and correlations, for each combination of symptom prevalence (0.05, 0.1, 0.3, 0.5, and 0.7) and correlation (0, 0.1, 0.4, 0.7, and 0.9), 100 cohorts with 10,000 individuals were randomly created. Diagnoses were made according to the DSM-IV-TR criteria. The associations between the diagnoses and their input symptoms were quantified with odds ratios and correlation coefficients. P-values from 100 cohorts for each combination of symptom prevalence and correlation were summarized. Results Three mental illness diagnoses were not significantly correlated with their own symptoms in all simulations, particularly when symptoms were not correlated, except for the symptom in the major criteria of major depressive episodes or dysthymic disorder. The symptoms for the diagnosis of major depressive episodes and dysthymic disorder were significantly correlated with these two diagnoses in some simulations, assuming 0.1, 0.4, 0.7, or 0.9 symptom correlations, except for one symptom. The overlap in the input symptoms for the diagnosis of major depressive episodes and dysthymic disorder also leads to significant correlations between these two diagnoses, assuming 0.1, 0.4, 0.7, and 0.9 correlations between input symptoms. Manic episodes are not significantly associated with the input symptoms of major depressive episodes and dysthymic disorder. Conclusion There are challenges to establish the causation between psychiatric symptoms and mental illness diagnoses. There is insufficient prevalence and incidence data to show all psychiatric symptoms exist or can be observed in patients. The diagnostic accuracy of symptoms to detect a disease cause is far from perfect. Assuming the symptoms of three mood disorders may present in patients, three diagnoses are not significantly associated with all psychiatric symptoms used to diagnose them. The diagnostic criteria of the three diagnoses have not been designed to guarantee significant associations between symptoms and diagnoses. Because statistical associations are important for making causal inferences, there may be a lack of causation between diagnoses and symptoms. Previous research has identified factors that lead to insignificant associations between diagnoses and symptoms, including biases due to data processing and a lack of epidemiological evidence to support the design of mental illness diagnostic criteria.

The Upper Limits of Risk Ratios and Recommendations for Reporting Risk Ratios, Odds Ratios, and Rate Ratios

Background Relative measures, including risk ratios (RRs) and odds ratios (ORs), are reported in many epidemiological studies. RRs represent how many times a condition is likely to develop when exposed to a risk factor. The upper limit of RRs is the multiplicative inverse of the baseline incidence. Ignoring the upper limits of RRs can lead to reporting exaggerated relative effect sizes. Objectives This study aims to demonstrate the importance of such upper limits for effect size reporting via equations, examples, and simulations and provide recommendations for the reporting of relative measures. Methods Equations to calculate RRs and their 95% confidence intervals (CIs) were listed. We performed simulations with 10,000 simulated subjects and three population variables: proportions at risk (0.05, 0.1, 0.3, 0.5, and 0.8), baseline incidence (0.05, 0.1, 0.3, 0.5, and 0.8), and RRs (0.5, 1.0, 5.0, 10.0, and 25.0). Subjects were randomly assigned with a risk based on the set of proportions-at-risk values. A disease occurred based on the baseline incidence among those not at risk. The incidence of those at risk was the product of the baseline incidence and the RRs. The 95% CIs of RRs were calculated according to Altman. Results The calculation of RR 95% CIs is not connected to the RR upper limits in equations. The RRs in the simulated populations at risk could reach the upper limits of RRs: multiplicative inverse of the baseline incidence. The upper limits to the derived RRs were around 1.25, 2, 3.3, 10, and 20, when the assumed baseline incidence rates were 0.8, 0.5, 0.3, 0.2, and 0.05, respectively. We demonstrated five scenarios in which the RR 95% CIs might exceed the upper limits. Conclusions Statistical significance does not imply the RR 95% CIs not exceeding the upper limits of RRs. When reporting RRs or ORs, the RR upper limits should be assessed. The rate ratio is also subject to a similar upper limit. In the literature, ORs tend to overestimate effect sizes. It is recommended to correct ORs that aim to approximate RRs assuming outcomes are rare. A reporting guide for relative measures, RRs, ORs, and rate ratios, is provided. Researchers are recommended to report whether the 95% CIs of relative measures, RRs, ORs, and rate ratios, overlap with the range of upper limits and discuss whether the relative measure estimates may exceed the upper limits.

Critical Hierarchical Appraisal and repOrting tool for composite measureS (CHAOS)

Background Composite measures are often used to represent certain concepts that cannot be measured with single variables and can be used as diagnoses, prognostic factors, or outcomes in clinical or health research. For example, frailty is a diagnosis confirmed based on the number of age-related symptoms and has been used to predict major health outcomes. However, undeclared assumptions and problems are prevalent among composite measures. Thus, we aim to propose a reporting guide and an appraisal tool for identifying these assumptions and problems. Methods We developed this reporting and assessment tool based on evidence and the consensus of experts pioneering research on index mining and syndrome mining. We designed a development framework for composite measures and then tested and revised it based on several composite measures commonly used in medical research, such as frailty, body mass index (BMI), mental illness diagnoses, and innovative indices mined for mortality prediction. We extracted review questions and reporting items from various issues identified by the development framework. This panel reviewed the identified issues, considered other aspects that might have been neglected in previous studies, and reached a consensus on the questions to be used by the reporting and assessment tool. Results We selected 19 questions in seven domains for reporting or critical assessment. Each domain contains review questions for authors and readers to critically evaluate the interpretability and validity of composite measures, which include candidate variable selection, variable inclusion and assumption declaration, data processing, weighting scheme, methods to aggregate information, composite measure interpretation and justification, and recommendations on the use. Conclusions For all seven domains, interpretability is central with respect to composite measures. Variable inclusion and assumptions are important clues to show the connection between composite measures and their theories. This tool can help researchers and readers understand the appropriateness of composite measures by exploring various issues. We recommend using this Critical Hierarchical Appraisal and repOrting tool for composite measureS (CHAOS) along with other critical appraisal tools to evaluate study design or risk of bias.

Frailty does not cause all frail symptoms: United States Health and Retirement Study

BACKGROUND

Frailty is associated with major health outcomes. However, the relationships between frailty and frailty symptoms haven't been well studied. This study aims to show the associations between frailty and frailty symptoms.

METHODS

The Health and Retirement Study (HRS) is an ongoing longitudinal biannual survey in the United States. Three of the most used frailty diagnoses, defined by the Functional Domains Model, the Burden Model, and the Biologic Syndrome Model, were reproduced according to previous studies. The associations between frailty statuses and input symptoms were assessed using odds ratios and correlation coefficients.

RESULTS

The sample sizes, mean ages, and frailty prevalence matched those reported in previous studies. Frailty statuses were weakly correlated with each other (coefficients = 0.19 to 0.38, p < 0.001 for all). There were 49 input symptoms identified by these three models. Frailty statuses defined by the three models were not significantly correlated with one or two symptoms defined by the same models (p > 0.05 for all). One to six symptoms defined by the other two models were not significantly correlated with each of the three frailty statuses (p > 0.05 for all). Frailty statuses were significantly correlated with their own bias variables (p < 0.05 for all).

CONCLUSION

Frailty diagnoses lack significant correlations with some of their own frailty symptoms and some of the frailty symptoms defined by the other two models. This finding raises questions like whether the frailty symptoms lacking significant correlations with frailty statuses could be included to diagnose frailty and whether frailty exists and causes frailty symptoms.

Diagnostic accuracy of symptoms for an underlying disease: a simulation study

Symptoms have been used to diagnose conditions such as frailty and mental illnesses. However, the diagnostic accuracy of the numbers of symptoms has not been well studied. This study aims to use equations and simulations to demonstrate how the factors that determine symptom incidence influence symptoms' diagnostic accuracy for disease diagnosis. Assuming a disease causing symptoms and correlated with the other disease in 10,000 simulated subjects, 40 symptoms occurred based on 3 epidemiological measures: proportions diseased, baseline symptom incidence (among those not diseased), and risk ratios. Symptoms occurred with similar correlation coefficients. The sensitivities and specificities of single symptoms for disease diagnosis were exhibited as equations using the three epidemiological measures and approximated using linear regression in simulated populations. The areas under curves (AUCs) of the receiver operating characteristic (ROC) curves was the measure to determine the diagnostic accuracy of multiple symptoms, derived by using 2 to 40 symptoms for disease diagnosis. With respect to each AUC, the best set of sensitivity and specificity, whose difference with 1 in the absolute value was maximal, was chosen. The results showed sensitivities and specificities of single symptoms for disease diagnosis were fully explained with the three epidemiological measures in simulated subjects. The AUCs increased or decreased with more symptoms used for disease diagnosis, when the risk ratios were greater or less than 1, respectively. Based on the AUCs, with risk ratios were similar to 1, symptoms did not provide diagnostic values. When risk ratios were greater or less than 1, maximal or minimal AUCs usually could be reached with less than 30 symptoms. The maximal AUCs and their best sets of sensitivities and specificities could be well approximated with the three epidemiological and interaction terms, adjusted R-squared ≥ 0.69. However, the observed overall symptom correlations, overall symptom incidence, and numbers of symptoms explained a small fraction of the AUC variances, adjusted R-squared ≤ 0.03. In conclusion, the sensitivities and specificities of single symptoms for disease diagnosis can be explained fully by the at-risk incidence and the 1 minus baseline incidence, respectively. The epidemiological measures and baseline symptom correlations can explain large fractions of the variances of the maximal AUCs and the best sets of sensitivities and specificities. These findings are important for researchers who want to assess the diagnostic accuracy of composite diagnostic criteria.

Why Mental Illness Diagnoses Are Wrong: A Pilot Study on the Perspectives of the Public

BACKGROUND

Mental illness diagnostic criteria are made based on assumptions. This pilot study aims to assess the public's perspectives on mental illness diagnoses and these assumptions.

METHODS

An anonymous survey with 30 questions was made available online in 2021. Participants were recruited via social media, and no personal information was collected. Ten questions focused on participants' perceptions regarding mental illness diagnoses, and 20 questions related to the assumptions of mental illness diagnoses. The participants' perspectives on these assumptions held by professionals were assessed.

RESULTS

Among 14 survey participants, 4 correctly answered the relationships of 6 symptom pairs (28.57%). Two participants could not correctly conduct the calculations involved in mood disorder diagnoses (14.29%). Eleven (78.57%) correctly indicated that 2 or more sets of criteria were available for single diagnoses of mental illnesses. Only 1 (7.14%) correctly answered that the associations between symptoms and diagnoses were supported by including symptoms in the diagnostic criteria of the diagnoses. Nine (64.29%) correctly answered that the diagnosis variances were not fully explained by their symptoms. The confidence of participants in the major depressive disorder diagnosis and the willingness to take medications for this diagnosis were the same (mean = 5.50, standard deviation [SD] = 2.31). However, the confidence of participants in the symptom-based diagnosis of non-solid brain tumor was significantly lower (mean = 1.62, SD = 2.33, p < 0.001).

CONCLUSION

Our study found that mental illness diagnoses are wrong from the perspectives of the public because our participants did not agree with all the assumptions professionals make about mental illness diagnoses. Only a minority of our participants obtained correct answers to the calculations involved in mental illness diagnoses. In the literature, neither patients nor the public have been engaged in formulating the diagnostic criteria of mental illnesses.

Interpretable Trials: Is Interpretability a Reason Why Clinical Trials Fail?

Background: There are clinical trials using composite measures, indices, or scales as proxy for independent variables or outcomes. Interpretability of derived measures may not be satisfying. Adopting indices of poor interpretability in clinical trials may lead to trial failure. This study aims to understand the impact of using indices of different interpretability in clinical trials.

Methods: The interpretability of indices was categorized as: fair-to-poor, good, and unknown. In the literature, frailty indices were considered fair to poor interpretability. Body mass index (BMI) was highly interpretable. The other indices were of unknown interpretability. The trials were searched at clinicaltrials.gov on October 2, 2018. The use of indices as conditions/diseases or other terms was searched. The trials were grouped as completed, terminated, active, and other status. We tabulated the frequencies of frailty, BMI, and other indices.

Results: There were 263,928 clinical trials found and 155,606 were completed or terminated. Among 2,115 trials adopting indices or composite measures as condition or disease, 244 adopted frailty and 487 used BMI without frailty indices. Significantly higher proportions of trials of unknown status used indices as conditions/diseases or other terms, compared to completed and terminated trials. The proportions of active trials using frailty indices were significantly higher than those of completed or terminated trials.

Discussion: Clinical trial databases can be used to understand why trials may fail. Based on the findings, we suspect that using indices of poor interpretability may be associated with trial failure. Interpretability has not been conceived as an essential criterion for outcomes or proxy measures in trials. We will continue verifying the findings in other databases or data sources and apply this research method to improve clinical trial design. To prevent patients from experiencing trials likely to fail, we suggest further examining the interpretability of the indices in trials.