Performance of ChatGPT on the Chinese Postgraduate Examination for Clinical Medicine: Survey Study

BACKGROUND

ChatGPT, an artificial intelligence (AI) based on large-scale language models, has sparked interest in the field of health care. Nonetheless, the capabilities of AI in text comprehension and generation are constrained by the quality and volume of available training data for a specific language, and the performance of AI across different languages requires further investigation. While AI harbors substantial potential in medicine, it is imperative to tackle challenges such as the formulation of clinical care standards; facilitating cultural transitions in medical education and practice; and managing ethical issues including data privacy, consent, and bias.

OBJECTIVE

The study aimed to evaluate ChatGPT's performance in processing Chinese Postgraduate Examination for Clinical Medicine questions, assess its clinical reasoning ability, investigate potential limitations with the Chinese language, and explore its potential as a valuable tool for medical professionals in the Chinese context.

METHODS

A data set of Chinese Postgraduate Examination for Clinical Medicine questions was used to assess the effectiveness of ChatGPT's (version 3.5) medical knowledge in the Chinese language, which has a data set of 165 medical questions that were divided into three categories: (1) common questions (n=90) assessing basic medical knowledge, (2) case analysis questions (n=45) focusing on clinical decision-making through patient case evaluations, and (3) multichoice questions (n=30) requiring the selection of multiple correct answers. First of all, we assessed whether ChatGPT could meet the stringent cutoff score defined by the government agency, which requires a performance within the top 20% of candidates. Additionally, in our evaluation of ChatGPT's performance on both original and encoded medical questions, 3 primary indicators were used: accuracy, concordance (which validates the answer), and the frequency of insights.

RESULTS

Our evaluation revealed that ChatGPT scored 153.5 out of 300 for original questions in Chinese, which signifies the minimum score set to ensure that at least 20% more candidates pass than the enrollment quota. However, ChatGPT had low accuracy in answering open-ended medical questions, with only 31.5% total accuracy. The accuracy for common questions, multichoice questions, and case analysis questions was 42%, 37%, and 17%, respectively. ChatGPT achieved a 90% concordance across all questions. Among correct responses, the concordance was 100%, significantly exceeding that of incorrect responses (n=57, 50%; P<.001). ChatGPT provided innovative insights for 80% (n=132) of all questions, with an average of 2.95 insights per accurate response.

CONCLUSIONS

Although ChatGPT surpassed the passing threshold for the Chinese Postgraduate Examination for Clinical Medicine, its performance in answering open-ended medical questions was suboptimal. Nonetheless, ChatGPT exhibited high internal concordance and the ability to generate multiple insights in the Chinese language. Future research should investigate the language-based discrepancies in ChatGPT's performance within the health care context.

Proteomic and functional analysis of HDL subclasses in humans and rats: a proof-of-concept study

BACKGROUND

The previous study investigated whether the functions of small, medium, and large high density lipoprotein (S/M/L-HDL) are correlated with protein changes in mice. Herein, the proteomic and functional analyses of high density lipoprotein (HDL) subclasses were performed in humans and rats.

METHODS

After purifying S/M/L-HDL subclasses from healthy humans (n = 6) and rats (n = 3) using fast protein liquid chromatography (FPLC) with calcium silica hydrate (CSH) resin, the proteomic analysis by mass spectrometry was conducted, as well as the capacities of cholesterol efflux and antioxidation was measured.

RESULTS

Of the 120 and 106 HDL proteins identified, 85 and 68 proteins were significantly changed in concentration among the S/M/L-HDL subclasses in humans and rats, respectively. Interestingly, it was found that the relatively abundant proteins in the small HDL (S-HDL) and large HDL (L-HDL) subclasses did not overlap, both in humans and in rats. Next, by searching for the biological functions of the relatively abundant proteins in the HDL subclasses via Gene Ontology, it was displayed that the relatively abundant proteins involved in lipid metabolism and antioxidation were enriched more in the medium HDL (M-HDL) subclass than in the S/L-HDL subclasses in humans, whereas in rats, the relatively abundant proteins associated with lipid metabolism and anti-oxidation were enriched in M/L-HDL and S/M-HDL, respectively. Finally, it was confirmed that M-HDL and L-HDL had the highest cholesterol efflux capacity among the three HDL subclasses in humans and rats, respectively; moreover, M-HDL exhibited higher antioxidative capacity than S-HDL in both humans and rats.

CONCLUSIONS

The S-HDL and L-HDL subclasses are likely to have different proteomic components during HDL maturation, and results from the proteomics-based comparison of the HDL subclasses may explain the associated differences in function.

U-shaped association between the triglyceride-glucose index and atrial fibrillation incidence in a general population without known cardiovascular disease

OBJECTIVE

The triglyceride-glucose (TyG) index has been shown to be a new alternative measure for insulin resistance. However, no study has attempted to investigate the association of the TyG index with incident atrial fibrillation (AF) in the general population without known cardiovascular diseases.

METHODS

Individuals without known cardiovascular diseases (heart failure, coronary heart disease, or stroke) from the Atherosclerosis Risk in Communities (ARIC) cohort were recruited. The baseline TyG index was calculated as the Ln [fasting triglycerides (mg/dL) × fasting glucose (mg/dL)/2]. The association between the baseline TyG index and incident AF was examined using Cox regression.

RESULTS

Of 11,851 participants, the mean age was 54.0 years; 6586 (55.6%) were female. During a median follow-up of 24.26 years, 1925 incidents of AF cases (0.78/per 100 person-years) occurred. An increased AF incidence with a graded TyG index was found by Kaplan‒Meier curves (P < 0.001). In multivariable-adjusted analysis, both < 8.80 (adjusted hazard ratio [aHR] = 1.15, 95% confidence interval [CI] 1.02, 1.29) and > 9.20 levels (aHR 1.18, 95% CI 1.03, 1.37) of the TyG index were associated with an increased risk of AF compared with the middle TyG index category (8.80-9.20). The exposure-effect analysis confirmed the U-shaped association between the TyG index and AF incidence (P = 0.041). Further sex-specific analysis showed that a U-shaped association between the TyG index and incident AF still existed in females but not in males.

CONCLUSIONS

A U-shaped association between the TyG index and AF incidence is observed in Americans without known cardiovascular diseases. Female sex may be a modifier in the association between the TyG index and AF incidence.

Association of motor index scores with fall incidence among community-dwelling older people

BACKGROUND

Several kinds of motor dysfunction have been studied for predicting future fall risk in community-dwelling older individuals. However, no study has tested the ability of the fine motor index (FINEA) and gross motor index (GROSSA) to predict the risk of falling, as well as the specific fall type.

OBJECTIVE

We investigated the associations of FINEA/GROSSA scores with fall risk, explained falls, and unexplained falls.

METHODS

A total of 6267 community-dwelling adults aged ≥ 50 years from the Irish Longitudinal Study on Aging (TILDA) cohort were included. First, the associations of FINEA and GROSSA scores with the history of total falls, explained falls and unexplained falls were assessed in a cross-sectional study and further verified in a prospective cohort after 2 years of follow-up by Poisson regression analysis.

RESULTS

We found that high FINEA and GROSSA scores were positively associated with almost all fall histories (FINEA scores: total falls: adjusted prevalence ratio [aPR] = 1.28, P = 0.009; explained falls: aPR = 1.15, P = 0.231; unexplained falls: aPR = 1.88, P < 0.001; GROSSA scores: total falls: aPR = 1.39, P < 0.001; explained falls: aPR = 1.28, P = 0.012; unexplained falls: aPR = 2.18, P < 0.001) in a cross-sectional study. After 2 years of follow-up, high FINEA scores were associated with an increased incidence of total falls (adjusted rate ratio [aRR] = 1.42, P = 0.016) and explained falls (aRR = 1.51, P = 0.020) but not with unexplained falls (aRR = 1.41, P = 0.209). High GROSSA scores were associated with an increased incidence of unexplained falls (aRR = 1.57, P = 0.041) and were not associated with either total falls (aRR = 1.21, P = 0.129) or explained falls (aRR = 1.07, P = 0.656). Compared with individuals without limitations in either the FINEA or GROSSA, individuals with limitations in both indices had a higher risk of falls, including total falls (aRR = 1.35, P = 0.002), explained falls (aRR = 1.31, P = 0.033) and unexplained falls (aRR = 1.62, P = 0.004).

CONCLUSION

FINEA scores were positively associated with accidental falls, while GROSSA scores were positively associated with unexplained falls. The group for whom both measures were impaired showed a significantly higher risk of both explained and unexplained falls. FINEA or GROSSA scores should be investigated further as possible tools to screen for and identify community-dwelling adults at high risk of falling.

Usefulness of B-Type Natriuretic Peptide for Predicting the Risk of Stroke in Patients With Heart Failure With Preserved Ejection Fraction

Background B-type natriuretic peptide (BNP) is a well-known biomarker for prognosis in heart failure with patients with preserved ejection fraction. However, the clinical predictive ability of BNP for the risk of stroke in HFpEF is not clear. Methods and Results A total of 799 patients with HFpEF from the TOPCAT (Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist) trial were included. Association of baseline BNP with risk of stroke was assessed using the Cox proportional hazard model. The discriminatory ability of BNP was expressed using the C index. The improvement in 5-year stroke prediction was assessed by C statistic, categorical net reclassification improvement index, and relative integrated discrimination improvement. A total of 34 (4.3%) patients among the 799 patients with HFpEF experienced stroke events over a median of 2.85 years of follow-up. The stroke group showed a higher BNP level than the nonstroke group (375 pg/mL versus 241 pg/mL, respectively; P=0.006). Higher BNP levels were associated with increased risk of stroke after multivariable adjustment (hazard ratio, 3.29 [95% CI, 1.51-7.16]) and had a moderate performance for stroke prediction (C index, 0.67). Adding BNP to CHADS₂/CHA₂DS₂-VASc/R₂CHADS₂ scores improved their predictive value for stroke (CHADS₂: C index, 0.67; BNP+CHADS₂: C index, 0.77; net reclassification improvement, 40.9%; integrated discrimination improvement, 3.0%; CHA₂DS₂-VASc: C index, 0.64; BNP+CHA₂DS₂-VASc: C index, 0.74; net reclassification improvement, 41.4%; integrated discrimination improvement, 2.2%; R₂CHADS₂: C index, 0.70; BNP+R₂CHADS₂: C index, 0.78; net reclassification improvement, 40.9%; integrated discrimination improvement, 3.2%). Conclusions BNP is associated with an increased risk of stroke in patients with HFpEF and may be a valuable biomarker for stroke prediction in HFpEF.

Efficacy and safety of HDL/apoA-1 mimetics on human and mice with atherosclerosis: a systematic review and meta-analysis

Funding Acknowledgements

Type of funding sources: Public grant(s) – National budget only. Main funding source(s): the National Natural Science Foundation of China

Background

Low high-density lipoprotein cholesterol (HDL-C) level as a residual risk factor of cardiovascular disease (CVD) is still causing concern, although using chemical drugs for raising HDL-C level failed. The effect of high-density lipoprotein/ apolipiproteinA-1(HDL/apoA-1) mimetics on atherosclerosis is controversial.

Aim

In this meta-analysis we analyzed the effect of high-density lipoprotein/ apolipiproteinA-1(HDL/apoA-1) mimetics on atherosclerotic lesion both in human and mice.

Methods

We systematically searched PubMed, Cochrane, Web of Science and EMBASE databases up to June 6, 2020 for eligible studies using wide search terms and included all the publications meet the including criteria. The methodological quality of the human studies was assessed using Review Manager (RevMan) software (version 5.3.). The methodological quality of the mice studies was assessed by using stair list. WMD(SMD) with 95% CI was used as a measure of the association between HDL/apoA-1 mimetics and plaque regression in human (in mice), after pooling data across trials in a random effect model. Sensitivity and subgroup analyses were used to explore sources of heterogeneity and the effect of potential confounders. STATA (version 14.0) was used to conduct all statistical analyses.

Results

We identified 15 randomized controlled trials in which 6 trails including 754 ACS (HDL/apoA-1 mimetics = 414, placebo = 340) patients used for efficacy analysis and all of 15 trails used for safety analysis and 17 controlled trials for animal study. The pooled results showed that the use of HDL/apoA-1 mimetics did not significant decreased the percent atheroma volume(p = 0.494) and total atheroma volume(p = 0.560) in patients with acute coronary syndrome (ACS). However, HDL/apoA-1 mimetics (or gene transfection) was significant associated with all of final percent lesion area, final lesion area and changes in lesion area (SMD, -1.75; 95% CI: -2.21∼-1.29, p = 0.000; SMD, -0.78; 95% CI: -1.18∼-0.38, p = 0.000; SMD: -2.06; 95% CI, -3.92∼-0.2, p = 0.03) in mice.

Conclusions

In human, HDL/apoA-1 mimetics cannot significantly improve atheroma volume in artery, although it is safe. However, in animal, the results suggest HDL/apoA-1 mimetics (or gene transfection) can decrease lesion area. So additional studies are needed to further investigate and explain the different efficacy of HDL/apoA-1 mimetic peptides between human and animal. Abstract Figure. Forest plots of human studies

Efficacy and Safety of High-Density Lipoprotein/Apolipoprotein A1 Replacement Therapy in Humans and Mice With Atherosclerosis: A Systematic Review and Meta-Analysis

Background: Although elevation of HDL-C levels by pharmaceutical drugs have no benefit of cardiovascular endpoint, the effect of high-density lipoprotein/apolipoprotein A1 (HDL/apoA-1) replacement therapy on atherosclerosis is controversial. The current meta-analysis analyzed the effects of HDL/apoA-1 replacement therapies on atherosclerotic lesions both in humans and mice.

Methods: The PubMed, Cochrane Library, Web of Science, and EMBASE databases were searched through June 6, 2020. The methodological quality of the human studies was assessed using Review Manager (RevMan, version 5.3.). The methodological quality of the mouse studies was assessed using a stair list. STATA (version 14.0) was used to perform all statistical analyses.

Results: Fifteen randomized controlled human trials and 17 animal studies were included. The pooled results showed that HDL/apoA-1 replacement therapy use did not significantly decrease the percent atheroma volume (p = 0.766) or total atheroma volume (p = 0.510) in acute coronary syndrome (ACS) patients (N = 754). However, HDL/apoA-1 replacement therapies were significantly associated with the final percent lesion area, final lesion area, and changes in lesion area (SMD, −1.75; 95% CI: −2.21~-1.29, p = 0.000; SMD, −0.78; 95% CI: −1.18~-0.38, p = 0.000; SMD: −2.06; 95% CI, −3.92~-0.2, p = 0.03, respectively) in mice.

Conclusions: HDL/apoA-1 replacement therapies are safe but do not significantly improve arterial atheroma volume in humans. The results in animals suggest that HDL/apoA-1 replacement therapies decrease the lesion area. Additional studies are needed to investigate and explain the differences in HDL/apoA-1 replacement therapy efficacies between humans and animals.

Trial registration number: Human pooled analysis: PROSPERO, CRD42020210772. prospectively registered.

Fine or Gross Motor Index as a Simple Tool for Predicting Cognitive Impairment in Elderly People: Findings from The Irish Longitudinal Study on Ageing (TILDA)

BACKGROUND

Several kinds of motor dysfunction can predict future cognitive impairment in elderly individuals. However, the ability of the fine motor index (FINEA) and gross motor index (GROSSA) to predict the risk of cognitive impairment has not been assessed.

OBJECTIVE

We investigated the associations between FINEA/GROSSA and cognitive impairment.

METHODS

The data of 4,745 participants from The Irish Longitudinal Study on Ageing (TILDA) were analyzed. Cognitive function was assessed using the Mini-Mental State Examination (MMSE). We first assessed the correlation between the FINEA/GROSSA and MMSE in a cross-sectional study. Then, we further investigated the predictive role of the incidence of cognitive impairment in a prospective cohort study.

RESULTS

We found that both FINEA and GROSSA were negatively correlated with MMSE in both the unadjusted (FINEA: B = -1.00, 95%confidence intervals (CI): -1.17, -0.83, t = -11.53, p < 0.001; GROSSA: B = -0.85, 95%CI: -0.94, -0.76, t = -18.29, p < 0.001) and adjusted (FINEA: B = -0.63, 95%CI: -0.79, -0.47, t = -7.77, p < 0.001; GROSSA: B = -0.57, 95%CI: -0.66, -0.48, t = -12.61, p < 0.001) analyses in a cross-sectional study. In a prospective cohort study, both high FINEA and high GROSSA were associated with an increased incidence of cognitive function impairment (FINEA: adjusted odds ratios (OR) = 2.35, 95%CI: 1.05, 5.23, p = 0.036; GROSSA adjusted OR = 3.00, 95%CI: 1.49, 6.03, p = 0.002) after 2 years of follow-up.

CONCLUSION

Higher FINEA and GROSSA scores were both associated with an increased incidence of cognitive impairment. FINEA or GROSSA might be a simple tool for identifying patients with cognitive impairment.