Al-Moghrabi D, Abu Arqub S, Maroulakos MP, Pandis N, Fleming PS. Can ChatGPT identify predatory biomedical and dental journals? A cross-sectional content analysis.
J Dent 2024;
142:104840. [PMID:
38219888 DOI:
10.1016/j.jdent.2024.104840]
[Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Revised: 01/09/2024] [Accepted: 01/11/2024] [Indexed: 01/16/2024] Open
Abstract
OBJECTIVES
To assess whether ChatGPT can help to identify predatory biomedical and dental journals, analyze the content of its responses and compare the frequency of positive and negative indicators provided by ChatGPT concerning predatory and legitimate journals.
METHODS
Four-hundred predatory and legitimate biomedical and dental journals were selected from four sources: Beall's list, unsolicited emails, the Web of Science (WOS) journal list and the Directory of Open Access Journals (DOAJ). ChatGPT was asked to determine journal legitimacy. Journals were classified into legitimate or predatory. Pearson's Chi-squared test and logistic regression were conducted. Two machine learning algorithms determined the most influential criteria on the correct classification of journals.
RESULTS
The data were categorized under 10 criteria with the most frequently coded criteria being the transparency of processes and policies. ChatGPT correctly classified predatory and legitimate journals in 92.5 % and 71 % of the sample, respectively. The accuracy of ChatGPT responses was 0.82. ChatGPT also demonstrated a high level of sensitivity (0.93). Additionally, the model exhibited a specificity of 0.71, accurately identifying true negatives. A highly significant association between ChatGPT verdicts and the classification based on known sources was observed (P <0.001). ChatGPT was 30.2 times more likely to correctly classify a predatory journal (95 % confidence interval: 16.9-57.43, p-value: <0.001).
CONCLUSIONS
ChatGPT can accurately distinguish predatory and legitimate journals with a high level of accuracy. While some false positive (29 %) and false negative (7.5 %) results were observed, it may be reasonable to harness ChatGPT to assist with the identification of predatory journals.
CLINICAL SIGNIFICANCE STATEMENT
ChatGPT may effectively distinguish between predatory and legitimate journals, with accuracy rates of 92.5 % and 71 %, respectively. The potential utility of large-scale language models in exposing predatory publications is worthy of further consideration.
Collapse