1
|
Martindale APL, Llewellyn CD, de Visser RO, Ng B, Ngai V, Kale AU, di Ruffano LF, Golub RM, Collins GS, Moher D, McCradden MD, Oakden-Rayner L, Rivera SC, Calvert M, Kelly CJ, Lee CS, Yau C, Chan AW, Keane PA, Beam AL, Denniston AK, Liu X. Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat Commun 2024; 15:1619. [PMID: 38388497 PMCID: PMC10883966 DOI: 10.1038/s41467-024-45355-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 01/22/2024] [Indexed: 02/24/2024] Open
Abstract
The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.
Collapse
Affiliation(s)
| | - Carrie D Llewellyn
- Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK
| | - Richard O de Visser
- Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK
| | - Benjamin Ng
- Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK
- Christ Church, University of Oxford, Oxford, UK
| | - Victoria Ngai
- University College London Medical School, London, UK
| | - Aditya U Kale
- Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
| | | | - Robert M Golub
- Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Gary S Collins
- Centre for Statistics in Medicine//UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - David Moher
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottowa, Canada
| | - Melissa D McCradden
- Department of Bioethics, The Hospital for Sick Children, Toronto, Canada
- Genetics & Genome Biology Research Program, Peter Gilgan Centre for Research & Learning, Toronto, Canada
- Division of Clinical and Public Health, Dalla Lana School of Public Health, Toronto, Canada
| | - Lauren Oakden-Rayner
- Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia
| | - Samantha Cruz Rivera
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Melanie Calvert
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- NIHR Applied Research Collaboration (ARC) West Midlands, University of Birmingham, Birmingham, UK
- NIHR Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, University of Birmingham, Birmingham, UK
| | | | | | - Christopher Yau
- Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK
- Health Data Research UK, London, UK
| | - An-Wen Chan
- Department of Medicine, Women's College Hospital. University of Toronto, Toronto, Canada
| | - Pearse A Keane
- NIHR Biomedical Research Centre at Moorfields, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK
| | - Andrew L Beam
- Department of Epidemiology, Harvard. T.H. Chan School of Public Health, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Alastair K Denniston
- Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- NIHR Biomedical Research Centre at Moorfields, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK
| | - Xiaoxuan Liu
- Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
| |
Collapse
|
2
|
Vaish A, Migliorini F, Vaishya R. Artificial intelligence in foot and ankle surgery: current concepts. ORTHOPADIE (HEIDELBERG, GERMANY) 2023; 52:1011-1016. [PMID: 37626240 PMCID: PMC10692015 DOI: 10.1007/s00132-023-04426-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 07/13/2023] [Indexed: 08/27/2023]
Abstract
The twenty-first century has proven that data are the new gold. Artificial intelligence (AI) driven technologies might potentially change the clinical practice in all medical specialities, including orthopedic surgery. AI has a broad spectrum of subcomponents, including machine learning, which consists of a subdivision called deep learning. AI has the potential to increase healthcare delivery, improve indications and interventions, and minimize errors. In orthopedic surgery. AI supports the surgeon in the evaluation of radiological images, training of surgical residents, and excellent performance of machine-assisted surgery. The AI algorithms improve the administrative and management processes of hospitals and clinics, electronic healthcare databases, monitoring the outcomes, and safety controls. AI models are being developed in nearly all orthopedic subspecialties, including arthroscopy, arthroplasty, tumor, spinal and pediatric surgery. The present study discusses current applications, limitations, and future prospective of AI in foot and ankle surgery.
Collapse
Affiliation(s)
- Abhishek Vaish
- Department of Orthopaedics and Joint Replacement Surgery, Indraprastha Apollo Hospital, Sarita Vihar, 110076, New Delhi, India
| | - Filippo Migliorini
- Department of Orthopaedic, Trauma, and Reconstructive Surgery, RWTH University Medical Centre of Aachen, Pauwelsstraße 30, 52064, Aachen, Germany.
- Department of Orthopaedic and Trauma Surgery, Academic Hospital of Bolzano (SABES-ASDAA), 39100 Bolzano, Italy.
| | - Raju Vaishya
- Department of Orthopaedics and Joint Replacement Surgery, Indraprastha Apollo Hospital, Sarita Vihar, 110076, New Delhi, India
| |
Collapse
|
3
|
Anastasio AT, Mills FB, Karavan MP, Adams SB. Evaluating the Quality and Usability of Artificial Intelligence-Generated Responses to Common Patient Questions in Foot and Ankle Surgery. FOOT & ANKLE ORTHOPAEDICS 2023; 8:24730114231209919. [PMID: 38027458 PMCID: PMC10666700 DOI: 10.1177/24730114231209919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2023] Open
Abstract
Background Artificial intelligence (AI) platforms, such as ChatGPT, have become increasingly popular outlets for the consumption and distribution of health care-related advice. Because of a lack of regulation and oversight, the reliability of health care-related responses has become a topic of controversy in the medical community. To date, no study has explored the quality of AI-derived information as it relates to common foot and ankle pathologies. This study aims to assess the quality and educational benefit of ChatGPT responses to common foot and ankle-related questions. Methods ChatGPT was asked a series of 5 questions, including "What is the optimal treatment for ankle arthritis?" "How should I decide on ankle arthroplasty versus ankle arthrodesis?" "Do I need surgery for Jones fracture?" "How can I prevent Charcot arthropathy?" and "Do I need to see a doctor for my ankle sprain?" Five responses (1 per each question) were included after applying the exclusion criteria. The content was graded using DISCERN (a well-validated informational analysis tool) and AIRM (a self-designed tool for exercise evaluation). Results Health care professionals graded the ChatGPT-generated responses as bottom tier 4.5% of the time, middle tier 27.3% of the time, and top tier 68.2% of the time. Conclusion Although ChatGPT and other related AI platforms have become a popular means for medical information distribution, the educational value of the AI-generated responses related to foot and ankle pathologies was variable. With 4.5% of responses receiving a bottom-tier rating, 27.3% of responses receiving a middle-tier rating, and 68.2% of responses receiving a top-tier rating, health care professionals should be aware of the high viewership of variable-quality content easily accessible on ChatGPT. Level of Evidence Level III, cross sectional study.
Collapse
Affiliation(s)
| | - Frederic Baker Mills
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| | - Mark P. Karavan
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| | - Samuel B. Adams
- Department of Orthopaedic Surgery, Duke University Medical Center, Durham, NC, USA
| |
Collapse
|
4
|
Gupta P, Kingston KA, O’Malley M, Williams RJ, Ramkumar PN. Advancements in Artificial Intelligence for Foot and Ankle Surgery: A Systematic Review. FOOT & ANKLE ORTHOPAEDICS 2023; 8:24730114221151079. [PMID: 36817020 PMCID: PMC9929923 DOI: 10.1177/24730114221151079] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/16/2023] Open
Abstract
Background There has been a rapid increase in research applying artificial intelligence (AI) to various subspecialties of orthopaedic surgery, including foot and ankle surgery. The purpose of this systematic review is to (1) characterize the topics and objectives of studies using AI in foot and ankle surgery, (2) evaluate the performance of their models, and (3) evaluate their validity (internal or external validation). Methods A systematic literature review was conducted using PubMed/MEDLINE and Embase databases in December 2022. All studies that used AI or its subsets machine learning (ML) and deep learning (DL) in the setting of foot and ankle surgery relevant to orthopaedic surgeons were included. Studies were evaluated for their demographics, subject area, outcomes of interest, model(s) tested, model(s)' performance, and validity (internal or external). Results A total of 31 studies met inclusion criteria: 14 studies investigated AI for image interpretation, 13 studies investigated AI for clinical predictions, and 4 studies were grouped as "other." Studies commonly explored AI for ankle fractures, calcaneus fractures, hallux valgus, Achilles tendon pathologies, plantar fasciitis, and sports injuries. For studies reporting the area under the receiver operating characteristic curve (AUC), AUCs ranged from 0.64 (poor) to 0.99 (excellent). Two studies (6.45%) reported external validation. Conclusion Applications of AI in the field of foot and ankle surgery are expanding, particularly for image interpretation and clinical predictions. Current model performances range from poor to excellent, and most studies lack external validation, demonstrating a need for further research prior to deploying AI-based clinical applications. Level of Evidence Level III, retrospective cohort study.
Collapse
Affiliation(s)
- Puneet Gupta
- Department of Orthopaedic Surgery, George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | | | - Martin O’Malley
- Hospital for Special Surgery, New York, NY, USA,Brooklyn Nets, National Basketball Association (NBA), Brooklyn, NY, USA
| | - Riley J. Williams
- Hospital for Special Surgery, New York, NY, USA,Brooklyn Nets, National Basketball Association (NBA), Brooklyn, NY, USA
| | - Prem N. Ramkumar
- Hospital for Special Surgery, New York, NY, USA,Brooklyn Nets, National Basketball Association (NBA), Brooklyn, NY, USA,Prem N. Ramkumar, MD, MBA, Hospital for Special Surgery, 535 E 70th St, New York, NY 10021-4898, USA.
| |
Collapse
|