Clinical coding and data quality in oculoplastic procedures

Inter-rater reliability of ACS-NSQIP colorectal procedure coding in Canada

BACKGROUND

The American College of Surgeons National Surgical Quality Improvement Project (ACS-NSQIP) uses Current Procedural Terminology (CPT) codes for risk-adjusted calculations. This study evaluates the inter-rater reliability of coding colorectal resections across Canada by ACS-NSQIP surgical clinical nurse reviewers (SCNR) and its impact on risk predictions.

METHODS

SCNRs in Canada were asked to code simulated operative reports. Percent agreement and free-marginal kappa correlation were calculated. The ACS-NSQIP risk calculator was utilized to illustrate its impact on risk prediction.

RESULTS

Responses from 44 of 150 (29.3 ​%) SCNRs revealed 3 to 6 different codes chosen per case, with agreement ranging from 6.7 ​% to 62.3 ​%. Free-marginal kappa correlation ranged from moderate agreement (0.53) to high disagreement (-0.17). ACS-NSQIP risk calculator predicted large absolute differences in risk for serious complications (0.2 ​%-13.7 ​%) and mortality (0.2 ​%-6.3 ​%).

CONCLUSION

This study demonstrated low inter-rater reliability in coding ACS-NSQIP colorectal procedures in Canada among SCNRs, impacting risk predictions.

Targeted education for clinicians and clinical coding staff improves the accuracy of clinical coding: A quality improvement project

The allocation of healthcare resources is reliant upon accurate information generated through clinical coding. Several factors contribute to coding inaccuracies, one of which is interpreting medical documentation. A lack of awareness among medical staff of the clinical coding process and the importance of detailed documentation exacerbates this problem. To investigate this further, 1 month of inpatient clinical coding data from a single hospital ward was reviewed by clinicians experienced in the coding and auditing process. If the reviewing clinician identified inaccuracies in the initial clinical coding, Healthcare Resource Group (HRG) codes were changed. Education sessions were then provided both to junior clinicians working on the hospital ward and to clinical coding staff and a further month of clinical coding data was again reviewed to assess for any difference after the sessions. HRG changes were made in 58.5% of 94 cases initially. Following the educational sessions, 20.5% of HRGs changed in 73 cases (p<0.0001), indicating more accurate initial clinical coding. There were also statistically significant reductions in the extent to which the primary and secondary diagnoses were changed. This study demonstrates that targeted education sessions for both junior clinicians and clinical coding staff can improve the accuracy of inpatient clinical coding.

Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores

Importance

Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR.

Objective

Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates.

Methods

Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation.

Results

For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.

Conclusions

In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Ophthalmology Operation Note Encoding with Open-Source Machine Learning and Natural Language Processing

INTRODUCTION

Accurate assignment of procedural codes has important medico-legal, academic, and economic purposes for healthcare providers. Procedural coding requires accurate documentation and exhaustive manual labour to interpret complex operation notes. Ophthalmology operation notes are highly specialised making the process time-consuming and challenging to implement. This study aimed to develop natural language processing (NLP) models trained by medical professionals to assign procedural codes based on the surgical report. The automation and accuracy of these models can reduce burden on healthcare providers and generate reimbursements that reflect the operation performed.

METHODS

A retrospective analysis of ophthalmological operation notes from two metropolitan hospitals over a 12-month period was conducted. Procedural codes according to the Medicare Benefits Schedule (MBS) were applied. XGBoost, decision tree, Bidirectional Encoder Representations from Transformers (BERT) and logistic regression models were developed for classification experiments. Experiments involved both multi-label and binary classification, and the best performing model was used on the holdout test dataset.

RESULTS

There were 1,000 operation notes included in the study. Following manual review, the five most common procedures were cataract surgery (374 cases), vitrectomy (298 cases), laser therapy (149 cases), trabeculectomy (56 cases), and intravitreal injections (49 cases). Across the entire dataset, current coding was correct in 53.9% of cases. The BERT model had the highest classification accuracy (88.0%) in the multi-label classification on these five procedures. The total reimbursement achieved by the machine learning algorithm was $184,689.45 ($923.45 per case) compared with the gold standard of $214,527.50 ($1,072.64 per case).

CONCLUSION

Our study demonstrates accurate classification of ophthalmic operation notes into MBS coding categories with NLP technology. Combining human and machine-led approaches involves using NLP to screen operation notes to code procedures, with human review for further scrutiny. This technology can allow the assignment of correct MBS codes with greater accuracy. Further research and application in this area can facilitate accurate logging of unit activity, leading to reimbursements for healthcare providers. Increased accuracy of procedural coding can play an important role in training and education, study of disease epidemiology and improve research ways to optimise patient outcomes.

Surgeon involvement in clinical coding to improve data accuracy and remuneration in a shoulder and elbow unit

BACKGROUND

Clinical coders are dependent on clear data regarding diagnoses and procedures to generate an accurate representation of clinical activity and ensure appropriate remuneration is received. The accuracy of this process may potentially be improved by collaboration with the surgical team.

METHODS

Between November 2017 and November 2019, 19 meetings took place between the Senior Clinical Fellow of our tertiary Shoulder & Elbow Unit and the coding validation lead of our Trust. At each meeting, the Clinical Fellow assessed the operative note of cases in which uncertainty existed as to the most suitable clinical codes to apply and selected the codes which most accurately represented the operative intervention performed.

RESULTS

Over a 24-month period, clinical coding was reviewed in 153 cases (range 3-14 per meeting, mean 8). Following review, the clinical coding was amended in 102 (67%) of these cases. A total of £115,160 additional income was generated as a result of this process (range £1677-£15,796 per meeting, mean £6061). Only 6 out of 28 (21%) cases initially coded as arthroscopic sub-acromial decompressions were correctly coded as such.

DISCUSSION

Surgeon input into clinical coding greatly improves data quality and increases remuneration received for operative interventions performed.

Real-Time Operative Coding for Endoscopic Sinonasal Procedures: Quality Improvement in Practice

Objective: investigate the impact of an intraoperative coding sticker (ICS) on the accuracy of coding in endoscopic sinonasal procedures. Methods: this was a two-cycle audit evaluating the accuracy (and financial impact) of intraoperative coding of sinonasal procedures at a single tertiary centre. An ICS was introduced following consultation with the coding department. The accuracy of coding was measured before (cycle 1) and after (cycle 2) the ICS was introduced to a pilot firm and compared to a control firm. The ICS was used in 35% of the pilot firm cases. Results: the accuracy of clinical coding for endoscopic sinus surgery was 60% in the first cycle. Switching to the ICS has improved the accuracy in that firm from 50% in first cycle to 70% in the second cycle (p = 0.936; Chi-squared test). The median reimbursement for endoscopic sinus surgery was equal in both cycles of £1493.00 per patient. However, inaccurate coding resulted in £109.92 excess tariff payment in first cycle and £130.96 deficiency in the second cycle. Users of ICS reported it to be easy to use for clinicians, staff and clinical coders, whilst minimizing human error. Conclusions: The integration of the ICS improves the coding in sinonasal procedures and offers low-fidelity option alternative to live coding on the computer. The accuracy was not statistically significant in the study possibly due to the low number of observations. This can allow a precise coding standard with reliable service remuneration.