Development and Validation of a Natural Language Processing Tool to Identify Injuries in Infants Associated With Abuse

Factors Associated with Abusive Head Trauma in Young Children Presenting to Emergency Medical Services Using a Large Language Model

OBJECTIVES

Abusive head trauma (AHT) is a leading cause of death in young children. Analyses of patient characteristics presenting to Emergency Medical Services (EMS) are often limited to structured data fields. Artificial Intelligence (AI) and Large Language Models (LLM) may identify rare presentations like AHT through factors not found in structured data. Our goal was to apply AI and LLM to EMS narrative documentation of young children to detect AHT.

METHODS

This is a retrospective cohort study of EMS transports of children <36 months of age with a diagnosis of head injury from the 2018-2019 ESO Research Data Collaborative. Non-abusive closed head injury (NA-CHI) was distinguished from AHT and child maltreatment (AHT-CAN) through 2 expert reviewers; kappa statistic (k) assessed inter-rater reliability. A Natural Language Processing (NLP) framework using an LLM augmented with expert derived n-grams was developed to identify AHT-CAN. We compared test characteristics (sensitivity, specificity, negative predictive value (NPV)) between this NLP framework to a Generative Pretrained Transformer (GPT) or n-grams only models to detect AHT-CAN. Association of specific word tokens with AHT-CAN was analyzed using Pearson's chi-square. Area Under the Receiver Operator Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC) are also reported.

RESULTS

There were 1082 encounters in our cohort; 1030 (95.2%) NA-CHI and 52 (4.8%) AHT-CAN. Inter-rater agreement was substantial (k = 0.71). The augmented NLP framework had a specificity and sensitivity of 72.4% and 92.3%, respectively with a NPV of 99.5%. In comparison, the GPT model had a sensitivity of 69.2%, specificity of 97.1% and NPV of 98.4% and n-grams alone had a sensitivity of 53.8%, specificity of 62.0%, NPV of 96.4%. AUROC was 0.91 and AUPRC was 0.52. A total of 44 n-grams and bi-grams were positively associated with AHT-CAN including "domestic," "various," "bruise," "cheek," "multiple," "doa," "not respond," "see EMS."

CONCLUSIONS

AI and LLMs have high sensitivity and specificity to detect AHT-CAN in EMS free-text narratives. Words associated with physical signs of trauma are strongly associated with AHT-CAN. LLMs augmented with a list of n-grams may help EMS identify signs of trauma that aid in the detection of AHT in young children.

Identifying Risk Factors of Children Who Suffered Physical Abuse: A Systematic Review

BACKGROUND

Approximately 25% of children in the United States experience child abuse or neglect, 18% of whom are physically abused. Physicians are often in a position to differentiate accidental trauma from physical child abuse. Therefore, the aim of this study was to review recent literature for risk factors associated with physical child abuse.

METHODS

In this systematic review, three electronic databases were searched for articles published in the past 10 years, using the terms "abuse," "risk factors," and "children," with associated variations. A total of 1,568 articles were identified. A sequential screening process was conducted by two independent reviewers in each phase, and 63 articles were included in the final analysis. Data extraction was conducted, and a narrative synthesis was conducted.

RESULTS

Sociodemographic risk factors of physical child abuse were younger age, male sex, African American or Hispanic race, nonprivate insurance, lower income, and lower maternal education. Other risk factors reported were previous reports of child abuse, birth defects, and developmental, musculoskeletal, intellectual, or mood disorders. Clinical and radiographic signs possibly indicative of child abuse included subdural hematoma, traumatic brain injury, retinal injury, bruising, superficial skin injury, lung injury, and fracture in skull, femur, clavicle, humerus, and foot.

CONCLUSION

The results of this systematic review provide insights into the potential risk factors that should be considered when assessing a child for physical abuse in the health care setting.

Routine Emergency Department Screening to Decrease Subsequent Physical Abuse

STUDY OBJECTIVE

Emergency department (ED) screening for child physical abuse has been widely implemented, with uncertain effects on child abuse identification. Our goal was to determine the effect of screening on referrals to child protective services (CPS) identifying abuse.

METHODS

We performed a retrospective cohort study of children younger than 6 years old with an ED encounter at 1 of 2 large health care systems, one of which implemented routine child abuse screening. The main outcome was initial (<2 days) or subsequent (3 to 180 days) referral to CPS identifying child abuse using linked records. We compared outcomes for the 2-year period after screening was implemented to the preperiod and nonscreening EDs using generalized estimating equations to adjust for sex, age, race/ethnicity, payor and prior ED encounters and clustered by center.

RESULTS

Of the 331,120 ED encounters, 41,589 (12.6%) occurred at screening EDs during the screening period. Screening was completed in 34,272 (82%) and was positive in 188 (0.45%). Overall, 7,623 encounters (2.3%) had a subsequent referral, of which 589 (0.2%) identified moderate or severe abuse. ED screening did not change initial (adjusted odds ratio [aOR]=1.01, 95% confidence interval [CI] 0.89 to 1.15) or subsequent referral to CPS when compared to the prescreening period (aOR=1.05, 95% CI 0.9 to 1.18) or to the nonscreening EDs (aOR=1.06, 95% CI 0.92 to 1.21).

CONCLUSION

Routine screening did not affect initial or subsequent referrals to CPS.

Estimating the Prevalence of Child Abuse and Neglect Among Adolescents in Primary Care Through Diagnoses Codes and Free-Text EHR Clinical Notes

INTRODUCTION

Adolescents' child abuse and neglect experiences are often under-documented in primary care, leading to missed opportunities for interventions. This study compares the prevalence of child abuse and neglect cases identified by diagnostic codes versus a natural language processing approach of clinical notes.

METHOD

We retrospectively analyzed data from 8,157 adolescents, using ICD-10 codes and a natural language processing algorithm to identify child abuse and neglect cases and applied topic modeling on clinical notes to extract prevalent topics.

RESULTS

The natural language processing approach identified more cases of child abuse and neglect cases (n = 294) compared to ICD-10 codes (n = 111). Additionally, topic modeling of clinical notes showed the multifaceted nature of child abuse and neglect as captured in clinical narratives.

DISCUSSION

Integrating natural language processing with ICD codes has the potential to enhance the identification and documentation of child abuse and neglect, which could lead to earlier and more targeted interventions and coordinated care.

Developing and Testing the Usability of a Novel Child Abuse Clinical Decision Support System: Mixed Methods Study

BACKGROUND

Despite the impact of physical abuse on children, it is often underdiagnosed, especially among children evaluated in emergency departments (EDs). Electronic clinical decision support (CDS) can improve the recognition of child physical abuse.

OBJECTIVE

We aimed to develop and test the usability of a natural language processing-based child abuse CDS system, known as the Child Abuse Clinical Decision Support (CA-CDS), to alert ED clinicians about high-risk injuries suggestive of abuse in infants' charts.

METHODS

Informed by available evidence, a multidisciplinary team, including an expert in user design, developed the CA-CDS prototype that provided evidence-based recommendations for the evaluation and management of suspected child abuse when triggered by documentation of a high-risk injury. Content was customized for medical versus nursing providers and initial versus subsequent exposure to the alert. To assess the usability of and refine the CA-CDS, we interviewed 24 clinicians from 4 EDs about their interactions with the prototype. Interview transcripts were coded and analyzed using conventional content analysis.

RESULTS

Overall, 5 main categories of themes emerged from the study. CA-CDS benefits included providing an extra layer of protection, providing evidence-based recommendations, and alerting the entire clinical ED team. The user-centered, workflow-compatible design included soft-stop alert configuration, editable and automatic documentation, and attention-grabbing formatting. Recommendations for improvement included consolidating content, clearer design elements, and adding a hyperlink with additional resources. Barriers to future implementation included alert fatigue, hesitancy to change, and concerns regarding documentation. Facilitators of future implementation included stakeholder buy-in, provider education, and sharing the test characteristics. On the basis of user feedback, iterative modifications were made to the prototype.

CONCLUSIONS

With its user-centered design and evidence-based content, the CA-CDS can aid providers in the real-time recognition and evaluation of infant physical abuse and has the potential to reduce the number of missed cases.

Natural Language Processing - A Surveillance Stepping Stone to Identify Child Abuse

OBJECTIVE

We aimed to refine a natural language processing (NLP) algorithm that identified injuries associated with child abuse and identify areas in which integration into a real-time clinical decision support (CDS) tool may improve clinical care.

METHODS

We applied an NLP algorithm in "silent mode" to all emergency department (ED) provider notes between July 2021 and December 2022 (n = 353) at 1 pediatric and 8 general EDs. We refined triggers for the NLP, assessed adherence to clinical guidelines, and evaluated disparities in degree of evaluation by examining associations between demographic variables and abuse evaluation or reporting to child protective services.

RESULTS

Seventy-three cases falsely triggered the NLP, often due to errors in interpreting linguistic context. We identified common false-positive scenarios and refined the algorithm to improve NLP specificity. Adherence to recommended evaluation standards for injuries defined by nationally accepted clinical guidelines was 63%. There were significant demographic differences in evaluation and reporting based on presenting ED type, insurance status, and race and ethnicity.

CONCLUSIONS

Analysis of an NLP algorithm in "silent mode" allowed for refinement of the algorithm and highlighted areas in which real-time CDS may help ED providers identify and pursue appropriate evaluation of injuries associated with child physical abuse.

Artificial Intelligence and Child Abuse and Neglect: A Systematic Review

All societies should carefully address the child abuse and neglect phenomenon due to its acute and chronic sequelae. Even if artificial intelligence (AI) implementation in this field could be helpful, the state of the art of this implementation is not known. No studies have comprehensively reviewed the types of AI models that have been developed/validated. Furthermore, no indications about the risk of bias in these studies are available. For these reasons, the authors conducted a systematic review of the PubMed database to answer the following questions: "what is the state of the art about the development and/or validation of AI predictive models useful to contrast child abuse and neglect phenomenon?"; "which is the risk of bias of the included articles?". The inclusion criteria were: articles written in English and dated from January 1985 to 31 March 2023; publications that used a medical and/or protective service dataset to develop and/or validate AI prediction models. The reviewers screened 413 articles. Among them, seven papers were included. Their analysis showed that: the types of input data were heterogeneous; artificial neural networks, convolutional neural networks, and natural language processing were used; the datasets had a median size of 2600 cases; the risk of bias was high for all studies. The results of the review pointed out that the implementation of AI in the child abuse and neglect field lagged compared to other medical fields. Furthermore, the evaluation of the risk of bias suggested that future studies should provide an appropriate choice of sample size, validation, and management of overfitting, optimism, and missing data.