Assessing the ability of an instrumental variable causal forest algorithm to personalize treatment evidence using observational data: the case of early surgery for shoulder fracture

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

BACKGROUND

Treatment variation from observational data has been used to estimate patient-specific treatment effects. Causal Forest Algorithms (CFAs) developed for this task have unknown properties when treatment effect heterogeneity from unmeasured patient factors influences treatment choice - essential heterogeneity.

METHODS

We simulated eleven populations with identical treatment effect distributions based on patient factors. The populations varied in the extent that treatment effect heterogeneity influenced treatment choice. We used the generalized random forest application (CFA-GRF) to estimate patient-specific treatment effects for each population. Average differences between true and estimated effects for patient subsets were evaluated.

RESULTS

CFA-GRF performed well across the population when treatment effect heterogeneity did not influence treatment choice. Under essential heterogeneity, however, CFA-GRF yielded treatment effect estimates that reflected true treatment effects only for treated patients and were on average greater than true treatment effects for untreated patients.

CONCLUSIONS

Patient-specific estimates produced by CFAs are sensitive to why patients in real-world practice make different treatment choices. Researchers using CFAs should develop conceptual frameworks of treatment choice prior to estimation to guide estimate interpretation ex post.

ICD-10 diagnosis codes in electronic health records do not adequately capture fracture complexity for proximal humerus fractures

BACKGROUND

The ability to do comparative effectiveness research (CER) for proximal humerus fractures (PHF) using data in electronic health record (EHR) systems and administrative claims databases was enhanced by the 10th revision of the International Classification of Diseases (ICD-10), which expanded the diagnosis codes for PHF to describe fracture complexity including displacement and the number of fracture parts. However, these expanded codes only enhance secondary use of data for research if the codes selected and recorded correctly reflect the fracture complexity. The objective of this project was to assess the accuracy of ICD-10 diagnosis codes documented during routine clinical practice for secondary use of EHR data.

METHODS

A sample of patients with PHFs treated by orthopedic providers across a large, regional health care system between January 1, 2016, and December 31, 2018, were retrospectively identified from the EHR. Four fellowship-trained orthopedic surgeons reviewed patient radiographs and recorded the Neer Classification characteristics of displacement, number of parts, and fracture location(s). The fracture characteristics were then reviewed by a trained coder, and the most clinically appropriate ICD-10 diagnosis code based on the number of fracture parts was assigned. We assessed congruence between ICD-10 codes documented in the EHR and radiograph-validated codes, and assessed sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for EHR-documented ICD-10 codes.

RESULTS

There were 761 patients with unilateral, closed PHF who met study inclusion criteria. On average, patients were 67 years of age and 77% were female. Based on radiograph review, 37% were 1-part fractures, 42% were 2-part, 11% were 3-part, and 10% were 4-part fractures. Of the EHR diagnosis codes recorded during clinical practice, 59% were "unspecified" fracture diagnosis codes that did not identify the number of fracture parts. Examination of fracture codes revealed PPV was highest for 1-part (PPV = 0.66, 95% confidence interval [CI] 0.60-0.72) and 4-part fractures (PPV = 0.67, 95% CI 0.13-1.00).

CONCLUSIONS

Current diagnosis coding practices do not adequately capture the fracture complexity needed to conduct subgroup analysis for PHF. Conclusions drawn from population studies or large databases using ICD-10 codes for PHF classification should be interpreted within this limitation. Future studies are warranted to improve diagnostic coding to support large observational studies using EHR and administrative claims data.