Response to "Logistic regression in signal detection: another piece added to the puzzle"

Exploring Completeness of Adverse Event Reports as a Tool for Signal Detection in Pharmacovigilance

BACKGROUND

Completeness of adverse event (AE) reports is an important component of quality for good pharmacovigilance practices. We aimed to evaluate the impact of incorporating a measure of completeness of AE reports on quantitative signal detection.

METHODS

An internal safety database from a global pharmaceutical company was used in the analysis. vigiGrade, an index score of completeness, was derived for each AE report. Data from various patient support programs (PSPs) were categorized based on average vigiGrade score per PSP. Performance of signal detection was compared between: (1) weighting and not weighting by vigiGrade score; and, (2) well documented and poorly documented PSPs using sensitivity, specificity, area under the receiver operating characteristics curve (AUC) and time-to-signal detection.

RESULTS

The ability to detect signals did not differ significantly when weighting by vigiGrade score [sensitivity (50% vs. 45%, p = 1), specificity (82.8% vs. 82.8%, p = 1), AUC (0.66 vs. 0.63, p = 0.051) or time-to-signal detection (HR 0.81, p = 0.63)] compared to not weighting. Well documented PSPs were better at detecting signals than poorly documented PSPs (AUC 0.66 vs. 0.52; p = 0.041) but time-to-signal detection did not differ significantly (HR 1.54, p = 0.42).

CONCLUSION

Completeness of AE reports did not significantly impact the ability to detect signals when weighting by vigiGrade score or restricting the database based on the level of completeness. While the vigiGrade helps provide quality assessments of AE reports and prioritize cases for review, our findings indicate the tool might not be useful for quantitative signal detection when used by itself.

A method for systematic discovery of adverse drug events from clinical notes

OBJECTIVE

Adverse drug events (ADEs) are undesired harmful effects resulting from use of a medication, and occur in 30% of hospitalized patients. The authors have developed a data-mining method for systematic, automated detection of ADEs from electronic medical records.

MATERIALS AND METHODS

This method uses the text from 9.5 million clinical notes, along with prior knowledge of drug usages and known ADEs, as inputs. These inputs are further processed into statistics used by a discriminative classifier which outputs the probability that a given drug-disorder pair represents a valid ADE association. Putative ADEs identified by the classifier are further filtered for positive support in 2 independent, complementary data sources. The authors evaluate this method by assessing support for the predictions in other curated data sources, including a manually curated, time-indexed reference standard of label change events.

RESULTS

This method uses a classifier that achieves an area under the curve of 0.94 on a held out test set. The classifier is used on 2,362,950 possible drug-disorder pairs comprised of 1602 unique drugs and 1475 unique disorders for which we had data, resulting in 240 high-confidence, well-supported drug-AE associations. Eighty-seven of them (36%) are supported in at least one of the resources that have information that was not available to the classifier.

CONCLUSION

This method demonstrates the feasibility of systematic post-marketing surveillance for ADEs using electronic medical records, a key component of the learning healthcare system.