Validation of infant immunization billing codes in administrative data

Validating pertussis data measures using electronic medical record data in Ontario, Canada 1986-2016

Background

Pertussis is a reportable disease in many countries, but ascertainment bias has limited data accuracy. This study aims to validate pertussis data measures using a reference standard that incorporates different suspected case severities, allowing for the impact of case severity on accuracy and detection to be explored.

Methods

We evaluated 25 pertussis detection algorithms in a primary care electronic medical record database between January 1, 1986 and December 30, 2016. We estimated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). We used sensitivity analyses to explore areas of uncertainty and evaluated reasons for lack of detection.

Results

The algorithm including all data measures achieved the highest sensitivity at 20.6%. Sensitivity increased to 100% after reclassifying symptom-only cases as non-cases, but the PPV remained low. Age at first episode was significantly associated with detection in half of the tested scenarios, and false negatives often had some history of immunization.

Conclusions

Sensitivity improved by reclassifying symptom-only cases but remained low unless multiple data sources were used. Results demonstrate a trade-off between PPV and sensitivity. EMRs can enhance detection through patient history and clinical note data. It is essential to improve case identification of older individuals with vaccination history to reduce ascertainment bias.

Developing a reference standard for pertussis by applying a stratified sampling strategy to electronic medical record data

PURPOSE

Pertussis surveillance remains essential in Canada, but ascertainment bias limits the accuracy of surveillance data. Introducing other sources to improve detection has highlighted the importance of validation. However, challenges arise due to low prevalence, and oversampling suspected cases can introduce partial verification bias. The aim of this study was to build a reference standard for pertussis validation studies that provides adequate analytic precision and minimizes bias.

METHODS

We used a stratified strategy to sample the reference standard from a primary care electronic medical record cohort. We incorporated abstractor notes into definite, possible, ruled-out, and no mention of pertussis classifications which were based on surveillance case definitions.

RESULTS

We abstracted eight hundred records from the cohort of 404,922. There were 208 (26%) definite and 261 (32.6%) possible prevalent pertussis cases. Classifications demonstrated a wide variety of case severities. Abstraction reliability was moderate to substantial based on Cohen's kappa and raw percent agreement.

CONCLUSIONS

When conducting validation studies for pertussis and other low prevalence diseases, this stratified sampling strategy can be used to develop a reference standard using limited resources. This approach mitigates verification and spectrum bias while providing sufficient precision and incorporating a range of case severities.

Trends and characteristics of Tdap vaccination during pregnancy in Ontario, Canada: a retrospective cohort study

BACKGROUND

In February 2018, Canada's National Advisory Committee on Immunization (NACI) recommended tetanus toxoid, reduced diphtheria toxoid and acellular pertussis (Tdap) vaccination during pregnancy to protect newborns against pertussis infection. We sought to describe pre- and postrecommendation trends in Tdap vaccination coverage among pregnant Ontario residents.

METHODS

Using linked health administrative databases, we conducted a population-based retrospective cohort study of all pregnant individuals who gave birth in Ontario hospitals between April 2012 and March 2020. We described Tdap vaccination patterns in pregnancy for the entire study period and before and after the NACI recommendation. We used log-binomial regression to identify characteristics associated with Tdap vaccination during pregnancy.

RESULTS

Among the 991 850 deliveries included, 7.0% of pregnant individuals received the Tdap vaccination during pregnancy. Vaccine coverage increased from 0.4% in 2011/12 to 29.2% in 2019/20. Coverage was highest among individuals who were older, had no previous live births, had adequate prenatal care and received maternity care primarily from a family physician. After adjustment, characteristics associated with lower coverage included younger maternal age, having a multiple birth, residing in a rural location and higher area material deprivation. In 2019/20, 71.0% of vaccinated individuals received the Tdap vaccination during the recommended gestational window (27-32 wk). Stratified analyses of the pre- and postrecommendation cohorts yielded similar findings to the main analyses with a few gradient differences after adjustment.

INTERPRETATION

During pregnancy, Tdap vaccination coverage increased substantially in Ontario between 2011/12 and 2019/20, most notably after recommendations for universal Tdap vaccination during pregnancy began in Canada. To further improve vaccine coverage in the obstetric setting, public health strategies should consider tailoring their programs to reach subpopulations with lower vaccine coverage.

Impact of the COVID-19 pandemic on routine immunization coverage in children under 2 years old in Ontario, Canada: A retrospective cohort study

BACKGROUND

The COVID-19 pandemic has caused a disruption in childhood immunization coverage around the world. This study aimed to determine the change in immunization coverage for children under 2 years old in Ontario, Canada, comparing time periods pre-pandemic to during the first year of the pandemic.

METHODS

Observational retrospective open cohort study, using primary care electronic medical record data from the University of Toronto Practice-Based Research Network (UTOPIAN) database, from January 2019 to December 2020. Children under 2 years old who had at least 2 visits recorded in UTOPIAN were included. We measured up-to-date (UTD) immunization coverage rates, overall and by type of vaccine (DTaP-IPV-Hib, PCV13, Rota, Men-C-C, MMR, Var), and on-time immunization coverage rates by age milestone (2, 4, 6, 12, 15, 18 months). We compared average coverage rates over 3 periods of time: January 2019-March 2020 (T1); March-July 2020 (T2); and August-December 2020 (T3).

RESULTS

12,313 children were included. Overall UTD coverage for all children was 71.0% in T1, dropped by 5.7% (95% CI: -6.2, -5.1) in T2, slightly increased in T3 but remained lower than in T1. MMR vaccine UTD coverage slightly decreased in T2 and T3 by approximately 2%. The largest decreases were seen at ages 15-month and 18-month old, with drops in on-time coverage of 14.7% (95% CI: -18.7, -10.6) and 16.4% (95% CI: -20.0, -12.8) respectively during T2. When stratified by sociodemographic characteristics, no specific subgroup of children was found to have been differentially impacted by the pandemic.

CONCLUSION

Childhood immunization coverage rates for children under 2 years in Ontario decreased significantly during the early period of the COVID-19 pandemic and only partially recovered during the rest of 2020. Public health and educational interventions for providers and parents are needed to ensure adequate catch-up of delayed/missed immunizations to prevent potential outbreaks of vaccine-preventable diseases.

Pediatric primary care in Ontario and Manitoba after the onset of the COVID-19 pandemic: a population-based study

BACKGROUND

There were large disruptions to health care services after the onset of the COVID-19 pandemic. We sought to describe the extent to which pandemic-related changes in service delivery and access affected use of primary care for children overall and by equity strata in the 9 months after pandemic onset in Manitoba and Ontario.

METHODS

We performed a population-based study of children aged 17 years or less with provincial health insurance in Ontario or Manitoba before and during the COVID-19 pandemic (Jan. 1, 2017-Nov. 28, 2020). We calculated the weekly rates of in-person and virtual primary care well-child and sick visits, overall and by age group, neighbourhood material deprivation level, rurality and immigrant status, and assessed changes in visit rates after COVID-19 restrictions were imposed compared to expected baseline rates calculated for the 3 years before pandemic onset.

RESULTS

Among almost 3 million children in Ontario and more than 300 000 children in Manitoba, primary care visit rates declined to 0.80 (95% confidence interval [CI] 0.77-0.82) of expected in Ontario and 0.82 (95% CI 0.79-0.84) of expected in Manitoba in the 9 months after the onset of the pandemic. Virtual visits accounted for 53% and 29% of visits in Ontario and Manitoba, respectively. The largest monthly decreases in visits occurred in April 2020. Although visit rates increased slowly after April 2020, they had not returned to prerestriction levels by November 2020 in either province. Children aged more than 1 year to 12 years experienced the greatest decrease in visits, especially for well-child care. Compared to prepandemic levels, visit rates were lowest among rural Manitobans, urban Ontarians and Ontarians in low-income neighbourhoods.

INTERPRETATION

During the study period, the pandemic contributed to rapid, immediate and inequitable decreases in primary care use, with some recovery and a substantial shift to virtual care. Postpandemic planning must consider the need for catch-up visits, and the long-term impacts warrant further study.

A Call for Caution in Use of Pertussis Vaccine Effectiveness Studies to Estimate Waning Immunity: A Canadian Immunization Research Network Study

BACKGROUND

Vaccine effectiveness (VE) studies provide essential evidence on waning vaccine-derived immunity, a major threat to pertussis control. We evaluated how study design affects estimates by comparing 2 case-control studies conducted in Ontario, Canada.

METHODS

We compared results from a test-negative design (TND) with a frequency-matched design (FMD) case-control study using pertussis cases from 2005-2015. In the first study, we identified test-negative controls from the public health laboratory that diagnosed cases and, in the second, randomly selected controls from patients attending the same physicians that reported cases, frequency matched on age and year. We compared characteristics of cases and controls using standardized differences.

RESULTS

In both designs, VE estimates for the early years postimmunization were consistent with clinical trials (TND, 84%; FMD, 89% at 1-3 years postvaccination) but diverged as time since last vaccination increased (TND, 41%; FMD, 74% by 8 years postvaccination). Overall, we observed lower VE and faster waning in the TND than the FMD. In the TND but not FMD, controls differed from cases in important confounders, being younger, having more comorbidities, and higher healthcare use. Differences between the controls of each design were greater than differences between cases. TND controls were more likely to be unvaccinated or incompletely vaccinated than FMD controls (P < .001).

CONCLUSIONS

The FMD adjusted better for healthcare-seeking behavior than the TND. Duration of protection from pertussis vaccines is unclear because estimates vary by study design. Caution should be exercised by experts, researchers, and decision makers when evaluating evidence on optimal timing of boosters.

Health Outcomes in Young Children Following Pertussis Vaccination During Pregnancy

BACKGROUND AND OBJECTIVES

Maternal immunization with tetanus, diphtheria, and acellular pertussis vaccine (Tdap) is routinely recommended in many countries as a strategy to protect young infants against severe pertussis infection; few studies have assessed whether prenatal exposure to the vaccine is associated with any longer-term adverse health effects in children. We evaluated the long-term safety of exposure to Tdap vaccination during pregnancy.

METHODS

Population-based retrospective cohort study conducted in Ontario, Canada using multiple linked province-wide health administrative databases. All live births between April 2012 and March 2017 were included, and children were followed for up to 6 years to ascertain study outcomes. Children exposed to prenatal Tdap were propensity score matched to unexposed children at a 1:5 ratio. Tdap vaccination during pregnancy was ascertained by using vaccine-specific fee codes. Immune-related (infectious diseases, asthma) and nonimmune-related (neoplasm, sensory disorders) outcomes and a nonspecific morbidity outcome (urgent or inpatient health service use) were evaluated from birth to end of follow-up.

RESULTS

Of 625 643 live births, 12 045 (1.9%) were exposed to Tdap in utero. There were no significant increased risks of adverse childhood outcomes and prenatal Tdap exposure; however, we observed inverse associations (adjusted incidence rate ratio [95% confidence interval]) with upper respiratory infections (0.94 [0.90-0.99]), gastrointestinal infections (0.85 [0.79-0.91]), and urgent and inpatient health service use (0.93 [0.91-0.96]).

CONCLUSIONS

Exposure to Tdap vaccination in pregnancy was not associated with any increased risk of adverse health outcomes in early childhood, supporting the long-term safety of Tdap administration in pregnancy.

A multisite study of pertussis vaccine effectiveness by time since last vaccine dose from three Canadian provinces: A Canadian Immunization Research Network study

BACKGROUND

Pertussis remains poorly controlled relative to other diseases targeted by childhood vaccination programs. We combined estimates from four population-based studies of pertussis vaccine effectiveness (VE) in three Canadian provinces using a meta-analytic approach to improve precision and explore regional variation in VE and durability of protection.

METHODS

Studies were conducted in Alberta, Manitoba, and Ontario over periods ranging from 1996 to 2015. Adjusted log odds ratios (OR; VE = 100*[1-OR]) of the effect of vaccination on pertussis risk were estimated by time since last vaccination in each study and pooled using DerSimonian and Laird random-effects models. We used the I² statistic to estimate between-study heterogeneity and assessed methodological and clinical heterogeneity through subgroup analyses of study design and age.

RESULTS

Data on 3,270 pertussis cases and 23,863 controls were available. Pertussis VE declined from 86% (95% CI 79%-90%, I² = 81.5%) at < 1 year since last vaccination to 51% (11%-74%, I² = 80.9%) by ≥ 8 years. Effect estimates were the most heterogeneous in the least and most elapsed time periods since last vaccine dose. This was attributable mostly to variation between provinces in the distribution of age groups and number of vaccine doses received within time periods, as well as study design and small numbers in the most elapsed time period.

INTERPRETATION

Consistent trends of decreasing pertussis VE with increasing time since last vaccination across three Canadian provinces indicate the need for immunization schedules and vaccine development to optimize protection for all individuals, especially for adolescents and young adults at greatest risk of infection.

Obstetric and perinatal health outcomes after pertussis vaccination during pregnancy in Ontario, Canada: a retrospective cohort study

BACKGROUND

In February 2018, Canada's National Advisory Committee on Immunization recommended maternal vaccination with tetanus-diphtheria-acellular pertussis (Tdap) vaccine during pregnancy to prevent severe pertussis infection in young infants. This study assessed the relation between maternal Tdap vaccination and obstetric and perinatal outcomes in Ontario.

METHODS

We performed a population-based cohort study of all births from April 2012 to March 2017 using multiple linked health administrative databases. We used Cox regression with a time-dependent exposure variable to estimate adjusted hazard ratios (HRs) for preterm birth (< 37 wk), very preterm birth (< 32 wk) and stillbirth. We assessed remaining outcomes (gestational hypertension, chorioamnionitis, postpartum hemorrhage, severe postpartum hemorrhage, being small for gestational age, neonatal intensive care unit stay > 24 h, composite neonatal morbidity) using log-binomial regression to generate adjusted risk ratios (RRs). We adjusted estimates for potential confounding using propensity score weighting.

RESULTS

Of 615 213 infants (live births and stillbirths), 11 519 were exposed to Tdap vaccination in utero. There was no increased risk for preterm birth (adjusted HR 0.98, 95% confidence interval [CI] 0.91-1.06), very preterm birth (adjusted HR 1.10, 95% CI 0.86-1.41), stillbirth (adjusted HR 1.15, 95% CI 0.82-1.60) or being small for gestational age (adjusted RR 0.96, 95% CI 0.90-1.02). The risks of a neonatal intensive care unit stay exceeding 24 hours (adjusted RR 0.82, 95% CI 0.76-0.88) and neonatal morbidity (adjusted RR 0.81, 95% CI 0.75-0.87) were decreased. There was no association with chorioamnionitis (adjusted RR 1.17, 95% CI 0.99-1.39), postpartum hemorrhage (adjusted RR 1.01, 95% CI 0.91-1.13) or severe postpartum hemorrhage (adjusted RR 0.79, 95% CI 0.55-1.13), but we observed a reduced risk of gestational hypertension (adjusted RR 0.87, 95% CI 0.78-0.96).

INTERPRETATION

Our results complement evidence that maternal Tdap vaccination is not associated with adverse outcomes in mothers or infants. Ongoing evaluation in Canada is needed as maternal Tdap vaccination coverage increases in coming years.

Vaccine coverage among children with epilepsy in two Canadian provinces: A Canadian immunization research network study

OBJECTIVES

Children with epilepsy are at increased risk of complications from vaccine-preventable infections, yet information on vaccine coverage in these children is scarce. We aimed to compare vaccine coverage among children with epilepsy to children without epilepsy.

STUDY DESIGN

We conducted a retrospective cohort study including all 2005-2013 births in Manitoba and Ontario, Canada, creating two cohorts: 2-year-olds and 7-year-olds (followed to age 2 and 7 years). We split each cohort into epilepsy and non-epilepsy subcohorts. We assessed vaccination coverage based on provincial schedules and determined timeliness of MMR (measles, mumps, rubella) dose 1 (recommended at 12 months) and DTaP (diphtheria, tetanus, pertussis) dose 4 (recommended at 18 months). We used logistic regression to calculate adjusted odds ratios (aORs) of the association between epilepsy and vaccination, combining both provincial estimates using random effects meta-analysis.

RESULTS

We included 16,558 2-year-olds (Manitoba, 653; Ontario, 15,905) and 13,004 7-year-olds (Manitoba, 483; Ontario, 12,521) with epilepsy. At age 2 years, the aOR for up-to-date vaccination among children with versus without epilepsy was 0.9 (95% confidence interval 0.8-1.1); at age 7 years it was 1.0 (0.9-1.1). Infants diagnosed with epilepsy before age 6 months were less likely to be up-to-date at age 2 years (0.9; 0.8-0.9), although this difference disappeared by age 7 years. Vaccine timeliness was similar between children with and without epilepsy for MMR dose 1 and DTaP dose 4.

CONCLUSIONS

Overall, this study suggests that children with epilepsy are not significantly under-vaccinated compared to their peers without epilepsy. As children with epilepsy are at a higher risk of complications from vaccine-preventable diseases, vaccination in children with epilepsy should be optimized, especially early in life, as these children may not be able to rely on herd protection.

Challenges Frequently Encountered in the Secondary Use of Electronic Medical Record Data for Research

The wide adoption of electronic medical records and subsequent availability of large amounts of clinical data provide a rich resource for researchers. However, the secondary use of clinical data for research purposes is not without limitations. In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a systematic review to identify current issues related to secondary use of electronic medical record data via MEDLINE and CINAHL databases. All articles published until June 2018 were included. Sixty articles remained after title and abstract review, and four domains of potential limitations were identified: (1) data quality issues, present in 91.7% of the articles reviewed; (2) data preprocessing challenges (53.3%); (3) privacy concerns (18.3%); and (4) potential for limited generalizability (21.7%). Researchers must be aware of the limitations inherent to the use of electronic medical record data for research and consider the potential effects of these limitations throughout the entire study process, from initial conceptualization to the identification of adequate sources that can provide data appropriate for answering the research questions, analysis, and reporting study results. Consideration should also be given to using existing data quality assessment frameworks to facilitate use of standardized data quality definitions and further efforts of standard data quality reporting in publications.

Comparing prescribing and dispensing databases to study antibiotic use: a validation study of the Electronic Medical Record Administrative data Linked Database (EMRALD)

BACKGROUND

Monitoring and studying community antibiotic use is a critical component in combating rising antimicrobial resistance.

OBJECTIVES

To validate an electronic medical record dataset containing antibiotic prescriptions and to quantify some important differences between prescribing and dispensing databases.

METHODS

We evaluated antibiotics prescribed and dispensed to patients ≥65 years of age during 2011-15. We compared the EMRALD prescribing database with the validated Ontario Drug Benefit (ODB) dispensing database. Using ODB as the gold standard and limiting to EMRALD physicians, we calculated sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) with 95% CIs. We also compared the relative change in antibiotic use prescribed by all physicians to this population over time between the databases using Poisson regression models.

RESULTS

In this population, 74% of all antibiotics dispensed were from non-EMRALD physicians. Trends in use were discordant over time. When we limited ODB to EMRALD prescribers only to assess the validity of EMRALD data, we observed good sensitivity and excellent specificity for correctly identifying antibiotics at 85% (95% CI 84%-85%) and 98% (95% CI 98%-98%), respectively. The PPV was 78% (95% CI 78%-78%) and the NPV was 99% (95% CI 99%-99%). All performance measures were higher among the highest prescribing physicians.

CONCLUSIONS

We demonstrated EMRALD is well suited for studying antibiotic prescribing by EMRALD physicians. However, due to the frequency with which patients receive antibiotic prescriptions from their non-primary care physicians, we caution against the use of non-population-based prescribing databases to infer antibiotic use rates or trends over time.

Receipt of routine preventive care among infant daughters and sons of immigrant mothers in Ontario, Canada: a retrospective cohort study

OBJECTIVES

To explore gender disparities in infant routine preventive care across maternal countries of birth (MCOB) and by mother tongue among infants of Indian-born mothers.

SETTING

Retrospective population-based administrative cohort in Ontario, Canada (births between 2002 and 2014).

PARTICIPANTS

350 366 (inclusive) healthy term singletons belonging to families with a minimum of one opposite gender child.

OUTCOME MEASURES

Fixed effects conditional logistic regression generated adjusted ORs (aORs) for a daughter being underimmunised and having an inadequate number of well-child visits compared with her brother, stratified by MCOB. Moderation by maternal mother tongue was assessed among children to Indian-born mothers.

RESULTS

Underimmunisation and inadequate well-child visits were common among both boys and girls, ranging from 26.5% to 58.2% (underimmunisation) and 10.5% to 47.8% (inadequate well-child visits). depending on the maternal birthplace. Girls whose mothers were born in India had 1.19 times (95% CI 1.07 to 1.33) the adjusted odds of inadequate well-child visits versus their brothers. This association was only observed among the Punjabi mother tongue subgroup (aOR: 1.26, 95% CI 1.08 to 1.47). In the Hindi mother tongue subgroup, girls had lower odds of underimmunisation than their brothers (aOR: 0.73, 95% CI 0.54 to 0.98).

CONCLUSIONS

Gender equity in routine preventive healthcare is mostly achieved among children of immigrants. However, daughters of Indian-born mothers whose mother tongue is Punjabi, appear to be at a disadvantage for well-child visits compared with their brothers. This suggests son preference may persist beyond the family planning stage among some Indian immigrants.

Assessing the completeness of infant and childhood immunizations within a provincial registry populated by parental reporting: A study using linked databases in Ontario, Canada

INTRODUCTION

In Ontario, Canada, parents have the responsibility to report their child's routine infant and childhood vaccines to the provincial immunization registry (the Digital Health Immunization Repository; DHIR) without healthcare provider validation. Despite its use in routine immunization coverage monitoring, no study has previously examined the completeness of immunization data within the DHIR.

METHODS

We assessed the completeness of DHIR immunizations, as compared to immunizations within the Electronic Medical Records-Primary Care (EMRPC) database, also known as EMRALD, a network of family physician electronic medical records (EMRs). We linked client records from the DHIR and EMRPC to a centralized population file. To create the study cohort, we examined children born during 2005-2008 and further defined the cohort based on those rostered to an EMRPC physician, visit criteria to ensure ongoing care by an EMRPC provider, and school attendance in Ontario at age 7. We calculated up-to-date (UTD) immunization coverage at age 7 for individual vaccines and overall using data from the DHIR and EMRPC separately, and compared the estimates.

RESULTS

The analytic cohort to assess DHIR data completeness included 2,657 children. Overall UTD coverage (all vaccines assessed) was 82.0% in the DHIR and 67.6% in EMRPC. UTD coverage was higher in the DHIR for all vaccines assessed individually, with the exception of meningococcal C conjugate vaccine (difference = 0.3%). After excluding two EMRPC sites with irregularities in immunization data, the difference in overall UTD coverage between systems decreased from 14.4% to 6.6% INTERPRETATION: These results validate the use of DHIR for coverage assessment but also suggest that bidirectional exchange of immunization information has the potential to increase immunization data completeness in both systems.

Adverse Events Following Immunization Among Children With Epilepsy: A Self-Controlled Case Series

BACKGROUND

In children with epilepsy, fever and illness are known triggers for seizure; therefore, clinicians and parents could be concerned that immunization-induced inflammation and fever could also trigger seizures. We sought to estimate the risk of emergency department (ED) visit or hospitalization for epilepsy/seizure and all causes after immunization in children younger than 7 years of age with epilepsy.

METHODS

We conducted a self-controlled case series of children diagnosed with epilepsy before their 7th birthday and immunized from 2005 to 2015 in Ontario (population 14.2 million) and Manitoba (population 1.3 million), Canada, using administrative healthcare data. We estimated the age- and season-adjusted relative incidence (aRI) of epilepsy/seizure-related and all-cause ED visits/hospitalizations during various risk periods 0-28 days after inactivated and live immunizations versus a control period 35-83 days postimmunization. Estimates from each province were analyzed separately and then combined in a random-effects meta-analysis.

RESULTS

The combined risk of epilepsy/seizure-related hospitalization/ED visit was increased 0-2 days after inactivated vaccines (aRI = 1.5, 95% confidence interval: 1.1-1.9) and 7-10 days after live vaccines (aRI = 1.9, 1.4-2.7). For all-cause ED visit/hospitalization, the combined aRI estimate was 0.9 (0.8-1.2) 0-2 days after inactivated vaccines and 1.3 (1.1-1.5) 7-10 days after live vaccines.

CONCLUSIONS

The risk of epilepsy/seizure-related ED visit/hospitalization was modestly increased among children with epilepsy during peak periods of fever and inflammation following inactivated and live vaccines. These risks must be balanced against the risk of complications from vaccine-preventable diseases.

Primary care performance of alternatively licenced physicians in Ontario, Canada: a cross-sectional study using administrative data

OBJECTIVES

Medical Regulatory Authorities (MRAs) provide licences to physicians and monitor those physicians once in practice to support their continued competence. In response to physician shortages, many Canadian MRAs developed alternative licensure routes to allow physicians who do not meet traditional licensure criteria to obtain licences to practice. Many physicians have gained licensure through alternative routes, but the performance of these physicians in practice has not been previously examined. This study compared the performance of traditionally and alternatively licenced physicians in Ontario using quality indicators of primary care. The purpose of this study was to examine the practice performance of alternatively licenced physicians and provide evaluative evidence for alternative licensure policies.

DESIGN

A cross-sectional retrospective examination of Ontario health administrative data was conducted using Poisson regression analyses to compare the performance of traditionally and alternatively licenced physicians.

SETTING

Primary care in Ontario, Canada.

PARTICIPANTS

All family physicians who were licenced in Ontario between 2000 and 2012 and who had complete medical billing data in 2014 were included (n=11 419).

OUTCOME MEASURES

Primary care quality indicators were calculated for chronic disease management, preventive paediatric care, cancer screening and hospital readmission rates using Ontario health administrative data.

RESULTS

Alternatively licenced physicians performed similarly to traditionally licenced physicians in many primary care performance measures. Minimal differences were seen across groups in indicators of diabetic care, congestive heart failure care, asthma care and cancer screening rates. Larger differences were found in preventive care for children less than 2 years of age, particularly for alternatively licenced physicians who entered Ontario from another Canadian province.

CONCLUSIONS

Our findings demonstrate that alternatively licenced physicians perform similarly to traditionally licenced physicians across many indicators of primary care. Our study also demonstrates the utility of administrative data for examining physician performance and evaluating medical regulatory policies and programmes.

Overweight and obesity in preschool aged children and risk of mental health service utilization

OBJECTIVE

To determine if overweight or obesity in preschool-age children is associated with mental health service utilization in later childhood.

SUBJECTS

Overall, 10,522 children 2 to <5 years, with no previous history of mental health service utilization, were identified from primary care electronic medical records (EMRs) across Ontario, Canada.

METHODS

This was a retrospective longitudinal cohort study. Height and weight data were extracted and body mass index z-scores (zBMI) were calculated using the World Health Organization Growth Standards. Mental health service utilization, between ages 5 and <19, was defined using administrative billing codes for mental health outpatient visits, emergency department visits, and hospitalizations. A multivariable Cox proportional hazards model was performed.

RESULTS

In total, 74.9% of children were healthy weight (zBMI between -2 and ≤1), 18.8% of children were at risk of overweight (zBMI between 1 and ≤2), 4.9% were overweight (zBMI > 2 and ≤3), and 1.5% had obesity (zBMI > 3). The median follow-up time was 2.2 years (IQR 1.0-4.2). The overall incidence rate of mental health service use was 44.5 events per 1000 person-years. The hazard ratio for girls with obesity was 2.73 (95% CI 1.62-4.60; p < 0.001) compared to girls with healthy weight. Compared to boys with healthy weight, boys 'at risk of overweight' and overweight were 1.22 (95% CI 1.03-1.44; p = 0.02) and 1.43 (95% CI 1.09-1.87; p = 0.01) times at higher risk of an incident mental health visit.

CONCLUSION

Our study shows an association between weight status in preschool school aged children and higher incidence of mental health service use in later childhood. This relationship was strongest in girls. Future research is needed to understand this relationship by mental health diagnosis, sex, and age.

Use of physician billing claims to identify infections in children

While medical records have detailed information, they are limited in reach to the availability and accessibility of those records. On the other hand, administrative data while limited in scope, have a much further reach in coverage of an entire population. However, few studies have validated the use of administrative data for identifying infections in pediatric populations. Pediatric patients from Ontario, Canada aged <18 years were randomly sampled from the Electronic Medical Record Administrative data Linked Database (EMRALD). Using physician diagnoses from the electronic medical record (EMR) as the reference standard, we determined the criterion validity of physician billing claims in administrative data for identifying infectious disease syndromes from 2012 to 2014. Diagnosis codes were assessed by infection category (respiratory, skin and soft tissue, gastrointestinal, urinary tract and otitis externa) and for all infections combined. Sensitivity analyses assessed the performance if patients had more than one reason to visit the physician. We analysed 2,139 patients and found 33.3% of all visits were for an infection, and respiratory infections accounted for 67.6% of the infections. When we combined all infection categories, sensitivity was 0.74 (95% CI 0.70-0.77), specificity was 0.95 (95% CI 0.93-0.96), positive predictive value (PPV) was 0.87 (95% CI 0.84-0.90), and negative predictive value (NPV) was 0.88 (95% CI 0.86-0.89). For respiratory infections, sensitivity was 0.77 (95% CI 0.73-0.81), specificity was 0.96 (95% CI 0.95-0.97), PPV was 0.85 (95% CI 0.81-0.88), and NPV was 0.94 (95% CI 0.92-0.95). Similar performance was observed for skin and soft tissue, gastrointestinal, urinary tract, and otitis externa infections, but with lower sensitivity. Performance measures were highest when the patient visited the physician with only one health complaint. We found when using linked EMR data as the reference standard, administrative billing codes are reasonably accurate in identifying infections in a pediatric population.

A matched case-control study to assess the association between non-steroidal anti-inflammatory drug use and thrombotic microangiopathy

Several case reports suggest that non-steroidal anti-inflammatory drug (NSAID) use is associated with development of thrombotic microangiopathy (TMA). We conducted a matched population-based case-control study using linked administrative healthcare data in Ontario, Canada to assess the relationship between TMA hospitalization and recent exposure to prescription NSAIDs versus acetaminophen (acetaminophen was chosen as the referent drug because it has no known association with TMA). Cases and controls were drawn from a source population of adults who filled a prescription for either NSAIDs or acetaminophen between 1991 and 2015 (restricted to adults with prescription drug benefits [n = 3.6 million]). We identified 44 eligible cases with a hospital admission for incident TMA and a recent prescription for NSAIDs or acetaminophen. We successfully matched 38 cases (1:4) to 152 controls without TMA on demographics, risk factors for TMA, and indications for NSAID use. Cases and controls were similar with respect to age (71 years) and sex (63% women); however, on average, cases had more comorbidities than controls (12 vs. 14; p<0.05) and more primary care visits in the year before the index date (19 vs. 12; p<0.05). Cases were significantly less likely than controls to have received a recent prescription for NSAIDs (19/38 [50%] vs. 115/152 [76%], respectively; adjusted odds ratio 0.37, 95% confidence interval: 0.16 to 0.84). Results were similar in several sensitivity analyses. Overall, the results of this study do not support a harmful association between NSAID use and the development of TMA.

T-cell receptor excision circle levels and safety of paediatric immunization: A population-based self-controlled case series analysis

T-cell receptor excision circle levels are a surrogate marker of T-cell production and immune system function. We sought to determine whether non-pathological levels of infant T-cell receptor excision circles were associated with adverse events following immunization. A self-controlled case series design was applied on a sample of 231,693 children who completed newborn screening for severe combined immunodeficiency in Ontario, Canada between August 2013 and December 2015. Exposures included routinely administered pediatric vaccines up to 15 months of age. Main outcomes were combined health services utilization for recognized adverse events following immunization. 1,406,981 vaccination events were included in the final dataset. 103,007 children received the Pneu-C-13 or Men-C-C vaccine and 97,998 received the MMR vaccine at 12 months of age. 67,725 children received the varicella immunization at 15 months. Our analysis identified no association between newborn T-cell receptor excision circle levels and subsequent health services utilization events following DTa-IPV-Hib, Pneu-C-13, and Men-C-C vaccinations at 2-month (RI 0.94[95%CI 0.87-1.02]), 4-month (RI 0.82[95%CI 0.75-0.9]), 6-month (RI 0.63[95%CI 0.57-0.7]) and 12-month (RI 0.49[95%CI 0.44-0.55]). We also found no trends in health services utilization following MMR (RI 1.43[95%1.34-1.52]) or varicella (RI 1.14[95%CI 1.05-1.23]) vaccination. Our findings provide further support for the safety of pediatric vaccinations.

Under-reporting of pertussis in Ontario: A Canadian Immunization Research Network (CIRN) study using capture-recapture

Introduction

Under-reporting of pertussis cases is a longstanding challenge. We estimated the true number of pertussis cases in Ontario using multiple data sources, and evaluated the completeness of each source.

Methods

We linked data from multiple sources for the period 2009 to 2015: public health reportable disease surveillance data, public health laboratory data, and health administrative data (hospitalizations, emergency department visits, and physician office visits). To estimate the total number of pertussis cases in Ontario, we used a three-source capture-recapture analysis stratified by age (infants, or aged one year and older) and adjusting for dependency between sources. We used the Bayesian Information Criterion to compare models.

Results

Using probable and confirmed reported cases, laboratory data, and combined hospitalizations/emergency department visits, the estimated total number of cases during the six-year period amongst infants was 924, compared with 545 unique observed cases from all sources. Using the same sources, the estimated total for those aged 1 year and older was 12,883, compared with 3,304 observed cases from all sources. Only 37% of infants and 11% for those aged 1 year and over admitted to hospital or seen in an emergency department for pertussis were reported to public health. Public health reporting sensitivity varied from 2% to 68% depending on age group and the combination of data sources included. Sensitivity of combined hospitalizations and emergency department visits varied from 37% to 49% and of laboratory data from 1% to 50%.

Conclusions

All data sources contribute cases and are complementary, suggesting that the incidence of pertussis is substantially higher than suggested by routine reports. The sensitivity of different data sources varies. Better case identification is required to improve pertussis control in Ontario.

Rotavirus vaccine coverage and factors associated with uptake using linked data: Ontario, Canada

BACKGROUND

In August 2011, Ontario, Canada introduced a rotavirus immunization program using Rotarix™ vaccine. No assessments of rotavirus vaccine coverage have been previously conducted in Ontario.

METHODS

We assessed vaccine coverage (series initiation and completion) and factors associated with uptake using the Electronic Medical Record Administrative data Linked Database (EMRALD), a collection of family physician electronic medical records (EMR) linked to health administrative data. Series initiation (1 dose) and series completion (2 doses) before and after the program's introduction were calculated. To identify factors associated with series initiation and completion, adjusted odds ratios (aOR) and 95% confidence intervals (95%CI) were calculated using logistic regression.

RESULTS

A total of 12,525 children were included. Series completion increased each year of the program (73%, 79% and 84%, respectively). Factors associated with series initiation included high continuity of care (aOR = 2.15; 95%CI, 1.61-2.87), maternal influenza vaccination (aOR = 1.55; 95%CI,1.24-1.93), maternal immmigration to Canada in the last five years (aOR = 1.47; 95% CI, 1.05-2.04), and having no siblings (aOR = 1.62; 95%CI,1.30-2.03). Relative to the first program year, infants were more likely to initiate the series in the second year (aOR = 1.71; 95% CI 1.39-2.10) and third year (aOR = 2.02; 95% CI 1.56-2.61) of the program. Infants receiving care from physicians with large practices were less likely to initiate the series (aOR 0.91; 95%CI, 0.88-0.94, per 100 patients rostered) and less likely to complete the series (aOR 0.94; 95%CI, 0.91-0.97, per 100 patients rostered). Additional associations were identified for series completion.

CONCLUSIONS

Family physician delivery achieved moderately high coverage in the program's first three years. This assessment demonstrates the usefulness of EMR data for evaluating vaccine coverage. Important insights into factors associated with initiation or completion (i.e. high continuity of care, smaller roster sizes, rural practice location) suggest areas for research and potential program supports.

Limitations of administrative data to identify measles cases in Ontario, Canada: a cautionary tale

OBJECTIVE

To explore the utility of laboratory data and determine the validity of healthcare administrative data for describing the frequency of measles in Ontario.

METHODS

We linked Ontario healthcare utilization administrative data to Public Health Ontario (PHO) laboratory data from 01 January 2006 to 30 November 2012.

RESULTS

The sensitivity of the administrative data was 54% and the positive predictive value was 1% when compared with 50 cases identified in laboratory data as a gold standard.

CONCLUSIONS

As measles is no longer endemic in Ontario, the high number of measles-coded healthcare visits found in Ontario health administration data exceeds the true number of cases. Great caution should be taken in using administrative data to calculate the incidence of measles in areas where it has been eliminated.

Manual review of electronic medical records as a reference standard for case definition development: a validation study

BACKGROUND

The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) previously carried out a validation study of case definitions for 8 chronic diseases (diabetes mellitus, hypertension, osteoarthritis, depression, dementia, chronic obstructive pulmonary disease, parkinsonism and epilepsy) using direct review of "raw" electronic medical record data. Although effective, this method is time-consuming and can present methodological and organizational challenges. We aimed to determine whether the processed and standardized data contained with the CPCSSN database might function as a reference standard for case definition validation.

METHODS

Using a traditional validation study design, we compared the case identification results of the chart reviews for the 8 chronic diseases with the results of a manual review of the CPCSSN processed data for the same conditions in the same patient sample. Patients were randomly sampled from the June 30, 2012 CPCSSN database, with oversampling of patients with rare conditions.

RESULTS

We analyzed data for 1906 patients. Manual review of the CPCSSN records for case ascertainment yielded sensitivity ranging from 77.5% (95% confidence interval [CI] 73.3%-81.6%) for depression to 97.2% (95% CI 95.4%-99.0%) for diabetes. Specificity was high for all definitions (range 93.1% [95% CI 91.4%-94.7%] to 99.4% [95% CI 99.0%-99.8%]). Positive predictive values and negative predictive values also showed high accuracy of the manual CPCSSN record review relative to review of the raw chart data.

INTERPRETATION

The use of CPCSSN records as the reference standard to validate case definitions substantially reduces the burden on sentinel physicians and clinic managers as well as on researchers while offering a reference standard that is a reasonable substitution for chart review.

How well do ICD-9 physician claim diagnostic codes identify confirmed pertussis cases in Alberta, Canada? A Canadian Immunization Research Network (CIRN) Study

Background

Rates of Bordetella pertussis have been increasing in Alberta, Canada despite vaccination programs. Waning immunity from existing acellular component vaccines may be contributing to this. Vaccine effectiveness can be estimated using a variety of data sources including diagnostic codes from physician billing claims, public health records, reportable disease and laboratory databases. We sought to determine if diagnostic codes from billing claims (administrative data) are adequately sensitive and specific to identify pertussis cases among patients who had undergone disease-specific laboratory testing.

Methods

Data were extracted for 2004–2014 from a public health communicable disease database that contained data on patients under investigation for B. pertussis (both those who had laboratory tests and those who were epidemiologically linked to laboratory-confirmed cases) in Alberta, Canada. These were deterministically linked using a unique lifetime person identifier to the provincial billing claims database, which contains International Classification of Disease version 9 (ICD-9) diagnostic codes for physician visits. We examined visits within 90 days of laboratory testing. ICD-9 codes 033 (whooping cough), 033.0 (Bordetella pertussis), 033.1 (B. parapertussis), 033.8 (whooping cough, other specified organism), and 033.9 (whooping cough, other unspecified organism) in any of the three diagnostic fields for a claim were classified as being pertussis-specific codes. We calculated sensitivity, specificity, positive (PPV) and negative (NPV) predictive values.

Results

We identified 22,883 unique patients under investigation for B. pertussis. Of these, 22,095 underwent laboratory testing. Among those who had a laboratory test, 2360 tested positive for pertussis. The sensitivity of a pertussis-specific ICD-9 code for identifying a laboratory-confirmed case was 38.6%, specificity was 76.9%, PPV was 16.0%, and NPV was 91.6%.

Conclusion

ICD-9 codes from physician billing claims data have low sensitivity and moderate specificity to identify laboratory-confirmed pertussis among persons tested for pertussis.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-017-2321-1) contains supplementary material, which is available to authorized users.

Effectiveness of pertussis vaccination and duration of immunity

BACKGROUND

A resurgence of pertussis cases among both vaccinated and unvaccinated people raises questions about vaccine effectiveness over time. Our objective was to study the effectiveness of the pertussis vaccine and characterize the effect of waning immunity and whole-cell vaccine priming.

METHODS

We used the test-negative design, a nested case-control study with test-negative individuals as controls. We constructed multivariable logistic regression models to estimate odds ratios (ORs). Vaccine effectiveness was calculated as (1 - OR) × 100. We assessed waning immunity by calculating the odds of developing pertussis per year since last vaccination and evaluated the relative effectiveness of priming with acellular versus whole-cell vaccine.

RESULTS

Between Dec. 7, 2009, and Mar. 31, 2013, data on 5867 individuals (486 test-positive cases and 5381 test-negative controls) were available for analysis. Adjusted vaccine effectiveness was 80% (95% confidence interval [CI] 71% to 86%) at 15-364 days, 84% (95% CI 77% to 89%) at 1-3 years, 62% (95% CI 42% to 75%) at 4-7 years and 41% (95% CI 0% to 66%) at 8 or more years since last vaccination. We observed waning immunity with the acellular vaccine, with an adjusted OR for pertussis infection of 1.27 (95% CI 1.20 to 1.34) per year since last vaccination. Acellular, versus whole-cell, vaccine priming was associated with an increased odds of pertussis (adjusted OR 2.15, 95% CI 1.30 to 3.57).

INTERPRETATION

We observed high early effectiveness of the pertussis vaccine that rapidly declined as time since last vaccination surpassed 4 years, particularly with acellular vaccine priming. Considering whole-cell vaccine priming and/or boosters in pregnancy to optimize pertussis control may be prudent.

Incidence of Hospitalization for Respiratory Syncytial Virus Infection amongst Children in Ontario, Canada: A Population-Based Study Using Validated Health Administrative Data

Importance

RSV is a common illness among young children that causes significant morbidity and health care costs.

Objective

Routinely collected health administrative data can be used to track disease incidence, explore risk factors and conduct health services research. Due to potential for misclassification bias, the accuracy of data-elements should be validated prior to use. The objectives of this study were to validate an algorithm to accurately identify pediatric cases of hospitalized respiratory syncytial virus (RSV) from within Ontario’s health administrative data, estimate annual incidence of hospitalization due to RSV and report the prevalence of major risk factors within hospitalized patients.

Study Design and Setting

A retrospective chart review was performed to establish a reference-standard cohort of children from the Ottawa region admitted to the Children’s Hospital of Eastern Ontario (CHEO) for RSV-related disease in 2010 and 2011. Chart review data was linked to Ontario’s administrative data and used to evaluate the diagnostic accuracy of algorithms of RSV-related ICD-10 codes within provincial hospitalization and emergency department databases. Age- and sex-standardized incidence was calculated over time, with trends in incidence assessed using Poisson regression.

Results

From a total of 1411 admissions, chart review identified 327 children hospitalized for laboratory confirmed RSV-related disease. Following linkage to administrative data and restriction to first admissions, there were 289 RSV patients in the reference-standard cohort. The best algorithm, based on hospitalization data, resulted in sensitivity 97.9% (95%CI: 95.5–99.2%), specificity 99.6% (95%CI: 98.2–99.8%), PPV 96.9% (95%CI: 94.2–98.6%), NPV 99.4% (95%CI: 99.4–99.9%). Incidence of hospitalized RSV in Ontario from 2005–2012 was 10.2 per 1000 children under 1 year and 4.8 per 1000 children aged 1 to 3 years. During the surveillance period, there was no identifiable increasing or decreasing linear trend in the incidence of hospitalized RSV, hospital length of stay and PICU admission rates. Among the Ontario RSV cohort, 16.3% had one or more major risk factors, with a decreasing trend observed over time.

Conclusion

Children hospitalized for RSV-related disease can be accurately identified within population-based health administrative data. RSV is a major public health concern and incidence has not changed over time, suggesting a lack of progress in prevention.