Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures

Effect of patient selection method on provider group performance estimates

BACKGROUND

Performance measurement at the provider group level is increasingly advocated, but different methods for selecting patients when calculating provider group performance have received little evaluation.

OBJECTIVE

We compared 2 currently used methods according to characteristics of the patients selected and impact on performance estimates.

RESEARCH DESIGN, SUBJECTS, AND MEASURES

We analyzed Medicare claims data for fee-for-service beneficiaries with diabetes ever seen at an academic multispeciality physician group in 2003 to 2004. We examined sample size, sociodemographics, clinical characteristics, and receipt of recommended diabetes monitoring in 2004 for the groups of patients selected using 2 methods implemented in large-scale performance initiatives: the Plurality Provider Algorithm and the Diabetes Care Home method. We examined differences among discordantly assigned patients to determine evidence for differential selection regarding these measures.

RESULTS

Fewer patients were selected under the Diabetes Care Home method (n=3558) than the Plurality Provider Algorithm (n=4859). Compared with the Plurality Provider Algorithm, the Diabetes Care Home method preferentially selected patients who were female, not entitled because of disability, older, more likely to have hypertension, and less likely to have kidney disease and peripheral vascular disease, and had lower levels of predicted utilization. Diabetes performance was higher under Diabetes Care Home method, with 67% versus 58% receiving >1 A1c tests, 70% versus 65% receiving ≥1 low-density lipoprotein (LDL) test, and 38% versus 37% receiving an eye examination.

CONCLUSIONS

The method used to select patients when calculating provider group performance may affect patient case mix and estimated performance levels, and warrants careful consideration when comparing performance estimates.

How good are the data? Feasible approach to validation of metrics of quality derived from an outpatient electronic health record

Although electronic health records (EHRs) promise to be efficient resources for measuring metrics of quality, they are not designed for such population-based analyses. Thus, extracting meaningful clinical data from them is not straightforward. To avoid poorly executed measurements, standardized methods to measure and to validate metrics of quality are needed. This study provides and evaluates a use case for a generally applicable approach to validating quality metrics measured electronically from EHR-based data. The authors iteratively refined and validated 4 outpatient quality metrics and classified errors in measurement. Multiple iterations of validation and measurement resulted in high levels of sensitivity and agreement versus the "gold standard" of manual review. In contrast, substantial differences remained for measurement based on coded billing data. Measuring quality metrics using an EHR-based electronic process requires validation to ensure accuracy; approaches to validation such as those described in this study should be used by organizations measuring quality from EHR-based information.

Health Resource Utilization and Direct Costs Associated with Angina for Patients with Coronary Artery Disease in a US Managed Care Setting

BACKGROUND

Angina is often a first symptom of coronary artery disease (CAD); however, the specific burden of illness for patients with CAD-associated angina in managed care has not been reported.

OBJECTIVE

To determine the clinical and cost burden of illness for patients with CAD-associated angina in a managed care environment.

STUDY DESIGN

A retrospective database analysis in a nationwide commercial managed care plan.

METHODS

This study included patients with International Classification of Diseases, Ninth Revision, Clinical Modification diagnostic or procedure codes for CAD between July 1, 2004, and June 30, 2006, who had data available for the period 6 months before and 12 months after the index date. The primary analyses for patients classified as having CAD with angina were based on a 3-algorithm patient-identification model (combined positive predictive value of 89%, 95% confidence interval, 0.79-0.95). Utilization measures for the 12-month postindex period, annual CAD-related direct costs, and total all-cause costs (ie, medical plus pharmacy) were determined. A generalized linear model was used to compare CAD-related costs and overall costs.

RESULTS

Of the 246,227 patients with CAD, the 3-algorithm model assigned 230,919 patients (93.8%) to the CAD-without-angina cohort and 15,308 (6.2%) to the CAD-with-angina cohort. Patients with angina were more likely than patients without angina to be hospitalized (41% vs 11%, respectively; P <.001), to visit the emergency department (34% vs 12%, respectively; P <.001), to have office visits (94% vs 79%, respectively; P <.001), and to have more revascularization procedures (35% vs 8%, respectively; P <.001). Average CAD-related inpatient costs were $9536 versus $2169, and pharmacy costs were $1499 versus $891, for patients with and without angina, respectively. Total average CAD-related medical and pharmacy costs for patients with angina were $14,851 versus $4449 for patients with CAD without angina, and the average all-cause per-patient cost was $28,590 versus $14,334, respectively.

CONCLUSION

Based on these results, US patients with CAD plus angina in a managed care setting use significantly more healthcare services and incur higher costs than patients who have CAD without angina. Revascularization procedures are a major driver of these increased costs for those with CAD and angina.

Electronic health records and quality of diabetes care

BACKGROUND

Available studies have shown few quality-related advantages of electronic health records (EHRs) over traditional paper records. We compared achievement of and improvement in quality standards for diabetes at practices using EHRs with those at practices using paper records. All practices, including many safety-net primary care practices, belonged to a regional quality collaborative and publicly reported performance.

METHODS

We used generalized estimating equations to calculate the percentage-point difference between EHR-based and paper-based practices with respect to achievement of composite standards for diabetes care (including four component standards) and outcomes (five standards), after adjusting for covariates and accounting for clustering. In addition to insurance type (Medicare, commercial, Medicaid, or uninsured), patient-level covariates included race or ethnic group (white, black, Hispanic, or other), age, sex, estimated household income, and level of education. Analyses were conducted separately for the overall sample and for safety-net practices.

RESULTS

From July 2009 through June 2010, data were reported for 27,207 adults with diabetes seen at 46 practices; safety-net practices accounted for 38% of patients. After adjustment for covariates, achievement of composite standards for diabetes care was 35.1 percentage points higher at EHR sites than at paper-based sites (P<0.001), and achievement of composite standards for outcomes was 15.2 percentage points higher (P=0.005). EHR sites were associated with higher achievement on eight of nine component standards. Such sites were also associated with greater improvement in care (a difference of 10.2 percentage points in annual improvement, P<0.001) and outcomes (a difference of 4.1 percentage points in annual improvement, P=0.02). Across all insurance types, EHR sites were associated with significantly higher achievement of care and outcome standards and greater improvement in diabetes care. Results confined to safety-net practices were similar.

CONCLUSIONS

These findings support the premise that federal policies encouraging the meaningful use of EHRs may improve the quality of care across insurance types.

Sensible use of observational clinical data

Observational data sets offer many potential advantages for medical research including their low cost, large size and generalisability. Because they are collected for clinical care and health care operations purposes, observational data sets have some limitations that must be considered in order to perform useful analyses. Sensible use of observational data sets can yield valuable insights, particularly when clinical trials are impractical.

Resident-initiated interventions to improve inpatient heart-failure management

BACKGROUND

Third-year internal medicine residents participating in a quality improvement rotation identified gaps between the Joint Commission's ORYX quality guidelines and clinical practices for the inpatient management of heart failure (HF) at the William S. Middleton Memorial Veterans Hospital. Residents focused on the performance metrics associated with tobacco-cessation counselling documentation, ejection fraction assessment and angiotensin-converting enzyme inhibitor/angiotensin receptor blocker prescriptions.

METHODS

After analysing data collected by the External Peer Review Program, residents reviewed the institution's admissions and discharge processes with the aim of improving quality and compliance. In redesigning these processes, residents created an admissions template and a discharge face sheet, and compared specific ORYX measure compliance rates before and after institution-wide implementation.

RESULTS

Following implementation of the tobacco-cessation admissions template, 100% of HF patients who used tobacco received documented cessation counselling, compared with 59% prior to intervention (p<0.01, n=32). Following implementation of the mandatory discharge face sheet, 97% of HF patients (compared with 92% preintervention, p>0.05) received comprehensive discharge instruction; LV function assessment went from 98% to 100% (p>0.05); and angiotensin-converting enzyme inhibitor/angiotensin receptor blocker prescription for left ventricular systolic dysfunction at discharge (or documentation of a contra-indication) went from 82% to 100% (p<0.01, n=48).

DISCUSSION

By implementing a standardised admissions template and a mandatory discharge face sheet, the hospital improved its processes of documentation and increased adherence to quality-performance measures. By strengthening residents' learning and commitment to quality improvement, the hospital created a foundation for future changes in the systems that affect patient care.

The marginal value of pre-visit paper reminders when added to a multifaceted electronic health record based quality improvement system

OBJECTIVE

We have reported that implementation of an electronic health record (EHR) based quality improvement system that included point-of-care electronic reminders accelerated improvement in performance for multiple measures of chronic disease care and preventive care during a 1-year period. This study examined whether providing pre-visit paper quality reminders could further improve performance, especially for physicians whose performance had not improved much during the first year.

DESIGN

Time-series analysis at a large internal medicine practice using a commercial EHR. All patients eligible for each measure were included (range approximately 100-7500).

MEASUREMENTS

The proportion of eligible patients in the practice who satisfied each of 15 quality measures after removing those with exceptions from the denominator. To analyze changes in performance for individual physicians, two composite measures were used: prescribing seven essential medications and completion of five preventive services.

RESULTS

During the year after implementing pre-encounter reminders, performance continued to improve for eight measures, remained stable for four, and declined for three. Physicians with the worst performance at the start of the pre-encounter reminders showed little absolute improvement over the next year, and most remained below the median performance for physicians in the practice.

CONCLUSIONS

Paper pre-encounter reminders did not appear to improve performance beyond electronic point-of-care reminders in the EHR alone. Lagging performance is likely not due to providers' EHR workflow alone, and trying to step backwards and use paper reminders in addition to point-of-care reminders in the EHR may not be an effective strategy for engaging slow adopters.

Data standards in diabetes patient registries

Widespread adoption of electronic health records (EHRs) and expansion of patient registries present opportunities to improve patient care and population health and advance translational research. However, optimal integration of patient registries with EHR functions and aggregation of regional registries to support national or global analyses will require the use of standards. Currently, there are no standards for patient registries and no content standards for health care data collection or clinical research, including diabetes research. Data standards can facilitate new registry development by supporting reuse of well-defined data elements and data collection systems, and they can enable data aggregation for future research and discovery. This article introduces standardization topics relevant to diabetes patient registries, addresses issues related to the quality and use of registries and their integration with primary EHR data collection systems, and proposes strategies for implementation of data standards in diabetes research and management.

Asian Americans have greater prevalence of metabolic syndrome despite lower body mass index

OBJECTIVE

To examine the relationship between body mass index (BMI) and metabolic syndrome for Asian Americans and non-Hispanic Whites (NHWs), given that existing evidence shows racial/ethnic heterogeneity exists in how BMI predicts metabolic syndrome.

RESEARCH DESIGN AND METHODS

Electronic health records of 43,507 primary care patients aged 35 years and older with self-identified race/ethnicity of interest (Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese or NHW) were analyzed in a mixed-payer, outpatient-focused health-care organization in the San Francisco Bay Area.

RESULTS

Metabolic syndrome prevalence is significantly higher in Asians compared with NHWs for every BMI category. For women at the mean age of 55 and BMI of 25  kg  m(-2), the predicted prevalence of metabolic syndrome is 12% for NHW women compared with 30% for Asians; similarly for men, the predicted prevalence of metabolic syndrome is 22% for NHWs compared with 43% of Asians. Compared with NHW women and men with a BMI of 25  kg  m(-2), comparable prevalence of metabolic syndrome was observed at BMI of 19.6  kg  m(-2) for Asian women and 19.9  kg  m(-2) for Asian men. A similar pattern was observed in disaggregated Asian subgroups.

CONCLUSIONS

In spite of the lower BMI values and lower prevalence of overweight/obesity than NHWs, Asian Americans have higher rates of metabolic syndrome over the range of BMI. Our results indicate that BMI ranges for defining overweight/obesity in Asian populations should be lower than for NHWs.

Importance of health system context for evaluating utilization patterns across systems

Measuring health services provided to patients can be difficult when patients see providers across multiple health systems and all visits are rarely captured in a single data source covering all systems where patients receive care. Studies that account for only one system will omit the out-of-system health-care use at the patient level. Combining data across systems and comparing utilization patterns across health systems creates complications for both aggregation and accuracy because data-generating processes (DGPs) tend to vary across systems. We develop a hybrid methodology for aggregation across systems, drawing on the strengths of the DGP in each system, and demonstrate its validity for answering research questions requiring cross-system assessments of health-care utilization. Positive and negative predictive probabilities can be useful to assess the impact of the hybrid methodology. We illustrate these issues comparing public sector (administrative records from the US Department of Veterans Affairs system) and private sector (billing records from the US Medicare system) patient level data to identify primary-care utilization. Understanding the context of a particular health system and its effect on the DGP is important in conducting effective valid evaluations.

Investigating concordance in diabetes diagnosis between primary care charts (electronic medical records) and health administrative data: a retrospective cohort study

BACKGROUND

Electronic medical records contain valuable clinical information not readily available elsewhere. Accordingly, they hold important potential for contributing to and enhancing chronic disease registries with the goal of improving chronic disease management; however a standard for diagnoses of conditions such as diabetes remains to be developed. The purpose of this study was to establish a validated electronic medical record definition for diabetes.

METHODS

We constructed a retrospective cohort using health administrative data from the Institute for Clinical Evaluative Sciences Ontario Diabetes Database linked with electronic medical records from the Deliver Primary Healthcare Information Project using data from 1 April 2006-31 March 2008 (N = 19,443). We systematically examined eight definitions for diabetes diagnosis, both established and proposed.

RESULTS

The definition that identified the highest number of patients with diabetes (N = 2,180) while limiting to those with the highest probability of having diabetes was: individuals with ≥2 abnormal plasma glucose tests, or diabetes on the problem list, or insulin prescription, or ≥2 oral anti-diabetic agents, or HbA1c ≥6.5%. Compared to the Ontario Diabetes Database, this definition identified 13% more patients while maintaining good sensitivity (75%) and specificity (98%).

CONCLUSIONS

This study establishes the feasibility of developing an electronic medical record standard definition of diabetes and validates an algorithm for use in this context. While the algorithm may need to be tailored to fit available data in different electronic medical records, it contributes to the establishment of validated disease registries with the goal of enhancing research, and enabling quality improvement in clinical care and patient self-management.

Consortium for oral health-related informatics: improving dental research, education, and treatment

Advances in informatics, particularly the implementation of electronic health records (EHR), in dentistry have facilitated the exchange of information. The majority of dental schools in North America use the same EHR system, providing an unprecedented opportunity to integrate these data into a repository that can be used for oral health education and research. In 2007, fourteen dental schools formed the Consortium for Oral Health-Related Informatics (COHRI). Since its inception, COHRI has established structural and operational processes, governance and bylaws, and a number of work groups organized in two divisions: one focused on research (data standardization, integration, and analysis), and one focused on education (performance evaluations, virtual standardized patients, and objective structured clinical examinations). To date, COHRI (which now includes twenty dental schools) has been successful in developing a data repository, pilot-testing data integration, and sharing EHR enhancements among the group. This consortium has collaborated on standardizing medical and dental histories, developing diagnostic terminology, and promoting the utilization of informatics in dental education. The consortium is in the process of assembling the largest oral health database ever created. This will be an invaluable resource for research and provide a foundation for evidence-based dentistry for years to come.

The use of databases and registries to enhance colonoscopy quality

Administrative databases, registries, and clinical databases are designed for different purposes and therefore have different advantages and disadvantages in providing data for enhancing quality. Administrative databases provide the advantages of size, availability, and generalizability, but are subject to constraints inherent in the coding systems used and from data collection methods optimized for billing. Registries are designed for research and quality reporting but require significant investment from participants for secondary data collection and quality control. Electronic health records contain all of the data needed for quality research and measurement, but that data is too often locked in narrative text and unavailable for analysis. National mandates for electronic health record implementation and functionality will likely change this landscape in the near future.

International classification of diseases, 10th edition, clinical modification and procedure coding system: descriptive overview of the next generation HIPAA code sets

Described are the changes to ICD-10-CM and PCS and potential challenges regarding their use in the US for financial and administrative transaction coding under HIPAA in 2013. Using author constructed derivative databases for ICD-10-CM and PCS it was found that ICD-10-CM's overall term content is seven times larger than ICD-9-CM: only 3.2 times larger in those chapters describing disease or symptoms, but 14.1 times larger in injury and cause sections. A new multi-axial approach ICD-10-PCS increased size 18-fold from its prior version. New ICD-10-CM and PCS reflect a corresponding improvement in specificity and content. The forthcoming required national switch to these new administrative codes, coupled with nearly simultaneous widespread introduction of clinical systems and terminologies, requires substantial changes in US administrative systems. Through coordination of terminologies, the systems using them, and healthcare objectives, we can maximize the improvement achieved and engender beneficial data reuse for multiple purposes, with minimal transformations.

Democratizing information creation from health care data for quality improvement, research, and education-the Montefiore Medical Center Experience

The National Research Council recently reviewed the capabilities of health care software implemented in the United States and described a health care information technology chasm that is threatening the medical community's ability to meet the health care quality goals enumerated in Institute of Medicine reports. Among the critical gaps is the inability of health care software systems to allow users to convert data into meaningful information supporting quality improvement, analysis, and research. In this article, the authors describe the Montefiore Medical Center's decade-long experience developing software for the purpose of converting data into useful information and integrating software use into the clinical culture. The program at Montefiore could serve as a potential national model.

The impact of electronic medical records data sources on an adverse drug event quality measure

OBJECTIVE

To examine the impact of billing and clinical data extracted from an electronic medical record system on the calculation of an adverse drug event (ADE) quality measure approved for use in The Joint Commission's ORYX program, a mandatory national hospital quality reporting system.

DESIGN

The Child Health Corporation of America's "Use of Rescue Agents-ADE Trigger" quality measure uses medication billing data contained in the Pediatric Health Information Systems (PHIS) data warehouse to create The Joint Commission-approved quality measure. Using a similar query, we calculated the quality measure using PHIS plus four data sources extracted from our electronic medical record (EMR) system: medications charged, medication orders placed, medication orders with associated charges (orders charged), and medications administered.

MEASUREMENTS

Inclusion and exclusion criteria were identical for all queries. Denominators and numerators were calculated using the five data sets. The reported quality measure is the ADE rate (numerator/denominator).

RESULTS

Significant differences in denominators, numerators, and rates were calculated from different data sources within a single institution's EMR. Differences were due to both common clinical practices that may be similar across institutions and unique workflow practices not likely to be present at any other institution. The magnitude of the differences would significantly alter the national comparative ranking of our institution compared to other PHIS institutions.

CONCLUSIONS

More detailed clinical information may result in quality measures that are not comparable across institutions due institution-specific workflow, differences that are exposed using EMR-derived data.

Electronic medical records, nurse staffing, and nurse-sensitive patient outcomes: evidence from California hospitals, 1998-2007

OBJECTIVE

To estimate the effects of electronic medical records (EMR) implementation on medical-surgical acute unit costs, length of stay, nurse staffing levels, nursing skill mix, nurse cost per hour, and nurse-sensitive patient outcomes.

DATA SOURCES

Data on EMR implementation came from the 1998-2007 HIMSS Analytics Databases. Data on nurse staffing and patient outcomes came from the 1998-2007 Annual Financial Disclosure Reports and Patient Discharge Databases of the California Office of Statewide Health Planning and Development (OSHPD).

METHODS

Longitudinal analysis of an unbalanced panel of 326 short-term, general acute care hospitals in California. Marginal effects estimated using fixed effects (within-hospital) OLS regression.

PRINCIPAL FINDINGS

EMR implementation was associated with 6-10 percent higher cost per discharge in medical-surgical acute units. EMR stage 2 increased registered nurse hours per patient day by 15-26 percent and reduced licensed vocational nurse cost per hour by 2-4 percent. EMR stage 3 was associated with 3-4 percent lower rates of in-hospital mortality for conditions.

CONCLUSIONS

Our results suggest that advanced EMR applications may increase hospital costs and nurse staffing levels, as well as increase complications and decrease mortality for some conditions. Contrary to expectation, we found no support for the proposition that EMR reduced length of stay or decreased the demand for nurses.

Review: electronic health records and the reliability and validity of quality measures: a review of the literature

Previous reviews of research on electronic health record (EHR) data quality have not focused on the needs of quality measurement. The authors reviewed empirical studies of EHR data quality, published from January 2004, with an emphasis on data attributes relevant to quality measurement. Many of the 35 studies reviewed examined multiple aspects of data quality. Sixty-six percent evaluated data accuracy, 57% data completeness, and 23% data comparability. The diversity in data element, study setting, population, health condition, and EHR system studied within this body of literature made drawing specific conclusions regarding EHR data quality challenging. Future research should focus on the quality of data from specific EHR components and important data attributes for quality measurement such as granularity, timeliness, and comparability. Finally, factors associated with poor or variability in data quality need to be better understood and effective interventions developed.

Using your electronic medical record for research: a primer for avoiding pitfalls

In Canada, use of electronic medical records (EMRs) among primary health care (PHC) providers is relatively low. However, it appears that EMRs will eventually become more ubiquitous in PHC. This represents an important development in the use of health care information technology as well as a potential new source of PHC data for research. However, care in the use of EMR data is required. Four years ago, researchers at the Centre for Studies in Family Medicine, The University of Western Ontario created an EMR-based research project, called Deliver Primary Health Care Information. Implementing this project led us to two conclusions about using PHC EMR data for research: first, additional time is required for providers to undertake EMR training and to standardize the way data are entered into the EMR and second, EMRs are designed for clinical care, not research. Based on these experiences, we offer our thoughts about how EMRs may, nonetheless, be used for research. Family physician researchers who intend to use EMR data to answer timely questions relevant to practice should evaluate the possible impact of the four questions raised by this paper: (i) why are EMR data different?; (ii) how do you extract data from an EMR?; (iii) where are the data stored? and (iv) what is the data quality? In addition, consideration needs to be given to the complexity of the research question since this can have an impact on how easily issues of using EMR data for research can be overcome.

The physician-patient relationship and a National Health Information network

The growing use of interoperable electronic health records is likely to have significant effects on the physician-patient relationship. This relationship involves two-way trust: of the physician in patients, and of the patients in their providers. Interoperable records opens up this relationship to further view, with consequences that may both enhance and undermine trust. On the one hand, physicians may learn (from additional records) that information from their patients is - or is not - to be trusted. On the other hand, patients may learn from the increased oversight made possible by electronic records that their trust in their physicians is - or is not - warranted. Release of information through new methods of surveillance may also undermine patient trust. The article concludes that because trust is fragile, attention to transparency and confidentiality in the use of interoperable electronic records is essential.

Dreams and nightmares: practical and ethical issues for patients and physicians using personal health records

Electronic health records for patients, personal health records (PHRs), have become increasingly popular among policy makers and purchasers, but uptake among patients and physicians has been relatively slow. PHRs have varying uses that might make them more or less appealing to different stakeholders. The three core uses for PHRs - promoting communication, data use, and patient responsibility - each raises a set of potential practical and financial dilemmas. But some ethical concerns are also at play, some of which are rarely recognized as values-based barriers to the use of PHRs. Recognizing these ethical issues, and addressing them explicitly in PHR design and policy making, would help PHRs to achieve their promise.

Measuring the effects of health information technology on quality of care: a novel set of proposed metrics for electronic quality reporting

BACKGROUND

Electronic health records (EHRs), in combination with health information exchange, are being promoted in the United States as a strategy for improving quality of care. No single metric set exists for measuring the effectiveness of these interventions. A set of quality metrics was sought that could be retrieved electronically and would be sensitive to the changes in quality that EHRs with health information exchange may contribute to ambulatory care.

METHODS

A literature search identified quality metric sets for ambulatory care. Two rounds of quantitative rating of individual metrics were conducted. Metrics were developed de novo to capture additional expected effects of EHRs with health information exchange. A 36-member national expert panel validated the rating process and final metric set.

RESULTS

Seventeen metric sets containing 1,064 individual metrics were identified; 510 metrics met inclusion criteria. Two rounds of rating narrowed these to 59 metrics and then to 18. The final 18 consisted of metrics for asthma, cardiovascular disease, congestive heart failure, diabetes, medication and allergy documentation, mental health, osteoporosis, and prevention. Fourteen metrics were developed de novo to address test ordering, medication management, referrals, follow-up after discharge, and revisits.

DISCUSSION

The novel set of 32 metrics is proposed as suitable for electronic reporting to capture the potential quality effects of EHRs with health information exchange. This metric set may have broad utility as health information technology becomes increasingly common with funding from the federal stimulus package and other sources. This work may also stimulate discussion on improving how data are entered and extracted from clinically rich, electronic sources, with the goal of more accurately measuring and improving care.

Evolving health information technology and the timely availability of visit diagnoses from ambulatory visits: a natural experiment in an integrated delivery system

Background

Health information technology (HIT) may improve health care quality and outcomes, in part by making information available in a timelier manner. However, there are few studies documenting the changes in timely availability of data with the use of a sophisticated electronic medical record (EMR), nor a description of how the timely availability of data might differ with different types of EMRs. We hypothesized that timely availability of data would improve with use of increasingly sophisticated forms of HIT.

Methods

We used an historical observation design (2004–2006) using electronic data from office visits in an integrated delivery system with three types of HIT: Basic, Intermediate, and Advanced. We calculated the monthly percentage of visits using the various types of HIT for entry of visit diagnoses into the delivery system's electronic database, and the time between the visit and the availability of the visit diagnoses in the database.

Results

In January 2004, when only Basic HIT was available, 10% of office visits had diagnoses entered on the same day as the visit and 90% within a week; 85% of office visits used paper forms for recording visit diagnoses, 16% used Basic at that time. By December 2006, 95% of all office visits had diagnoses available on the same day as the visit, when 98% of office visits used some form of HIT for entry of visit diagnoses (Advanced HIT for 67% of visits).

Conclusion

Use of HIT systems is associated with dramatic increases in the timely availability of diagnostic information, though the effects may vary by sophistication of HIT system. Timely clinical data are critical for real-time population surveillance, and valuable for routine clinical care.

Using technology to promote gastrointestinal outcomes research: a case for electronic health records

Electronic health records (EHRs) have been shown to reduce medication errors, improve patient outcomes, and create administrative efficiencies. Numerous public and private efforts are currently underway to achieve universal EHR adoption in the United States by the year 2014. EHRs hold a great potential to integrate clinical care and research by allowing input of clinical data in a structured format, facilitating electronic data capture for clinical trials and providing linkage with genomic information. The goal of this article is to inform the academic gastrointestinal community about the research opportunities created by the widespread adoption of EHRs and present a systematic approach in utilizing EHR-derived data for observational, experimental, or translational studies.

Sensitivity of International Classification of Diseases codes for hyponatremia among commercially insured outpatients in the United States

Background

Administrative claims are a rich source of information for epidemiological and health services research; however, the ability to accurately capture specific diseases or complications using claims data has been debated. In this study, the authors examined the validity of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes for the identification of hyponatremia in an outpatient managed care population.

Methods

We analyzed outpatient laboratory and professional claims for patients aged 18 years and older in the National Managed Care Benchmark Database from Integrated Healthcare Information Services. We obtained all claims for outpatient serum sodium laboratory tests performed in 2004 and 2005, and all outpatient professional claims with a primary or secondary ICD-9-CM diagnosis code of hyponatremia (276.1).

Results

A total of 40,668 outpatient serum sodium laboratory results were identified as hyponatremic (serum sodium < 136 mmol/L). The sensitivity of ICD-9-CM codes for hyponatremia in outpatient professional claims within 15 days before or after the laboratory date was 3.5%. Even for severe cases (serum sodium ≤ 125 mmol/L), sensitivity was < 30%. Specificity was > 99% for all cutoff points.

Conclusion

ICD-9-CM codes in administrative data are insufficient to identify hyponatremia in an outpatient population.

Automatic classification of foot examination findings using clinical notes and machine learning

We examine the feasibility of a machine learning approach to identification of foot examination (FE) findings from the unstructured text of clinical reports. A Support Vector Machine (SVM) based system was constructed to process the text of physical examination sections of in- and out-patient clinical notes to identify if the findings of structural, neurological, and vascular components of a FE revealed normal or abnormal findings or were not assessed. The system was tested on 145 randomly selected patients for each FE component using 10-fold cross validation. The accuracy was 80%, 87% and 88% for structural, neurological, and vascular component classifiers, respectively. Our results indicate that using machine learning to identify FE findings from clinical reports is a viable alternative to manual review and warrants further investigation. This application may improve quality and safety by providing inexpensive and scalable methodology for quality and risk factor assessments at the point of care.

The Clinical Outcomes Assessment Toolkit: a framework to support automated clinical records-based outcomes assessment and performance measurement research

The Clinical Outcomes Assessment Toolkit (COAT) was created through a collaboration between the University of California, Los Angeles and Brigham and Women's Hospital to address the challenge of gathering, formatting, and abstracting data for clinical outcomes and performance measurement research. COAT provides a framework for the development of information pipelines to transform clinical data from its original structured, semi-structured, and unstructured forms to a standardized format amenable to statistical analysis. This system includes a collection of clinical data structures, reusable utilities for information analysis and transformation, and a graphical user interface through which pipelines can be controlled and their results audited by nontechnical users. The COAT architecture is presented, as well as two case studies of current implementations in the domain of prostate cancer outcomes assessment.

Facilitating Clinical Outcomes Assessment through the automated identification of quality measures for prostate cancer surgery

OBJECTIVES

The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the postoperative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer.

DESIGN

An application was developed with the Clinical Outcomes Assessment Toolkit that uses an information pipeline of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the University of California at Los Angeles Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed.

MEASUREMENTS

The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared, and a detailed failure analysis was conducted to identify errors caused by poor data quality versus system shortcomings.

RESULTS

Accuracies for Gleason score were 99.7%, tumor stage 99.1%, and margin status 97.2%, for an overall accuracy of 98.67%. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3% but accounted for 25.9% of the total errors.

CONCLUSION

Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of using a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from postoperative prostate cancer pathology reports.

Pay for performance for the allergist-immunologist: Potential promise and problems

Pay for performance is one component of a growing movement in this country to better align incentives so as to increase the quality of medical care and reduce unnecessary costs. The Physician Consortium for Quality Improvement is a national multistakeholder organization that is developing performance measures. Concerns regarding performance measures include technical concerns (eg, validity, risk adjustment, sample size, and accuracy considerations), concerns regarding the cost of implementation, and concerns regarding unintended consequences. Some data support the effectiveness of pay-for-performance programs, but more and better-designed studies are needed to rigorously assess the effectiveness of pay-for-performance programs. The Institute of Medicine 2006 report supports pay for performance "as a stimulus to foster comprehensive and system-wide improvements in the quality of healthcare." This report also recommends that implementation of pay-for-performance programs should be carefully monitored to be sure that the stated goals are being achieved and that unintended consequences are recognized as early as possible. It is important for the allergist-immunologist to be an active participant in the evolving paradigm of pay for performance, advocating for the best interests of patients and providers alike.