Clinical research databases--a historical review

Error Rates of Data Processing Methods in Clinical Research: A Systematic Review and Meta-Analysis of Manuscripts Identified Through PubMed

Background

In clinical research, prevention of systematic and random errors of data collected is paramount to ensuring reproducibility of trial results and the safety and efficacy of the resulting interventions. Over the last 40 years, empirical assessments of data accuracy in clinical research have been reported in the literature. Although there have been reports of data error and discrepancy rates in clinical studies, there has been little systematic synthesis of these results. Further, although notable exceptions exist, little evidence exists regarding the relative accuracy of different data processing methods. We aim to address this gap by evaluating error rates for 4 data processing methods.

Methods

A systematic review of the literature identified through PubMed was performed to identify studies that evaluated the quality of data obtained through data processing methods typically used in clinical trials: medical record abstraction (MRA), optical scanning, single-data entry, and double-data entry. Quantitative information on data accuracy was abstracted from the manuscripts and pooled. Meta-analysis of single proportions based on the Freeman-Tukey transformation method and the generalized linear mixed model approach were used to derive an overall estimate of error rates across data processing methods used in each study for comparison.

Results

A total of 93 papers (published from 1978 to 2008) meeting our inclusion criteria were categorized according to their data processing methods. The accuracy associated with data processing methods varied widely, with error rates ranging from 2 errors per 10,000 fields to 2,784 errors per 10,000 fields. MRA was associated with both high and highly variable error rates, having a pooled error rate of 6.57% (95% CI: 5.51, 7.72). In comparison, the pooled error rates for optical scanning, single-data entry, and double-data entry methods were 0.74% (0.21, 1.60), 0.29% (0.24, 0.35) and 0.14% (0.08, 0.20), respectively.

Conclusions

Data processing and cleaning methods may explain a significant amount of the variability in data accuracy. MRA error rates, for example, were high enough to impact decisions made using the data and could necessitate increases in sample sizes to preserve statistical power. Thus, the choice of data processing methods can likely impact process capability and, ultimately, the validity of trial results.

Comparing Medical Record Abstraction (MRA) Error Rates in an Observational Study to Pooled Rates Identified in the Data Quality Literature

Background

Medical record abstraction (MRA) is a commonly used method for data collection in clinical research, but is prone to error, and the influence of quality control (QC) measures is seldom and inconsistently assessed during the course of a study. We employed a novel, standardized MRA-QC framework as part of an ongoing observational study in an effort to control MRA error rates. In order to assess the effectiveness of our framework, we compared our error rates against traditional MRA studies that had not reported using formalized MRA-QC methods. Thus, the objective of this study was to compare the MRA error rates derived from the literature with the error rates found in a study using MRA as the sole method of data collection that employed an MRA-QC framework.

Methods

Using a moderator meta-analysis employed with Q-test, the MRA error rates from the meta-analysis of the literature were compared with the error rate from a recent study that implemented formalized MRA training and continuous QC processes.

Results

The MRA process for data acquisition in clinical research was associated with both high and highly variable error rates (70 - 2,784 errors per 10,000 fields). Error rates for the study using our MRA-QC framework were between 1.04% (optimistic, all-field rate) and 2.57% (conservative, populated-field rate) (or 104 - 257 errors per 10,000 fields), 4.00 - 5.53 percentage points less than the observed rate from the literature (p<0.0001).

Conclusions

Review of the literature indicated that the accuracy associated with MRA varied widely across studies. However, our results demonstrate that, with appropriate training and continuous QC, MRA error rates can be significantly controlled during the course of a clinical research study.

Informatics infrastructure in a rural pediatric clinical trials network: Matching specific clinical research needs with best practices and industry guidelines

Children have historically been underrepresented in randomized controlled trials and multi-center studies. This is particularly true for children who reside in rural and underserved areas. Conducting multi-center trials in rural areas presents unique informatics challenges. These challenges call for increased attention towards informatics infrastructure and the need for development and application of sound informatics approaches to the collection, processing, and management of data for clinical studies. By modifying existing local infrastructure and utilizing open source tools, we have been able to successfully deploy a multi-site data coordinating and operations center. We report our implementation decisions for data collection and management for the IDeA States Pediatric Clinical Trial Network (ISPCTN) based on the functionality needed for the ISPCTN, our synthesis of the extant literature in data collection and management methodology, and Good Clinical Data Management Practices.

The need for data management standards in public health nursing: A narrative review and case study

BACKGROUND

Data management is the key to the success of all projects and research. The ability to safely store, manipulate, and decipher data in real time is invaluable. Currently data management standards in public health are non-existent. Since the invention of computers real-time data retrieval and analysis has been possible but underutilized by researchers in the field. Historically, most small research studies and field-based projects have utilized spreadsheets for data management, which often proves problematic as the project grows. However, a viable and superior alternative exists in relational databases, such as REDCap. Relational databases allow for easier concatenation of multiple legacy datasets, facilitate data entry with surveys that incorporate branching logic, and allow for real time data entry in the field without the need for WIFI.

METHODS

One example of a public health project being transitioned from spreadsheet data management to a relational database is the Farmworker Family Health Program based out of the Lillian Carter Center for Global Health & Social Responsibility at Emory University's Nell Hodgson Woodruff School of Nursing. The data management transition from spreadsheets to REDCap has provided the team with unique insight into the data that has been collected in the 30 years the program has been running.

CONCLUSION

Through this case study, we identify the need for and recommend that those in public health nursing utilize relational databases when collecting data during research studies or as electronic medical records for field clinics.

The role of patient records in research: A bibliometric analysis of publications from an academic medical center in Taiwan

BACKGROUND

This study aimed to assess the use of medical record items in clinical research in one large academic medical center in Taiwan.

METHODS

A descriptive survey design was adopted to collect the data. Articles published in 2018 by Taipei Veterans General Hospital (TVGH) staff as the first author were obtained. The types of specialties and types of research were analyzed. To understand the conditions for the use of medical records, the retrospective research using hospital's medical records were analyzed. Each article was read in entirety to realize the use and number of patients and the medical record items.

RESULTS

Among the 362 articles first-authored by TVGH staff in 2018, 219 (60.4%) were classified as clinical studies, 60 (16.6%) as basic studies, 53 (14.6%) as database studies, and 30 (8.2%) as other categories. About 50% of the retrospective research using TVGH medical records had a case number <100 (67 cases, 49.6%) with an average number of 41 cases and 13 studies (9.6%) had a case number >1000. Analysis of the number of medical record items used in 135 retrospective research studies based on TVGH medical records showed that 118 (87.4%) used basic patient information. In addition to basic information, notes written by professionals were used most frequently (73 cases, 54.0%), whereas medication information was used in 50 cases (37.0%); laboratory test data were used in 49 cases (36.2%); and body measurements was used in 27 cases (20%).

CONCLUSION

More than one-third of publications utilized medical records, but the patient numbers and record items in use were relatively limited. In the era of digitalization and big data analytics, the potential of medical records in research deserves attention. Investment in establishing a more accessible database of medical records to access nonstructural, descriptive medical records could be considered.

Extracting Medical Information from Paper COVID-19 Assessment Forms

OBJECTIVE

This study examines the validity of optical mark recognition, a novel user interface, and crowdsourced data validation to rapidly digitize and extract data from paper COVID-19 assessment forms at a large medical center.

METHODS

An optical mark recognition/optical character recognition (OMR/OCR) system was developed to identify fields that were selected on 2,814 paper assessment forms, each with 141 fields which were used to assess potential COVID-19 infections. A novel user interface (UI) displayed mirrored forms showing the scanned assessment forms with OMR results superimposed on the left and an editable web form on the right to improve ease of data validation. Crowdsourced participants validated the results of the OMR system. Overall error rate and time taken to validate were calculated. A subset of forms was validated by multiple participants to calculate agreement between participants.

RESULTS

The OMR/OCR tools correctly extracted data from scanned forms fields with an average accuracy of 70% and median accuracy of 78% when the OMR/OCR results were compared with the results from crowd validation. Scanned forms were crowd-validated at a mean rate of 157 seconds per document and a volume of approximately 108 documents per day. A randomly selected subset of documents was reviewed by multiple participants, producing an interobserver agreement of 97% for documents when narrative-text fields were included and 98% when only Boolean and multiple-choice fields were considered.

CONCLUSION

Due to the COVID-19 pandemic, it may be challenging for health care workers wearing personal protective equipment to interact with electronic health records. The combination of OMR/OCR technology, a novel UI, and crowdsourcing data-validation processes allowed for the efficient extraction of a large volume of paper medical documents produced during the COVID-19 pandemic.

Fear N. Identifying Veterans Using Electronic Health Records in the United Kingdom: A Feasibility Study

There is a lack of quantitative evidence concerning UK (United Kingdom) Armed Forces (AF) veterans who access secondary mental health care services-specialist care often delivered in high intensity therapeutic clinics or hospitals-for their mental health difficulties. The current study aimed to investigate the utility and feasibility of identifying veterans accessing secondary mental health care services using National Health Service (NHS) electronic health records (EHRs) in the UK. Veterans were manually identified using the Clinical Record Interactive Search (CRIS) system-a database holding secondary mental health care EHRs for an NHS Trust in the UK. We systematically and manually searched CRIS for veterans, by applying a military-related key word search strategy to the free-text clinical notes completed by clinicians. Relevant data on veterans' socio-demographic characteristics, mental disorder diagnoses and treatment pathways through care were extracted for analysis. This study showed that it is feasible, although time consuming, to identify veterans through CRIS. Using the military-related key word search strategy identified 1600 potential veteran records. Following manual review, 693 (43.3%) of these records were verified as "probable" veterans and used for analysis. They had a median age of 74 years (interquartile range (IQR): 53-86); the majority were male (90.8%) and lived alone (38.0%). The most common mental diagnoses overall were depressive disorders (22.9%), followed by alcohol use disorders (10.5%). Differences in care pathways were observed between pre and post national service (NS) era veterans. This feasibility study represents a first step in showing that it is possible to identify veterans through free-text clinical notes. It is also the first to compare veterans from pre and post NS era.

Rates and Associated Factors of Secondary Mental Health Care Utilisation among Ex-Military Personnel in the United States: A Narrative Review

Little is known about ex-serving military personnel who access secondary mental health care. This narrative review focuses on studies that quantitatively measure secondary mental health care utilisation in ex-serving personnel from the United States. The review aimed to identify rates of mental health care utilisation, as well as the factors associated with it. The electronic bibliographic databases OVID Medline, PsycInfo, PsycArticles, and Embase were searched for studies published between January 2001 and September 2018. Papers were retained if they included ex-serving personnel, where the majority of the sample had deployed to the recent conflicts in Iraq or Afghanistan. Fifteen studies were included. Modest rates of secondary mental health care utilisation were found in former military members-for mean percentage prevalence rates, values ranged from 12.5% for at least one psychiatric inpatient episode, to 63.2% for at least one outpatient mental health appointment. Individuals engaged in outpatient care visits most often, most likely because these appointments are the most commonly offered source of support. Post-traumatic stress disorder, particularly re-experiencing symptoms, and comorbid mental health problems were most consistently associated with higher mental health care utilisation. Easily accessible interventions aimed at facilitating higher rates of help seeking in ex-serving personnel are recommended.

Analysis of professional competencies for the clinical research data management profession: implications for training and professional certification

OBJECTIVE

To assess and refine competencies for the clinical research data management profession.

MATERIALS AND METHODS

Based on prior work developing and maintaining a practice standard and professional certification exam, a survey was administered to a captive group of clinical research data managers to assess professional competencies, types of data managed, types of studies supported, and necessary foundational knowledge.

RESULTS

Respondents confirmed a set of 91 professional competencies. As expected, differences were seen in job tasks between early- to mid-career and mid- to late-career practitioners. Respondents indicated growing variability in types of studies for which they managed data and types of data managed.

DISCUSSION

Respondents adapted favorably to the separate articulation of professional competencies vs foundational knowledge. The increases in the types of data managed and variety of research settings in which data are managed indicate a need for formal education in principles and methods that can be applied to different research contexts (ie, formal degree programs supporting the profession), and stronger links with the informatics scientific discipline, clinical research informatics in particular.

CONCLUSION

The results document the scope of the profession and will serve as a foundation for the next revision of the Certified Clinical Data Manager TM exam. A clear articulation of professional competencies and necessary foundational knowledge could inform the content of graduate degree programs or tracks in areas such as clinical research informatics that will develop the current and future clinical research data management workforce.

A hybrid solution for extracting structured medical information from unstructured data in medical records via a double-reading/entry system

BACKGROUND

Healthcare providers generate a huge amount of biomedical data stored in either legacy system (paper-based) format or electronic medical records (EMR) around the world, which are collectively referred to as big biomedical data (BBD). To realize the promise of BBD for clinical use and research, it is an essential step to extract key data elements from unstructured medical records into patient-centered electronic health records with computable data elements. Our objective is to introduce a novel solution, known as a double-reading/entry system (DRESS), for extracting clinical data from unstructured medical records (MR) and creating a semi-structured electronic health record database, as well as to demonstrate its reproducibility empirically.

METHODS

Utilizing the modern cloud-based technologies, we have developed a comprehensive system that includes multiple subsystems, from capturing MRs in clinics, to securely transferring MRs, storing and managing cloud-based MRs, to facilitating both machine learning and manual reading, and to performing iterative quality control before committing the semi-structured data into the desired database. To evaluate the reproducibility of extracted medical data elements by DRESS, we conduct a blinded reproducibility study, with 100 MRs from patients who have undergone surgical treatment of lung cancer in China. The study uses Kappa statistic to measure concordance of discrete variables, and uses correlation coefficient to measure reproducibility of continuous variables.

RESULTS

Using the DRESS, we have demonstrated the feasibility of extracting clinical data from unstructured MRs to create semi-structured and patient-centered electronic health record database. The reproducibility study with 100 patient's MRs has shown an overall high reproducibility of 98 %, and varies across six modules (pathology, Radio/chemo therapy, clinical examination, surgery information, medical image and general patient information).

CONCLUSIONS

DRESS uses a double-reading, double-entry, and an independent adjudication, to manually curate structured data elements from unstructured clinical data. Further, through distributed computing strategies, DRESS protects data privacy by dividing MR data into de-identified modules. Finally, through internet-based computing cloud, DRESS enables many data specialists to work in a virtual environment to achieve the necessary scale of processing thousands MRs within days. This hybrid system represents probably a workable solution to solve the big medical data challenge.

Development and evaluation of a de-identification procedure for a case register sourced from mental health electronic records

Background

Electronic health records (EHRs) provide enormous potential for health research but also present data governance challenges. Ensuring de-identification is a pre-requisite for use of EHR data without prior consent. The South London and Maudsley NHS Trust (SLaM), one of the largest secondary mental healthcare providers in Europe, has developed, from its EHRs, a de-identified psychiatric case register, the Clinical Record Interactive Search (CRIS), for secondary research.

Methods

We describe development, implementation and evaluation of a bespoke de-identification algorithm used to create the register. It is designed to create dictionaries using patient identifiers (PIs) entered into dedicated source fields and then identify, match and mask them (with ZZZZZ) when they appear in medical texts. We deemed this approach would be effective, given high coverage of PI in the dedicated fields and the effectiveness of the masking combined with elements of a security model. We conducted two separate performance tests i) to test performance of the algorithm in masking individual true PIs entered in dedicated fields and then found in text (using 500 patient notes) and ii) to compare the performance of the CRIS pattern matching algorithm with a machine learning algorithm, called the MITRE Identification Scrubber Toolkit – MIST (using 70 patient notes – 50 notes to train, 20 notes to test on). We also report any incidences of potential breaches, defined by occurrences of 3 or more true or apparent PIs in the same patient’s notes (and in an additional set of longitudinal notes for 50 patients); and we consider the possibility of inferring information despite de-identification.

Results

True PIs were masked with 98.8% precision and 97.6% recall. As anticipated, potential PIs did appear, owing to misspellings entered within the EHRs. We found one potential breach. In a separate performance test, with a different set of notes, CRIS yielded 100% precision and 88.5% recall, while MIST yielded a 95.1% and 78.1%, respectively. We discuss how we overcome the realistic possibility – albeit of low probability – of potential breaches through implementation of the security model.

Conclusion

CRIS is a de-identified psychiatric database sourced from EHRs, which protects patient anonymity and maximises data available for research. CRIS demonstrates the advantage of combining an effective de-identification algorithm with a carefully designed security model. The paper advances much needed discussion of EHR de-identification – particularly in relation to criteria to assess de-identification, and considering the contexts of de-identified research databases when assessing the risk of breaches of confidential patient information.

Electronic medical records (EMRs), epidemiology, and epistemology: reflections on EMRs and future pediatric clinical research

Electronic medical records (EMRs) are increasingly common in pediatric patient care. EMR data represent a relatively novel and rich resource for clinical research. The fact, however, that pediatric EMR data are collected for the purposes of clinical documentation and billing rather than research creates obstacles to their use in scientific investigation. Particular issues include accuracy, completeness, comparability between settings, ease of extraction, and context of recording. Although these problems can be addressed through standard strategies for dealing with partially accurate and incomplete data, a longer-term solution will involve work with pediatric clinicians to improve data quality. As research becomes one of the explicit purposes for which pediatricians collect EMR data, the pediatric clinician will play a central role in future pediatric clinical research.

Use of a large diabetes electronic medical record system in India: clinical and research applications

BACKGROUND

The diabetes electronic medical record (DEMR) has emerged as an effective information management tool with the potential to improve diabetes care and research. This study reports on the usefulness of the DEMR system at Dr. Mohan's Diabetes Specialities Centre (DMDSC), Chennai, India, for clinical and research purposes.

METHODS

The DEMR, set up in 1996 at DMDSC, connects data of nine centers/clinics in different geographical areas in Southern India. The present data analysis is based on a total of 226,228 patients registered in the DEMR system at DMDSC between the years 1991 and 2010.

RESULTS

The DEMR included data of 139,906 male and 86,322 female patients, of whom 92.6% had type 2 diabetes mellitus (T2DM), 1.4% had type 1 diabetes mellitus (T1DM), and the rest had other types. Patients with T2DM had higher prevalence rates of neuropathy (33.1% vs 13.0%, p < .001), microalbuminuria (25.5% vs 20.0%, p < .001), coronary artery disease (17.5% vs 9.2%, p < .001) and peripheral vascular disease (3.9% vs 2.8%, p = .017) compared with T1DM patients, while prevalence of diabetic retinopathy was similar (37.9% vs 35.7%, p = .06). Prevalence of microvascular and macrovascular complications of diabetes increased with increasing glycated hemoglobin levels (p for trend < .001) and increasing diabetes duration (p for trend < .001).

CONCLUSIONS

The DEMR helps track diabetes care and is a valuable tool for research.

Implementing Single Source: the STARBRITE proof-of-concept study

OBJECTIVE

Inefficiencies in clinical trial data collection cause delays, increase costs, and may reduce clinician participation in medical research. In this proof-of-concept study, we examine the feasibility of using point-of-care data capture for both the medical record and clinical research in the setting of a working clinical trial. We hypothesized that by doing so, we could increase reuse of patient data, eliminate redundant data entry, and minimize disruption to clinic workflow.

DESIGN

We developed and used a point-of-care electronic data capture system to record data during patient visits. The standards-based system was used for clinical research and to generate the clinic note for the medical record. The system worked in parallel with data collection procedures already in place for an ongoing multicenter clinical trial. Our system was iteratively designed after analyzing case report forms and clinic notes, and observing clinic workflow patterns and business procedures. Existing data standards from CDISC and HL7 were used for database insertion and clinical document exchange.

RESULTS

Our system was successfully integrated into the clinic environment and used in two live test cases without disrupting existing workflow. Analyses performed during system design yielded detailed information on practical issues affecting implementation of systems that automatically extract, store, and reuse healthcare data.

CONCLUSION

Although subject to the limitations of a small feasibility study, our study demonstrates that electronic patient data can be reused for prospective multicenter clinical research and patient care, and demonstrates a need for further development of therapeutic area standards that can facilitate researcher use of healthcare data.

Severity of Injury is Underestimated in the Absence of Autopsy Verification

BACKGROUND

Trauma disproportionately affects young, productive citizens. To decrease the preventable death rate and morbidity, and to save society some of the estimated 230 million dollars per day revenue loss attributable to these injuries, trauma systems were developed. New York State instituted a population-based regional trauma registry to enter all patients meeting appropriate International Classification of Diseases, Ninth Revision diagnostic codes.

METHODS

We evaluated the registry records deaths in a single New York State trauma region. We compared the medical records used for registry entry to the autopsy records from the County Medical Examiner's Office to determine accuracy of diagnostic coding. On the basis of autopsy data, the records were then recoded and the extent of the trauma rescored.

RESULTS

One hundred thirty-four deaths from 1993 to 1998 were recorded. Twelve records (9%) were accurately entered. One hundred twenty-two records had 452 errors. The mean Injury Severity Score (ISS), based on the medical record face sheet, was 29.93. The revised ISS, based on autopsy review, was 34.44 (p = 0.0108, two-tailed t test). The 95% confidence interval of the difference was 1.05 to 7.96. Z scores were -14.36 before autopsy review and -13.21 after autopsy review (p = 0.4395, not significant). We demonstrated a significantly higher ISS in the patients who died when the autopsy findings were included for coding. This information was not available from the medical record.

CONCLUSION

To accurately compare trauma center performance and injury severity, the inclusion of autopsy data is critically important. Present state law does not permit sharing of this information with the trauma centers. When comparing mortality rates of New York State trauma centers, data must be carefully interpreted.

Structure, functions, and activities of a research support informatics section

The authors describe a research group that supports the needs of investigators seeking data from an electronic medical record system. Since its creation in 1972, the Regenstrief Medical Records System has captured and stored more than 350 million discrete coded observations on two million patients. This repository has become a central data source for prospective and retrospective research. It is accessed by six data analysts--working closely with the institutional review board--who provide investigators with timely and accurate data while protecting patient and provider privacy and confidentiality. From January 1, 1999, to July 31, 2002, data analysts tracked their activities involving 47,559 hours of work predominantly for physicians (54%). While data retrieval (36%) and analysis (25%) were primary activities, data analysts also actively collaborated with researchers. Primary objectives of data provided to investigators were to address disease-specific (35.4%) and drug-related (12.2%) questions, support guideline implementation (13.1%), and probe various aspects of clinical epidemiology (5.7%). Outcomes of these endeavors included 117 grants (including 300,000 US dollars per year salary support for data analysts) and 139 papers in peer-reviewed journals by investigators who rated the support provided by data analysts as extremely valuable.

Informatics and Geriatric Psychiatry

The rapidly growing discipline of medical informatics is changing the face of clinical practice and research. The author reviews current efforts toward the development of electronic medical record systems. A successful system must provide satisfactory solutions to five major requirements: a user interface acceptable to varied health care personnel, data storage and transmission standards that will allow communication with other systems, a coding system that is flexible but accommodates complex queries, a multilevel security structure and audit trail, and unique identifiers for patients and providers. These issues have not been fully resolved, and replacement of paper charts with fully computerized records is many years away, but the application of readily available computer tools to geriatric psychiatry can yield immediate benefits. The author describes a supplemental record system that provides improved organization of clinical information as well as powerful search capabilities.

Clinical research in a hospital--from the lone rider to teamwork

Clinical research of high international standard is very demanding and requires clinical data of high quality, software, hardware and competence in research design and statistical treatment of data. Most busy clinicians have little time allocated for clinical research and this increases the need for a potent infrastructure. This paper describes how the Norwegian Radium Hospital, a specialized cancer hospital, has reorganized the clinical research process. This includes a new department, the Clinical Research Office, which serves the formal framework, a central Diagnosis Registry, clinical databases and multicentre studies. The department assists about 120 users, mainly clinicians. Installation of a network software package with over 10 programs has strongly provided an internal standardization, reduced the costs and saved clinicians a great deal of time. The hospital is building up about 40 diagnosis-specific clinical databases with up to 200 variables registered. These databases are shared by the treatment group and seem to be important tools for quality assurance. We conclude that the clinical research process benefits from a firm infrastructure facilitating teamwork through extensive use of modern information technology. We are now ready for the next phase, which is to work for a better external technical framework for cooperation with other institutions throughout the world.

The paper-based patient record: is it really so bad?

In a recent review of literature, a committee of the American Institute of Medicine found much support of the weakness of the paper-based patient record. Inspired by these results, a local survey was held among practicing clinicians to test, whether they could subscribe the Committee's conclusions. The clinicians turned out to be far more positive about the quality of the paper-based patient record. Possible explanations for this discrepancy are discussed, as well as the question, whether the results of the Committee's review may be used as a basis for the implementation of computer-based patient records.