Parents' Active Role and ENgagement in The review of their Stillbirth/perinatal death 2 (PARENTS 2) study: a mixed-methods study of implementation

OBJECTIVE

When a formal review of care takes places after the death of a baby, parents are largely unaware it takes place and are often not meaningfully involved in the review process. Parent engagement in the process is likely to be essential for a successful review and to improve patient safety. This study aimed to evaluate an intervention process of parental engagement in perinatal mortality review (PNMR) and to identify barriers and facilitators to its implementation.

DESIGN

Mixed-methods study of parents' engagement in PNMR.

SETTING

Single tertiary maternity unit in the UK.

PARTICIPANTS

Bereaved parents and healthcare professionals (HCPs).

INTERVENTIONS

Parent engagement in the PNMR (intervention) was based on principles derived through national consensus and qualitative research with parents, HCPs and stakeholders in the UK.

OUTCOMES

Recruitment rates, bereaved parents and HCPs' perceptions.

RESULTS

Eighty-one per cent of bereaved parents approached (13/16) agreed to participate in the study. Two focus groups with bereaved parents (n=11) and HCP (n=7) were carried out postimplementation to investigate their perceptions of the process.Overarching findings were improved dialogue and continuity of care with parents, and improvements in the PNMR process and patient safety. Bereaved parents agreed that engagement in the PNMR process was invaluable and helped them in their grieving. HCP perceived that parent involvement improved the review process and lessons learnt from the deaths; information to understand the impact of aspects of care on the baby's death were often only found in the parents' recollections.

CONCLUSIONS

Parental engagement in the PNMR process is achievable and useful for parents and HCP alike, and critically can improve patient safety and future care for mothers and babies. To learn and prevent perinatal deaths effectively, all hospitals should give parents the option to engage with the review of their baby's death.

Realizing One's Own Subjectivity: Assessors' Perceptions of the Influence of Training on Their Conduct of Workplace-Based Assessments

PURPOSE

Assessor training is essential for defensible assessments of physician performance, yet research on the effectiveness of training programs for promoting assessor consistency has produced mixed results. This study explored assessors' perceptions of the influence of training and assessment tools on their conduct of workplace-based assessments of physicians.

METHOD

In 2017, the authors used a constructivist grounded theory approach to interview 13 physician assessors about their perceptions of the effects of training and tool development on their conduct of assessments.

RESULTS

Participants reported that training led them to realize that there is a potential for variability in assessors' judgments, prompting them to change their scoring and feedback behaviors to enhance consistency. However, many participants noted they had not substantially changed their numerical scoring. Nonetheless, most thought training would lead to increased standardization and consistency among assessors, highlighting a "standardization paradox" in which participants perceived a programmatic shift toward standardization but minimal changes in their own ratings. An "engagement effect" was also found in which participants involved in both tool development and training cited more substantial learnings than participants involved only in training.

CONCLUSIONS

Findings suggest that training may help assessors recognize their own subjectivity when judging performance, which may prompt behaviors that support rigorous and consistent scoring but may not lead to perceptible changes in assessors' numeric ratings. Results also suggest that participating in tool development may help assessors align their judgments with the scoring criteria. Overall, results support the continued study of assessor training programs as a means of enhancing assessor consistency.

Peer-assisted debriefing of multisource feedback: an exploratory qualitative study

BACKGROUND

The Manitoba Physician Achievement Review (MPAR) is a 360-degree feedback assessment that physicians undergo every 7 years to retain licensure. Deliberate reflection on feedback has been demonstrated to encourage practice change. The MPAR Reflection Exercise (RE), a peer-assisted debriefing tool, was developed whereby the physician selects a peer with whom to review and reflect on feedback, committing to change. This qualitative study explores how physicians who had undergone the MPAR used the RE, what areas of change are identified and committed to, and what they perceived as the role of reflection in the MPAR process.

METHODS

The MPAR RE was piloted out to a cohort of MPAR-reviewed physicians. Thematic analysis was conducted on completed exercises (n = 61). Semi-structured interviews were conducted with individuals (n = 6) who completed the MPAR RE until saturation was reached.

RESULTS

Physicians reviewed feedback with a range of peers, including colleagues, staff, and spouses. Many physicians were surprised by feedback, both positive and negative, but interviewees found the RE useful in processing feedback. Areas where physicians committed to change were diverse, covering all CanMEDS roles. Most physicians identified themselves as being successful in implementing change, though time, habit, and structures were cited as barriers.

CONCLUSIONS

Peer-assisted debriefing can assist reflection of multisource feedback. It is easy to implement, is not resource-intensive, and feedback implies that it is effective at promoting change. Participants, with the aid of peers, identified areas for change, developed approaches for change, and largely thought themselves successful at implementing changes. Areas of change included all seven CanMEDS roles.

Effect of patient safety incident review and reflection in an extended morbidity and mortality meeting

BACKGROUND

We modified the departmental mortality and morbidity (M&M) meetings to evaluate whether patient safety incident review as a part of this meeting was associated with reduced patient safety incidents.

METHOD

A pilot programme of peer review of patient safety incidents (PSI) supported by education relevant to that event and follow-on action plan was introduced as a part of an extended morbidity and mortality meeting in a university hospital orthopaedic department. The pilot programme was conducted over six months (January 2012-June 2012). This programme involved junior and senior doctors including consultants although multidisciplinary groups were invited to attend. We investigated PSI rate/1000 hospital admissions for trauma and elective surgery, which were collected prospectively and independently between Jan 2011 to June 2013. We noted if the incident was caused by a medical or a nursing error and compared PSI rates.

RESULTS

Rates of PSI (33/1000) were 7.8 times higher in trauma cases (80.2/1000) than in elective admissions (11.2/1000). There was 18% reduction in trauma and 27% reduction in planned elective admissions. The rate increased after the pilot programme finished but there was still a 7% reduction compared to the pre-pilot period. This study found a significant reduction in the PSI rate for medical error but no change in the rate of nursing error.

CONCLUSION

This continuous reflection, education and action process, where safety events are reviewed as a part of the extended morbidity and mortality meeting, is associated with reduction of patient safety incidents. We recommend that PSI reflection should be introduced in Mortality and Morbidity meetings with mandated attendance of the entire multidisciplinary health care team.

Validation of the INCEPT: A Multisource Feedback Tool for Capturing Different Perspectives on Physicians' Professional Performance

INTRODUCTION

Multisource feedback (MSF) instruments are used to and must feasibly provide reliable and valid data on physicians' performance from multiple perspectives. The "INviting Co-workers to Evaluate Physicians Tool" (INCEPT) is a multisource feedback instrument used to evaluate physicians' professional performance as perceived by peers, residents, and coworkers. In this study, we report on the validity, reliability, and feasibility of the INCEPT.

METHODS

The performance of 218 physicians was assessed by 597 peers, 344 residents, and 822 coworkers. Using explorative and confirmatory factor analyses, multilevel regression analyses between narrative and numerical feedback, item-total correlations, interscale correlations, Cronbach's α and generalizability analyses, the psychometric qualities, and feasibility of the INCEPT were investigated.

RESULTS

For all respondent groups, three factors were identified, although constructed slightly different: "professional attitude," "patient-centeredness," and "organization and (self)-management." Internal consistency was high for all constructs (Cronbach's α ≥ 0.84 and item-total correlations ≥ 0.52). Confirmatory factor analyses indicated acceptable to good fit. Further validity evidence was given by the associations between narrative and numerical feedback. For reliable total INCEPT scores, three peer, two resident and three coworker evaluations were needed; for subscale scores, evaluations of three peers, three residents and three to four coworkers were sufficient.

DISCUSSION

The INCEPT instrument provides physicians performance feedback in a valid and reliable way. The number of evaluations to establish reliable scores is achievable in a regular clinical department. When interpreting feedback, physicians should consider that respondent groups' perceptions differ as indicated by the different item clustering per performance factor.

Performance results for a workstation-integrated radiology peer review quality assurance program

OBJECTIVE

To assess review completion rates, RADPEER score distribution, and sources of disagreement when using a workstation-integrated radiology peer review program, and to evaluate radiologist perceptions of the program.

DESIGN

Retrospective review of prospectively collected data.

SETTING

Large private outpatient radiology practice.

PARTICIPANTS

Radiologists (n = 66) with a mean of 16.0 (standard deviation, 9.2) years of experience.

INTERVENTIONS

Prior studies and reports of cases being actively reported were randomly selected for peer review using the RADPEER scoring system (a 4-point scale, with a score of 1 indicating agreement and scores of 2-4 indicating increasing levels of disagreement).

MAIN OUTCOME MEASURES

Assigned peer review completion rates, review scores, sources of disagreement and radiologist survey responses.

RESULTS

Of 31 293 assigned cases, 29 044 (92.8%; 95% CI 92.5-93.1%) were reviewed. Discrepant scores (score = 2, 3 or 4) were given in 0.69% (95% CI 0.60-0.79%) of cases and clinically significant discrepancy (score = 3 or 4) was assigned in 0.42% (95% CI 0.35-0.50%). The most common cause of disagreement was missed diagnosis (75.2%; 95% CI 66.8-82.1%). By anonymous survey, 94% of radiologists felt that peer review was worthwhile, 90% reported that the scores they received were appropriate and 78% felt that the received feedback was valuable.

CONCLUSION

Workstation-based peer review can increase completion rates and levels of radiologist acceptance while producing RADPEER scores similar to those previously reported. This approach may be one way to increase radiologist engagement in peer review quality assurance.

Assessing doctors' competencies using multisource feedback: validating a Japanese version of the Sheffield Peer Review Assessment Tool (SPRAT)

OBJECTIVE

To assess the validity and reliability of the Sheffield Peer Review Assessment Tool (SPRAT) Japanese version for evaluating doctors' competencies using multisource feedback.

METHODS

SPRAT, originally developed in the UK, was translated and validated in three phases: (1) an existing Japanese version of SPRAT was back-translated into English; (2) two expert panel meetings were held to develop and assure content validity in a Japanese setting; (3) the newly devised Japanese SPRAT instrument was tested by a multisource feedback survey, validity was tested using principal component factor analysis, and reliability was assessed using generalisability and decision studies based on generalisability theory.

RESULTS

86 doctors who had been practising for between 2 and 33 years participated as assessees and were evaluated with the SPRAT tool. First, the doctors identified 1019 potential assessors who were each sent SPRAT forms (response rate, 81%). The mean number of assessors per doctor was 9.7 (SD=2.5). The decision study showed that 95% CIs of ± 0.5 were achieved with only 5 assessors. 85 of the 86 doctors achieved scores that could be placed with 95% CI above the 4 expected standard. Doctors received lower scores from more senior assessors (p<0.001) and higher scores from those they had known longer (p<0.001). Scores also varied with the job role (p<0.05).

CONCLUSIONS

Following translation and content validation, the Japanese instrument behaved similarly to the UK tool. Assessor selection remains a primary concern, as the assessment scores are affected by the seniority of the assessor, the length of the assessor-assessee working relationship, and the assessor's job role. Users of the SPRAT tool need to be aware of these limitations when administering the instrument.

Evaluation of collaborative therapy review to improve care of heart failure patients

OBJECTIVES

As more demands are placed on primary care providers, new innovative models are required to optimize heart failure (HF) care. The purpose of this study was to evaluate a collaborative therapy review (CTR) program that was implemented to improve guideline-based therapy among HF outpatients.

STUDY DESIGN AND METHODS

We screened patient lists of 18 PCPs at the Portland Veterans Affairs Medical Center to identify patients with an ICD-9 code for HF. The charts of patients with ejection fractions (EFs) < 40% were then abstracted in more detail. The CTR team reviewed each patient and provided specific guideline-based recommendations. The team then gave specific recommendations to providers through the electronic medical record system. We categorized recommendations relating to drug or device therapies, or need for laboratory testing, and calculated provider acceptance rates by recommendation type.

RESULTS

Of the 641 patients reviewed, 156 patients had detailed chart reviews. We found opportunities for improvement in care in 70 (45%) patients who received 100 recommendations. Among the 100 recommendations, 62 (55%) were for guideline-based drugs, 12 (17%) were for consideration of device therapy, and 26 (24%) were to update lab tests or echocardiograms. Eighty percent of the recommendations were acted on within 90 days.

CONCLUSIONS

The CTR program was able to facilitate guideline-based management for HF patients by identifying treatment gaps and making specific guideline-based recommendations to PCPs. While further evaluations are needed, this approach may serve as an efficient method of leveraging the expertise of specialty-trained clinicians to optimize patient care.

The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review

PURPOSE

The use of multisource feedback (MSF) or 360-degree evaluation has become a recognized method of assessing physician performance in practice. The purpose of the present systematic review was to investigate the reliability, generalizability, validity, and feasibility of MSF for the assessment of physicians.

METHOD

The authors searched the EMBASE, PsycINFO, MEDLINE, PubMed, and CINAHL databases for peer-reviewed, English-language articles published from 1975 to January, 2013. Studies were included if they met the follow ing inclusion criteria: used one or more MSF instruments to assess physician performance in practice; reported psychometric evidence of the instrument(s) in the form of reliability, generalizability coefficients, and construct or criterion-related validity; and provided information regarding the administration or feasibility of the process in collecting the feedback data.

RESULTS

Of the 96 full-text articles assessed for eligibility, 43 articles were included. The use of MSF has been shown to be an effective method for providing feedback to physicians from a multitude of specialties about their clinical and nonclinical (i.e., professionalism, communication, interpersonal relationship, management) performance. In general, assessment of physician performance was based on the completion of the MSF instruments by 8 medical colleagues, 8 coworkers, and 25 patients to achieve adequate reliability and generalizability coefficients of α ≥ 0.90 and Ep ≥ 0.80, respectively.

CONCLUSIONS

The use of MSF employing medical colleagues, coworkers, and patients as a method to assess physicians in practice has been shown to have high reliability, validity, and feasibility.

Effects of 2 educational interventions on the management of hypertensive patients in primary health care

INTRODUCTION

Experimental studies on the effectiveness of educational interventions to improve patient care are scarce, especially for low-resources settings. This study investigated the effects of 2 educational interventions on the treatment of hypertensive patients in primary health care in Brazil.

METHODS

Forty-one physicians were randomly assigned either to an "active educational intervention" (21 physicians) or to a "passive educational intervention" (20 physicians). The former comprised 1 small group discussion of routine practices, 1 outreach visit, and 3 reminders. The latter consisted of delivery of printed guidelines. Measures of quality of treatment provided for hypertensive patients (181 patients of physicians from the active intervention; 136 patients of physicians from the passive intervention) were obtained through patient interview and charts review, before and 3 months after the intervention. Chi-square and independent t-tests were performed for comparison between the conditions.

RESULTS

The groups did not differ before the study. After the intervention, the active intervention group outperformed the passive intervention group in several measures, such as improved prescription of antihypertensive drugs (80% of patients of physicians from the active intervention vs 51% patients of physicians from the passive intervention; p < .01), prescription of aspirin (18% vs 6%; p < .01) and hypolipidemic drugs for high-risk patients (39% vs 21%; p < .01), dietary counseling (76% vs 61%; p < .01), guidance on cardiovascular risk (20% vs 3%; p < .01). Patient outcomes did not differ.

DISCUSSION

A multifaceted intervention based on review of practices improved treatment of hypertensive patients in a low-resource setting whereas delivery of guidelines did not help. None of the interventions affected patient outcomes.

Workplace-based assessment for vocational registration of international medical graduates

INTRODUCTION

Medical regulatory authorities need efficient and effective methods of ensuring the competence of immigrating international medical graduates (IMGs). Not all IMGs who apply for specialist vocational registration will have directly comparable qualifications to those usually accepted. As general licensure examinations are inappropriate for these doctors, workplace-based assessment (WBA) techniques would appear to provide a solution. However, there is little published data on such outcomes.

METHODS

All cases of WBA (n = 81) used for vocational registration of IMGs in New Zealand between 2008 and 2013 were collated and analyzed.

RESULTS

The successful completion rate of IMGs through the pathway was 87%. The majority (64%) undertook the year of supervised practice and the final assessment in a provincial center. For those unsuccessful in the pathway, inadequate clinical knowledge was the most common deficit found, followed by poor clinical reasoning.

DISCUSSION

A WBA approach for assessing readiness of IMGs for vocational registration is feasible. The constructivist theoretical perspective of WBA has particular advantages in assessing the standard of practice for experienced practitioners working in narrow scopes than traditional methods of assessment. The majority of IMGs undertook both the clinical year and the assessment in provincial hospitals, thus providing a workforce for underserved areas.

Revival of a core public health function: state- and urban-based maternal death review processes

This article reviews some of the current challenges for maternal death review in the United States, describes key findings from an assessment of U.S. capacity for conducting maternal death reviews, and introduces a new Maternal Mortality Initiative that aims to develop standardized guidelines for state- or city-based maternal deaths review processes.

Implementing peer evaluation of handoffs: associations with experience and workload

BACKGROUND

Although peer evaluation can be used to evaluate in-hospital handoffs, few studies have described using this strategy.

OBJECTIVE

Our objective was to assess feasibility of an online peer handoff evaluation and characterize performance over time among medical interns.

DESIGN

The design was a prospective cohort study.

PATIENTS

Subjects were medical interns from residency program rotating at 2 teaching hospitals.

MEASUREMENTS

Measurements were performance on an end-of-rotation evaluation of giving and receiving handoffs.

RESULTS

From July 2009 to March 2010, 31 interns completed 60% (172/288) of peer evaluations. Ratings were high across domains (mean, 8.3-8.6). In multivariate regression controlling for evaluator and evaluatee, statistically significant improvements over time were observed for 4 items compared to the first 3 months of the year: 1) communication skills (season 2, +0.34 [95% confidence interval (CI), 0.08-0.60], P = 0.009); 2) listening behavior (season 2, +0.29 [95% CI, 0.04-0.55], P = 0.025); 3) accepting professional responsibility (season 3, +0.37 [95% CI, 0.08-0.65], P = 0.012); and 4) accessing the system (season 2, +0.21 [95% CI, 0.03-0.39], P = 0.023). Ratings were also significantly lower when interns were postcall in written sign-out quality (8.21 vs 8.39, P = 0.008) and accepting feedback (8.25 vs 8.42, P = 0.006). Ratings from a community hospital rotation, with a lower census than the teaching hospital, were significantly higher for overall performance and 7 of 12 domains (P < 0.05 for all). Significant evaluator effects were observed.

CONCLUSIONS

Although there is evidence of leniency, peer evaluation of handoffs demonstrate increases over time and associations with workload such as postcall status. This suggests the importance of examining how workload impacts handoffs in the future.

American Board of Medical Specialties Maintenance of Certification: theory and evidence regarding the current framework

The American Board of Medical Specialties Maintenance of Certification Program (ABMS MOC) is designed to provide a comprehensive approach to physician lifelong learning, self-assessment, and quality improvement (QI) through its 4-part framework and coverage of the 6 competencies previously adopted by the ABMS and the Accreditation Council for Graduate Medical Education (ACGME). In this article, the theoretical rationale and exemplary empiric data regarding the MOC program and its individual parts are reviewed. The value of each part is considered in relation to 4 criteria about the relationship of the competencies addressed within that part to (1) patient outcomes, (2) physician performance, (3) validity of the assessment or educational methods utilized, and (4) learning or improvement potential. Overall, a sound theoretical rationale and a respectable evidence base exists to support the current structure and elements of the MOC program. However, it is incumbent on the ABMS and ABMS member boards to continue to examine their programs moving forward to assure the public and the profession that they are meeting expectations, are clinically relevant, and provide value to patients and participating physicians, and to refine and improve them as ongoing research indicates.

A medical staff peer review system in a public teaching hospital--an internal quality improvement tool

Peer review of the quality of care of the medical staff in a healthcare delivery system, properly executed and utilized, can bring about changes that improve the quality and safety of patient care, enhance clinical performance, and augment physician education. Although all healthcare facilities are mandated to conduct peer reviews, the process of how it is conducted, reported, and utilized varies widely. In 2007, our institution, a large public teaching acute care facility, developed and implemented an electronic Medical Staff Peer Review System (MS-PRS) that replaced the existing paper-based system and created a centralized database for all peer review activities. Despite limited resources and mounting known challenges, we have developed and implemented a system that includes 100% mortality reviews, an ongoing random review for reappointment and operative procedures, and morbidity peer reviews. Parallel to the 4-year implementation of the system, we observed a steady, significant downward trend in the medical malpractice claim rate, which can be attributable in part to the implementation of MS-PRS. In this paper, we share our experiences in the development, outcomes, challenges encountered, and lessons learned from MS-PRS and provide our recommendations to similar institutions for the development of such a system.

Innovations in performance assessment: a criterion based performance assessment for advanced practice nurses using a synergistic theoretical nursing framework

PURPOSE

Health care organizations that employ advanced practice nurses are challenged to evaluate practice at this advanced level. Current evaluation methods tend to inter-mingle basic nursing competencies with competencies found in medical practice and organizational objectives that are typically derived from human resources departments. This article describes the development of a criterion-based job performance assessment for advanced nursing practice using a framework rooted in a nursing theory.

METHOD

A needs analysis; review of the literature, adaptation of nursing's Synergy Model, and input from various stakeholders guided the development of a generic job description. This job description progressed into a criterion-based performance assessment. Construct validity was tested using a questionnaire administered to a convenience sample of 9 practicing advanced practice nurses, 2 nurse executives, 1 PhD nurse educator, and 1 physician.

CONCLUSION

Autonomy, job satisfaction, and quality improvement for advanced practice nurses are fostered by a review process that defines roles and competencies specific to advanced nursing practice. Peer review, a concept contributing to this process is explored as a means to monitor and improve practice.

Peer reviews: taking on new meanings

Peer reviews in nursing are historically used to gauge performance within an individual's scope of practice or as a tool to evaluate a sentinel or adverse event. Quality of care measures, clinical pertinence, and evaluating standards of care have begun as parallel strategies to replace the former uses in assuring the right care at the right time in the right setting.

Twelve tips for implementing a successful peer assessment

Following a multi-institutional research project, the authors formulated 12 tips for the successful implementation of peer assessment in a health-based setting. These tips relate to planning, delivery and feedback. They also offer a wide range of practical guidance and anecdotal evidence relating to peer assessment generally.

Peer review - a safety and quality improvement initiative in a general practice

BACKGROUND

A general practice in inner city Melbourne (Victoria), committed to ensuring quality standards of clinical care, developed a process for peer review of their doctors' performance. The aim was to ensure that there was a robust and fair process for evaluation of doctor performance from both a safety point of view, and from the perspective of contribution to team based practice.

OBJECTIVE

This article describes the process and outcomes of this appraisal process.

DISCUSSION

From the springboard of weekly clinical meetings which address critical incidents and near misses, the practice doctors developed an annual process of formal performance review incorporating hard and soft indicators of clinical performance and compliance with professional and practice standards. This type of activity falls within the scope of quality improvement in general practice.

How do physicians assess their family physician colleagues' performance?: creating a rubric to inform assessment and feedback

INTRODUCTION

The Colleges of Physicians and Surgeons of Alberta and Nova Scotia (CPSNS) use a standardized multisource feedback program, the Physician Achievement Review (PAR/NSPAR), to provide physicians with performance assessment data via questionnaires from medical colleagues, coworkers, and patients on 5 practice domains: consultation communication, patient interaction, professional self-management, clinical competence, and psychosocial management of patients. Physicians receive a confidential report; the intent is practice improvement. However, research indicates that feedback from medical colleagues appears to be less understood than that from coworkers or patients, due to a lack of specificity and concerns regarding feedback credibility. The purpose of this study was to determine how physicians make decisions about performance ratings for family physician (FP) colleagues in the 5 practice domains.

METHODS

This was an exploratory qualitative study using focus groups-one with 11 family physicians and one with 12 specialists-who had served as NSPAR "medical colleague'' reviewers. We analyzed focus group transcripts using content analysis.

RESULTS

Family and specialist physicians provided examples of behaviors indicative of both high- and low-scoring performance for items within the 5 practice domains. From these, an assessment rubric was created to inform both external reviewers and the physicians being reviewed of performance expectations. Reviewers reported using varied sources of information to make assessments, including shared patients, medical records, referral letters, feedback from others, and self-reference.

DISCUSSION

The CPSNS has used the assessment rubric to create an online resource to inform medical colleague assessment and enhance the usefulness of their NSPAR scores. Further research will be required to determine its impact.

Generalisability in unbalanced, uncrossed and fully nested studies

OBJECTIVES

There is growing interest in multi-source, multi-level feedback for measuring the performance of health care professionals. However, data are often unbalanced (e.g. there are different numbers of raters for each doctor), uncrossed (e.g. raters rate the doctor on only one occasion) and fully nested (e.g. raters for a doctor are unique to that doctor). Estimating the true score variance among doctors under these circumstances is proving a challenge.

METHODS

Extensions to reliability and generalisability (G) formulae are introduced to handle unbalanced, uncrossed and fully nested data to produce coefficients that take into account variances among raters, ratees and questionnaire items at different levels of analysis. Decision (D) formulae are developed to handle predictions of minimum numbers of raters for unbalanced studies. An artificial dataset and two real-world datasets consisting of colleague and patient evaluations of doctors are analysed to demonstrate the feasibility and relevance of the formulae. Another independent dataset is used for validating D predictions of G coefficients for varying numbers of raters against actual G coefficients. A combined G coefficient formula is introduced for estimating multi-sourced reliability.

RESULTS

The results from the formulae indicate that it is possible to estimate reliability and generalisability in unbalanced, fully nested and uncrossed studies, and to identify extraneous variance that can be removed to estimate true score variance among doctors. The validation results show that it is possible to predict the minimum numbers of raters even if the study is unbalanced.

DISCUSSION

Calculating G and D coefficients for psychometric data based on feedback on doctor performance is possible even when the data are unbalanced, uncrossed and fully nested, provided that: (i) variances are separated at the rater and ratee levels, and (ii) the average number of raters per ratee is used in calculations for deriving these coefficients.

Commentary: Urgently needed: a safe place for self-assessment on the path to maintaining competence and improving performance

Traditional continuing medical education (CME), necessary for keeping physicians current and competent, is insufficient in translating physician practice into better patient outcomes. CME, then, must be transformed from a system of episodic interventions to a more personalized, contextual, flexible, and targeted process within a continuing professional development framework. The core of this transformation must be a formal process of physician self-assessment. Unfortunately, health care providers tend toward inaccurate self-assessment, regardless of training, specialty, or manner of self-assessment. Therefore, the development of an external validation system conducted by credible, informal peer review in a safe environment is essential. Clinicians must be able to access practice and patient data without concerns about accuracy, timeliness, confidentiality, attribution, or unintended consequences. New analytical tools are also needed to illuminate the data compilations and present them in compelling, individualized, and comparative formats. However, such developmental work will not be possible without strong community leadership coordinating a collaboration of resources and a sharing of data. Ensuring physician competence has long been an issue for medical societies, state licensing boards, and others invested in improving patient care. Now it's time to get serious. Current efforts at massive health care reform provide the perfect opportunity to fully integrate physician self-assessment and performance improvement into the larger health care system through a continuing professional development model. Practitioners in CME have been far too complacent with the current practices. A shift to a focused and dedicated sense of urgency must occur to ensure physicians' continuous learning and change.

Results of a peer review process: the distribution of codes by examining dentists in the Republic of Ireland 2006-2007

The Health Service Executive (HSE) appointed 20 examining dentists in April 2006 under contract for one year as part of a probity assurance initiative by peer review in the Dental Treatment Services Scheme (DTSS) in the Republic of Ireland.

AIM

The aim of the study was to analyse the distribution of codes assigned to the reports drawn up by the examining dentists.

METHODS

At the end of the year's contract, each examining dentist forwarded an end of contract report of their activity, including the distribution of codes issued, to the HSE. These were correlated into a national summary of examining dentist activity, from which the data used in the study was extracted. A total of 11 different codes were used, varying from an indication of agreement between the examining dentist and the contracting dentist (code A) to a significant disagreement (code D).

RESULTS

The vast majority (94.5%) of reports on the clinical examination of patients, drawn up by examining dentists, were in broad agreement with the treatment or estimate of the responsible contracting dentists. A total of 622 contracting dentists received such reports. The small minority of reports (4.8%) where there was a significant disagreement related to a small number of dentists (47 dentists).

CONCLUSIONS

The study provides evidence that most contracting dentists were not a probity risk. The author suggests that any future probity assurance initiative should focus on areas of high risk rather than random selection of patients/contracting dentists. An enhanced advisory role for the examining dentist is recommended.

[PAR index of adolescents with congenital missing teeth pre- and post-orthodontic treatment]

OBJECTIVE

To assess the outcome of orthodontic treatment for adolescents with congenital missing teeth using the PAR (Peer Assessment Ration) index.

METHODS

The PAR index was adopted to evaluate and compare the teeth arrangement, molar occlusion, overjet, overbite and centerline condition of 20 adolescents with congenital missing teeth before and after orthodontic treatments.

RESULTS

The initial PAR scores and weighted PAR total scores decreased significantly after Orthodontic treatments (P < 0.01), except for the centerline. The PAR total scores decreased by 14.25 +/- 7.91 (80.90% +/- 11.09%). The weighted PAR total scores decreased by 15.45 +/- 8.16 (75.34% +/- 18.41%).

CONCLUSION

PAR index can well reflect the improvement of occlusion characters by orthodontic treatment for adolescents with congenital missing teeth. But further improvement is needed for clinical application.

Is rural radiation oncology practice quality as good as the big smoke? Results of the Australian radiotherapy single machine unit trial

Radiotherapy utilization rates in rural Australia are suboptimal, with one solution being the building of single machine units (SMUs). One concern raised with such an approach is the quality of care delivered in SMUs. The Australian and Victorian governments have established two SMUs in the state of Victoria, with each SMU operated as a satellite service of a major 'hub' site. We report on the planned evaluation of practice quality. Radiation oncologist (RO) clinical practice was externally audited using the Royal Australian and New Zealand College of Radiologists Peer Review Audit instrument. This tool splits RO clinical practice into documentation/quality assurance (QA) criteria and decision-making criteria. Over the four sites, 130 patients were randomly selected for audit. At hub sites, 79.6% of all criteria audited were adequate, compared with 84.4% of criteria audited at SMUs (P = 0.0002). This difference was largely because of better adherence to documentation/QA criteria at the SMU sites. RO decision-making and protocol adherence were routinely very high and consistent with other clinical practice audits. There were no significant differences between hubs and SMUs for adherence to decision-making criteria; however, the few potential deficiencies in patient care identified occurred only at the hub sites. In at least one of these cases, potential suboptimal management was as a direct result of inadequate documentation. This audit found that SMUs provide as high a standard of radiotherapeutic care as larger hub departments. The findings also emphasize the need for all departments to target clinical documentation.

Peer review in nursing and midwifery: a literature review

AIMS AND OBJECTIVES

The Clinical Education Project investigated clinical education in nursing and midwifery settings. The aim of this phase was to investigate and evaluate the processes and outcomes of clinical assessment of preregistration nurses and midwives, focusing on the assessment interview, and to evaluate the feasibility of introducing peer review of the clinical assessment interview in acute clinical settings.

BACKGROUND

Peer review is common in many professional areas. The literature describes various applications of peer review and makes recommendations for its use. However, there is a shortage of studies investigating the use of peer review in nursing and midwifery education and practice.

DESIGN

The project involved a systematic literature review and a qualitative exploratory study. This article describes the first part of the study: a systematic literature review of peer review. The second part of the study is reported elsewhere.

METHODS

The systematic literature review investigated international articles written since 1994 that contained information on peer review in pre/post registration nursing and midwifery within higher education or practice.

RESULTS

From the available literature, 52 specific initiatives were analysed. The majority of articles originated in America and involved nursing staff working in secondary care settings. Fifty-one articles had missing information varying from not stating the sample size to not including information about evaluations.

CONCLUSIONS

The literature review found that whilst peer review is commonplace in nursing and midwifery practice, there is a lack of robust literature about its use. Relevance to clinical practice. Peer review in clinical settings such as nursing and midwifery can facilitate the sharing of good practice and personal and professional growth. It allows participants to learn from each other and gain insight into their development.

Do panels vary when assessing intrapartum adverse events? The reproducibility of assessments by hospital risk management groups

INTRODUCTION

A national audit project, Scotland-wide Learning from Intrapartum Critical Events (SLICE), included local assessment of quality of care in cases of perinatal death and neonatal encephalopathy due to intrapartum events. Concerns had been raised about interobserver variation in case assessment by different panels. We therefore studied the extent of agreement and disagreement between assessment panels, and examined the areas in which agreement and disagreement tended to occur.

METHODS

8 cases were randomly selected from all 42 cases identified during a 6-month period (1 January-1 July 2005). Each case was independently reviewed by three panels: the local hospital clinical risk-management group and two specially convened external panels. Panels assessed quality of care in three areas: admission assessment, recognition of incident, and method and timing of delivery. Predefined standards of care were provided for these three areas. Panels were also asked to assess the overall quality of care.

RESULTS

For each area of care, agreement between the two external panels was lowest. The lowest levels of agreement between panels were seen in assessment of overall care (50% crude agreement between external panel 1 and the hospital (kappa = 0.24, AC(1) = 0.36); 29% crude agreement between external panels 1 and 2 (kappa = -0.11, AC(1) = 0.1); 47% crude agreement between external panel 2 and the hospital (kappa = 0.36, AC(1) = 0.46). The lowest level of agreement among all three panels was also in the assessment of overall care (crude agreement 48%; kappa = 0.16, AC(1) = 0.34).

CONCLUSION

Moderate to substantial agreement among the three panels was achieved for the three areas in which explicit standards were provided. Therefore, a systematic approach to analysis of adverse events in perinatal care improves reproducibility.

Statistical sampling in a legislative framework for peer review of medicare services

This article discusses problems addressed in developing an efficient way of identifying levels of inappropriate professional practice in delivery of Medicare services, using statistical sampling within a legislative peer-review scheme. An efficient alternative to the current sampling methodology is proposed.

Effects of visitation among allied health professionals

OBJECTIVE

Visitation is a method for external peer review. The goal is to improve the quality of patient care by giving feedback on quality of competence and performance of a professional during a practice visit. Feedback is offered as recommendations for improvement. This study aims to evaluate the effects of visitation and to determine which factors are related to the effectiveness of visitation.

PARTICIPANTS

Members of seven allied health professions in the Netherlands: dieticians, exercise therapists, physiotherapists, dental hygienists, occupational therapists, podiatrists, and radiology assistants.

DESIGN

Evaluation questionnaires were sent to 151 allied health professionals who had participated in visitation. The questions included all practice management aspects that had been assessed during the practice visit. The effects of visitation were studied at three levels: change in awareness of weak and strong aspects of competence and performance, intention to carry out recommendations, and actual improvements.

RESULTS

Results showed effects of visitation on all three levels. Respondents intended to carry out two-thirds of the recommendations. Visitation led to a better awareness of weak points on 36% of the aspects and better awareness of strong points on 53% of the aspects of practice management. Young respondents reported more changes in awareness than older respondents. Actual improvements were carried out on 33% of the aspects.

CONCLUSIONS

Visitation is an effective method to stimulate quality improvement in allied health professionals. Although changes in awareness more often occurred in younger respondents, actual improvements were made by all respondents.

Registered Nurse Peer Evaluation in the Perioperative Setting

ANNUAL PERFORMANCE evaluations can be difficult to prepare and may rely, in part, on anecdotal information. PERIOPERATIVE RNs at CHRISTUS St Patrick Hospital, Lake Charles, La, developed and implemented a peer evaluation as part of nurses' annual performance evaluations. THE EVALUATION FORMS created were considered to be useful and fair by both staff members and managers.

Audience response system technology improves accuracy and reliability of trauma outcome judgments

BACKGROUND

Peer-review judgments are necessary for effective trauma performance improvement (PI), but may be influenced by peer pressure and the tendency to vote with the majority. Incorporation of Audience Response System (ARS) technology into trauma PI should result in improved outcome assessments.

METHODS

We compared 30 months of nonanonymous trauma care judgments with 30 months of anonymous judgments obtained with the use of a keypad-based ARS. Statistical methods included the chi2 test and the Wilcoxon rank sum test.

RESULTS

Use of the ARS resulted in a 28% reduction in deaths judged nonpreventable and a 24% reduction in trauma care judged to be appropriate (p < 0.0001). Unanimous outcome judgments were also significantly reduced (p < 0.0001).

CONCLUSIONS

Outcome judgments obtained anonymously were significantly more divergent and less positive than those obtained nonanonymously. Anonymously derived outcome judgments may provide a better opportunity to identify adverse outcomes and thereby potentially improve trauma PI and trauma care.