Establishing the content validity of a new emergency department patient-reported experience measure (ED PREM): a Delphi study

Validation of a national leadership framework to promote and protect quality residential aged care: protocol for a Delphi study

INTRODUCTION

Australia's ageing population is driving an increased demand for residential aged care services, yet concerns about the quality and safety of such care remain. The recent Royal Commission into Aged Care Quality and Safety identified various limitations relating to leadership within these services. While some competency frameworks exist globally, there is a need for sector-specific leadership competencies in the Australian residential aged care setting to promote and protect quality of care.

METHODS AND ANALYSIS

This study uses the Delphi technique to establish the content validity of a national leadership framework (RCSM-QF) for promoting and protecting the quality of residential aged care in Australia. Participants will be identifiable experts through current employment within, policy development for or research with the aged care sector. The survey will ask participants to rate the relevance, importance and clarity of RCSM-QF items and their corresponding descriptions and seek suggestions for revisions or additional items. Content validity will be assessed using the Content Validity Index, with items meeting specific criteria retained, revised, or removed.

ETHICS AND DISSEMINATION

Ethics approval has been sought via the James Cook University Human Research Ethics Committee (HREC) to ensure the well-being and convenience of participants while mitigating potential recruitment challenges. Data will be prepared for submission to an appropriate peer-reviewed journal and presentation at relevant academic conferences.

Psychometric validation of a patient-reported experience measure for older adults attending the emergency department: the PREM-ED 65 study

INTRODUCTION

Optimising emergency department (ED) patient experience is vital to ensure care quality. However, there are few validated instruments to measure the experiences of specific patient groups, including older adults. We previously developed a draft 82-item Patient Reported Experience Measure (PREM-ED 65) for adults ≥65 attending the ED. This study aimed to derive a final item list and provide initial validation of the PREM-ED 65 survey.

METHODS

A cross-sectional study involving patients in 18 EDs in England. Adults aged 65 years or over, deemed eligible for ED discharge, were recruited between May and August 2021 and asked to complete the 82-item PREM at the end of the ED visit and 7-10 days post discharge. Test-retest reliability was assessed 7-10 days following initial attendance. Analysis included descriptive statistics, including per-item proportions of responses, hierarchical item reduction, exploratory factor analysis (EFA), reliability testing and assessment of criterion validity.

RESULTS

Five hundred and ten initial surveys and 52 retest surveys were completed. The median respondent age was 76. A similar gender mix (men 47.5% vs women 50.7%) and reason for attendance (40.3% injury vs 49.0% illness) was observed. Most participants self-reported their ethnicity as white (88.6%).Hierarchical item reduction identified 53/82 (64.6%) items for exclusion, due to inadequate engagement (n=33), ceiling effects (n=5), excessive inter-item correlation (n=12) or significant differential validity (n=3). Twenty-nine items were retained.EFA revealed 25 out of the 29 items demonstrating high factor loadings (>0.4) across four scales with an Eigenvalue >1. These scales were interpreted as measuring 'relational care', 'the ED environment', 'staying informed' and 'pain assessment'. Cronbach alpha for the scales ranged from 0.786 to 0.944, indicating good internal consistency. Test-retest reliability was adequate (intraclass correlation coefficient 0.67). Criterion validity was fair (r=0.397) when measured against the Friends and Families Test question.

CONCLUSIONS

Psychometric testing demonstrates that the 25-item PREM-ED 65 is suitable for administration to adults ≥65 years old up to 10 days following ED discharge.

Development of a Patient-Reported Experience Measure Tool for Ambulatory Patients With Acute Unexpected Needs: The APEX Questionnaire

Background: The aim of this study was to develop a patient-reported experience measure (PREM) for comparing the experience of care received by ambulatory patients with acute unexpected needs presenting in emergency departments (EDs), walk-in clinics, and primary care practices. Methods: The Ambulatory Patient EXperience (APEX) questionnaire was developed using a 5-phase mixed-methods approach. The questionnaire was pretested by asking potential users to rate its clarity, usefulness, redundancy, content and face validities, and discrimination on a 9-point scale (1 = strongly disagree to 9 = strongly agree). The pre-final version was then tested in a pilot study. Results: The final questionnaire is composed of 61 questions divided into 7 sections. In the pretest (n = 25), median responses were 8 and above for all dimensions assessed. In the pilot study, 63 participants were enrolled. Adjusted results show that access, cleanliness, and feeling treated with respect and dignity by nurses and physicians were significantly better in the clinics than in the ED. Conclusion: We developed a questionnaire to assess and compare experience of ambulatory care in different clinical settings.

Development of the Parental Experience with Care for Children with Serious Illnesses (PRECIOUS) quality of care measure

BACKGROUND

Parent-reported experience measures are part of pediatric Quality of Care (QoC) assessments. However, existing measures were not developed for use across multiple healthcare settings or throughout the illness trajectory of seriously ill children. Formative work involving in-depth interviews with parents of children with serious illnesses generated 66 draft items describing key QoC processes. Our present aim is to develop a comprehensive parent-reported experience measure of QoC for children with serious illnesses and evaluate its content validity and feasibility.

METHODS

For evaluating content validity, we conducted a three-round Delphi expert panel review with 24 multi-disciplinary experts. Next, we pre-tested the items and instructions with 12 parents via cognitive interviews to refine clarity and understandability. Finally, we pilot-tested the full measure with 30 parents using self-administered online surveys to finalize the structure and content.

RESULTS

The Delphi expert panel review reached consensus on 68 items. Pre-testing with parents of seriously ill children led to consolidation of some items. Pilot-testing supported feasibility of the measure, resulting in a comprehensive measure comprising 56 process assessment items, categorized under ten subthemes and four themes: (1) Professional qualities of healthcare workers, (2) Supporting parent-caregivers, (3) Collaborative and holistic care, and (4) Efficient healthcare structures and standards. We named this measure the PaRental Experience with care for Children with serIOUS illnesses (PRECIOUS).

CONCLUSIONS

PRECIOUS is the first comprehensive measure and has the potential to standardize assessment of QoC for seriously ill children from parental perspectives. PRECIOUS allows for QoC process evaluation across contexts (such as geographic location or care setting), different healthcare workers, and over the illness trajectory for children suffering from a range of serious illnesses.

Development of a low-cost congenital abdominal wall defect simulator (wall-go) for undergraduate medical education: a validation study

BACKGROUND

Congenital Anomalies were responsible for 303,000 deaths in the neonatal period, according to the WHO, they are among the world's top 20 causes of morbidity and mortality. Expensive simulators demonstrate several diseases, but few are related to congenital anomalies. This study aims to develop, validate, and evaluate low-cost simulator models (WALL-GO) of the most common abdominal wall defects, gastroschisis, and omphalocele, to enable diagnosis through an accessible tool with study value and amenable to replication.

METHODS

Market research was conducted to find materials to build low-cost models. The researchers built the model and underwent validation assessment of the selected experts who scored five or more in the adapted Fehring criteria. The experts were assessed through a 5-point Likert scale to 7 statements (S1-7). Statements were assigned values according to relevance in face and transfer validities. Concomitantly, the model was also evaluated by students from 1st to 5th year with the same instruments. Content Validity Indexes (CVIs) were considered validated between groups with concordance greater than 90%. Text feedback was also collected. Each statement was subjected to Fisher's Exact Test.

RESULTS

Gastroschisis and omphalocele model costs were US $15 and US $27, respectively. In total, there were 105 simulator evaluators. 15 experts were selected. Of the 90 students, there were 16 (1st year), 22 (2nd), 16 (3rd), 22 (4th), and 14 (5th). Students and experts obtained CVI = 96.4% and 94.6%, respectively. The CVIs of each statement were not significantly different between groups (p < 0,05).

CONCLUSIONS

The WALL-GO models are suitable for use and replicable at a manufacturable low cost. Mannequins with abdominal wall defects are helpful in learning to diagnose and can be applied in teaching and training health professionals in developing and low-income countries.

Application of the Delphi Method for Content Validity Analysis of a Questionnaire to Determine the Risk Factors of the Chemsex

Chemsex is understood as "the intentional use of stimulant drugs to have sex for an extended time among gay, bisexual, and other men who have sex with men". It is a public health problem because of the increased incidence of cases and because of the consequences on the physical and mental health of those who practice it.

AIM

This study aimed to analyze, with the help of the Delphi method, the content validity of a new instrument to assess the risk of behaviors associated with the chemsex phenomenon.

METHOD

First, a bank of items identified from the literature was elaborated. Secondly, 50 experts with knowledge of the chemsex phenomenon at the national level were contacted. A Delphi group was formed with them to carry out two rounds of item evaluation. The linguistic evaluation (comprehension and appropriateness) was assessed using a Likert scale from 1 to 5 for each item. Items that did not reach a mean score of 4 were eliminated. Content assessment was calculated using each item's content validity index (CVI) and Aiken's V (VdA). A minimum CVI and VdA value of 0.6 was established to include the items in the questionnaire.

RESULTS

A total of 114 items were identified in the literature. In the first round of Delphi evaluation, 36 experts evaluated the items. A total of 58 items were eliminated for obtaining a CVI or VdA of less than 0.6, leaving 56 items. In a second Delphi round, 30 experts re-evaluated the 56 selected items, where 4 items were eliminated for being similar, and 10 items were also eliminated for not being relevant to the topic even though they had values higher than 0.6, leaving the scale finally composed of 52 items.

CONCLUSION

A questionnaire has been designed to assess the risk of behaviors associated with the chemsex phenomenon. The items that make up the questionnaire have shown adequate content and linguistic validity. The Delphi method proved to be a helpful technique for the proposed objective.

Involving patients and caregivers to develop items for a new patient-reported experience measure for older adults attending the emergency department. Findings from a nominal group technique study

CONTEXT

Patient experience is an important component of high-quality care and is linked to improved clinical outcomes across a range of different conditions. Patient-reported experience measures (PREMs) are psychometrically validated instruments designed to identify where strengths and vulnerabilities in care exist. Currently, there is no validated instrument available to measure patient experience among people aged over 65 years attending the emergency department (ED).

OBJECTIVE

This paper aims to describe the process of generating, refining and prioritising candidate items for inclusion in a new PREM measuring older adults' experiences in ED (PREM-ED 65).

DESIGN

One hundred and thirty-six draft items were generated via a systematic review, interviews with patients and focus groups with ED staff exploring older adults' experiences in the ED. A 1-day multiple stakeholder workshop was then convened to refine and prioritise these items. The workshop entailed a modified nominal groups technique exercise comprised of three discrete parts-(i) item familiarisation and comprehension assessment, (ii) initial voting and (iii) final adjudication.

SETTING AND PARTICIPANTS

Twenty-nine participants attended the stakeholder workshop, conducted in a nonhealthcare setting (Buckfast Abbey). The average age of participants was 65.6 years. Self-reported prior experiences of emergency care among the participants included attending the ED as a patient (n = 16, 55.2%); accompanying person (n = 11, 37.9%) and/or as a healthcare provider (n = 7, 24.1%).

RESULTS

Participants were allocated time to familiarise themselves with the draft items, suggest any improvements to the item structure or content, and suggest new items. Two additional items were proposed by participants, yielding a total of 138 items for prioritisation. Initial prioritisation deemed most items 'critically important' (priority 7-9 out of 9, n = 104, 75.4%). Of these, 70 items demonstrated suitable inter-rater agreement (mean average deviation from the median < 1.04) and were recommended for automatic inclusion. Participants then undertook final adjudication to include or exclude the remaining items, using forced choice voting. A further 29 items were included. Thirty-nine items did not meet the criteria for inclusion.

CONCLUSIONS

This study has generated a list of 99 prioritised candidate items for inclusion in the draft PREM-ED 65 instrument. These items highlight areas of patient experience that are particularly important to older adults accessing emergency care. This may be of direct interest to those looking to improve the patient experience for older adults in the ED. For the final stage of development, psychometric validation amongst a real-world population of ED patients is now planned.

PATIENT AND PUBLIC CONTRIBUTION

Initial item generation was informed using qualitative research, including interviews with patients in the ED. The opinions of patients and members of the public were integral to achieving outcomes from the prioritisation meeting. The lay chair of the Royal College of Emergency Medicine participated in the meeting and reviewed the results of this study.

Validation of the accident and emergency experience questionnaire: a cross-sectional survey

BACKGROUND

Patient feedback is an important way for healthcare providers to understand patient experience and improve the quality of care effectively and facilitate patient-centered care in the healthcare system. This study aimed to suggest a validated instrument by evaluating the psychometric properties of the Accident and Emergency Experience Questionnaire (AEEQ) for measuring patient experience in the accident and emergency department (AED) service among the adult Chinese population.

METHODS

Attendances aged 18 or above from all public hospitals with AEDs during 16-30 June 2016 were targeted and a cross-sectional telephone survey was conducted using AEEQ. Preliminary AEEQ consisted of 92 items, including 53 core evaluative items and 19 informative items, and the other 20 items covered socio-demographics, self-perceived health status, and free open-ended comments on AED service. Psychometric properties of the evaluative items were evaluated for practicability, content and structure validity, internal consistency, and test-retest reliability in this study.

RESULTS

A total of 512 patients were recruited with a response rate of 54% and a mean age of 53.2 years old. The exploratory factor analysis suggested removing 7 items due to weak factor loadings and high cross-loading and then leaving 46 items grouped into 5 dimensions, which were care and treatment (14 items), environment and facilities (16 items), information on medication and danger signals (5 items), clinical investigation (3 items), and overall impression (8 items) to represent patient experience on AED service. The internal consistency and test-retest reliability were high with Cronbach's alpha coefficient and Spearman's correlation coefficient of the suggested scale of 0.845 and 0.838, respectively.

CONCLUSION

The AEEQ is a valid and reliable instrument to evaluate the AED service which helps to build the engagement platform for promoting patient-centered care between patients and frontline healthcare professionals and improving healthcare quality in the future.