Qualitative assessment of the primary care outcomes questionnaire: a cognitive interview study

Development and content validity testing of a colonoscopy-specific patient-reported experience measure: the Patient Experience Colonoscopy Scale (PECS)

BACKGROUND

In endoscopic care, favourable patient experiences before, during and after a colonoscopy are essential for the patient's willingness to repeat the procedure. To ensure that significant experiences are measured, patients should be involved in creating the measurement instruments. Thus, the aim of the present study was to develop a colonoscopy-specific PREM by (1) operationalising patient experiences before, during and after a colonoscopy procedure and (2) evaluating its content validity.

METHODS

The colonoscopy-specific PREM was developed in two stages: (1) operationalisation with item generation and (2) content validity testing. A previously developed conceptual model, based on a systematic literature review that illustrates patients' (n = 245) experiences of undergoing a colonoscopy, formed the theoretical basis. To assess the degree to which the PREM reflected patients' experiences before, during and after a colonoscopy procedure, content validity was tested-through face validity with healthcare professionals (n = 4) and cognitive interviews with patients (n = 14) having experienced a colonoscopy. Content validity index (CVI) was calculated to investigate the relevance of the items.

RESULTS

The Patient Experience Colonoscopy Scale (PECS) is a colonoscopy-specific PREM consisting of five different constructs: health motivation, discomfort, information, a caring relationship and understanding. Each construct was defined and generated into a pool of items (n = 77). After face-validity assessment with healthcare professionals, a draft 52-item version of the PECS was ready for content validity testing by the patients. During cognitive interviews the patients contributed valuable insights that led to rewording and removal of items. Results from the CVI suggest that the PECS and its content are relevant (I-CVI range 0.5-1, S-CVI/Ave = 0.86). The final PECS consists of 30 items representing a colonoscopy-specific PREM.

CONCLUSION

The PECS is a new 30-item PREM instrument designed for adult elective colonoscopy patients after they have undergone the procedure. Each item in the PECS derives from a conceptual model based on a systematic literature review. Patients and healthcare professionals were involved in developing the PECS, which measures colonoscopy-specific patient experiences before, during and after the procedure. The content validity testing positively contributed to the development of the PECS. Psychometric properties need to be evaluated further.

The Feasibility and Validity of Preference-Based Quality of Life Measures With Informal Carers: A Think-Aloud Study

OBJECTIVES

A range of preference-based quality of life (QoL) measures have been proposed for use with informal carers. Qualitative evaluation of validity and feasibility of the measures is an important step in understanding whether measures will work as intended. At present, little is known about the performance of different types of preference-based QoL measures with informal carers. The objective of this study was to qualitatively assess the feasibility, content validity (including face validity), and acceptability of 5 QoL measures (the Carer Experience Scale, CarerQoL-7D, ASCOT-C, ICECAP-A, and EQ-5D-5L) with informal carers.

METHODS

A total of 24 "think-aloud" interviews were conducted with a cross-section of carers of adults in the United Kingdom. This think-aloud process was followed by semistructured discussion to probe issues of validity and feasibility in more detail. The interview data were transcribed, coded to identify the frequency of errors in completing the QoL measures and thematically analyzed to study the validity, feasibility, and acceptability of the measures.

RESULTS

Few errors (3%-7% per item) were identified in completing each of the measures with little distinct pattern. Most participants found the measures to be concise, clear, and relevant. Challenges included relevance, context, time period, missing items, multiple questions, and response options. Informal carers generally expressed a preference for using a care-related QoL measure.

CONCLUSIONS

Existing preference-based QoL measures have encouraging validity and feasibility within a mixed sample of informal carers, with minor challenges raised. These challenges ought to be considered, alongside the decision context, when administering QoL measures in this context.

The person-based development and realist evaluation of a pre-consultation form for GP consultations

Background

Use of telephone, video and e-consultations is increasing. These can make consultations more transactional, potentially missing patients' concerns. This study aimed to develop a complex intervention to address patients' concerns more comprehensively in general practice and test the feasibility of this in a cluster-randomised framework.The complex intervention used two technologies: a patient-completed pre-consultation form used at consultation opening and a doctor-provided summary report provided at consultation closure. This paper reports on the development and realist evaluation of the pre-consultation questionnaire.

Methods

A person-based approach was used to develop the pre-consultation form. An online questionnaire system was designed to allow patient self-completion of a form which could be shared with GPs. This was tested with 45 patients in three rounds, with iterative adjustments made based on feedback after each round.Subsequently, an intervention incorporating the pre-consultation form with the summary report was then tested in a cluster-randomised framework with 30 patients per practice in six practices: four randomised to intervention, and two to control. An embedded realist evaluation was carried out. The main feasibility study results are reported elsewhere.

Results

Intervention Development: 15 patients were recruited per practice. Twelve patients, six GPs and three administrators were interviewed and 32 changes were made iteratively in three rounds. Recruitment rates (proportion of patients responding to the text) increased from 15% in round one to 50% in round three.Realist evaluation: The pre-consultation form was most useful for people comfortable with technology and with hidden concerns or anxiety about the consultation. It resulted in more issues being discussed and support provided, more effective use of time and greater patient satisfaction.

Conclusions

The person-based approach was successful. The pre-consultation form uncovers more depth and improves satisfaction in certain consultations and patients. Technological improvements are required before this could be rolled out more widely.

Creation and validation of the pictorial ecological momentary well-being instrument (EMOWI) for adolescents

PURPOSE

Adolescence is characterized by the ongoing maturation of emotion-regulation skills and increased emotional reactivity. There is a need for a measurement tool suitable to the Ecological Momentary Assessment methodology, to better capture within-day variations in well-being, and provide fine-grained data that can help understand how environments, behaviors, and health intersect. This paper presents the development and evaluation of the Ecological MOmentary Well-Being Instrument for adolescents, designed for use in EMA.

METHODS

A mixed-methods study was conducted, using both qualitative and quantitative approaches, to develop and assess the EMOWI. A literature review, pictorial production by graphic designers, and qualitative interviews with French and Canadian professionals and adolescents helped design and evaluate the scale face validity. Quantitative evaluation of dimensionality, reliability, and validity was conducted in two samples of French 8th graders.

RESULTS

The resulting 8-item EMOWI showed excellent face validity. Confirmatory factor analysis supported a single factor hypothesis (RMSEA = 0.072). Internal consistency (Cronbach's alpha = 0.85) and intraday test-retest reliability (ICC = 0.83) were high. Correlations with existing scales were consistent with preset hypotheses. Ceiling effects were evidenced for all items, yet not on the global score. Quantitative estimations were similar for the verbal and pictorial versions, but qualitative findings argued in favor of the pictorial version.

CONCLUSION

The 8-item pictorial EMOWI is a short and innovative instrument to measure momentary well-being in adolescents aged 12 to 17 years. Its strong psychometric properties and its acceptability among adolescents make it an excellent candidate instrument for the Ecological Momentary Assessment of well-being in this population.

Using rapid cycle tests of change to develop the Carers Assistive Technology Experience Questionnaire: a cognitive interview study in the UK

We describe the use of rapid cycle tests of change to pretest and develop a Carers Assistive Technology Experience Questionnaire for a survey of informal carers of persons with dementia. The Plan-Do-Study-Act (PDSA) cycle is a commonly used improvement process in healthcare settings. We used this method for conducting rapid cycle tests of change through cognitive interviews to pretest the questionnaire. The items for the questionnaire were developed based on an earlier systematic review and qualitative study. PDSA cycles were used incrementally with learning from each cycle used to inform subsequent changes to the questionnaire prior to testing on the next participant.

DESIGN

Qualitative with use of cognitive interviews through rapid cycle tests of change.

SETTING

UK.

RESULTS

Nine participants were recruited based on eligibility criteria and purposive sampling. Cognitive interviewing using think aloud and concurrent verbal probing was used to test the comprehension, recall, decision and response choice of participants to the questionnaire. Seven PDSA cycles involving the participants helped identify problems with the questionnaire items, instructions, layout and grouping of items. Participants used a laptop, smartphone and/or tablet computer for testing the electronic version of the questionnaire and one participant also tested the paper version. A cumulative process of presenting items in the questionnaire, anticipating problems with specific items and learning from the unanticipated responses from participants through rapid cycle tests of change allowed rich learning and reflection to progressively improve the questionnaire.

CONCLUSION

Using rapid cycle tests of change in the pretesting questionnaire phase of research provided a structure for conducting cognitive interviews. Learning and reflections from the rapid testing and revisions made to the questionnaire helped improve the process of reaching the final version of the questionnaire, that the authors were confident would measure what was intended, rapidly and with less respondent burden.

Cultural adaptation and content validity of a Chinese translation of the 'Person-Centered Primary Care Measure': findings from cognitive debriefing

Objectives

To develop an equivalent Chinese translation of the Person-Centered Primary Care Measure (PCPCM) and to establish its cultural adaptability and content validity through cognitive debriefing.

Design

The original English PCPCM was first translated into Chinese by double forward-translation by professional translators. The reconciliated Chinese version was then doubly back-translated into English by two other professional translators blinded to the forward-translation. On affirmation on its linguistic equivalence with the developers of the original English PCPCM, the reconciliated Chinese PCPCM was sent for cognitive debriefing with 20 Chinese-speaking primary care subjects by a trained interviewer using structured probing questions to collect their opinions on the clarity, comprehensibility and relevance of each item and response option in the Measure.

Setting

Subjects were invited from a primary care clinic in Hong Kong to undergo the cognitive debriefing interviews. The interviews were divided into four groups chronologically to allow revision of the items to be made in between.

Participants

Ten males and 10 females above the age of 18 completed the cognitive interviews. They were all Cantonese-speaking Chinese recruited by convenience sampling. Subjects with cognitive impairment, could not read Chinese, too old or too sick to complete the interviews were excluded from the study.

Results

An average of 3.3 min (range 3–4 min) was required for the subjects to self-complete the Measure. All items were generally perceived to be easily understood and relevant. Modifications were made to items with the content validity index (CVI) on clarity or understanding <0.8 in each round of the interviews or if a majority of the subjects suggested rewording. Revisions were made to two items in the Chinese PCPCM throughout the whole cognitive debriefing process before the final version was confirmed. The average CVI on clarity of the Chinese PCPCM items ranged from 0.75 to 1. The average CVI on understanding ranged from 0.7 to 1. The average CVI on relevance ranged from 0.55 to 1.

Conclusions

The content validity of the PCPCM was ascertained in terms of its clarity, understandability and relevance to allow further testing of its psychometric properties in a larger Chinese population.

Cognitive testing of the PHQ-9 for depression screening among pregnant and postpartum women in Kenya

BACKGROUND

African women face high rates of depression, particularly during pregnancy or postpartum or after a recent HIV diagnosis. The Patient Health Questionnaire-9 (PHQ-9) depression screening tool has been quantitatively validated and extensively used to identify depression and link individuals to care. However, qualitative work is necessary to identify important opportunities to improve PHQ-9 question comprehension and performance among Kenyan women.

METHODS

We administered the Kiswahili or English PHQ-9 (based on preference) to 29 pregnant and postpartum women in Thika, Kenya. Following administration, we conducted cognitive interviews with a purposive sample of 20 women. We used analytic memos and data matrices to identify themes around scale acceptability, comprehension, and decision and response processes.

RESULTS

Most participants preferred to answer the PHQ-9 in Kiswahili (N = 15; 52%). Among the 20 interview participants, 12 (60%) had scores ≥5, indicating depressive symptoms. Overall, participants found the scale acceptable as an interviewer-administered tool. Participants reported few problems related to comprehension but had difficulty answering items not relevant to their lives (e.g., "watching television") and double-barreled items (e.g., "poor appetite or overeating"). They were hesitant to endorse items related to "duties as a wife and mother" and suicidal ideation. Most participants had difficulty distinguishing between response options of "several days" and "more than half the days".

CONCLUSIONS

We detected several problems related to PHQ-9 comprehension, decision processes, and response processes. We provide recommended changes to instructions and item wording to improve PHQ-9 validity among Kenyan women.

Methods to Assess Cannabis Consumption in Population Surveys: Results of Cognitive Interviewing

The Cannabis Act legalized the possession and sale of nonmedical cannabis in Canada on October 17, 2018. Evaluating the impact of cannabis legalization requires a more thorough understanding than is provided by most existing measures of cannabis use. The aim of this study was to pretest a range of cannabis consumption measures used in a population-based survey and to share insights gained in the process. Cognitive interviewing was conducted among 10 cannabis users aged ≥16 years. Comprehension and self-reporting of consumption types and amounts, sources of purchase, and cannabinoid levels were examined. Findings revealed areas for improvement in a number of survey items, including unclear wording and reference images. Identified issues were used to improve the survey for use in the International Cannabis Policy Study. The authors discuss important principles (e.g., use of visual cues, user-selected units, and time frames) that should be adopted when assessing cannabis use in population-based studies.

Patient understanding of two commonly used patient reported outcome measures for primary care: a cognitive interview study

Background

Standardised generic patient-reported outcome measures (PROMs) which measure health status are often unresponsive to change in primary care. Alternative formats, which have been used to increase responsiveness, include individualised PROMs (in which respondents specify the outcomes of interest in their own words) and transitional PROMs (in which respondents directly rate change over a period). The objective of this study was to test qualitatively, through cognitive interviews, two PROMs, one using each respective format.

Methods

The individualised PROM selected was the Measure Yourself Medical Outcomes Profile (MYMOP). The transitional PROM was the Patient Enablement Instrument (PEI). Twenty patients who had recently attended the GP were interviewed while completing the questionnaires. Interview data was analysed using a modification of Tourangeau’s model of cognitive processing: comprehension, response, recall and face validity.

Results

Patients found the PEI simple to complete, but for some it lacked face validity. The transitional scale was sometimes confused with a status scale and was problematic in situations when the relevant GP appointment was part of a longer episode of care. Some patients reported a high enablement score despite verbally reporting low enablement but high regard for their GP, which suggested hypothesis-guessing. The interpretation of the PEI items was inconsistent between patients.

MYMOP was more difficult for patients to complete, but had greater face validity than the PEI. The scale used was open to response-shift: some patients suggested they would recalibrate their definition of the scale endpoints as their illness and expectations changed.

Conclusions

The study provides information for both users of PEI/MYMOP and developers of individualised and transitional questionnaires.

Users should heed the recommendation that MYMOP should be interview-administered, and this is likely to apply to other individualised scales. The PEI is open to hypothesis-guessing and may lack face-validity for a longer episode of care (e.g. in patients with chronic conditions). Developers should be cognisant that transitional scales can be inconsistently completed: some patients forget during completion that they are measuring change from baseline. Although generic questionnaires require the content to be more general than do disease-specific questionnaires, developers should avoid questions which allow broad and varied interpretations.

Electronic supplementary material

The online version of this article (10.1186/s12875-018-0850-2) contains supplementary material, which is available to authorized users.

Primary Care Outcomes Questionnaire: psychometric testing of a new instrument

Background

Patients attend primary care for many reasons and to achieve a range of possible outcomes. There is currently no Patient Reported Outcome Measure (PROM) designed to capture these diverse outcomes, and trials of interventions in primary care may thus fail to detect beneficial effects.

Aim

This study describes the psychometric testing of the Primary Care Outcomes Questionnaire (PCOQ), which was designed to capture a broad range of outcomes relevant to primary care.

Design and setting

Questionnaires were administered in primary care in South West England.

Method

Patients completed the PCOQ in GP waiting rooms before a consultation, and a second questionnaire, including the PCOQ and seven comparator PROMs, after 1 week. Psychometric testing included exploratory factor analysis on the PCOQ, internal consistency, correlation coefficients between domain scores and comparator measures, and repeated measures effect sizes indicating change across 1 week.

Results

In total, 602 patients completed the PCOQ at baseline, and 264 (44%) returned the follow-up questionnaire. Exploratory factor analysis suggested four dimensions underlying the PCOQ items: health and wellbeing, health knowledge and self-care, confidence in health provision, and confidence in health plan. Each dimension was internally consistent and correlated as expected with comparator PROMs, providing evidence of construct validity. Patients reporting an improvement in their main problem exhibited small to moderate improvements in relevant domain scores on the PCOQ.

Conclusion

The PCOQ was acceptable, feasible, showed strong psychometric properties, and was responsive to change. It is a promising new tool for assessment of outcomes of primary care interventions from a patient perspective.