The Validation of a Case-Based, Cumulative Assessment and Progressions Examination

OBJECTIVE

To assess content and criterion validity, as well as reliability of an internally developed, case-based, cumulative, high-stakes third-year Annual Student Assessment and Progression Examination (P3 ASAP Exam).

METHODS

Content validity was assessed through the writing-reviewing process. Criterion validity was assessed by comparing student scores on the P3 ASAP Exam with the nationally validated Pharmacy Curriculum Outcomes Assessment (PCOA). Reliability was assessed with psychometric analysis comparing student performance over four years.

RESULTS

The P3 ASAP Exam showed content validity through representation of didactic courses and professional outcomes. Similar scores on the P3 ASAP Exam and PCOA with Pearson correlation coefficient established criterion validity. Consistent student performance using Kuder-Richardson coefficient (KR-20) since 2012 reflected reliability of the examination.

CONCLUSION

Pharmacy schools can implement internally developed, high-stakes, cumulative progression examinations that are valid and reliable using a robust writing-reviewing process and psychometric analyses.

Content validity of global measures for at-work productivity in patients with rheumatic diseases: an international qualitative study

OBJECTIVES

To identify from a patient's perspective, difficulties and differences in the comprehension of five global presenteeism measures in patients with inflammatory arthritis and OA across seven countries.

METHODS

Seventy patients with a diagnosis of inflammatory arthritis or OA in paid employment were recruited from seven countries across Europe and Canada. Patients were randomly allocated to be cognitively debriefed on 3/5 global measures [Work Productivity Scale - Rheumatoid Arthritis, Work Productivity and Activity Impairment Questionnaire (WPAI), Work Ability Index, Quality and Quantity questionnaire, and WHO Health and Work Performance Questionnaire (HPQ)], with the WPAI debriefed in all patients as a standard measure of comparison between countries and patients. NVivo was used to code the data into four themes: construct and anchor, time recall, reference frame, and attribution.

RESULTS

Discrepancies were found in the interpretation of the word performance (HPQ) between countries, with Romania and Sweden relating performance to sports rather than work. Seventy percent of patients considered that a 7-day recall (WPAI) can accurately represent how their disease affects work productivity. The compared to normal reference (Quality and Quantity questionnaire) was reportedly too ambiguous, and the comparison with colleagues (HPQ), made many feel uncomfortable. Overall, 29% of patients said the WPAI was the most relevant to them, making it the most favoured measure.

CONCLUSION

Overall, patients across countries agree that the construct of work productivity in the last 7 days can accurately reflect the impact of disease while at work. Some current constructs to assess at-work productivity are not interchangeable between languages.

Development of the Flu-PRO: a patient-reported outcome (PRO) instrument to evaluate symptoms of influenza

BACKGROUND

To develop content validity of a comprehensive patient-reported outcome (PRO) measure following current best scientific methodology to standardize assessment of influenza (flu) symptoms in clinical research.

METHODS

Stage I (Concept Elicitation): 1:1 telephone interviews with influenza-positive adults (≥18 years) in the US and Mexico within 7 days of diagnosis. Participants described symptom type, character, severity, and duration. Content analysis identified themes and developed the draft Flu-PRO instrument. Stage II (Cognitive Interviewing): The Flu-PRO was administered to a unique set of influenza-positive adults within 14 days of diagnosis; telephone interviews addressed completeness, respondent interpretation of items and ease of use.

RESULTS

Samples: Stage I: N = 46 adults (16 US, 30 Mexico); mean (SD) age: 38 (19), 39 (14) years; % female: 56%, 73%; race: 69% White, 97% Mestizo. Stage II: N = 34 adults (12 US, 22 Mexico); age: 37 (14), 39 (11) years; % female: 50%, 50%; race: 58% White, 100% Mestizo.

SYMPTOMS

Symptoms identified by >50%: coughing, weak or tired, throat symptoms, congestion, headache, weakness, sweating, chills, general discomfort, runny nose, chest (trouble breathing), difficulty sleeping, and body aches or pains. No new content was uncovered during Stage II; participants easily understood the instrument.

CONCLUSIONS

Results show the 37-item Flu-PRO is a content valid measure of influenza symptoms in adults with a confirmed diagnosis of influenza. Research is underway to evaluate the suitability of the instrument for children and adolescents. This work can form the basis for future quantitative tests of reliability, validity, and responsiveness to evaluate the measurement properties of Flu-PRO for use in clinical trials and epidemiology studies.

Concept Elicitation Within Patient-Powered Research Networks: A Feasibility Study in Chronic Lymphocytic Leukemia

OBJECTIVES

To explore the feasibility of using social media-based patient networks to gather qualitative data on patient-reported outcome (PRO) concepts relevant to chronic lymphocytic leukemia (CLL).

METHODS

Between August and November 2013, US-residing members of the PatientsLikeMe online CLL patient community completed open-ended web-based surveys designed to elicit descriptions of CLL symptoms, impacts, and treatment-related perceptions. Qualitative telephone follow-up interviews were conducted with a subsample of respondents. Survey responses and interview transcripts were coded for qualitative analysis using Atlas.ti.

RESULTS

Fifty survey responses were included in the analyses. Participants were age 60.5 ± 6.9 years, 54% female, and 96% white. When surveyed, 20% were receiving current treatment, 16% were in remission, and 64% were treatment-naïve. Among respondents, 369 descriptions of CLL symptoms were coded. Fatigue-related symptoms were expressed most frequently, with 54% reporting "fatigue," "tiredness," or both in their responses. These concepts were followed by night sweats (38%), swollen lymph nodes (32%), and frequent infections (28%). Among impacts of CLL, worry and fear (66% of respondents), depressed feelings (52%), and work limitations (50%) were noted most frequently.

CONCLUSIONS

Survey results identified constitutional symptoms of CLL included in existing PRO instruments and the literature. Although the findings suggest that qualitative data obtained through social media applications can be potentially useful in supporting concept identification for newly developed PRO instruments, they also indicate that online approaches alone may not be sufficient to achieve efficient and exhaustive concept elicitation. Further research is needed to identify whether the results can support content validity in the same way as established qualitative research methods.

Exploring Fear of Childbirth in the United States Through a Qualitative Assessment of the Wijma Delivery Expectancy Questionnaire

OBJECTIVE

To explore women's experiences while completing the Wijma Delivery Expectancy Questionnaire (W-DEQ), an instrument used to measure fear of childbirth, and to analyze the readability and applicability of the instrument within a diverse population of women in the United States.

DESIGN

Qualitative descriptive study using focus groups with women who were pregnant or had given birth in the last 5 years.

SETTING

Urban health center in the Detroit metropolitan area.

PARTICIPANTS

Participants included 22 women who participated in three focus groups.

METHODS

Focus groups were used to collect data, which were analyzed using content analysis. The Fry Readability Graph, in computer form, was used to rate the readability of the instrument.

RESULTS

Women in the focus groups identified many themes that were consistent with previous research. However, the women indicated many new factors that contributed to their fear that were previously unidentified by the W-DEQ, including fear of abandonment by their clinicians and fear of how the structure of the maternity care system affects care during childbirth.

CONCLUSION

The findings from the focus groups challenge the utility and appropriateness of the W-DEQ for use as a screening tool to identify women who are pregnant and experiencing FOC in a U.S.

The German version of the Camberwell Assessment of Need for the Elderly (CANE): evaluation of content validity and adaptation to the German-speaking context

BACKGROUND

The Camberwell Assessment of Need for the Elderly (CANE) was developed for the assessment of physical-, psychological-, and environment-related needs in the elderly. The aim of this study was to revise and adapt the German version of the CANE with regard to the content validity of the instrument.

METHODS

Following a multistage approach, face-to-face interviews using the CANE, an expert survey and a multidisciplinary consensus conference were conducted in order to evaluate the frequency and relevance of met and unmet needs in the German elderly population, and to modify the content of the CANE for the German-speaking countries.

RESULTS

In Germany, unmet physical needs including physical health, medication, eyesight/hearing/communication, mobility/falls, self-care, and continence were found to have top priority closely followed by social needs (company, intimate relationships, daytime activities, information, and abuse/neglect). Psychological needs were the lowest ranked care category. Experts' proposals for the improvement of the German version of the CANE were collected. All findings were discussed and integrated in the multidisciplinary consensus conference with the result of a revised and adapted CANE that is applicable in the German-speaking context.

CONCLUSIONS

The provision of an adapted and improved German version of the CANE may substantially contribute to a comprehensive and valid assessment of needs in the elderly population. The results of this study represent an important basis for comprehensive needs assessment in the elderly in the theoretical and practical field of healthcare and health services research.

Content Validity Testing of the Maternal Fetal Triage Index

OBJECTIVE

To describe the development and content validity testing of the Maternal Fetal Triage Index (MFTI), a standardized tool for obstetric triage.

DESIGN

Online survey.

PARTICIPANTS

Participants included 15 registered nurses, 15 certified nurse-midwives, and 15 physicians from across the United States who provided maternity care.

METHODS

A convenience sample of experienced clinicians was used as content validators for the MFTI. An item content validity index (I-CVI) was computed for the tool's items and a scale content validity index (S-CVI) was computed for the tool's scale based on the responses submitted via the online survey. Two rounds of content validation occurred.

RESULTS

In the first round of testing, a total of 12 of 61 items in the MFTI did not meet the I-CVI threshold of greater than 0.78 because of disagreement about clinical condition (75%) or priority level placement (25%). In the second round of testing, all but 3 of the 69 content items in the revised version of the MFTI had I-CVI thresholds greater than 0.78. These 3 items were related to vital sign values. The overall S-CVI score calculated for Round 2 only was 0.95, which was greater than the threshold of 0.90.

CONCLUSION

The results of the content validity testing of multidisciplinary validators suggest that the MFTI is a valid tool for use in obstetric triage and evaluation settings.

[Content validation of the questionnaire “The Management of Aggression in People with Dementia Attitude Questionnaire German Version” (MAPDAQ-G)]

BACKGROUND

Dementia is one of the most common diseases of aging and leads to an increased need for care. Caregivers' attitudes about aggression can influence their interaction with people with dementia. To examine this further, “The Management of Aggression in People with Dementia Attitude Questionnaire” (MAPDAQ) was developed for use in nursing homes in the United Kingdom. In Switzerland, the MAPDAQ has been translated into a German Version (MAPDAQ-G), but not yet tested for its validity and reliability.

AIM

Therefore the present study examined the content and face validity, comprehensibility and interpretability of MAPDAQ-G in a Swiss context.

METHODS

The review was based on expert surveys and interviews with caregivers in nursing homes and psychiatric hospitals.

RESULTS

According to experts, the optimised MAPDAQ-G is comprehensive (88 %) and shows good content validity (I-CVI = 0,6 – 0,1; S-CVI-Ave = 0,88). 15 items have been adapted and further validated by 16 nurses. Finally, apart from two items, the MAPDAQ-G is understood by nurses and can be interpreted consistently.

CONCLUSIONS

A questionnaire of this sort allows nurses to reflect in practice upon their own attitude with regard to the management of aggression in people with dementia and to influence the quality of care. The MAPDAQ-G should be statistically tested for validity and reliability using a larger sample.

[Not Available]

PURPOSE

To prepare a Canadian French translation of the PEDro Scale under the proposed name l'Échelle PEDro, and to examine the validity of its content.

METHODS

A modified approach of Vallerand's cross-cultural validation methodology was used, beginning with a parallel back-translation of the PEDro scale by both professional translators and clinical researchers. These versions were reviewed by an initial panel of experts (P1), who then created the first experimental version of l'Échelle PEDro. This version was evaluated by a second panel of experts (P2). Finally, 32 clinical researchers evaluated the second experimental version of l'Échelle PEDro, using a 5-point clarity scale, and suggested final modifications.

RESULTS

The various items on the final version of l'Échelle PEDro show a high degree of clarity (from 4.0 to 4.7 on the 5-point scale).

CONCLUSION

The four rigorous steps of the translation process have produced a valid Canadian French version of the PEDro scale.

The impact of upper facial lines and psychological impact of crow's feet lines: content validation of the Facial Line Outcomes (FLO-11) Questionnaire

BACKGROUND

Treatments for upper facial lines (UFL), the most visible sign of aging, are of interest to patients and clinicians alike. Patient-reported outcomes (PROs) are valuable in evaluating the impact of such treatments; however, regulatory recommendations have stipulated that the patient perspective be central in developing these assessments.

OBJECTIVES

(1) To evaluate the content validity of the Facial Lines Outcomes Questionnaire, a PRO instrument developed to assess upper facial line impacts, according to the regulatory guidance of the United States Food and Drug Administration and (2) assess whether it adequately measures the psychological impacts associated with crow's feet lines (CFL) (lateral canthal lines) from the patient perspective.

METHODS

Two patient groups participated in face-to-face qualitative interviews. One group included patients with UFL (Group 1, n = 25 interviews), and the other included patients specifically with CFL (Group 2, n = 41 interviews). Each interview consisted of a concept elicitation and cognitive debriefing phase.

RESULTS

Interviews with both groups elicited all key concepts of the instrument, including "bothered by facial lines"; "looking older"; "looking less attractive"; and looking "tired," "stressed," or "angry." Most Group 2 patients (n = 35, 85%) agreed that the instrument adequately assessed the psychological impacts associated with CFL. During cognitive debriefing, the majority of patients in both groups agreed the instrument was understandable, comprehensive, and easy to complete.

CONCLUSIONS

The Facial Line Outcomes Questionnaire is an appropriate and valid tool to assess the impact of UFL and the psychological impacts associated with CFL.

Development and content validity testing of a patient-reported outcomes questionnaire for the assessment of hereditary angioedema in observational studies

BACKGROUND

Hereditary Angioedema (HAE), a rare genetic disease, manifests as intermittent, painful attacks of angioedema. Attacks vary in frequency and severity and include skin, abdominal and life-threatening laryngeal swellings. This study aimed to develop a patient reported outcome (PRO) tool for the assessment of HAE attacks, including their management and impact on patients' lives, for use in clinical studies, or by physicians in general practice.

METHODS

The results of open-ended face to face concept elicitation interviews with HAE patients in Argentina (n = 10) and the US (n = 33) were used to develop the first draft questionnaire of the HAE patient reported outcomes questionnaire (HAE PRO). Subsequently, in-depth cognitive debriefing interviews were performed with HAE patients in the UK (n = 10), Brazil (n = 10), Germany (n = 11) and France (n = 12). Following input from eight multinational clinical experts further cognitive interviews were conducted in the US (n = 12) and Germany (n = 12). Patients who experienced abdominal, cutaneous or laryngeal attacks of varying severity levels were included in all rounds of interviews. Across the rounds of interviews patients discussed their HAE attack symptoms, impacts and treatments. Cognitive debriefing interviews explored patient understanding and relevance of questionnaire items. All interviews were conducted face to face following a pre-defined semi-structured interview guide in the patient's native language.

RESULTS

Patients reported a variety of HAE symptoms, attack triggers, warning signs, attack impacts and treatment options which were used to develop the HAE PRO. The HAE PRO was revised and refined following input from patients and clinical experts. The final 18-item HAE PRO provides an assessment of the HAE attack experience including symptoms, impacts, treatment requirements, healthcare resource use and loss of productivity caused by HAE attacks.

CONCLUSIONS

Patient and expert input has contributed to the development of a content valid questionnaire that assesses concepts important to HAE patients globally. HAE patients across cultures consider the HAE PRO a relevant and appropriate assessment of HAE attacks and treatment.

Content validity testing of questions for a patient satisfaction with general anesthesia care instrument

INTRODUCTION

Patient satisfaction with any health care experience is the result of a complex set of interactions between the patient and the health system in which care is received. Measuring patient satisfaction allows quantification of the congruence between the expectations of care and the care that is received. It is important to quantify satisfaction with care because it involves the patient in the care experience and decreases the potential gap between expected and actual care delivered over time. Despite the benefits of measuring satisfaction with anesthesia care, this outcome has been historically understudied possibly because of a lack of reliable tools that measure the construct.

PURPOSE

The purpose of this study was to test the content validity of items derived from an integrated review of studies that measured patient satisfaction with anesthesia care.

METHODS

The content validity of the questions was tested using an expert panel. A total of 13 nurse anesthetists consented to participate in the study as expert panel members. Each expert panel member evaluated the items for content validity on a 4-point scale. Both an Individual Content Validity Index (I-CVI) and a Scale Content Validity Index (S-CVI) were calculated.

RESULTS

All I-CVI scores were greater than 0.8. The range of I-CVI was 0.833 to 1.0. The average scale Content Validity Index (S-CVI/ave) was 0.979.

CONCLUSION

High content validity of all items was supported by an expert panel of anesthesia providers. The high level of agreement among raters related to the content of the general anesthesia statements supports progressing to the next step in instrument development.

Content validity to support the use of a computer-based phonological awareness screening and monitoring assessment (Com-PASMA) in the classroom

PURPOSE

This study investigated the content validity of a computer-based phonological awareness (PA) screening and monitoring assessment (Com-PASMA) designed to evaluate school-entry PA abilities. Establishing content validity by confirming that test items suitably 'fit' and sample a spectrum of difficulty levels is critical for ensuring educators can deduce accurate information to comprehensively differentiate curricular reading instruction.

METHOD

Ninety-five children, inclusive of 21 children with spoken language impairment, participated in a 1-year longitudinal study whereby the Com-PASMA was administered at the start, middle and end of the school year.

RESULT

Estimates of content validity using Rasch Model analysis demonstrated that: (1) rhyme oddity and initial phoneme identity tasks were most appropriate at school-entry and sampled a spectrum of difficulty levels, (2) more challenging phoneme level tasks (e.g. final phoneme identity, phoneme blending, phoneme deletion and phoneme segmentation) became increasingly appropriate and differentiated between high- and low-ability students by the middle and end of the first year of school and (3) letter-knowledge tasks were appropriate but declined in their ability to differentiate student ability as the year progressed.

CONCLUSION

Findings demonstrate that the Com-PASMA has sufficient content validity to measure and differentiate between the PA abilities of 5-year-old children on entry to school.

Acceptance: what's in a name? A content analysis of acceptance instruments in individuals with chronic pain

Instruments to assess chronic pain acceptance have been developed and used. However, whether and to what extent the content of the items reflects acceptance remain uninvestigated. A content analysis of 13 instruments that aim to measure acceptance of chronic pain was performed. A coding scheme was used that consisted of 3 categories representing the key components of acceptance, that is, disengagement from pain control, pain willingness, and engagement in activities other than pain control. The coding scheme consisted of 5 additional categories in order to code items that do not represent acceptance, that is, controlling pain, pain costs, pain benefits, unclear, and no fit. Two coders rated to what extent the items of acceptance instruments belonged to one or more of these categories. Results indicated that acceptance categories were not equally represented in the acceptance instruments. Of note, some instruments had many items in the category controlling pain. Further analyses revealed that the meaning of acceptance differs among different instruments and among different versions of the same instrument. This study illustrates the importance of content validity when developing and evaluating self-report instruments.

PERSPECTIVE

This article investigated the content validity of questionnaires designed to measure acceptance in individuals with chronic pain. Knowledge about the content of the instruments will provide further insight into the features of acceptance and how to measure them.

Criterion analysis and content validity for standardized behavioral tests in a detector-dog breeding program

Many working-dog programs assess behavior during a dog's first year of life with the aim of predicting success in the field. However, decisions about which tests to administer are frequently made on the basis of tradition or intuition. This study reports results from a survey given to U.S.A.'s Transportation Security Administration (TSA) detection-dog handlers (N = 34). We categorized and summarized handlers' responses regarding traits they felt were important for work. We used this criterion analysis to examine the content validity of the TSA's puppy tests. Results indicate that 13 of 15 traits that are currently being measured are relevant. However, several traits not currently measured were identified as being highly important, notably "play" and off-duty "calmness." These results provide support that the TSA tests are measuring traits relevant to operational search team performance but also highlight other traits that may be profitable to assess in this and other detection-dog programs.

Developing an interactive mobile phone self-report system for self-management of hypertension. Part 2: content validity and usability

Self-management support tools using technology may improve adherence to hypertension treatment. There is a need for user-friendly tools facilitating patients' understanding of the interconnections between blood pressure, wellbeing and lifestyle. This study aimed to examine comprehension, comprehensiveness and relevance of items, and further to evaluate the usability and reliability of an interactive hypertension-specific mobile phone self-report system. Areas important in supporting self-management and candidate items were derived from five focus group interviews with patients and healthcare professionals (n = 27), supplemented by a literature review. Items and response formats were drafted to meet specifications for mobile phone administration and were integrated into a mobile phone data-capture system. Content validity and usability were assessed iteratively in four rounds of cognitive interviews with patients (n = 21) and healthcare professionals (n = 4). Reliability was examined using a test-retest. Focus group analyses yielded six areas covered by 16 items. The cognitive interviews showed satisfactory item comprehension, relevance and coverage; however, one item was added. The mobile phone self-report system was reliable and perceived easy to use. The mobile phone self-report system appears efficiently to capture information relevant in patients' self-management of hypertension. Future studies need to evaluate the effectiveness of this tool in improving self-management of hypertension in clinical practice.

Psychometric properties and validation of the Italian version of the Mainz pain staging system as a tool for pain-patients referral selection

RATIONALE, AIMS AND OBJECTIVES

Indications are lacking on which patient to refer to pain facilities. Pain-chronicity stage and outcome prognosis may be used for such aims. The Mainz pain-staging system (MPSS) classifies pain patients in three chronicity stages that respectively require more extensive management. We explored the psychometric and validation properties of its Italian version towards its application as screening/referral tool.

METHODS

I-MPSS was administered to n=120 mixed non-cancer-pain outpatients. Psychometric analyses and formal validation included: content validity, by assessing the hypothesis of an existing relationship between the I-MPSS classes and criteria derived from an operational case definition of chronic pain; construct validity, by principle component analysis (PCA); the autonomous construct of the I-MPSS was assessed by the strength of the Spearman correlation between its classes and the brief pain inventory (BPI) items; and reliability, by applying Cronbach's alpha statistics. Associations between psychosocial moderators and the I-MPSS were assessed applying χ(2) analyses.

RESULTS

Quantitative and qualitative analyses showed significant differences between I-MPSS classes for health care and drug utilization; BPI item scores significantly differed between the classes; Spearman correlation between I-MPSS classes and BPI items was mostly moderate or mild. PCA and scree test identified four components accounting for 63.7% of the variance. Cronbach's alpha was 0.842.

CONCLUSIONS

The I-MPSS showed satisfactory psychometric and validation properties. With adequate feasibility, it enabled the screening of mixed non-cancer-pain outpatients in three chronicity/prognostic stages. Results are sufficient to warrant its use for a subsequent impact study as a prognostic model and screening tool for referring pain patients.

An instrument for evaluating clinical teaching in Japan: content validity and cultural sensitivity

BACKGROUND

Many instruments for evaluating clinical teaching have been developed but almost all in Western countries. None of these instruments have been validated for the Asian culture, and a literature search yielded no instruments that were developed specifically for that culture. A key element that influences content validity in developing instruments for evaluating the quality of teaching is culture. The aim of this study was to develop a culture-specific instrument with strong content validity for evaluating clinical teaching in initial medical postgraduate training in Japan.

METHODS

Based on data from a literature search and an earlier study we prepared a draft evaluation instrument. To ensure a good cultural fit of the instrument with the Asian context we conducted a modified Delphi procedure among three groups of stakeholders (five education experts, twelve clinical teachers and ten residents) to establish content validity, as this factor is particularly susceptible to cultural factors.

RESULTS

Two rounds of Delphi were conducted. Through the procedure, 52 prospective items were reworded, combined or eliminated, resulting in a 25-item instrument validated for the Japanese setting.

CONCLUSIONS

This is the first study describing the development and content validation of an instrument for evaluating clinical teaching specifically tailored to an East Asian setting. The instrument has similarities and differences compared with instruments of Western origin. Our findings suggest that designers of evaluation instruments should consider the probability that the content validity of instruments for evaluating clinical teachers can be influenced by cultural aspects.

The development and initial validation of a clinical tool for patients' preferences on patient participation--The 4Ps

AIMS

To report on the development and initial testing of a clinical tool, The Patient Preferences for Patient Participation tool (The 4Ps), which will allow patients to depict, prioritize, and evaluate their participation in health care.

BACKGROUND

While patient participation is vital for high quality health care, a common definition incorporating all stakeholders' experience is pending. In order to support participation in health care, a tool for determining patients' preferences on participation is proposed, including opportunities to evaluate participation while considering patient preferences.

METHODS

Exploratory mixed methods studies informed the development of the tool, and descriptive design guided its initial testing. The 4Ps tool was tested with 21 Swedish researcher experts (REs) and patient experts (PEs) with experience of patient participation. Individual Think Aloud interviews were employed to capture experiences of content, response process, and acceptability.

RESULTS

'The 4Ps' included three sections for the patient to depict, prioritize, and evaluate participation using 12 items corresponding to 'Having Dialogue', 'Sharing Knowledge', 'Planning', and 'Managing Self-care'. The REs and PEs considered 'The 4Ps' comprehensible, and that all items corresponded to the concept of patient participation. The tool was perceived to facilitate patient participation whilst requiring amendments to content and layout.

CONCLUSIONS

A tool like The 4Ps provides opportunities for patients to depict participation, and thus supports communication and collaboration. Further patient evaluation is needed to understand the conditions for patient participation. While The 4Ps is promising, revision and testing in clinical practice is required.

Expediting Clinical and Translational Research via Bayesian Instrument Development

Developing valid and reliable instruments is crucial but costly and time-consuming in health care research and evaluation. The Food and Drug Administration and the National Institutes of Health have set up guidelines for developing patient-reported outcome instruments. However, the guidelines are not applicable to cases of small sample sizes. Instead of using an exact estimation procedure to examine psychometric properties, our Bayesian Instrument Development (BID) method integrates expert data and participant data into a single seamless analysis. Using a novel set of priors, we use simulated data to compare BID to classical instrument development procedures and test the stability of BID. To display BID to non-statisticians, a graphical user interface based on R and WINBUGS is developed and demonstrated with data on a small sample of heart failure patients. Costs were saved by eliminating the need for unnecessary continuation of data collection for larger samples as required by the classical instrument development approach.

Examining the diagnostic criteria for Internet addiction: Expert validation

BACKGROUND/PURPOSE

Internet addiction is the coming problem around the world. The diagnostic criteria for Internet addiction among adolescents (DC-IA-A) has become a widely used measure for assessing the presence of Internet addiction in Taiwan. This study examined the diagnosis criteria for Internet addiction in adolescents by expert evaluation.

METHODS

Twenty psychiatrists rated the adequacy of each criterion in DC-IA-A. The content validity and homogeneity reliability proposed by Aiken were calculated.

RESULTS

The coefficients content validity and homogeneity reliability showed twenty psychiatrists agreed on each of DC-IA-A as relevant to the diagnosis of Internet addiction, though several criteria need improvements. Two criteria "excessive time spent on Internet activities and leaving the Internet" and "excessive effort spent on activities necessary to obtain access to the Internet" should be omitted, and the criteria of "tolerance" should be modified.

CONCLUSION

The diagnostic criteria for Internet addiction among adolescents should be revised to meet the real condition of this population.

Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures

BACKGROUND

The US Food and Drug Administration's guidance for industry document on patient-reported outcomes (PRO) defines content validity as "the extent to which the instrument measures the concept of interest" (FDA, 2009, p. 12). According to Strauss and Smith (2009), construct validity "is now generally viewed as a unifying form of validity for psychological measurements, subsuming both content and criterion validity" (p. 7). Hence, both qualitative and quantitative information are essential in evaluating the validity of measures.

METHODS

We review classical test theory and item response theory (IRT) approaches to evaluating PRO measures, including frequency of responses to each category of the items in a multi-item scale, the distribution of scale scores, floor and ceiling effects, the relationship between item response options and the total score, and the extent to which hypothesized "difficulty" (severity) order of items is represented by observed responses.

RESULTS

If a researcher has few qualitative data and wants to get preliminary information about the content validity of the instrument, then descriptive assessments using classical test theory should be the first step. As the sample size grows during subsequent stages of instrument development, confidence in the numerical estimates from Rasch and other IRT models (as well as those of classical test theory) would also grow.

CONCLUSION

Classical test theory and IRT can be useful in providing a quantitative assessment of items and scales during the content-validity phase of PRO-measure development. Depending on the particular type of measure and the specific circumstances, the classical test theory and/or the IRT should be considered to help maximize the content validity of PRO measures.

Using Cognitive Interviews to Assess the Cultural Validity of State and Trait Measures of Mindfulness among Zen Buddhists

Although Western psychological mindfulness shares many common features with Buddhist mindfulness, subtle differences in the way in which it is practiced and assessed may have important implications. Therefore, the primary goal of this qualitative study was to evaluate the cultural validity of the Five Facet Mindfulness Questionnaire (FFMQ) and Toronto Mindfulness Scale (TMS) by using cognitive interviews among a sample of Buddhist clergy and laypersons to assess their perceptions of these two scales. Participants were 14 Zen Buddhists (7 laypersons, 6 Zen priests, and 1 in priest the ordination process) recruited from a monastery in the Pacific Northwestern U.S. Each participant completed a cognitive interview using the FFMQ and TMS. We developed a coding schema to identify and categorize participant responses, and then applied the final coding framework to all 14 interviews. Results revealed perceived concerns and strengths of each scale, as well as concerns regarding content deemed missing from both scales and general issues related to mindfulness self-assessment. These findings suggest that Buddhist and Western psychological conceptualizations of mindfulness may have important differences.

Discriminant content validity: a quantitative methodology for assessing content of theory-based measures, with illustrative applications

OBJECTIVES

In studies involving theoretical constructs, it is important that measures have good content validity and that there is not contamination of measures by content from other constructs. While reliability and construct validity are routinely reported, to date, there has not been a satisfactory, transparent, and systematic method of assessing and reporting content validity. In this paper, we describe a methodology of discriminant content validity (DCV) and illustrate its application in three studies.

METHODS

Discriminant content validity involves six steps: construct definition, item selection, judge identification, judgement format, single-sample test of content validity, and assessment of discriminant items. In three studies, these steps were applied to a measure of illness perceptions (IPQ-R) and control cognitions.

RESULTS

The IPQ-R performed well with most items being purely related to their target construct, although timeline and consequences had small problems. By contrast, the study of control cognitions identified problems in measuring constructs independently. In the final study, direct estimation response formats for theory of planned behaviour constructs were found to have as good DCV as Likert format.

CONCLUSIONS

The DCV method allowed quantitative assessment of each item and can therefore inform the content validity of the measures assessed. The methods can be applied to assess content validity before or after collecting data to select the appropriate items to measure theoretical constructs. Further, the data reported for each item in Appendix S1 can be used in item or measure selection. Statement of contribution What is already known on this subject? There are agreed methods of assessing and reporting construct validity of measures of theoretical constructs, but not their content validity. Content validity is rarely reported in a systematic and transparent manner. What does this study add? The paper proposes discriminant content validity (DCV), a systematic and transparent method of assessing and reporting whether items assess the intended theoretical construct and only that construct. In three studies, DCV was applied to measures of illness perceptions, control cognitions, and theory of planned behaviour response formats. Appendix S1 gives content validity indices for each item of each questionnaire investigated. Discriminant content validity is ideally applied while the measure is being developed, before using to measure the construct(s), but can also be applied after using a measure.