Content Validity and Inter-rater Reliability of the Music Interventions in Pediatric DoC Behavior Observation Record

Aligned with best practice guidelines for patients presenting with a disorder of consciousness (DoC), music therapy interventions with this population aim to increase arousal and awareness to support emergence to consciousness. There is a significant evidence base supporting music therapy for adults with a DoC; however, there are currently no published tools that systematically capture behavioral responses of this population during rehabilitative music therapy interventions. Further, the developmentally specific response to severe brain injury in the pediatric population means pediatric-specific research is required. The Music Interventions in Pediatric DoC Behavior Observation Record (Music Behavior Record [MBR]) was developed to objectively record responses during music therapy interventions for children presenting with a DoC. To establish content validity and inter-rater reliability, a pragmatic pilot study was undertaken. Results established that the MBR has content validity with 100% agreement among participants. Overall fair-substantial inter-rater reliability in >70% of the behavioral responses recorded in the MBR indicate the MBR is an early but promising tool to objectively capture responses during music therapy interventions. The use of the MBR may ultimately support clinical advancement and intervention research to optimize consciousness recovery for the pediatric DoC population.

Content validity of a psychological e-health program of self-control and motivation for adults with excess weight

Excess weight requires multidisciplinary treatment, including psychology in face-to-face and e-health interventions. The need to analyse interventions' content validity has become evident. The objective is to assess content validity of a psychological e-health program of self-control and motivation for adults with excess weight. Six health professionals (a nutritionist, a personal trainer, physician and three psychologists) evaluated the content validity of the program. The results showed that the item-level content validity index (I-CVI) was 1 (the maximum value) for 11 of the 22 activities that comprise the program. This index fluctuated between 0.83 and 0.85 for the other activities, and the scale-level content validity index was greater than 0.90 for the four modules. In terms of relevance, effectiveness and appropriateness, 90%, 65% and 60% of the activities, respectively, obtained the maximum possible I-CVI, while the index was acceptable for the remainder of the activities. This is the first study about content validity of a psychological program for adults with excess weight in Mexico and it supported that the content program was suitable, and it may be used as a part of multidisciplinary treatments. This study also highlights the importance of involving an expert review in the development of intervention programs. This intervention program may be useful in the context of a multidisciplinary intervention; however, it is important to conduct studies to evaluate its efficacy.

An Evidence-Based Somatic Acupressure Intervention Protocol for Managing the Breast Cancer Fatigue-Sleep Disturbance-Depression Symptom Cluster: Development and Validation following the Medical Research Council Framework

BACKGROUND

Somatic acupoint stimulation (SAS) has been frequently utilised as a promising intervention for individual cancer-related symptom management, such as fatigue, sleep disturbance and depression. However, research evidence regarding the role of SAS in mitigating the fatigue-sleep disturbance-depression symptom cluster (FSDSC) has been scant. This study was conducted to develop an evidence-based SAS intervention protocol that can be further implemented in a Phase II randomized controlled trial (RCT) to manage the FSDSC in breast cancer survivors.

METHODS

The Medical Research Council Framework for Developing and Evaluating Complex Intervention (MRC framework) was employed to guide the development procedures of the SAS intervention protocol, including the identification of an existing evidence base, the identification of theories and practice standards, and the validation of the SAS intervention protocol. A content validity study was performed through an expert panel to assess the scientific and practical appropriateness of the SAS intervention protocol. The content validity index (CVI), including item-level CVI and protocol-level CVI, were calculated to evaluate the consensus level of the expert panel.

RESULTS

Key components of the SAS protocol, including the acupoint formula, the SAS modality, technique, intensity and frequency were identified for both a true and placebo SAS intervention based on the best available research evidence retrieved from systematic reviews, clinical trials, and relevant theories, particularly regarding the inflammatory process, yin-yang theory, zang-fu organs and meridians theory, and acupressure practical standards. The true SAS intervention was determined as daily self-administered acupressure on specific acupoints for seven weeks. The placebo SAS was designed as light acupressure on non-acupoints with the same frequency and duration as the true SAS. Excellent content validity was achieved after one round of expert panel assessment, with all the key components of the true and placebo SAS protocols rated as content valid (CVI ranged from 0.86 to 1.00).

CONCLUSIONS

A research-informed, theory-driven and practically feasible SAS intervention protocol for the FSDSC management in breast cancer survivors was developed following the MRC framework. The feasibility and acceptability of the SAS intervention will be further tested in breast cancer survivors through a Phase II RCT.

The Hip Disability and Osteoarthritis Outcome Score-Physical Function Shortform Does Not Adequately Represent Physical Functioning in Patients Undergoing Total Hip Arthroplasty

OBJECTIVES

A frequently used patient-reported outcome measure for assessing physical functioning in patients with hip problems is the 5-item Hip Disability and Osteoarthritis Outcome Score-Physical Function Shortform (HOOS-PS). Nevertheless, its content validity (whether this instrument adequately reflects the construct of physical functioning) is unknown. This study aimed to assess the content validity of the HOOS-PS.

METHODS

A quantitative and qualitative research approach was used. Physical functioning was defined as the ability to perform activities that require physical actions, ranging from self-care to more complex activities that require a combination of skills, often within a social context. Patients (n = 51) and experts (n = 25) completed questionnaires regarding the relevance, comprehensiveness, and comprehensibility of the HOOS-PS. Semistructured interviews (n = 5) explored issues identified in the quantitative data in more depth. Thematic content analysis was conducted using a coding frame.

RESULTS

One of the 5 items was considered relevant to measure physical functioning. Comprehensiveness was considered insufficient by both patients and experts. Furthermore, comprehensibility was considered inadequate. Several items were found ambiguous or double barreled. Regarding interpretability, floor or ceiling effects were not found.

CONCLUSIONS

This study showed concerns about the content validity of the HOOS-PS: most items are considered not relevant, the HOOS-PS is considered not comprehensive, and several items are considered not comprehensible. These findings challenge the applicability of the HOOS-PS in clinical practice, research, value-based healthcare, and benchmarking.

Assessing the Content Validity of the Revised Health of the Nation Outcome Scales (HoNOS 2018)

The Health of the Nation Outcome Scales (HoNOS) comprises 12 scales that cover the kinds of problems that may be experienced by working-age adults in contact with specialised mental health services. Drawing on 20 years' experience in clinical practice, a collaborative, international review of the HoNOS was undertaken and a revised measure (known as the HoNOS 2018) was published. In this study, 32 experts from Australia, England and New Zealand completed an anonymous web-based survey to assess the relevance, comprehensiveness and comprehensibility (aspects of content validity) of the HoNOS 2018. The experts rated 11 of the 12 HoNOS 2018 scales as 'important' or 'very important' for determining the overall clinical severity (item-level content validity index or I-CVI ≥ 0.75). Evaluations of the scales' ability to capture change, comprehensiveness and comprehensibility were more variable, but generally positive. Experts' comments provided further insights into this variability; for example, they noted that some scales combine multiple phenomena, which can result in ambiguity in item wording and assessment challenges. Results from this study suggest that the revisions have not altered the importance of the scales. Given the measure's breadth of content, training remains important for ensuring rating fidelity. Inter-rater reliability and utility testing are indicated.

Assessing the content validity of the revised Health of the Nation Outcome Scales 65+: the HoNOS Older Adults

AIMS AND METHOD

Recently, the Health of the Nation Outcome Scales 65+ (HoNOS65+) were revised. Twenty-five experts from Australia and New Zealand completed an anonymous web-based survey about the content validity of the revised measure, the HoNOS Older Adults (HoNOS OA).

RESULTS

All 12 HoNOS OA scales were rated by most (≥75%) experts as 'important' or 'very important' for determining overall clinical severity among older adults. Ratings of sensitivity to change, comprehensibility and comprehensiveness were more variable, but mostly positive. Experts' comments provided possible explanations. For example, some experts suggested modifying or expanding the glossary examples for some scales (e.g. those measuring problems with relationships and problems with activities of daily living) to be more older adult-specific.

CLINICAL IMPLICATIONS

Experts agreed that the HoNOS OA measures important constructs. Training may need to orient experienced raters to the rationale for some revisions. Further psychometric testing of the HoNOS OA is recommended.

Establishing the Content Validity of a Student Pharmacist Patient Counseling Competency Assessment in Oncology

Objective. The goal of this project was to establish content validity and describe internal consistency of a patient counseling competency assessment instrument used to evaluate student pharmacists practicing in an oncology setting.Methods. The study involved a modified e-Delphi panel of oncology clinical pharmacy specialists, clinical pharmacy generalists, and oncology pharmacy residents. Iterative rounds of the e-Delphi process were conducted until consensus was reached on most instrument items. Consensus was defined as agreement by at least 75% of participants that an item was or was not important.Results. The modified e-Delphi process included three rounds of responses from 13 panelists and resulted in a 35-item instrument with consensus reached on 33/35 (94%) of the items. All participants indicated that the assessment result options allowed them to indicate the student's level of competency either extremely well or very well.Conclusion. A modified e-Delphi method was used to validate a reliable instrument for the assessment of student pharmacist counseling abilities in an oncology setting. Similar methodology should be considered during the development of student assessment tools, especially for high-impact student pharmacist activities such as chemotherapeutic medication counseling.

Content Validity Assessment of the Revised Illness Perception Questionnaire in CKD Using Qualitative Methods

BACKGROUND

The Revised Illness Perception Questionnaire (IPQ-R) measures individuals' unique perceptions of their illness. While psychometric properties of the IPQ-R have been demonstrated in many disease populations, its content validity has not been extensively studied in non-dialysis chronic kidney disease (CKD). Unique features of CKD (e.g., few symptoms in early stages) may impact the measurement of illness perceptions. The purpose of this study was to explore the IPQ-R content validity in a sample of CKD patients.

METHODS

Thirty-one participants completed the IPQ-R and were interviewed regarding their subscale scores (timeline, consequences, personal control, treatment control, coherence, cyclical, and emotions). Participants' agreement with their scores was tallied and assessed qualitatively for themes related to the content validity of the measure.

RESULTS

Individual participant agreement with their subscale scores averaged 79% (range: 29-100%). Subscale agreement varied: timeline (100%), consequences, coherence, and emotion (83% each), cyclical (75%), personal control (65%), and treatment control (64%). A qualitative exploration of disagreement responses revealed concerns with the relevance and comprehensibility of personal control and treatment control.

CONCLUSIONS

Some IPQ-R subscales may pose content validity concerns in the non-dialysis CKD population. Item modification for comprehensibility (personal control) and relevance (treatment control) should be considered. Future studies should explore the impact of a patient's symptom experience on IPQ-R validity, especially in populations like CKD with a higher proportion of asymptomatic patients.

Think-aloud analysis of commonly used screening instruments for Internet use disorders: The CIUS, the IGDT-10, and the BSMAS

Background

Despite the constant publication of new screening instruments for Internet use disorders (IUD), little is known about their content validity. This study aimed to identify potential mismatches between the items' intention and young adults' interpretation of these items when answering three screening instruments that are commonly used in research and clinical settings: The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS).

Methods

In total, 30 vocational students (50% female, age = 21.3; SD = 2.1) took part in individual think-aloud interviews. All participants were asked to report their thoughts while completing the CIUS. In addition, participants who reported online games (OG) as their main Internet activity (n = 11) answered the IGDT-10. Participants who reported social networks (SN) as their main Internet activity (n = 18) answered the BSMAS. One participant used OG and SN equally and completed both screening instruments. All interviews were audio-recorded, transcribed, and content-analysed.

Results

Overall, four potential sources for errors were identified: (1) High scorings were often not congruent with the underlying diagnostic criteria. In particular, such discrepancies were found in items aimed to assess dysfunctional emotional regulation strategies and cognitive involvement. (2) Some participants were uncertain which time frame or Internet activity should be considered in their answers. (3) Long and complex items led to the building of mean values or the choice of the middle answer category. (4) Some wordings were perceived to be outdated and difficult to understand.

Discussion

These findings might help to provide recommendations on how to improve screening instruments for IUD. Most important, items should more clearly distinguish between Internet use as a "normal" leisure activity and Internet use that leads to functional impairments in daily life.

Face and content validity, acceptability, feasibility, and implementability of a novel outcome measure for children with life-limiting or life-threatening illness in three sub-Saharan African countries

BACKGROUND

The Children's Palliative Care Outcome Scale (C-POS) is the first measure developed for children with life-limiting and -threatening illness. It is essential to determine whether the measure addresses what matters to children, and if they can comprehend and respond to its items.

AIM

To determine the face and content validity, comprehensiveness, comprehensibility, acceptability and feasibility, and implementability of the C-POS.

DESIGN

Mixed methods (1) Content validation: mapping C-POS items onto an evidence-based framework from prior evidence; (2) Comprehensiveness, comprehensibility, acceptability feasibility, and implementability: qualitative in-depth and cognitive interviews with a purposive sample of children and young people (n = 6), family caregivers (n = 16), and health workers (n = 12) recruited from tertiary facilities in Kenya, South Africa, and Uganda.

RESULTS

(1) C-POS content mapped on to palliative care domains for (a) children (i.e. physical (e.g. symptoms), social (e.g. play/socialize), psychological (e.g. happy)) and (b) families (i.e. psychological (e.g. worry), social (e.g. information), and help and advice). (2) C-POS items were well understood by children and their caregivers, acceptable, and relevant. Completion time was a median of 10 min, patients/caregivers and health workers reported that using the C-POS improved their communication with children and young people. Methodological and content issues included: (i) conceptual gap in the spiritual/existential domain; (ii) further consideration of developmental, age-appropriate items in the social and psychological domains, and (iii) linguistic complexity and difficulty in proxy rating.

CONCLUSION

C-POS items capture the core symptoms and concerns that matter to children and their families. C-POS is feasible, comprehensible, and acceptable for use in clinical settings; areas for further development and improvement are identified.

Item-Level Psychometric Analysis of the Psychosocial Processes at Work Scale (PROPSIT) in Workers

The structural attributes and correlates of items have an effect on their composite scores and exploring them strengthens the content validity of a measure adapted to another context. The objective of this study was to evaluate the item properties of a measure of psychosocial work factors (PWFs). Data were collected through a web platform from 188 Peruvian working adults (men = 101, 50.5%) holding various professions and jobs. The instrument was the Psychosocial Processes at Work Scale (PROPSIT), adapted for the Peruvian context. The distributional characteristics, the efficiency of its response options and its correlates with engagement, occupational self-efficacy, general stress and psychological distress (explored with a coefficient of maximum information and another of monotonic association) were analyzed. It was found that the items were asymmetrically distributed, without statistical normality and with a response tendency at low (for psychosocial risk factors (PSRFs)) and medium (favorable psychosocial resources) levels. The number of efficient response options was lower (approximately five options) than the original structure (seven options). The monotonic associations with gender and age were essentially zero and theoretically converged with the external constructs, except for some items related to job demands. The contributions of the results to the content validity of the PROPSIT and the orientation of working hypotheses about PROPSIT item constructs and measures of work effects are discussed.

Development and Validation of the Tuberculosis Risk Score for Smokers (TBRSS)

Tuberculosis (TB) remains a significant public health issue worldwide. However, the effectiveness of TB screening programmes among smokers is still questionable. There is a need for a simple, reliable, and validated screening system for this risk population. This study aimed to develop and validate the tuberculosis risk score for smokers (TBRSS) in Kelantan, Malaysia. A case–control study was conducted on 159 patients (smokers with and without TB) between January and July 2020. Simple and multiple logistic regressions were applied to determine the variables to be included in the risk score. The cut-off points to determine a score indicating low or high risk for TB disease were obtained based on the receiver operating characteristics curve. Content validation was carried out through interviews with eight experts to measure each variable′s relevancy. The face validation was conducted among 20 health clinic staff. Seven variables were selected for inclusion in the risk score. The chosen cut-off point was 16 (out of 43), with 91% and 78% sensitivity and specificity, respectively. The scale-level content validity index was 0.83, while the face validity index scores for each element ranged between 0.85 and 1.00. The TBRSS can be considered a validated screening tool for use in screening TB disease risk among smokers, which potentially may lead to an increased detection of TB disease in the community.

Supporting Canadian Nursing Students to Write the NCLEX-RN Exam: A Three-Phased Mixed Methods Descriptive Design

BACKGROUND

In 2015, the College of Nurses of Ontario, replaced the Canadian Registered Nurse Examination with the NCLEX-RN exam as entry-to-practice. Faculty in a college-university partnership searched for products to provide nursing students with focused practice in writing exams modelled on the Canadian NCLEX-RN test plan.

PURPOSE

The aim of this three-phased evaluation study was to test and validate NCLEX-RN exam preparation materials newly developed for the Canadian context.

METHODS

A mixed methods descriptive design was used to capture subjective perspectives and objective measures. After ethical approval was obtained, 13 students assessed the e-learning platform's usability. Eight faculty/clinical experts assessed the content validity of materials using a content validity index (CVI) at both item (I-CVI), and scale (S-CVI) levels. Lastly, 72 completed tests served as the basis for assessing psychometric properties of selected test items.

RESULTS

Materials were assessed as useful and easy to use and navigate. I-CVIs ranged between 0.5 to 1.0 with none falling below 0.5 while S-CVIs were above the standard for acceptability of greater than 0.8 with none falling below 0.9. Overall test reliability measured by the Kuder-Richardson formula was 0.73. Many items assessed for difficulty (64%) showed a proportion of correct responses within desired ranges, and most point-biserial indices ranged from fair to very good.

CONCLUSION

Strong evidence supported the usability and content validity of the materials assessed. Item difficulty and discrimination analyses were within acceptable ranges. Suggestions for improvements were offered. Predictive analysis should form the basis of future research in this area.

Development of central venous access device-associated skin impairment assessment instrument

AIM

To develop an assessment tool to assess the severity and healing of skin impairment with the central venous access device.

DESIGN

Delphi technique.

METHODS

The instrument domain list was developed through a systematic literature review and semi-structured interviews. Experts from China evaluated the relevance and significance of these items in assessing the degree of skin impairment surrounding central venous access device sites through two Delphi rounds. The APA Style JARS checklist for this article was used.

RESULTS

For the systematic literature review, 28 articles were included to develop the wound assessment instrument. From the articles and interview contents, 15 criteria were selected based on reporting frequency. After further screening via in-depth discussion, the central venous access devices associated with the skin impairment assessment tool were refined to include 14 major domains. Through a two-phase Delphi process, 71 items in 12 domains were ultimately retained.

Content validation of common measures of functioning for young children against the International Classification of Functioning, Disability and Health and Code and Core Sets relevant to neurodevelopmental conditions

LAY ABSTRACT

Young children who have developmental delay, autism, or other neurodevelopmental conditions can have difficulties doing things in different areas of their life. What they can and cannot do is called their level of functioning. There are lots of assessment measures that aim to assess functioning. But, we are not sure if these measures assess all the things we need to know about these children's functioning. Other research has identified lists of items (codes) that need to be assessed to understand functioning for young children with different neurodevelopmental conditions fully. These lists include body functions (the things a child's body or brain can do), activities and participation (the activities and tasks a child does) and environmental factors (parts of the environment that can influence functioning). In this study, we looked at the items from these lists assessed by different functioning measures to see how they compared to what should be assessed. The measures that we looked at covered 21%-57% of all the codes and 19%-63% of the codes for lists specific to different conditions. Most of the measures focused on activity and participation codes, and they rarely assessed environmental factors. Knowing which codes and how much of the lists the measures assess can help researchers, clinicians and policymakers to choose measures that are more appropriate for young children with neurodevelopmental conditions.

Measuring Health Status in Long-Term Residential Care: Adapting the Veterans RAND 12 Item Health Survey (VR-12©)

Objectives: Measuring the perceived mental and physical health status of older adults living in long-term residential care (LTRC) is central to patient-centered care. This study examined the qualitative content validity of the Veterans RAND 12 Item Health Survey (VR-12) for LTRC and, based on the findings, the authors developed an adapted version of the generic patient-reported outcome measure for this population.Methods: Content validity was evaluated in two steps: (1) initial resident feedback (n = 9) and research team consensus discussions and (2) cognitive interviews with residents (n = 18) and a research team consensus discussion. The cognitive interviews examined comprehension, acceptability, and relevance of the VR-12 items.Results: Two VR-12 items had limited acceptability in the LTRC setting, the reference to "work" in items was irrelevant to residents, and the lack of a frame of reference ("During the past week … ") impacted comprehension of several items.Conclusions: Study findings informed the development of an adapted version of the VR-12 for older adults living in Canadian LTRC homes and provided content validity evidence regarding its relevance and appropriateness for this population.Clinical implications: Measuring the health status of older adults living in LTRC can help to monitor changes in health status over time and support person-centered care.

Healthy ageing through participation in community situated activities: A scoping review of assessment instruments to support occupational therapy practice

Introduction

The occupational therapy profession has an essential role to play in healthy ageing that includes enabling participation, a construct that according to The International Classification of Functioning, Disability and Health (ICF), incorporates an environmental context. Environmental barriers and enablers of participation in community‐situated activities for people over the age of 65 have been identified. To support practice, occupational therapists require assessments with demonstrated content validity including comprehensive coverage of the construct. The purpose of this scoping review study was to investigate what instruments are available to assess community participation for people over the age of 65 that included environmental factors.

Methods

A scoping review of the literature was conducted, utilising the Joanna Briggs Institute (JBI) scoping review methodology. The evidence source was review articles and inclusion criteria were that they reviewed instruments to assess participation that could be used for people over the age of 65. Items extracted from included instruments were evaluated against a preset list of community‐participation and environment categories that had been developed from the ICF.

Results

Twenty‐three review studies met inclusion criteria and from these 240 instruments were extracted. Twenty instruments were retained after exclusions and from these, 540 instrument items were extracted. Of these, 280 (47%) were coded as community‐participation, and only 20 (3.4%) as environment items. Fourteen of the instruments included no environment items.

Conclusions

No instrument was identified that comprehensively assessed community participation including the related environmental factors. Such an instrument is required to enable occupational therapy practitioners to support healthy ageing. The development of such an instrument will strengthen the profession's capacity to develop new ways of delivering services to older adults in line with emerging ways that aged care will be delivered and to advance its essential role in healthy ageing.

Analysis of the Actions of Net Zone Approach in Padel: Validation of the NAPOA Instrument

To carry out research that analyzes performance indicators through observational methodology, it is necessary to have validated tools. The purpose of this study was to design and validate a tool to ascertain the characteristics of the strokes that padel pairs use to reach the net and their consequences in the two subsequent shots of the game. Eleven experts, who had to meet four of the five inclusion criteria established by the researchers, participated in the process. Aiken’s V coefficient and confidence intervals were used to analyze content validity and Cronbach’s α coefficient to calculate reliability. The adequacy and wording of the sixteen variables initially designed were evaluated. Four variables were eliminated due to obtaining values <0.90 in Aiken’s V coefficient in adequacy. The rest of the variables were modified in their wording, according to the qualitative evaluations of the experts, or were considered correct. The reliability of the instrument was acceptable (α = 0.89). The NAPOA instrument is novel, as it is of interest to analyze the characteristics of the strokes that padel players use to achieve the offensive position.

Translation, Cultural Adaptation, and Content Validation of the Palliative Care Self-Efficacy Scale for Use in the Swedish Context

The Palliative Care Self-Efficacy Scale (PCSE) is a valid instrument in English for assessing healthcare professionals’ self-efficacy in providing palliative care; it has not been translated into Swedish. The aim of this study was to describe the translation, cultural adaptation, and content-validation process of the PCSE scale. In this study, forward and backward translations, pretesting including an expert panel (n = 7), and cognitive interviewing with possible healthcare professionals (physicians, nurses, and assistant nurses) (n = 10) were performed. Experts in palliative care rated items on a Likert scale based on their understandability, clarity, sensitivity, and relevance. The item-level content validity index (I-CVI) and modified kappa statistics were calculated. Healthcare professionals were interviewed using the think-aloud method. The translation and validation process resulted in the final version of the PCSE scale. The average I-CVI for sensitivity was evaluated and approved. Most of the items were approved for clarity, some items lacked understandability, but a majority of the items were considered relevant. The healthcare professionals agreed that the items in the questionnaire evoked emotions, but were relevant for healthcare professionals. Thus, the Palliative Care Self-Efficacy scale is relevant for assessing healthcare professionals’ self-efficacy in palliative care in a Swedish care context. Further research using psychometric tests is required.

The Development and Preliminary Psychometric Properties of the TeleWrite: A Telehealth-Based Handwriting Assessment for School-Aged Children

This article describes the development and preliminary psychometric properties of the TeleWrite, a handwriting assessment tool designed to measure the rate, accuracy, and fluency of children's handwriting for children in first through third grade administered via telehealth. A series of three pilot studies were completed to determine initial interrater reliability, content validity, and clinical utility of the TeleWrite. A study with eight children was used to determine preliminary interrater reliability testing using a beta version of the TeleWrite with (n = 9) raters. The reliability coefficient was obtained using intraclass correlation (ICC) and yielded α = 0.92 for total scores, CI [0.815, 0.983] and the Cronbach's α measure for internal consistency was α = 0.95 (excellent). A clinical utility survey of (n = 55) pediatric occupational therapists was used to obtain their clinical impression regarding the usefulness and accuracy. Representing twelve different countries and twenty-two U.S. states, the respondents indicated that 61% (n = 33) are quite or extremely likely to use the TeleWrite in their practice and most participants (64%; n = 35) indicated that TeleWrite was quite/extremely accurate in measuring handwriting performance. Third, a content validity study with nine handwriting experts reviewed and concluded that all ten subtests of the TeleWrite tool are considered essential. All preliminary studies strengthen the development of the TeleWrite assessment to advance to construct validation of the tool.

Development and validation of a novel patient-reported outcome measure in people with episodic migraine and chronic migraine: The Activity Impairment in Migraine Diary

Objective

To evaluate the content validity and psychometric properties of the Activity Impairment in Migraine Diary (AIM‐D).

Background

Measuring treatment effects on migraine impairment requires a psychometrically sound patient‐reported outcome (PRO) measure developed consistent with U.S. Food and Drug Administration guidance.

Methods

The AIM‐D was created from concepts that emerged during qualitative interviews with five clinicians experienced in treating migraine and concept elicitation (CE) interviews with 40 adults with episodic migraine (EM) or chronic migraine (CM). The initial version was refined based on three waves of cognitive interviews with 38 adults with EM or CM and input from a panel of clinical and measurement experts. The AIM‐D was psychometrically evaluated using data from 316 adults with EM or CM who participated in a 13‐week prospective observational study. Study participants completed PRO assessments including the AIM‐D and a daily headache diary. Exploratory and confirmatory factor analysis were used to determine the factor structure. The reliability, validity, and responsiveness of the AIM‐D were assessed. Additional PRO measures including the Patient Global Impression – Severity (PGI‐S), Migraine Specific Quality of Life Questionnaire, Version 2.1 Role Function‐Restrictive domain, and Headache Impact Test were used for psychometric evaluation of the AIM‐D.

Results

Based on CE interviews with adults with migraine and input from an expert panel, activity impairment was identified as the target in the preliminary conceptual framework, which had two domains: performance of daily activities (PDAs) and physical impairment (PI). Revision of the draft AIM‐D through multiple rounds of cognitive interviews and expert panel meetings resulted in a content valid 11‐item version. Exploratory factor analysis supported both one‐ and two‐domain structures for the AIM‐D, which were further supported by confirmatory factor analysis (factor loadings all >0.90). The AIM‐D domains (PDA and PI) and total score showed high internal consistency reliability (Cronbach's alpha 0.95–0.97), acceptable test–retest reliability for weekly average scores (intraclass correlation coefficient >0.60 for participants with no change in PGI‐S between baseline and week 2), and good convergent and known‐groups validity. There was evidence of responsiveness based on changes in PGI‐S score and monthly migraine days.

Conclusion

The AIM‐D is a content valid and psychometrically sound measure designed to evaluate activity impairment and is suitable for use in clinical trials of preventive treatments for EM or CM.

Measuring the patient experience of chronic rhinosinusitis with nasal polyposis: qualitative development of a novel symptom diary

Background

This qualitative study assessed the experience of patients with chronic rhinosinusitis with nasal polyposis (NP) to inform the development of a novel symptom diary for clinical study use.

Methods

Concept elicitation and cognitive interviews were conducted with patients who had a physician‐verified diagnosis of NP and a history of intranasal corticosteroid use. Concepts were identified via open‐ended and follow‐up questions. Relative symptom/impact disturbance level was assessed using a scale of 0 (not at all disturbing) to 10 (extremely disturbing).

Results

Patients (n = 30) attributed numerous symptoms and impacts to NP; the most prevalent and disturbing were nasal congestion (identified by 100% of patients; average disturbance rating = 7.9), nasal blockage/obstruction (97%; 8.2), difficulty with sense of smell (97%; 7.6), facial pressure (90%; 6.2), postnasal drip (87%; 6.5), runny nose (87%; 6.2), facial pain (80%; 6.3), and headache (77%; 6.5). These symptoms, along with the impact of NP on sleep and daily activities, were included in the Nasal Polyposis Symptom Diary (NPSD). Cognitive interviews confirmed that patients understood the NPSD items and could select a response reflective of their experience at its worst over the past 24 hours using a four‐point scale (none, mild, moderate, or severe).

Conclusion

The most relevant and disturbing symptoms, according to patients with NP, were included in the NPSD. Interviews confirmed the suitability of NPSD in capturing the daily experience of patients. These findings support the content validity of the NPSD as a suitable tool for capturing NP symptoms and impacts.

Five Recommendations to Address the Limitations of Patient-Reported Outcome Measures

Patient-reported outcome measures (PROMs) are reported by the patient and designed to capture patients' unique perspectives of their symptoms, quality of life, function, disability, and overall health status. Despite their important role in the health care landscape, it is important to recognize that PROMs have numerous shortcomings. These include weaknesses in the development of tools and interpretation of scale values, which can lead to variable patient reporting and dissimilarities in study results, potentially influencing the effectiveness of findings. This Viewpoint makes recommendations for how to interpret and best use PROMs, in spite of their shortcomings. J Orthop Sports Phys Ther 2021;51(12):562-565. Epub 16 Nov 2021. doi:10.2519/jospt.2021.10836.

Development of the Multiple Myeloma Symptom and Impact Questionnaire: A New Patient-Reported Outcome Instrument to Assess Symptom and Impacts in Patients With Multiple Myeloma

OBJECTIVES

This study aimed to develop and assess the content validity of a patient-reported outcomes (PROs) instrument to measure symptoms and impacts experienced by patients with active multiple myeloma (MM).

METHODS

The PRO instrument was developed using an iterative, mixed-methods approach. The list of concepts was generated based on a review of existing evidence (qualitative studies and literature) and post hoc psychometric evaluations of 2 PRO instruments in 3 clinical trials. A total of 30 adult patients with MM from the United States participated in hybrid concept elicitation/cognitive debriefing interviews to validate the content validity of the newly developed PRO instrument. Translatability assessment was completed in 8 languages.

RESULTS

The item generation process resulted in 17 symptom and 9 impact concepts for evaluation. The concept elicitation interviews and analysis were based on the first 25 participants; evidence of saturation was observed. The cognitive debriefing interviews and analysis were based on the last 23 participants across 4 waves of interviews. On the basis patient feedback, 10 items were removed, and 1 item was added to the PRO instrument. The translatability assessment resulted in 1 minor revision. The multiple myeloma symptom and impact questionnaire (MySIm-Q) includes 11 symptom and 6 impact concepts, organized within 8 hypothesized subdomains, with each concept measured using a 5-point verbal rating scale and a 7-day recall period.

CONCLUSIONS

The MySIm-Q instrument was developed using rigorous and mixed methodology and with direct input from patients who received a diagnosis of MM. The MySIm-Q has good content validity and is culturally relevant for use in global clinical trials.

Patient-Reported Experience Measures in outpatient rheumatology care: a systematic review

Objectives

There is a growing acceptance of the need for routine implementation of patient-reported experience measures (PREMs) in health care. Rheumatology patients, as frequent and long-term users of care, stand to benefit from collection of experience-related data. The aim of this study was to perform a systematic review to identify and critically appraise the development and psychometric validation of PREMs in rheumatology.

Methods

Six databases were searched systematically from inception to 14 December 2020: MEDLINE, EMBASE, PsycINFO, SCOPUS, Cochrane and Google Scholar. We included articles in English that described the use or development of PREMs, with results of psychometric testing, in an adult outpatient rheumatology context. This study is registered with PROSPERO (CRD42021233819). Articles were appraised using the COnsensus Based Standards for the selection of health status Measurement Instruments (COSMIN) (i) Risk of Bias checklist and (ii) criteria for good measurement properties.

Results

The search yielded 3809 publications, and six studies met inclusion criteria. All the included studies on PREM development fulfilled COSMIN standards for 'doubtful' or 'inadequate' quality of instrument development. One study fulfilled a 'sufficient' rating for content validity, and the remainder fulfilled 'inconsistent' ratings. During validity testing, studies fulfilled between one and four of the eight COSMIN checklist criteria for good measurement properties.

Conclusion

Methodological concerns regarding instrument development and validation limit the generalizability of the existing six validated PREMs in use in rheumatology contexts. There is a need for further well-designed studies to validate existing and new PREMs in this area.