1
|
Zhu Z, Tomassetti AJ, Dalal RS, Schrader SW, Loo K, Sabat IE, Alaybek B, Zhou Y, Jones C, Fyffe S. A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique. ORGANIZATIONAL RESEARCH METHODS 2021. [DOI: 10.1177/10944281211011529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliability, and in particular test-retest reliability, estimates in policy-capturing studies. We found that only 164 of 955 policy-capturing studies (i.e., 17.17%) reported a test-retest reliability estimate. We then conducted a reliability generalization meta-analysis on policy-capturing studies that did report test-retest reliability estimates—and we obtained an average reliability estimate of .78. We additionally examined 16 potential methodological and substantive antecedents to test-retest reliability (equivalent to moderators in validity generalization studies). We found that test-retest reliability was robust to variation in 14 of the 16 factors examined but that reliability was higher in paper-and-pencil studies than in web-based studies and was higher for behavioral intention judgments than for other (e.g., attitudinal and perceptual) judgments. We provide an agenda for future research. Finally, we provide several best-practice recommendations for researchers (and journal reviewers) with regard to (a) reporting test-retest reliability, (b) designing policy-capturing studies for appropriate reportage, and (c) properly interpreting test-retest reliability in policy-capturing studies.
Collapse
Affiliation(s)
- Ze Zhu
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | | | - Reeshad S. Dalal
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | | | - Kevin Loo
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | - Isaac E. Sabat
- Department of Psychology, Texas A&M University, College Station, TX, USA
| | - Balca Alaybek
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | - You Zhou
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | - Chelsea Jones
- Department of Psychology, George Mason University, Fairfax, VA, USA
| | - Shea Fyffe
- Department of Psychology, George Mason University, Fairfax, VA, USA
| |
Collapse
|
2
|
Jones PW. Estimation and application of the minimum clinically important difference in COPD. THE LANCET RESPIRATORY MEDICINE 2014; 2:167-9. [PMID: 24621674 DOI: 10.1016/s2213-2600(14)70038-4] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Affiliation(s)
- Paul W Jones
- St George's, University of London, London SW17 0RE, UK.
| |
Collapse
|
3
|
Hardie GE, Brown JK, Gold WM. Adrenergic responsiveness: FEV1 and symptom differences in Whites and African Americans with mild asthma. J Asthma 2007; 44:621-8. [PMID: 17943572 DOI: 10.1080/02770900701540481] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
Decision-making about inhaler use is, in part, determined by the ability of asthmatic patients to compare their symptoms over time and to recall the previous response to the bronchodilator during an episode of asthma. The perception of airway symptoms across varied ethnic and cultural groups are poorly understood. Study purpose was (1) to determine if African Americans and Whites with mild asthma could accurately perceive bronchodilation and (2) to identify the word descriptors they used to describe their breathing. Sixteen African American and 16 White patients (34.5 +/- 9.7 years old, mean+/-SD) with mild atopic asthma (FEV1 > or =70% predicted normal) were given increasing doses of an inhaled bronchodilator (Albuterol) after a methacholine challenge. Albuterol (180 microg) was given, by spacer, at 15 min intervals until the FEV1 increased < 5%. Borg, VAS, and Word Descriptors were collected at baseline and after each dose of Albuterol. Baseline FEV1 after Methacholine provocation was 1.94 +/- .39 L for African Americans and 2.13 +/- .70 L for Whites. After 180 microg and again after 360 microg Albuterol, FEV1 increased to 2.88 +/- 0.48 L for African Americans and 3.37 +/- 0.91 L for Whites. But after 540 microg Albuterol, FEV1 decreased significantly (16%) to 2.42 +/- 1.19 L for African Americans while increasing only slightly to 3.47 +/- 0.95 L for Whites. After this dose, 10/16 African Americans felt "tight at the base of throat" (p < 0.01); 7/16 felt "speech-voice-tight" (p < 0.03) suggesting persistent airway discomfort despite marked improvement in FEV1, Borg and VAS scores compared with baseline values. Word descriptors by African Americans' are a more reliable measure of airway symptoms compared to FEV1, Borg or VAS.
Collapse
Affiliation(s)
- Grace E Hardie
- San Francisco State University, San Francisco, California 94132, USA.
| | | | | |
Collapse
|
4
|
Abstract
Health-related quality-of-life scores depend on patients' judgments about their condition and its effect on them. Asthma is a long-term disease, and it is possible that patients may learn to accept some of its effects. We tested the hypothesis that patients come to accept some asthma effects and then discount them as being important when rating their health. We asked patients which of the asthma effects listed in the St. George's Respiratory Questionnaire were acceptable to them. This enabled us to calculate a score for effects of asthma that patients experienced, yet were acceptable to them. The comparative validity of the Current, Acceptable and Unacceptable St George's Respiratory Questionnaire scores was examined by testing their correlations with a variety of asthma-related variables. Eighty patients participated, mean age 50 years, mean forced expiratory volume in 1 second (FEV1) 73 (SD 24)% predicted. Acceptability of St George's Respiratory Questionnaire items was related to the previously identified severity weights (rho = -0.65, p < 0.0001), but some severe effects were acceptable to some patients. Patients who accepted higher levels of health impairment were older with more severe disease. Unacceptable health was less well correlated with asthma-related variables than was current health. We conclude that patients accept some asthma effects but do not appear to discount them when using a detailed health status questionnaire.
Collapse
Affiliation(s)
- Elizabeth A Barley
- Department of Medicine, Respiratory Medicine, Cardiac and Vascular Sciences, St. George's, University of London, London, United Kingdom.
| | | |
Collapse
|