Expediting Clinical and Translational Research via Bayesian Instrument Development

Bayesian Hierarchical Factor Analysis for Efficient Estimation across Race/Ethnicity

Patient reported outcomes are gaining more attention in patient-centered health outcomes research and quality of life studies as important indicators of clinical outcomes, especially for patients with chronic diseases. Factor analysis is ideal for measuring patient reported outcomes. If there is heterogeneity in the patient population and when sample size is small, differential item functioning and convergence issues are challenges for applying factor models. Bayesian hierarchical factor analysis can assess health disparity by assessing for differential item functioning, while avoiding convergence problems. We conducted a simulation study and used an empirical example with American Indian minorities to show that fitting a Bayesian hierarchical factor model is an optimal solution regardless of heterogeneity of population and sample size.

Confirmatory Factor Analysis Alternative: Free, Accessible CBID Software

New software that performs Classical and Bayesian Instrument Development (CBID) is reported that seamlessly integrates expert (content validity) and participant data (construct validity) to produce entire reliability estimates with smaller sample requirements. The free CBID software can be accessed through a website and used by clinical investigators in new instrument development. Demonstrations are presented of the three approaches using the CBID software: (a) traditional confirmatory factor analysis (CFA), (b) Bayesian CFA using flat uninformative prior, and (c) Bayesian CFA using content expert data (informative prior). Outcomes of usability testing demonstrate the need to make the user-friendly, free CBID software available to interdisciplinary researchers. CBID has the potential to be a new and expeditious method for instrument development, adding to our current measurement toolbox. This allows for the development of new instruments for measuring determinants of health in smaller diverse populations or populations of rare diseases.

A novel method for expediting the development of patient-reported outcome measures and an evaluation across several populations

Item response theory (IRT) models provide an appropriate alternative to the classical ordinal confirmatory factor analysis (CFA) during the development of patient-reported outcome measures (PROMs). Current literature has identified the assessment of IRT model fit as both challenging and underdeveloped (Sinharay & Johnson, 2003; Sinharay, Johnson, & Stern, 2006). This study evaluates the performance of Ordinal Bayesian Instrument Development (OBID), a Bayesian IRT model with a probit link function approach, through applications in two breast cancer-related instrument development studies. The primary focus is to investigate an appropriate method for comparing Bayesian IRT models in PROMs development. An exact Bayesian leave-one-out cross-validation (LOO-CV) approach (Vehtari & Lampinen, 2002) is implemented to assess prior selection for the item discrimination parameter in the IRT model and subject content experts' bias (in a statistical sense and not to be confused with psychometric bias as in differential item functioning) toward the estimation of item-to-domain correlations. Results support the utilization of content subject experts' information in establishing evidence for construct validity when sample size is small. However, the incorporation of subject experts' content information in the OBID approach can be sensitive to the level of expertise of the recruited experts. More stringent efforts need to be invested in the appropriate selection of subject experts to efficiently use the OBID approach and reduce potential bias during PROMs development.

A novel method for expediting the development of patient-reported outcome measures and an evaluation of its performance via simulation

BACKGROUND

Developing valid and reliable patient-reported outcome measures (PROMs) is a critical step in promoting patient-centered health care, a national priority in the U.S. Small populations or rare diseases often pose difficulties in developing PROMs using traditional methods due to small samples.

METHODS

To overcome the small sample size challenge while maintaining psychometric soundness, we propose an innovative Ordinal Bayesian Instrument Development (OBID) method that seamlessly integrates expert and participant data in a Bayesian item response theory (IRT) with a probit link model framework. Prior distributions obtained from expert data are imposed on the IRT model parameters and are updated with participants' data. The efficiency of OBID is evaluated by comparing its performance to classical instrument development performance using actual and simulation data. RESULTS AND DISCUSSION : The overall performance of OBID (i.e., more reliable parameter estimates, smaller mean squared errors (MSEs) and higher predictive validity) is superior to that of classical approaches when the sample size is small (e.g. less than 100 subjects). Although OBID may exhibit larger bias, it reduces the MSEs by decreasing variances. Results also closely align with recommendations in the current literature that six subject experts will be sufficient for establishing content validity evidence. However, in the presence of highly biased experts, three experts will be adequate.

CONCLUSIONS

This study successfully demonstrated that the OBID approach is more efficient than the classical approach when the sample size is small. OBID promises an efficient and reliable method for researchers and clinicians in future PROMs development for small populations or rare diseases.

Response to Sijtsma and van der Ark (2015): "Conceptions of reliability revisited and practical recommendations"

Sijtsma and van der Ark present a broad set of models and methods for reliability estimation, and their discussion of similarities and differences provides clear information for nurse researchers to move forward in their instrument development projects. In particular, we applaud the authors' clear exposition of the factor analytic model and its utility for providing a framework for unifying reliability and validity. However, we do not want to be constrained only to the point estimates. We also need to ascertain the uncertainty in the point estimate-usually in the form of a 95% confidence interval-or, as the Bayesians refer to, a credible interval. Another issue not discussed by Sijtsma and van der Ark is conditional standard errors of measurement along the score scale measuring latent traits or true scores. In our response, practical tools for estimating intervals and a brief discussion of conditional standard errors of measurement are presented.