Content validity of a health science evidence-based practice questionnaire (HS-EBP) with a web-based modified Delphi approach

Instruments to Assess Evidence-Based Practice Among Health Care Professionals: A Systematic Review

BACKGROUND

The use of measurement instruments to assess the use of Evidence-Based Practice by health professionals has been frequently reported in studies.

AIMS

This systematic review aimed to summarize, describe, and evaluate the measurement properties of the instruments that evaluate the use of Evidence-Based Practice in health professionals.

METHODS

The search was carried out in four databases considering three groups of search terms: evidence-based practice, evaluation, and measurement proprieties. Studies were included that described the use of instruments to assess Evidence-Based Practice in health professionals, with the full-text publication, which analyzed the measurement properties, in English. The methodological quality of the studies was evaluated using COnsensus-based Standards for the selection of health Measurement INstruments.

RESULTS

In total, 6,429 were found and only 92 were eligible for data analysis. Forty new instruments were identified most were developed for nursing and physical therapist. The investigators performed at least 1 type of validity test on 73% of the instruments. Reliability was tested at 90%, through internal consistency. Responsiveness was tested on less than half of the instruments (30%). Most of the instruments identified are reliable and valid to measure evidence-based practice in health professionals.

CONCLUSION

Although the Fresno Test remains the most complete instrument, and adequate for use. The COnsensus-based Standards for the selection of health Measurement INstruments checklist classified 7 (seven) instruments as suitable for the target audience.

Translating and Validating the Vietnamese Version of the Health Sciences Evidence-Based Practice Questionnaire

No validated instrument is available for assessing the evidence-based practice capacity of Vietnamese health professionals. This study aimed to translate and validate the Health Sciences Evidence-Based Practice questionnaire (HS-EBP) from English to Vietnamese and ascertain its psychometric properties. Data were collected from two obstetric hospitals in Vietnam. Participants: A total of 343 midwives were randomly selected. The HS-EBP questionnaire was translated by a group of bilingual experts into Vietnamese (HS-EBP-V). Content validity was assessed by two experts. Internal consistency and test-retest reliabilities were assessed using Cronbach's α and intraclass correlation (ICC), respectively. Construct validity was assessed using the contrasted groups approach. As a result, the content validity index of the HS-EBP-V reached 1.0. For the individual subscales, Cronbach's α was 0.92-0.97 and ICC was between 0.45 and 0.66. The validity of the contrasted-groups approach showed discrimination by a significant difference in the subscale scores among diploma holders compared with bachelor's degree holders (p < 0.001). The validation of the HS-EBP questionnaire indicated satisfactory psychometric properties. The results indicate that the HS-EBP is a reliable and valid instrument which assesses the competencies of as well as facilitators of and barriers to the five steps of EBP among midwives. The HS-EBP-V was deemed a reliable and validated tool for assessing the competency and application of EBP among Vietnamese healthcare professionals.

Adaptação transcultural e validação do “health sciences evidence based practice questionnaire” para o português do Brasil

Introdução: o instrumento “Health Sciences Evidence Based Practice Questionnaire” avalia as diferentes dimensões da construção da prática baseada em evidência como um processo para responder às situações decorrentes da prática clínica diária. Objetivos: adaptar culturalmente o instrumento Health Sciences Evidence Based Practice Questionnaire para a língua portuguesa do Brasil e avaliar suas qualidades psicométricas. Método: Estudo metodológico. A primeira etapa compreendeu o processo de tradução e adaptação transcultural. Na segunda etapa, o instrumento Questionário de Prática Baseada em Evidências e Efetividade Clínica foi aplicado para a validação convergente, utilizando-se o índice de correlação de Pearson, nível de significância 0,05 e realizada Análise Fatorial Confirmatória (AFC), de acordo com o modelo de equações estruturais. A confiabilidade foi avaliada pelo teste-reteste, utilizando o coeficiente de correlação r de Pearson. A consistência interna obtida por meio do coeficiente alpha de Cronbach. Resultados: validade de conteúdo obtida na terceira rodada, título final: Questionário HS-EBP- Prática Baseada em Evidência nas Ciências da Saúde, aplicado em 305 profissionais da saúde. A confiabilidade foi verificada por meio do coeficiente alfa de Cronbach (0,970), similar à versão original, e da estabilidade teste-reteste pelo índice de correlação r de Pearson (r = 0,766) que indicou uma correlação forte. Para a validação convergente foi aplicado o índice de correlação de Pearson (r = 0,683) positivo e com correlação moderada entre os instrumentos. Teste Qui-Quadrado foi significativo (<0,001) para todos os modelos obtidos, em todos os modelos testados. Conclusão: o instrumento mostrou-se válido e confiável para reprodutibilidade na versão brasileira.

The design and development of an experience measure for a peer community moderated forum in a digital mental health service

Online digital mental health communities can contribute to users' mental health positively and negatively. Yet the measurement of experience, outcomes and impact mechanisms relating to digital mental health communities is difficult to capture. In this paper we demonstrate the development of an online experience measure for a specific children and young people's community forum inside a digital mental health service. The development of the Peer Online Community Experience Measure (POCEM) is informed by a multi-phased design: (i) item reduction through Estimate-Talk-Estimate modified Delphi methods, (ii) user testing with think-aloud protocols and (iii) a pilot study within the digital service community to explore observational data within the platform. Experts in the field were consulted to help reduce the items in the pool and to check their theoretical coherence. User testing workshops helped to inform the usability appearance, wording, and purpose of the measure. Finally, the pilot results highlight completion rates, differences in scores for age and roles and "relate to others", as the most frequent domain mechanism of support for this community. Outcomes frequently selected show the importance of certain aspects of the community, such as safety, connection, and non-judgment previously highlighted in the literature. Experience measures like this one could be used as indicators of active therapeutic engagement within the forum community and its content but further research is required to ascertain its acceptability and validity. Multi-phased approaches involving stakeholders and user-centred design activities enhances the development of digitally enabled measurement tools.

Rasch Analysis of Self-Reported Adherence to Patient-Centered Physical Therapy Scale among Japanese Physical Therapists: Cross-Sectional Study

This study primarily aimed to develop a shorter version of the self-reported adherence to patient-centered physical therapy (s-SAPCPTS) by using Rasch analysis and secondarily aimed to preliminarily investigate the relationship between the s-SAPCPTS scores and demographics (i.e., age, sex, final academic degree (non-postgraduate degrees or postgraduate degrees), and practice environment). In an online anonymous survey, 110 Japanese physical therapists completed the self-reported adherence to patient-centered physical therapy and provided data on their demographics. Through the Rasch analysis, items were excluded in a stepwise manner, until certain pre-established criteria of the unidimensionality were satisfied. Subsequently, a conversion table for the Rasch score was developed. Furthermore, multiple regression analysis was conducted by using the independent variables age, sex, and final academic degree. Using the Kruskal-Wallis test, we compared the Rasch s-SAPCPTS scores among four practice environments. Consequently, the seven-item s-SAPCPTS was developed by excluding seven items through the Rasch analysis. Postgraduate degree was a statistically significant contributing factor for Rasch s-SAPCPTS scores (p = 0.038, β = 0.20). The Kruskal-Wallis test demonstrated statistically significant differences in the Rasch s-SAPCPTS scores among the four practice environments (p = 0.006). In conclusion, the seven-item s-SAPCPTS was developed with the preliminary evidence of construct validity. It was also found that the final academic degree and practice environment could be the contributing factors of s-SAPCPTS scores.

Cross-cultural Adaptation of the Health Sciences Evidence-based Practice Questionnaire into Japanese and Its Test-Retest Reliability in Undergraduate Students

Objectives:

The Health Sciences Evidence-Based Practice (HS-EBP) questionnaire was recently developed for measuring five constructs of evidence-based clinical practice among Spanish health professionals by applying content and construct validity investigation. The current study aims to undertake a cross-cultural adaptation of the HS-EBP into Japanese and to investigate the internal consistency and test–retest reliability of the Japanese HS-EBP among undergraduate students of nursing and physical and occupational therapies.

Methods:

Cross-cultural adaptation was undertaken by following Beaton’s five-step process. Subsequently, the Japanese HS-EBP test–retest reliability was assessed with a 2-week interval. Participants were recruited from among third and fourth grade undergraduate students of nursing and physical and occupational therapies with clinical training experience.

Results:

Pilot testing included 30 participants (11 nursing students, 11 physical therapy students, 8 occupational therapy students). Consequently, we developed the Japanese HS-EBP to be understandable for undergraduate students of nursing and physical and occupational therapies. Data from 52 participants who completed test–retest reliability questionnaires demonstrated adequate test–retest reliability in the total scores of Domains 1, 3, 4, and 5 [intraclass correlation coefficients were (ICC)=0.74, 0.70, 0.75, and 0.74, respectively]; the exception was Domain 2, which had an ICC of 0.66. Internal consistency (Cronbach’s α) was adequate for Domains 1–5, for which α was 0.87, 0.94, 0.86, 0.93, and 0.95, respectively.

Conclusions:

This study developed the Japanese version of HS-EBP and provided preliminary evidence of adequate internal consistency and test–retest reliability in most domains for undergraduate students of nursing and physical and occupational therapies.

Health Sciences-Evidence Based Practice Questionnaire (HS-EBP): Normative Data and Differential Profiles in Spanish Osteopathic Professionals

The main goal of this study was to obtain normative data of the scores of the Health-Sciences Evidence Based Practice (HS-EBP) questionnaire, and to analyse evidence-based practice (EBP) among potential clusters of osteopathy professionals in Spain. An online descriptive cross-sectional study has been applied. A total number of 443 Spanish practicing osteopaths answered a survey including the 5 dimensions of the HS-EBP questionnaire and sociodemographic, training, and practice variables using the “LimeSurvey” online platform. Results point out that the median scores for each five HS-EBP questionnaire dimensions were 95.00, 86.00, 78.00, 84.00 and 62.00 considering that the range of possible scores in each of the dimensions was: from 12 to 120 in dimensions 1, 4 and 5; from 14–140 in dimension 2; and from 10–100 in dimension 3. A clustering algorithm extracted 6 different profiles across the five HS-EBP latent dimensions: low scores in all dimensions (cluster 1); low scores in all dimensions but with medium scores in dimension 1 (cluster 2); mixed pattern of scores, low in dimensions 2 and 5 and medium in the rest of the dimensions; medium scores in all dimensions (cluster 4); high scores in all dimensions and low scores in dimension 5 (cluster 5); and high scores in all dimensions (cluster 6). Significant relationship was found among the response patterns in the clusters and: academic degree level, EBP training and training level, and work time invested in healthcare activity, research and teaching activity. These results allow a description of the actual level of EBP and differential profiles of Osteopathy care practice in Spain. Knowledge of normative scores of the HS-EBP questionnaire and identification of different predictors of Spanish osteopaths’ EBP, e.g., academic degree, EBP training and training level, work time invested in healthcare activity, research, and teaching activity, and having a working relationship with an accredited educational centre, enable a comprehensive evaluation of the EBP of osteopathic professionals and can also be useful for developing and implementing formative intervention programs for improving EBP practice in osteopathic practice.

Developing and Validating an Adjustment Scale: The Adaptation Status Assessment of Drug-Resistant Tuberculosis Patients

Purpose

Drug-resistant tuberculosis (DR-TB) remains a major global public health issue. For DR-TB patients, effective adaptation is crucial to prevent disease progression, improve health outcomes and decrease mortality. To date, there is no appropriate tool for evaluating the adaptation status of DR-TB patients. In this work, we aim to develop an adjustment scale for DR-TB patients (AS-DRTBP) and to evaluate its psychometric properties.

Patients and Methods

The development of the AS-DRTBP was based on the theory of the Roy adaptation model (RAM). The scale was designed through a literature review, in-depth individual interviews, a Delphi survey, and pilot testing. In total, 433 patients with DR-TB were recruited to validate the instrument. The split-half reliability coefficient, Cronbach’s alpha coefficient, and test-retest reliability coefficient were calculated to assess the reliability of the instrument. Content validity, construct validity and concurrent validity tests were applied to calculate the validity of the instrument.

Results

The final AS-DRTBP consisted of four dimensions and 26 items. The Cronbach’s alpha coefficient, split-half reliability coefficient and test-retest reliability coefficient were 0.893, 0.954, and 0.853, respectively. The content validity index was 0.92. Four factors that explained 64.605% of the total variance were also further determined by confirmatory factor analysis (CFA). The CFA results showed that the fitting effect of the model was appropriate (CMIN/DF = 1.681, GFI = 0.832, AGFI = 0.799, RMSEA = 0.055, SRMR = 0.0684). The AS-DRTBP and adjustment scale had correlation in the total score, and the correlation coefficient was 0.355 (p<0.05).

Conclusion

The findings of this study demonstrate that the AS-DRTBP is a reliable and valid instrument for measuring the adaptation status of patients with DR-TB, allowing health providers to comprehend the adaptive level of DR-TB patients and thus laying the foundation for interventions to help these patients achieve a physiologically, psychologically and socially optimal outcome.

Cross-cultural adaptation, internal consistency, test-retest reliability and feasibility of the German version of the evidence-based practice inventory

Background

A psychometrically robust measurement instrument is prerequisite to tailor and monitor interventions aiming to improve evidence-based practice (EBP). The recently developed “Evidence-based Practice Inventory” (EBPI) questionnaire (five dimensions) provides a sound inventory for a comprehensive assessment of adherence to EBP, and identification of barriers and facilitators for EBP. The aims of this study were to establish a German language version of the EBPI and to examine the instrument’s reliability in a diverse sample of healthcare professionals.

Methods

The English version of the EBPI was translated, adopted and subsequently test-retest reliability of the German language EBPI was examined in a nationwide online survey. Participants working in Germany were invited to complete the questionnaire twice.

For each EBPI dimension, internal consistency reliability (Cronbach’s alpha) and the relative test-retest reliability (intraclass correlation coefficient, ICC) were calculated. The standard error of measurement, limits of agreement and minimal detectable change values were estimated to quantify measurement error.

Results

A German language version of the EBPI was established. In the online survey, the EBPI was initially completed by 889 healthcare professionals. At follow-up, 344 individuals (39%) completed the questionnaire (74% female; mean work experience: 13.6 years).

The ICCs for the five dimensions varied between 0.78 and 0.86. The standard error of measurement varied between 6.5 and 8.8% of the respective dimension scale range, and the limits of agreement between 24 and 37%. For internal consistency reliability, alpha varied between 0.64 and 0.90. There were neither floor nor ceiling effects, nor any other relevant feasibility issues.

Conclusions

The German language EBPI can be used to assess EBP adherence of healthcare professionals in clinical practice, and to identify barriers and facilitators for an EBP conform behaviour. Results on test-retest reliability indicate that the EBPI produces reliable scores when used for group comparisons, but the questionnaire seems insufficiently reliable for individual measurements over time. Methods of item response theory or Rasch measurement theory should be used for further evaluation and revision of the EBPI, informed by the results of this study.

Trial registration

German Clinical Trials Register (DRKS00013792). Registered 19 January 2018.

Electronic supplementary material

The online version of this article (10.1186/s12913-019-4273-0) contains supplementary material, which is available to authorized users.

Health Sciences-Evidence Based Practice questionnaire (HS-EBP) for measuring transprofessional evidence-based practice: Creation, development and psychometric validation

Introduction

Most of the EBP measuring instruments available to date present limitations both in the operationalisation of the construct and also in the rigour of their psychometric development, as revealed in the literature review performed. The aim of this paper is to provide rigorous and adequate reliability and validity evidence of the scores of a new transdisciplinary psychometric tool, the Health Sciences Evidence-Based Practice (HS-EBP), for measuring the construct EBP in Health Sciences professionals.

Methods

A pilot study and a subsequent two-stage validation test sample were conducted to progressively refine the instrument until a reduced 60-item version with a five-factor latent structure. Reliability was analysed through both Cronbach’s alpha coefficient and intraclass correlations (ICC). Latent structure was contrasted using confirmatory factor analysis (CFA) following a model comparison aproach. Evidence of criterion validity of the scores obtained was achieved by considering attitudinal resistance to change, burnout, and quality of professional life as criterion variables; while convergent validity was assessed using the Spanish version of the Evidence-Based Practice Questionnaire (EBPQ-19).

Results

Adequate evidence of both reliability and ICC was obtained for the five dimensions of the questionnaire. According to the CFA model comparison, the best fit corresponded to the five-factor model (RMSEA = 0.049; CI 90% RMSEA = [0.047; 0.050]; CFI = 0.99). Adequate criterion and convergent validity evidence was also provided. Finally, the HS-EBP showed the capability to find differences between EBP training levels as an important evidence of decision validity.

Conclusions

Reliability and validity evidence obtained regarding the HS-EBP confirm the adequate operationalisation of the EBP construct as a process put into practice to respond to every clinical situation arising in the daily practice of professionals in health sciences (transprofessional). The tool could be useful for EBP individual assessment and for evaluating the impact of specific interventions to improve EBP.