An instrument for evaluating clinical teaching in Japan: content validity and cultural sensitivity

Psychometric Properties Evaluation of Persian Version of the Maastricht Clinical Teaching Questionnaire

Background

Students are a significant source of data for the evaluation of clinical instructors' performance. This study was undertaken with the aim of adaptation and validation of the Persian version of the Maastricht Clinical Teaching Questionnaire (MCTQ). The main objective of the researchers in this study was to evaluation the psychometric properties of the Persian version of the Maastricht Clinical Training Questionnaire (P-MCTQ) in Iran, considering cultural and social differences.

Materials and Methods

This methodological study was conducted from 2019 to 2021 at four teaching hospitals affiliated with Sabzevar University of Medical Sciences, Iran. Qualitative and quantitative face and content validity, and construct validity methods were used for the validity evaluation. Stability and internal consistency methods were used, respectively, for the reliability evaluation of the questionnaire. Exploratory Factor Analysis (EFA) stage, 264 students studying in the fields of nursing, midwifery, anesthesiology, operating room, emergency medicine, and laboratory sciences completed the P-MCTQ.

Results

The value of scale-content validity index (0.92) is indicative of the overall content validity of the questionnaire. EFA extracted a single-factor structure that could explain the overall variance of the clinical education structure at about 76.61%. The alfa and intraclass correlation values were equal to 0.98 and 0.82, respectively, indicating the excellent internal consistency and high overall stability of the questionnaire.

Conclusions

The P-MCTQ is a valid and reliable tool for the evaluation of the teaching performance of clinical instructors.

Competency lists for urban general practitioners/family physicians using the modified Delphi method

BACKGROUND

In recent years, the growing global urbanization and urban population have resulted in the emergence of various health problems unique to urban areas. Therefore, training general practitioners and family physicians who can tackle the complex health problems of urban areas and improve the health of urban people is one of the most important issues of our time. However, findings on competencies for urban general practitioners (GP) and family physicians (FP) were limited. This study aimed to identify their comprehensive and content-validated list of competencies.

METHODS

We used the modified Delphi method to develop a content-validated competency list. First, we analyzed and synthesized the competencies extracted from the literature review using qualitative thematic analysis methods to create an initial competency list of 34 items. We then assembled 39 expert panelists in four groups of study participants: physicians, nurses, patients, and medical education specialists. The expert panelists were asked to indicate their level of agreement with the lists and provide revised comments on the description of each competency via a web-based questionnaire. Their responses were analyzed quantitatively and qualitatively by the research team and used to revise the list. These processes were repeated, and the survey was completed when it was determined that consensus had been reached.

RESULTS

Three rounds of Delphi were conducted. 39 responded in the first round, 38 in the second round, and 36 in the third round. The initial list of competencies was revised and consolidated from 34 to 14 items in the first round, bringing the total to 20 items along with six new items proposed by the panelists. In the second round, it was revised and consolidated into a list of 18 items. In the third round, all 18 items were considered to have been agreed upon by the panelists, so the survey was closed.

CONCLUSION

We identified a comprehensive 18-item list of competencies for urban GP/FP in a content-validated manner. Several are newly discovered competencies in this study. The findings of this study will be useful for the future training of urban GP/FP and for solving urban health problems.

How culture affects validity: understanding Japanese residents' sense-making of evaluating clinical teachers

OBJECTIVES

Traditionally, evaluation is considered a measurement process that can be performed independently of the cultural context. However, more recently the importance of considering raters' sense-making, that is, the process by which raters assign meaning to their collective experiences, is being recognised. Thus far, the majority of the discussion on this topic has originated from Western perspectives. Little is known about the potential influence of an Asian culture on raters' sense-making. This study explored residents' sense-making associated with evaluating their clinical teachers within an Asian setting to better understand contextual dependency of validity.

DESIGN

A qualitative study using constructivist grounded theory.

SETTING

The Japanese Ministry of Health, Labour and Welfare has implemented a system to monitor the quality of clinical teaching within its 2-year postgraduate training programme. An evaluation instrument was developed specifically for the Japanese setting through which residents can evaluate their clinical teachers.

PARTICIPANTS

30 residents from 10 Japanese teaching hospitals with experience in evaluating their clinical teachers were sampled purposively and theoretically.

METHODS

We conducted in-depth semistructured individual interviews. Sensitising concepts derived from Confucianism and principles of response process informed open, axial and selective coding.

RESULTS

Two themes and four subthemes were constructed. Japanese residents emphasised the awareness of their relationship with their clinical teachers (1). This awareness was fuelled by their sense of hierarchy (1a) and being part of the collective society (1b). Residents described how the meaning of evaluation (2) was coloured by their perceived role as senior (2a) and their experienced responsibility for future generations (2b).

CONCLUSIONS

Japanese residents' sense-making while evaluating their clinical teachers appears to be situated and affected by Japanese cultural values. These findings contribute to a better understanding of a culture's influence on residents' sense-making of evaluation instruments and the validity argument of evaluation.

Validity and reliability of the Greek version of the Pain Beliefs and Perceptions Inventory for patients with chronic pain

Background/aims

The Pain Beliefs and Perceptions Inventory has been used widely to evaluate pain perceptions and beliefs of patients with chronic pain. This is a cross-cultural adaptation of the instrument into Greek. The purpose of the study was to investigate: the face and content validity; the factor structure; the concurrent validity; the discriminant validity; and the internal consistency and the test–retest reliability of the Pain Beliefs and Perceptions Inventory for people with chronic pain.

Methods

A total of 174 patients with chronic low back pain, neck pain and/or pain of limbs lasting for at least 6 months and with a physiotherapy referral participated in the study. Exploratory factor analysis by a principal axis factoring with direct oblimin rotation was chosen to examine the factor structure of the Pain Beliefs and Perceptions Inventory. The concurrent validity was assessed using correlations by Spearman's rho correlation coefficient among the Pain Beliefs and Perceptions Inventory and the Short Form-McGill Pain Questionnaire, the Present Pain Index, the visual analogue scale and the Brief Pain Inventory. The discriminant validity was examined by Spearman's rho correlation coefficient among the Pain Beliefs and Perceptions Inventory and Social Desirability Scale. The reliability of the instrument was examined using Cronbach's α internal consistency coefficients and intraclass correlation coefficients.

Results

Exploratory factor analysis confirmed the four factor structure of the Pain Beliefs and Perceptions Inventory that explained 66.353% of the total variance. Concurrent validity was determined through examination of correlations between the Pain Beliefs and Perceptions Inventory and other validated constructs (eg Short Form-McGill Pain Questionnaire Affective factor with Total Pain Beliefs and Perceptions Inventory Spearman's rho=0.257, P<0.001, Brief Pain Inventory General Activity and Pain Beliefs and Perceptions Inventory Constancy factor Spearman's rho=0.522, P<0.05, Brief Pain Inventory relations with other people and Pain Beliefs and Perceptions Inventory Constancy factor Spearman's rho=0.512, P<0.05). The discriminant validity of the Pain Beliefs and Perceptions Inventory was confirmed by examining correlations between the Pain Beliefs and Perceptions Inventory with the Marlowe–Crowne Social Desirability Scale. The questionnaire was internally consistent (α=0.89–0.96) and its stability was good (intraclass correlation coefficients=0.73–0.82).

Conclusions

The Greek translation of the Pain Beliefs and Perceptions Inventory is a reliable and valid instrument to evaluate Greek patients with chronic pain.

Effective Clinical Teaching Practice and Associated Factors Among Midwifery Educators in Public Universities of Ethiopia: Institution-Based Cross-Sectional Study

BACKGROUND

Although effective clinical teaching realistically improves learners to be competent, in Ethiopia only 31.6% of midwifery graduates have passed the national proficiency standard for graduation. Likewise, research evidence is lacking on effective clinical teaching practice of midwifery educators in Ethiopia.

OBJECTIVE

This study aimed to assess effective clinical teaching practice and associated factors among midwifery educators in public universities of Ethiopia.

METHODS

Institution-based cross-sectional study was conducted in April 2019 on 424 midwifery educators selected by simple random sampling technique. Data were collected through email by using a semi-structured, pretested, self-administered questionnaire, entered into Epi-data version 4.2, and analyzed by SPSS version 23. Bivariate and multivariable logistic regression analyses were done to test the association. The odds ratio at 95% confidence interval (CI) and P-value <0.05 were used to ascertain statistical significance.

RESULTS

Only 178 (47.3%) of participants had effective clinical teaching practice. Clinical teaching experience (AOR= 4.72; 95% CI=1.97, 11.29), training on higher diploma in teaching profession (AOR=2.78; 95% CI=1.61, 4.82), clinical teaching workshop (AOR=4; 95% CI=1.98, 8.05), students per clinical unit (AOR= 3.71; 95% CI= 1.66, 8.26), giving objectives of clinical learning for students (AOR= 4.74; 95% CI=2.43, 9.25), using performance-based assessment tool (AOR =1.82; 95% CI=1.04, 3.16) and having good interest in clinical teaching (AOR=8.63; 95% CI=2.91, 25.56) were factors positively associated with effective clinical teaching practice.

CONCLUSION

Less than half of midwifery educators of Ethiopian Public Universities had effective clinical teaching practice. The educators' clinical teaching experience, training, and adherence to clinical teaching standards were identified to affect effective clinical teaching positively. Therefore, educators should adhere to the standards for effective clinical teaching and receive training on effective teaching.

Lists of potential diagnoses that final-year medical students need to consider: a modified Delphi study

BACKGROUND

Contrastive learning is known to be effective in teaching medical students how to generate diagnostic hypotheses in clinical reasoning. However, there is no international consensus on lists of diagnostic considerations across different medical disciplines regarding the common signs and symptoms that should be learned as part of the undergraduate medical curriculum. In Japan, the national model core curriculum for undergraduate medical education was revised in 2016, and lists of potential diagnoses for 37 common signs, symptoms, and pathophysiology were introduced into the curriculum. This study aimed to validate the list of items based on expert consensus.

METHODS

The authors used a modified Delphi method to develop consensus among a panel of 23 expert physician-teachers in clinical reasoning from across Japan. The panel evaluated the items on a 5-point Likert scale, based on whether a disease should be hypothesized by final-year medical students considering given signs, symptoms, or pathophysiology. They also added other diseases that should be hypothesized. A positive consensus was defined as both a 75% rate of panel agreement and a mean of 4 or higher with a standard deviation of less than 1 on the 5-point scale. The study was conducted between September 2017 and March 2018.

RESULTS

This modified Delphi study identified 275 basic and 67 essential other than basic items corresponding to the potential diagnoses for 37 common signs, symptoms, and pathophysiology that Japanese medical students should master before graduation.

CONCLUSIONS

The lists developed in the study can be useful for teaching and learning how to generate initial hypotheses by encouraging students' contrastive learning. Although they were focused on the Japanese educational context, the lists and process of validation are generalizable to other countries for building national consensus on the content of medical education curricula.

Psychometric Properties of the Persian Questionnaire for Evaluation of Clinical Teaching at Outpatient Settings

PURPOSE

To assess the validity and reliability of the Persian Outpatient Clinical Teaching Evaluation (OCTE) questionnaire, in order to evaluate clinical teaching for undergraduate medical students.

METHODS

This was a methodological study conducted in Tabriz University of Medical Sciences in 2018. To assess the validity of the questionnaire, ten academic staff who had received master's degrees in medical education were chosen by convenience sampling. The first draft of the Persian questionnaire was derived from the Ministry of Health's clinical teaching standards booklet. The questionnaire consisted of just 15 obligatory items in the first draft of the questionnaire. Subsequently, the questionnaire was modified to be used in teaching outpatient clinics. Content validity indices were calculated. Subsequently, the modified questionnaires were given to 92 academic staff in Imam-Reza Hospital's outpatient clinics. We applied principle component analysis (PCA) and Varimax rotation for exploratory factor analysis (EFA). In order to confirm the EFA, confirmatory factor analysis (CFA) was used. Cronbach's alpha method, the intra-class correlation coefficient (ICC) and test-retest reliability were used to assess the reliability.

RESULTS

The overall content validity ratio (CVR), content validity index (CVI), and impact score (IS) were 0.78, 0.79, and 3.26, respectively. Out of 92 academic staff, 85 participated. The Keizer-Meyer-Olkin (KMO) measure of sampling adequacy for data was 0.726 (p-value=0.0001). After the EFA, the 10-item questionnaire loaded on three factors. The internal consistency of the questionnaire was established by Cronbach's alpha of 0.89 and the ICC was 0.94. Furthermore, there was a strong correlation with a 2-week retest interval.

CONCLUSION

The study demonstrated that the Persian clinical teaching questionnaire had optimal psychometric properties. Thus, application of this questionnaire at outpatient settings can be helpful.

Measuring social interdependence in collaborative learning: instrument development and validation

BACKGROUND

In health professions education, several collaborative learning approaches have been used. As collaborative learning has a theoretical background of social interdependence theory, a theory informed and valid instrument to measure social interdependence is required to evaluate and compare several learning approaches. The aim of this study was to develop an instrument (the SOcial interdependence in Collaborative learning Scale; SOCS) to measure students' perceived social interdependence in collaborative learning and validate it.

METHODS

We conducted a modified Delphi procedure among stakeholders to develop the content validity of the instrument. To establish construct validity, we performed a confirmatory factor analysis, and we estimated reliability.

RESULTS

Two rounds of Delphi were conducted to develop the instrument. Confirmatory factor analysis yielded a three-factor model with 15 items, which provided an excellent fit with CMIN/df = 1.838, GFI = 0.924, CFI = 0.951, RMSEA = 0.061, and PCLOSE = 0.121. Alpha-coefficients for all factors indicated high internal consistency of all the factors.

CONCLUSION

This study describes the development and construct validation of the SOCS for measuring social interdependence in collaborative learning. This instrument will provide teachers and schools with feedback about their classroom environment.

Attributes of Effective Clinical Teachers in Dental Hygiene Education

Many health professions have identified characteristics of an effective clinical instructor and used this information to create tools to measure clinical teaching effectiveness (CTE) in their disciplines. However, minimal research has been conducted to explore the attributes of an effective clinical instructor in dental hygiene education. The aim of this study was to identify the attributes of dental hygiene clinical instructors associated with effective teaching. This study used a Delphi research design to iteratively survey an expert panel of 14 dental hygiene instructors in dental hygiene programs from 13 U.S. states in 2019. A cumulative overall response rate of 89.3% was achieved after conducting three survey rounds. The results showed statistically significant consensus on 91 CTE characteristics. Participants agreed that effective clinical instructors should possess the ability to clearly communicate, foster interpersonal relationships, encourage a climate of mutual respect, and act as good role models who are emotionally intelligent, caring, trustworthy, fair, honest, and supportive. There was also agreement that effective clinical instructors were those who created a safe learning environment centered on collaboration, motivation, guidance, and positive corrective feedback. These results were similar to those identified in other health professions. However, there is a need for dental hygiene education to develop its own CTE assessment tool for use in professional development.

Evaluating the Reliability and Validity of the Maastricht Clinical Teaching Questionnaire in Bahrain

Objectives

The Maastricht Clinical Teaching Questionnaire (MCTQ) was developed to evaluate clinical teachers’ supervisory skills during undergraduate clinical rotations. Evidence exists supporting the reliability and validity of this questionnaire. Our study sought to examine the reliability and validity of the MCTQ in a Middle Eastern context.

Methods

Between 2016 and 2017, we evaluated a total of 549 medical students in their final year who were undergoing clinical rotations using the MCTQ. The construct validity was assessed using exploratory and confirmatory factor analysis. Internal consistency reliability was measured using Cronbach’s alpha. Factor scores were compared with other outcomes to explore the relationship with other relevant variables.

Results

A four-factor model demonstrated an adequate fit with the data. The findings showed good internal consistency reliability. The following results were obtained for the four-factor model: chi-square divided by degrees of freedom was 5.026, and the comparative index, goodness of fit index, normalized fit index, and non-normalized fit index were all above 0.800 (0.955, 0.858, 0.950, and 0.952, respectively). The standardized root mean square residual was 0.016, and the root mean square error of approximation score was 0.086. Acceptable reliability was achieved with 10 evaluations per teacher. We observed a strong correlation between factors and overall judgment.

Conclusions

Our study suggests that the MCTQ is a valid and reliable instrument to evaluate teachers’ performance during clinical rotations in Bahrain.

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review

PURPOSE

To collect and examine-using an argument-based validity approach-validity evidence of questionnaire-based tools used to assess physicians' clinical, teaching, and research performance.

METHOD

In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians' professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians' clinical, teaching, and research performance. Using Kane's validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications.

RESULTS

They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane's validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing.

CONCLUSIONS

Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians' clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

Measuring assessment standards in undergraduate medical programs: Development and validation of AIM tool

Objective:

To develop a tool to evaluate faculty perceptions of assessment quality in an undergraduate medical program.

Methods:

The Assessment Implementation Measure (AIM) tool was developed by a mixed method approach. A preliminary questionnaire developed through literature review was submitted to a panel of 10 medical education experts for a three-round ‘Modified Delphi technique'. Panel agreement of > 75% was considered the criterion for inclusion of items in the questionnaire. Cognitive pre-testing of five faculty members was conducted. Pilot study was done with 30 randomly selected faculty members. Content validity index (CVI) was calculated for individual items (I-CVI) and composite scale (S-CVI). Cronbach's alpha was calculated to determine the internal consistency reliability of the tool.

Results:

The final AIM tool had 30 items after the Delphi process. S-CVI was 0.98 with the S-CVI/Avg method and 0.86 by S-CVI/UA method, suggesting good content validity. Cut-off value of < 0.9 I-CVI was taken as criterion for item deletion. Cognitive pre-testing revealed good item interpretation. Cronbach's alpha calculated for the AIM was 0.9, whereas Cronbach's alpha for the four domains ranged from 0.67 to 0.80.

Conclusions:

‘AIM' is a relevant and useful instrument with good content validity and reliability of results, and may be used to evaluate the teachers´ perceptions about assessment quality.

What do Japanese residents learn from treating dying patients? The implications for training in end-of-life care

BACKGROUND

How medical residents' experiences with care for dying patients affect their emotional well-being, their learning outcomes, and the formation of their professional identities is not fully understood. We examine residents' emotional states and learning occurring during the provision of care to dying patients and specifically discuss the impact of providing end-of-life (EOL) care on professional identity formation.

METHODS

Semi-structured interviews were conducted with 13 residents who had graduated in the last 3 to 5 years. Thematic theoretical analysis was applied, and key themes were developed based on Kolb's experiential learning cycle.

RESULTS

Eight key themes emerged from the analysis. The residents experienced dilemmas in confronting the reality of medical uncertainty as well as a disruption of emotional state and self-efficacy. Although the residents felt a sense of helplessness and guilt, they were able to reflect on strategies for handling medical care that focused on patients and that required a truly sincere attitude. They also contemplated the importance of palliative care and communication with patients, patients' family members and medical staff. Building on these experiences, the residents rebuilt a sense of awareness that allowed them to directly engage with the type of medical care that they are likely to be called upon to perform in the future as the population continues to age.

CONCLUSIONS

This study revealed Japanese residents' perceptions, emotions and learning processes in caring for dying patients by applying Kolb's experiential learning theory. The findings of this study may illuminate valuable pieces of knowledge for future education in EOL care.

Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context

AIM

Many instruments for evaluating clinical teaching have been developed, albeit most in Western countries. This study aims to develop a validated cultural and local context sensitive instrument for clinical teachers in an East Asian setting (Japan), Japanese Clinical Teacher Evaluation Sheet (JaCTES).

METHODS

A multicenter, cross-sectional evaluation study was conducted. We collected a total of 1368 questionnaires on 304 clinical teachers, completed by residents in 16 teaching hospitals. The construct validity was examined by conducting a factor analysis and using structural equation modeling (SEM). We also assessed the reliability using generalizability analysis and decision study.

RESULTS

Exploratory factor analysis resulted in three-factor (role model, teaching activities, and accessibility) model including 18 items. Confirmatory factor analysis was performed, using SEM. The comparative fit index was 0.931 and the root mean square error of approximation was 0.087, meaning an acceptable goodness of fit for this model. To obtain a reliable dependability-coefficient of at least 0.70 or higher, 5-8 resident responses are necessary.

DISCUSSION AND CONCLUSION

JaCTES is the first reported instrument with validity evidence of content and internal structure and high feasibility in Japan, an East Asian setting. Medical educators should be aware of the local context and cultural aspects in evaluating clinical teachers.

What Do You Think You Are Measuring? A Mixed-Methods Procedure for Assessing the Content Validity of Test Items and Theory-Based Scaling

The valid measurement of latent constructs is crucial for psychological research. Here, we present a mixed-methods procedure for improving the precision of construct definitions, determining the content validity of items, evaluating the representativeness of items for the target construct, generating test items, and analyzing items on a theoretical basis. To illustrate the mixed-methods content-scaling-structure (CSS) procedure, we analyze the Adult Self-Transcendence Inventory, a self-report measure of wisdom (ASTI, Levenson et al., 2005). A content-validity analysis of the ASTI items was used as the basis of psychometric analyses using multidimensional item response models (N = 1215). We found that the new procedure produced important suggestions concerning five subdimensions of the ASTI that were not identifiable using exploratory methods. The study shows that the application of the suggested procedure leads to a deeper understanding of latent constructs. It also demonstrates the advantages of theory-based item analysis.