Development of script concordance test for assessment of clinical reasoning in nursing: Lessons learned regarding construct validity

Background

The script concordance test (SCT) has been shown to be an effective tool to assess the clinical reasoning skills of nursing students. Various nursing studies have demonstrated the construct validity of this test. However, studies on the barriers that may impede construct validity during the development process are limited.

Objective

This evaluation describes the barriers to the development of SCT for Bachelor's nursing students and the lessons learned regarding construct validity.

Methods

We conducted a descriptive evaluation of the SCT development and a validation process was performed. The evaluation was based on written comments during the assessment (N = 327), a Student's Perspective Questionnaire (N = 100), and student feedback during three live review sessions (N = 27).

Results

Despite consideration of the guidelines during SCT development, we encountered three main barriers that may impede construct validity. We undertook the necessary efforts to recruit an appropriate expert panel. We overestimated the experts' and students' understanding of the SCT methodology. Additionally, four potential causes of invalid item construction were identified. These possible causes were 'questionable intervention, hypothesis, or investigation', 'blurred data in new information', 'regression to the middle', and 'misinterpretation of the midpoint'.

Conclusion

The three lessons learned are as follows: 1) The recruitment of an appropriate expert panel must not be underestimated. Besides clinical expertise, experts need training in SCT methodology, including awareness of possible pitfalls; 2) SCT training is a prerequisite for SCT as an assessment; and 3) student feedback may offer a deeper understanding of potential hidden script errors and causes for misinterpretation of SCT. Further studies are necessary to identify additional causes which may impede the construct validity of SCT in nursing education.

Psychometric properties of the persian version of the nursing clinical reasoning scale

AIM

The aim of this study was to translate the Nursing Clinical Reasoning Scale (NCRS) into Persian and evaluate its psychometric properties.

DESIGN

This study was a methodological and cross-sectional study.

METHODS

This methodological study was conducted in 2020 in a teaching hospital. After obtaining necessary permission from its developers, NCRS was translated into Persian through the method proposed by the World Health Organization. Then, its face, content, and construct validity and reliability were assessed. For construct validity assessment through exploratory and confirmatory factor analyses, 300 nurses (two 150-nurse samples), who had randomly been selected, completed the instrument. Reliability also assessed through the internal consistency and the stability methods. Data were analysed using the SPSS (v. 20.0) and the AMOS (v. 5.0) software.

RESULTS

The content validity indices of NCRS and its items were 0.97 and more than 0.79, respectively. Exploratory factor analysis revealed an assessment and confirmation factor and an implementation and reflection factor for the scale which together explained 57.30% of the total variance. Confirmatory factor analysis also confirmed this two-factor structure (χ2 /df = 2.11, NNFI = 0.952, RMSEA = 0.053, CFI = 0.91, GFI = 0.94, IFI = 0.95, and NFI = 0.96). The Cronbach's alpha and the intraclass correlation coefficient values of the scale were 0.96 and 0.94, respectively.

PUBLIC CONTRIBUTION

The Persian NCRS can help nursing policy makers and mentors identify the need for developing nurses' and nursing students' CR skills and implement need-based educational courses to improve these skills. Moreover, it helps determine whether the educational programmes are effective in improving nurses' CR skills and clinical competence.

Scoring Methods in Script Concordance Tests: An Exploratory Psychometric Study

BACKGROUND

Despite the increasingly popular role of script concordance test (SCT) scoring methods in the evaluation of clinical reasoning, studies examining these methods in nursing are relatively scarce. This study explored the psychometric properties of five SCT scoring methods.

METHOD

An SCT was administered to 12 experts and 43 learners. Scores were calculated using five methods and descriptive statistics. Differences in scores were assessed with the Mann-Whitney U test, and Spearman correlation coefficients were calculated for the different methods.

RESULTS

The median scores of both experts and learners differed substantially according to the scoring method used. Learners' scores were statistically different from experts' scores (p < .01) for each method. Spearman coefficients (range, 0.44 to 0.95) were positive for the different methods.

CONCLUSION

Further research is needed to refine the influence of SCT scoring methods for use in certifying assessment of clinical reasoning in nursing. [J Nurs Educ. 2023;62(10):549-555.].

Assessment of Clinical Reasoning While Attending Critical Care Postsimulation Reflective Learning Conversation: A Scoping Review

BACKGROUND

The critical care environment is characterized with a high level of workload, complexity, and risk of committing practice mistakes. To avoid clinical errors, health care professionals should be competent with effective clinical reasoning skills. To develop effective clinical reasoning skills, health care professionals should get the chance to practice and be exposed to different patient experiences. To minimize safety risks to patients and health care professionals, clinical reasoning with a focus on reflective learning conversation opportunities can be practiced in simulated settings.

OBJECTIVES

To explore the most valid and reliable tools to assess clinical reasoning while attending adult critical care-related simulation-based courses in which reflective learning conversations are used.

METHODS

A scoping review was conducted following Joanna Briggs Institute and Preferred Reporting Items for Systematic Reviews Extension for Scoping Reviews. Eight electronic databases were searched, and full-text review was completed for 26 articles.

RESULTS

The search resulted in no studies conducted to measure clinical reasoning while attending adult critical care-related, simulation-based courses in which the reflective learning conversation method was embedded.

DISCUSSION

This highlights the need to evaluate current available clinical reasoning tools or develop new tools within the context of adult critical care simulation where reflective learning forms a key part of the simulation procedures.

Clinical reasoning in undergraduate paramedicine: utilisation of a script concordance test

INTRODUCTION

Clinical reasoning is a complex cognitive and metacognitive process paramount to patient care in paramedic practice. While universally recognised as an essential component of practice, clinical reasoning has been historically difficult to assess in health care professions. Is the Script Concordance Test (SCT) an achievable and reliable option to test clinical reasoning in undergraduate paramedic students?

METHODS

This was a single institution observational cohort study designed to use the SCT to measure clinical reasoning in paramedic students. Clinical vignettes were constructed across a range of concepts with varying shades of clinical ambiguity. A reference panel mean scores of the test were compared to that of students. Test responses were graded with the aggregate scoring method with scores awarded for both partially and fully correct responses.

RESULTS

Eighty-three student paramedic participants (mean age: 21.8 (3.5) years, 54 (65%) female, 27 (33%) male and 2 (2%) non-binary) completed the SCT. The difference between the reference group mean score of 80 (5) and student mean of score of 65.6 (8.4) was statistically significant (p < 0.001).

DISCUSSION

Clinical reasoning skills are not easily acquired as they are a culmination of education, experience and the ability to apply this in the context to a specific patient. The SCT has shown to be reliable and effective in measuring clinical reasoning in undergraduate paramedics as it has in other health professions such as nursing and medicine. More investigation is required to establish effective pedogeological techniques to optimise clinical reasoning in student and novice paramedics who are devoid of experience.

Development and validation of a script concordance test to assess biosciences clinical reasoning skills: A cross-sectional study of 1st year undergraduate nursing students

BACKGROUND

Developing evaluative measures that assess clinical reasoning remains a major challenge for nursing education. A thorough understanding of biosciences underpins much of nursing practice and is essential to allow nurses to reason effectively. A gap in clinical reasoning can lead to unintended harm. The Script Concordance test holds promise as a measure of clinical reasoning in the context of uncertainty, situations common in nursing practice. The aim of this study is to develop and validate a test for first year undergraduate nursing students that will evaluate how bioscience knowledge is used to clinically reason.

METHODS

An international team, teaching biosciences to undergraduate nurses constructed a test integrating common clinical cases with a series of related test items: diagnostic, investigative and treatment. An expert panel (n = 10) took the test and commented on authenticity/ambiguities/omissions etc. This step is crucial for validity and for scoring of the student test. The test was administered to 47 first year undergraduate nursing students from the author sites. Students rated educational aspects of the tool both quantitatively and qualitatively. Statistical and content analyses inform the findings.

FINDINGS

Results indicate that the test is reliable and valid, differentiating between experts and students. Students demonstrated an ability to identify relevant data, link this to their bioscience content and predict outcomes (mean score = 50.78 ± 8.89). However, they lacked confidence in their answers when the scenarios appeared incomplete to them.

CONCLUSION

Nursing practice is dependent on a thorough understanding of biosciences and the ability to clinically reason. Script concordance tests can be used to promote both competencies. This method of evaluation goes further than probing factual knowledge. It also explores capacities of data interpretation, critical analysis, and clinical reasoning. Evaluating bioscience knowledge and real-world situations encountered in practice is a unique strength of this test.

Evaluating the Clinical Reasoning of Student Health Professionals in Placement and Simulation Settings: A Systematic Review

(1) Background: Clinical reasoning is essential to the effective practice of autonomous health professionals and is, therefore, an essential capability to develop as students. This review aimed to systematically identify the tools available to health professional educators to evaluate students' attainment of clinical reasoning capabilities in clinical placement and simulation settings. (2) Methods: A systemic review of seven databases was undertaken. Peer-reviewed, English-language publications reporting studies that developed or tested relevant tools were included. Searches included multiple terms related to clinical reasoning and health disciplines. Data regarding each tool's conceptual basis and evaluated constructs were systematically extracted and analysed. (3) Results: Most of the 61 included papers evaluated students in medical and nursing disciplines, and over half reported on the Script Concordance Test or Lasater Clinical Judgement Rubric. A number of conceptual frameworks were referenced, though many papers did not reference any framework. (4) Conclusions: Overall, key outcomes highlighted an emphasis on diagnostic reasoning, as opposed to management reasoning. Tools were predominantly aligned with individual health disciplines and with limited cross-referencing within the field. Future research into clinical reasoning evaluation tools should build on and refer to existing approaches and consider contributions across professional disciplinary divides.

Clinical reasoning in nursing students: A concept analysis

AIM

The aim of this analysis is to clarify the concept of clinical reasoning in nursing students.

BACKGROUND

Sound clinical reasoning is the most important skill required in professional nursing and understanding of this concept is emphasized as a basis for clinical reasoning development in nursing education curricula.

DESIGN

Rodgers' concept analysis method was used to achieve a clear and understandable definition.

DATA SOURCE

Resources published from 2000 to 2020 were identified via electronic databases.

REVIEW METHODS

A review of the literature was completed, and the data were analyzed to identify the Surrogate and related terms, attributes, antecedents and consequences of the concept.

RESULTS

This concept is a holistic and recursive cognitive process that has a dynamic and flexible nature to perceive the patient's condition, select the best practice to respond to the situation, and learn from the situation. Clinical reasoning in nursing students emerges despite professional standards; discipline-specific knowledge, cognitive perception, critical thinking, learning experiences, and intuitive ability, and the requirements of the professional system affect its establishment in the nursing discipline. Clinical reasoning is the cognitive process underlying clinical judgment, appropriate decision making, improvement of nursing quality, metacognitive awareness, and professional competence in nursing, whose achievement, generally, paves the way for nursing professionalization and development that are important steps toward independence in the nursing profession.

CONCLUSIONS

The present concept analysis clarifies the concept of clinical reasoning as a complex thinking process that should be considered as a fundamental thinking skill in nursing program.

The validity and reliability of clinical judgement and decision-making skills assessment in nursing: A systematic literature review

OBJECTIVES

To appraise the validity and reliability of approaches to assessing the clinical decision-making skills of nurses, and use findings to inform the assessment of students as they transition to newly qualified nurses.

DESIGN

The preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines were used to conduct the review.

DATA SOURCES

Medline, CINAHL and the British Nursing Index were searched from inception to November 2019.

REVIEW METHODS

Studies were grouped according to their assessment approach following a competency framework with findings presented as a narrative synthesis.

RESULTS

38 articles were included in the review which assessed clinical decision-making in a variety of settings; clinical practice, simulation, written examinations and self-assessment. Multi-level rubric and checklist approaches demonstrated good validity and reliability in practice and simulation settings, and the former was effective at differentiating between students at different stages of their training. Written, case study examinations were also effective at assessing clinical decision-making, although an optimum structure for their presentation was not possible to discern. Students tended to score themselves more highly than faculty staff when undertaking rubric-based self-assessments.

CONCLUSIONS

Findings suggest that the best approach to assess clinical decision-making for final year students is to use several low-stakes, snap-shot summative assessments in practice environments, which are marked using a multi-level observational rubric. To assure reliability, it is recommended that a small team of expert practice assessors undergo regular training and peer review, have protected time to complete their assessor role and are appropriately supported.

Script Concordance and the Think-Aloud Approach Through the Lens of Cognitive Apprenticeship Theory

BACKGROUND

Script concordance (SC) and the think-aloud (TA) approach informed by cognitive apprenticeship theory can promote clinical reasoning.

PURPOSE

The purpose of this study was to determine whether SC with the TA approach based on the tenets of cognitive apprenticeship theory facilitated clinical reasoning in senior-level prelicensure nursing students in a baccalaureate program.

METHODS

This study was executed in a didactic pediatric nursing course using a quantitative, nonexperimental design. Creation of the SC/TA learning activity was grounded in cognitive apprenticeship theory and based on the SC testing/TA testing method.

RESULTS

Repeated-measures analysis of variance revealed a difference (P = .001 and .003) between earlier and later SC scores for the fall 2018 cohort.

CONCLUSION

The SC/TA teaching/learning activity informed by cognitive apprenticeship theory offers a leveled approach to foster clinical reasoning in prelicensure nursing education.

L’apprentissage du raisonnement clinique infirmier dans le cadre d’un dispositif éducatif numérique basé sur la concordance de scripts

Contexte : Un des défis actuels des responsables de la formation des futurs infirmiers est de concevoir des stratégies éducatives pour favoriser le développement optimal du raisonnement clinique infirmier (RCI).But : L’étude visait à développer, mettre à l’essai et évaluer un dispositif éducatif numérique basé sur la concordance de scripts auprès d’étudiants en sciences infirmières. Méthodes : Un devis de recherche-développement a permis de soutenir les étapes de réalisation du dispositif éducatif. Une validation de contenu a été intégrée au devis à l’aide d’une technique Delphi modifiée. Les choix de réponses des étudiants et des experts aux questions du dispositif ont été compilés. Un questionnaire en ligne et des entrevues de groupe focalisées ont été utilisés auprès d’étudiants afin d’évaluer l’acceptabilité et l’utilisabilité du dispositif éducatif. Résultats : Cinq experts-collaborateurs ont validé le contenu du dispositif éducatif. La version finale comptait 81 questions liées à 22 vignettes. Douze experts-panélistes et quarante-cinq étudiants ont répondu aux questions du dispositif éducatif. Les indices alpha (α) de Cronbach ont oscillé entre 0,87 et 0,90. Globalement, les étudiants ont apprécié l’utilisation du dispositif éducatif, dont la rétroaction variée et formative d’experts à des situations apparentées à la vie professionnelle. Conclusion : Le dispositif éducatif repose sur l’utilisation d’un test de concordance de scripts numérisé auquel des rétroactions d’experts sont intégrées à des fins d’apprentissage du RCI. Essentiel, l’engagement des experts-panélistes inscrit la conception d’un tel dispositif éducatif dans une démarche de co-construction et de validation du matériel pédagogique.

Examining response process validity of script concordance testing: a think-aloud approach

OBJECTIVES

This study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.

METHODS

A set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning).

RESULTS

Two hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%).

CONCLUSIONS

The findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.

An Integrative Review of Clinical Reasoning Teaching Strategies and Outcome Evaluation in Nursing Education

BACKGROUND

This study explored teaching strategies used to promote clinical reasoning in nursing education and identify outcomes used to evaluate effectiveness.

METHOD

The integrative review framework by Whittemore and Knafl (2005) was used in this study.

RESULTS

Strategies included simulation (n = 10); active learning strategies such as case studies, reflection, journaling, and collaborative learning (n = 13); teaching strategies such as think aloud or case studies with simulation (n = 12); and experiential or clinical experiences (n = 2). Although most studies used exams, worksheets (n = 19), or observations by clinical instructors (n = 2) to measure effectiveness, some studies (n = 8) also used surveys to measure student satisfaction, engagement, or perception; three studies measured the student's ability to use the instrument.

CONCLUSION

Improving clinical thinking requires development of innovative, effective teaching strategies. Instruments that can accurately evaluate teaching and learning strategies are needed to advance this educational initiative and improve quality of care.

Clinical Reasoning: A State of the Science Report

Clinical reasoning is the cognitive process that nurses use to gather and incorporate information into a larger bank of personal knowledge. This incorporated information guides therapeutic actions, and helps determine client care. Since the process guides therapeutic actions regarding client care, failure to use the process effectively leads to poor clinical decision-making, inappropriate actions, or inaction. Because of the criticality of this process, this paper presents an analysis of the literature that reveals the current state of the science of clinical reasoning, identifies gaps in knowledge, and elucidates areas for future research. A systematic review of the databases the Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Educational Resources Information Center (ERIC), PsychInfo, the Education Full Text (H.W. Wilson), and PubMed revealed 873 articles on the topic of clinical reasoning. Quality appraisal narrowed the field to 27 pieces of literature. Appendix A gives the State of the Science Coding Sheet used to identify the selections used in this research. Appendix B contains a summary of this literature. Although analysis of this literature shows that three theories exist on how to utilize most effectively the clinical reasoning process presently; a clear consistent definition is lacking. Additional research should focus on closing gaps that exist in defining the process, understanding the process, establishing linkages to non-clinical reasoning processes, and developing measures to both develop and accurately measure clinical reasoning.

Criterion scores, construct validity and reliability of a web-based instrument to assess physiotherapists' clinical reasoning focused on behaviour change: 'Reasoning 4 Change'

Background and aim: 'Reasoning 4 Change' (R4C) is a newly developed instrument, including four domains (D1-D4), to assess clinical practitioners' and students' clinical reasoning with a focus on clients' behaviour change in a physiotherapy context. To establish its use in education and research, its psychometric properties needed to be evaluated. The aim of the study was to generate criterion scores and evaluate the reliability and construct validity of a web-based version of the R4C instrument. Methods: Fourteen physiotherapy experts and 39 final-year physiotherapy students completed the R4C instrument and the Pain Attitudes and Beliefs Scale for Physiotherapists (PABS-PT). Twelve experts and 17 students completed the R4C instrument on a second occasion. The R4C instrument was evaluated with regard to: internal consistency (five subscales of D1); test-retest reliability (D1-D4); inter-rater reliability (D2-D4); and construct validity in terms of convergent validity (D1.4, D2, D4). Criterion scores were generated based on the experts' responses to identify the scores of qualified practitioners' clinical reasoning abilities. Results: For the expert and student samples, the analyses demonstrated satisfactory internal consistency (α range: 0.67-0.91), satisfactory test-retest reliability (ICC range: 0.46-0.94) except for D3 for the experts and D4 for the students. The inter-rater reliability demonstrated excellent agreement within the expert group (ICC range: 0.94-1.0). The correlations between the R4C instrument and PABS-PT (r range: 0.06-0.76) supported acceptable construct validity. Conclusions: The web-based R4C instrument shows satisfactory psychometric properties and could be useful in education and research. The use of the instrument may contribute to a deeper understanding of physiotherapists' and students' clinical reasoning, valuable for curriculum development and improvements of competencies in clinical reasoning related to clients' behavioural change.

Examining the effects of gaming and guessing on script concordance test scores

INTRODUCTION

In a script concordance test (SCT), examinees are asked to judge the effect of a new piece of clinical information on a proposed hypothesis. Answers are collected using a Likert-type scale (ranging from -2 to +2, with '0' indicating no effect), and compared with those of a reference panel of 'experts'. It has been argued, however, that SCT may be susceptible to the influences of gaming and guesswork. This study aims to address some of the mounting concern over the response process validity of SCT scores.

METHOD

Using published datasets from three independent SCTs, we investigated examinee response patterns, and computed the score a hypothetical examinee would obtain on each of the tests if he 1) guessed random answers and 2) deliberately answered '0' on all test items.

RESULTS

A simulated random guessing strategy led to scores 2 SDs below mean scores of actual respondents (Z-scores -3.6 to -2.1). A simulated 'all-0' strategy led to scores at least 1 SD above those obtained by random guessing (Z-scores -2.2 to -0.7). In one dataset, stepwise exclusion of items with modal panel response '0' to fewer than 10% of the total number of test items yielded hypothetical scores 2 SDs below mean scores of actual respondents.

DISCUSSION

Random guessing was not an advantageous response strategy. An 'all-0' response strategy, however, demonstrated evidence of artificial score inflation. Our findings pose a significant threat to the SCT's validity argument. 'Testwiseness' is a potential hazard to all testing formats, and appropriate countermeasures must be established. We propose an approach that might be used to mitigate a potentially real and troubling phenomenon in script concordance testing. The impact of this approach on the content validity of SCTs merits further discussion.

Monitoring progression of clinical reasoning skills during health sciences education using the case method - a qualitative observational study

BACKGROUND

Outcome- or competency-based education is well established in medical and health sciences education. Curricula are based on courses where students develop their competences and assessment is also usually course-based. Clinical reasoning is an important competence, and the aim of this study was to monitor and describe students' progression in professional clinical reasoning skills during health sciences education using observations of group discussions following the case method.

METHODS

In this qualitative study students from three different health education programmes were observed while discussing clinical cases in a modified Harvard case method session. A rubric with four dimensions - problem-solving process, disciplinary knowledge, character of discussion and communication - was used as an observational tool to identify clinical reasoning. A deductive content analysis was performed.

RESULTS

The results revealed the students' transition over time from reasoning based strictly on theoretical knowledge to reasoning ability characterized by clinical considerations and experiences. Students who were approaching the end of their education immediately identified the most important problem and then focused on this in their discussion. Practice knowledge increased over time, which was seen as progression in the use of professional language, concepts, terms and the use of prior clinical experience. The character of the discussion evolved from theoretical considerations early in the education to clinical reasoning in later years. Communication within the groups was supportive and conducted with a professional tone.

CONCLUSIONS

Our observations revealed progression in several aspects of students' clinical reasoning skills on a group level in their discussions of clinical cases. We suggest that the case method can be a useful tool in assessing quality in health sciences education.