Degrees of freedom analysis in educational research and decision-making: leveraging qualitative data to promote excellence in bioinformatics training and education

The Assessment Evaluation Rubric: Promoting Learning and Learner-Centered Teaching through Assessment in Face-to-Face or Distanced Higher Education

It is common to create courses for the higher education context that accomplish content-driven teaching goals and then develop assessments (quizzes and exams) based on the target content. However, content-driven assessment can tend to support teaching- or teacher-centered instruction. Adult learning and educational psychology theories suggest that instead, assessment should be aligned with learning, not teaching, objectives. To support the alignment of assessments with instruction in higher education, the Assessment Evaluation Rubric (AER) was developed. The AER can be utilized to guide the development and evaluation/revision of assessments that are already used. The AER describes, or permits the evaluation of, four features of an assessment: its general alignment with learning goal(s), whether the assessment is intended to/effective as formative or summative, whether some systematic approach to cognitive complexity is reflected, and whether the assessment (instructions as well as results) itself is clearly interpretable. Each dimension (alignment, utility, complexity, and clarity) has four questions that can be rated as present/absent. Other rating methods can also be conceptualized for the AER’s 16 questions, depending on the user’s intent. Any instructor can use the AER to evaluate their own assessments and ensure that they—or new assessments in development—will promote learning and learner-centered teaching. As instructors shift from face-to-face toward virtual or hybrid teaching models, or as they shift online instruction (back) to face-to-face teaching, it creates an ideal opportunity to ensure that assessment is optimizing learning and is valid for instructional decision-making.

The Preparation of Stewards with the Mastery Rubric for Stewardship: Re-Envisioning the Formation of Scholars and Practitioners

A steward of the discipline was originally defined as “someone who will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application”. This construct was articulated to support and strengthen doctoral education. The purpose of this paper is to expand the construct of stewardship so that it can be applied to both scholars and non-academic practitioners, and can be initiated earlier than doctoral education. To accomplish and justify this, we describe a general developmental trajectory supporting cross-curriculum teaching for stewardship of a discipline as well as of a profession. We argue that the most important features of stewardship, comprising the public trust for the future of their discipline or profession, are obtainable by all practitioners, and are not limited to those who have completed doctoral training. The developmental trajectory is defined using the Mastery Rubric construct, which requires articulating the knowledge, skills, and abilities (KSAs) to be targeted with a curriculum; recognizable stages of performance of these KSAs; and performance level descriptors of each KSA at each stage. Concrete KSAs of stewardship that can be taught and practiced throughout the career (professional or scholarly) were derived directly from the original definition. We used the European guild structure’s stages of Novice, Apprentice, Journeyman, and Master for the trajectory, and through a consensus-based standard setting exercise, created performance level descriptors featuring development of Bloom’s taxonometric cognitive abilities (see Appendix A) for each KSA. Together, these create the Mastery Rubric for Stewardship (MR-S). The MR-S articulates how stewardly behavior can be cultivated and documented for individuals in any disciplinary curriculum, whether research-intensive (preparing “scholars”) or professional (preparing members of a profession or more generally for the work force). We qualitatively assess the validity of the MR-S by examining its applicability to, and concordance with professional practice standards in three diverse disciplinary examples: (1) History; (2) Statistics and Data Science; and (3) Neurosciences. These domains differ dramatically in terms of content and methodologies, but students in each discipline could either continue on to doctoral training and scholarship, or utilize doctoral or pre-doctoral training in other professions. The MR-S is highly aligned with the practice standards of all three of these domains, suggesting that stewardship can be meaningfully cultivated and utilized by those working in or outside of academia, supporting the initiation of stewardship prior to doctoral training and for all students, not only those who will earn PhDs or be scholars first and foremost. The MR-S can be used for curriculum development or revision in order to purposefully promote stewardship at all levels of higher education and beyond. The MR-S renders features of professional stewardship accessible to all practitioners, enabling formal and informal, as well as self-directed, development and refinement of a professional identity.

The Mastery Rubric for Bioinformatics: A tool to support design and evaluation of career-spanning education and training

As the life sciences have become more data intensive, the pressure to incorporate the requisite training into life-science education and training programs has increased. To facilitate curriculum development, various sets of (bio)informatics competencies have been articulated; however, these have proved difficult to implement in practice. Addressing this issue, we have created a curriculum-design and -evaluation tool to support the development of specific Knowledge, Skills and Abilities (KSAs) that reflect the scientific method and promote both bioinformatics practice and the achievement of competencies. Twelve KSAs were extracted via formal analysis, and stages along a developmental trajectory, from uninitiated student to independent practitioner, were identified. Demonstration of each KSA by a performer at each stage was initially described (Performance Level Descriptors, PLDs), evaluated, and revised at an international workshop. This work was subsequently extended and further refined to yield the Mastery Rubric for Bioinformatics (MR-Bi). The MR-Bi was validated by demonstrating alignment between the KSAs and competencies, and its consistency with principles of adult learning. The MR-Bi tool provides a formal framework to support curriculum building, training, and self-directed learning. It prioritizes the development of independence and scientific reasoning, and is structured to allow individuals (regardless of career stage, disciplinary background, or skill level) to locate themselves within the framework. The KSAs and their PLDs promote scientific problem formulation and problem solving, lending the MR-Bi durability and flexibility. With its explicit developmental trajectory, the tool can be used by developing or practicing scientists to direct their (and their team's) acquisition of new, or to deepen existing, bioinformatics KSAs. The MR-Bi is a tool that can contribute to the cultivation of a next generation of bioinformaticians who are able to design reproducible and rigorous research, and to critically analyze results from their own, and others', work.

A developmental trajectory supporting the evaluation and achievement of competencies: Articulating the Mastery Rubric for the nurse practitioner (MR-NP) program curriculum

BACKGROUND

Advanced practice registered nursing (APRN) competencies exist, but there is no structure supporting the operationalization of the competencies by APRN educators. The development of a Mastery Rubric (MR) for APRNs provides a developmental trajectory that supports educational institutions, educators, students, and APRNs. A MR describes the explicit knowledge, skills, and abilities as performed by the individual moving from novice (student) through graduation and into the APRN career.

METHOD

A curriculum development tool, the Mastery Rubric (MR), was created to structure the curriculum and career of the nurse practitioner (NP), the MR-NP. Cognitive task analysis (CTA) yielded the first of the three required elements for any MR: a list of knowledge, skills, and abilities (KSAs) to be established through the curriculum. The European guild structure and Bloom's taxonomy of cognitive behaviors provided the second element of the MR, the specific developmental stages that are relevant for the curriculum. The Body of Work method of standard setting was used to create the third required element of the MR, performance level descriptors (PLDs) for each KSA at each of these stages. Although the CTA was informed by the competencies, it was still necessary to formally assess the alignment of competencies with the resulting KSAs; this was achieved via Degrees of Freedom Analysis (DoFA). Validity evidence was obtained from this Analysis and from the DoFA of the KSAs' alignment with principles of andragogy, and with learning outcomes assessment criteria. These analyses are the first time the national competencies for the NP have been evaluated in this manner.

RESULTS

CTA of the 43 NP Competencies led to seven KSAs that support a developmental trajectory for instruction and documenting achievement towards independent performance on the competencies. The Competencies were objectively evaluable for the first time since their publication due to the psychometric validity attributes of the PLD-derived developmental trajectory. Three qualitatively distinct performance levels for the independent practitioner make the previously implicit developmental requirements of the competencies explicit for the first time.

DISCUSSION

The MR-NP provides the first articulated and observable developmental trajectory for the NP competencies, during and beyond the formal curriculum. A focus on psychometric validity was brought to bear on how learners would demonstrate their development, and ultimately their achievement, of the competencies. The MR-NP goes beyond the competencies with trajectories and PLDs that can engage both learner and instructor in this developmental process throughout the career.

A new pan-European Train-the-Trainer programme for bioinformatics: pilot results on feasibility, utility and sustainability of learning

Demand for training life scientists in bioinformatics methods, tools and resources and computational approaches is urgent and growing. To meet this demand, new trainers must be prepared with effective teaching practices for delivering short hands-on training sessions—a specific type of education that is not typically part of professional preparation of life scientists in many countries. A new Train-the-Trainer (TtT) programme was created by adapting existing models, using input from experienced trainers and experts in bioinformatics, and from educational and cognitive sciences. This programme was piloted across Europe from May 2016 to January 2017. Preparation included drafting the training materials, organizing sessions to pilot them and studying this paradigm for its potential to support the development and delivery of future bioinformatics training by participants. Seven pilot TtT sessions were carried out, and this manuscript describes the results of the pilot year. Lessons learned include (i) support is required for logistics, so that new instructors can focus on their teaching; (ii) institutions must provide incentives to include training opportunities for those who want/need to become new or better instructors; (iii) formal evaluation of the TtT materials is now a priority; (iv) a strategy is needed to recruit, train and certify new instructor trainers (faculty); and (v) future evaluations must assess utility. Additionally, defining a flexible but rigorous and reliable process of TtT ‘certification’ may incentivize participants and will be considered in future.