Admission interviews: questions of utility and cost in masters of physical therapy programs in the United States

The development and psychometric properties of a measure to assess the written submission of an admissions application

Introduction

Health care programs evaluate prospective applicants using cognitive and non-cognitive criteria. The aim of this research was to develop and evaluate the psychometric properties of a measure to evaluate the non-cognitive criteria of admissions applications.

Method

A Masters of Occupational Therapy Written Submission Measure (MOTWSM) was developed and evaluated over 3 phases, using applicants’ written statements, resumes, and reference letters. Participants included 50 students who completed an occupational therapy program for determination of internal consistency and test-retest reliability. Additionally, 195 written submissions selected from the applicants who were admitted, waitlisted, and not admitted to the program were evaluated to determine inter-rater reliability using a two-way ANOVA. Analysis of 195 submissions using a one-way ANOVA determined the measure’s discriminative validity.

Findings

Results indicated test-retest reliability of 0.95 and internal consistency reliability of 0.76. Inter-rater reliability reported a Cronbach’s alpha coefficient of 0.86 using a horizontal scoring method. Good discriminative validity was established.

Conclusion

The MOTWSM is a reliable and valid measure that can be used to evaluate the non-cognitive criteria of admissions applications in health profession programs. Use of this measure can facilitate selection of the highest caliber of students.

Personality oriented job analysis to identify non-cognitive factors for a doctor of physical therapy program in the United States

It aimed to conduct a personality oriented job analysis to identify non-cognitive factors that may predict successful performance or performance difficulties in doctorate in physical therapy (DPT) students. The study employed focus groups and a survey with 9 DPT subject matter experts. Focus group participants including 3 DPT faculty members and 4 recent graduates of the DPT program identified 22 non-cognitive factors. Out of them, 15 factors were possibly associated with successful performance and 7 factors were possibly associated with performance difficulties. The questionnaire employing the Combination Job Analysis Method resulted in 12 factors which could be used in selection, and 10 which could be incorporated into training. The present study employed an established job analysis method using subject matter experts to identify a broad array of factors that go beyond what previous studies have examined, and which may predict success or difficulties in a DPT program.

A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions

While medical educators appear to believe that admission to the medical school should be governed, at least in part, by human judgement, there has been no systematic presentation of evidence suggesting it improves selection. From a fair testing perspective, legal, ethical, and psychometric considerations, all dictate that the scientific evidence regarding human judgement in selection should be given consideration. To investigate the validity of using human judgements in admissions, multi-disciplinary meta-analytic research evidence from the wider literature is combined with studies from within medical education to provide evidence regarding the fairness and validity of using interviews and holistic review in medical school admissions. Fourteen studies, 6 of which are meta-analytic studies that summarized 292 individual studies, were included in the final review. Within these studies, a total of 33 studies evaluated the reliability of the traditional interview. These studies reveal that the interview has low to moderate reliability (~.42) which significantly limits its validity. This is confirmed by over 100 studies examining interview validity which collectively show interview scores to be moderately correlated with important outcome variables (corrected value ~.29). Meta-analyses of over 150 studies demonstrate that mechanical/formula-based selection decisions produce better results than decisions made with holistic/clinical methods (human judgement). Three conclusions regarding the use of interviews and holistic review are provided by these meta-analyses. First, it is clear that the traditional interview has low reliability and that this significantly limits its validity. Second, the reliable variance from interview scores appears moderately predictive of outcomes that are relevant to consider in medical school admission. And third, the use of holistic review as a method of incorporating human judgement is not a valid alternative to mechanical/statistical approaches as the evidence clearly indicates that mechanistic methods are more predictive, reliable, cost efficient, and transparent.

Fitness for Purpose of Occupational Therapy Graduates: Two Different Perspectives

A research study that looked at fitness for purpose of 312 occupational therapy and physiotherapy graduates was carried out between 1996 and 1999. The research was commissioned by the Department of Health and managed jointly by the Universities of East Anglia and Southampton. A particular emphasis of the research was the influence of joint undergraduate education on the performance of graduates in first post. A full report of the research can be found in Wiles et al (1999).

Recommendations arising from the research report led to debate with service managers about the meaning of the results and thus the conclusions drawn. This paper was therefore prepared to explore the different perceptions of an educator and a service manager with regard to three themes arising from the research. These were the selection of students for occupational therapy education, the expectations of new graduates about their first post and the transition from student to therapist.

Computerized test versus personal interview as admission methods for graduate nursing studies: A retrospective cohort study

The purpose of this study was to compare the predictive validity, economic efficiency, and faculty staff satisfaction of a computerized test versus a personal interview as admission methods for graduate nursing studies. A mixed method study was designed, including cross-sectional and retrospective cohorts, interviews, and cost analysis. One hundred and thirty-four students in the Master of Nursing program participated. The success of students in required core courses was similar in both admission method groups. The personal interview method was found to be a significant predictor of success, with cognitive variables the only significant contributors to the model. Higher satisfaction levels were reported with the computerized test compared with the personal interview method. The cost of the personal interview method, in annual hourly work, was 2.28 times higher than the computerized test. These findings may promote discussion regarding the cost benefit of the personal interview as an admission method for advanced academic studies in healthcare professions.

The predictive value of criteria for student admission to dentistry

INTRODUCTION

In this study, we investigated the value of an individual selection procedure to choose dental students for the Faculty of Dental Medicine at the University of Witten/Herdecke, Germany. The selection procedure was divided into the following four steps: step one consisted of a formal application; step two consisted of a practical test; step three consisted of a personal interview; and step four consisted of a final selection by an independent committee.

MATERIAL AND METHODS

Data from 196 students were collected from the university files. The following data were compared: the grades of the final secondary school certificate (Abiturnoten) were compared with the grades of the Natural Science Examination (naturwissenschaftliche Vorprüfung), first Dental Examination (zahnärztliche Vorprüfung) and State Board Examination (Staatsexamen); specialisation or non-specialisation at secondary school in natural sciences was compared with the grades of the Natural Science Examination and the first Dental Examination; the results of the practical test of the selection procedure were compared with the grades of the practical examination in the first Dental Examination; the interview recommendations to the committee were compared with the grades of the first Dental Examination and the State Board Examination.

RESULTS

The results showed a positive correlation (R = 0.34; P < 0.001) between the secondary school grades and the Natural Science Examination grades. The correlation was weaker between the grades of the first Dental Examination and State Board Examination. No correlation was found between the specialisation during secondary school in biology, chemistry and physics or non-specialisation, and the results of the Natural Science Examination and first Dental Examination. A weak correlation was found between the grades the candidates obtained in the practical test of the selection procedure and the first Dental Examination.

The Value of Admission Interviews in Selecting Accelerated Second-Degree Baccalaureate Nursing Students

Accelerated undergraduate nursing programs have attracted more qualified applicants than can be accommodated, yet higher-than-desired attrition rates continue for a variety of reasons (e.g., pace of program, poor lifestyle choices, role concerns). The financial effects of the loss of students from these programs has gone largely undocumented and is intensified by the knowledge that a better admission decision could have resulted in another nurse in the workforce. In this article, the inclusion of structured personal interviews into the admission decision-making process is discussed. Specifically, do interviews provide relevant data in making admission decisions? Attempts to reduce subjectivity through the development of a structured interview guide, the training of paired interview teams, and the institution of a "post-mortem" technique to guide revisions are also described. Interviews do result in denial of applicants who seem appropriate based on paper portfolio criteria, allowing for inclusion of other potentially successful applicants.

The effectiveness of academic admission interviews: an exploratory meta-analysis

Admission to health-related professions is very competitive and selecting candidates with the best prospects for success is critical. A variety of measures are used to assess candidates to predict success. The purpose of this research was to assess the effectiveness of using selection interviews for admissions. Meta-analysis was applied to a sample of 20 studies examined in a comprehensive review article on the use of interviews in healthcare academic disciplines. Nineteen of these studies examined the relationship between performance in an interview situation and academic performance, while 10 examined the relationship between performance in an interview situation and clinical performance. A separate meta-analysis was conducted for each category of performance measure. The mean sample-size-effect size for studies examining the predictive power of interviews for academic success was 0.06 (95% confidence intervals 0.03-0.08), indicating a very small effect. The sample of studies was homogeneous using a fixed-effect model. The sample of studies for predicting clinical success had a mean effect size of 0.17 (95% confidence intervals 0.11-0.22), indicating modest positive predictive power. Using a random-effects model, this sample of studies was also homogeneous. Future research should investigate a larger sample of primary studies.

1.1 Student selection and the influence of their clinical and academic environment on learning

Student selection and recruitment play a vital role in the successful outcome of dental education. To identify key issues and practices in selection and recruitment, the group assessed current processes, philosophies and practices from a range of different educational systems, although it was not possible to gather data from all countries or continents within the timeframe provided. Furthermore, the group explored the effect of the educational learning environment on the successful outcome of teaching dental students. It is clear that a wide variety of practices and philosophies exist and are used in different parts of the world. Measuring the success of any given process used for student selection remains a challenge. In some parts of the world, certain practices have become an integral part of the tertiary educational system, and have been applied in a similar way by many or all of the dental schools in that country. In other countries, methods vary from one dental school to another, often reflecting differences in the structure and philosophy of the educational system. There was great variation in the combinations of selection criteria used and in student recruitment strategies. However, it was clear that there was much to be gained by learning from the experiences of other dental schools in student selection. Lessons learned, best practices and philosophies used and supporting value systems proved to be very helpful for benchmarking the processes used. In the discussion of student selection, a number of important questions were raised which deserve further thought and reflection both in the ongoing debate and as part of the ever-changing world of dental education. Important new matters that require more debate and research include: a) ethical issues, including the nature of funding from the student perspective, and the concern that in some regions dentistry may become a profession only for the elite or wealthy students. b) Health standards of students entering dental school. c) How realistic is the applicant's sense of dentistry as a profession? d) How accurate is the students' sense of their career opportunities and the employment market upon graduation? Finally, the over-arching question remains, how valid, reliable and predictable are existing selection practices? Will it be practical and meaningful to standardize methods used, or will exchanging ideas as part of this global debate assist the thought process of dental leaders to improve selection practices by learning from the experiences of other schools in different parts of the world? The processes of open debate, sharing ideas and opinions and identifying sound practices across the globe is a powerful catalyst for developing innovative answers to the complex problems posed by student selection and recruitment. A 'virtual' global process with wide input from as many dental schools as possible should improve the efficacy of student selection, and allow dental educators to identify the 'potential' of prospective students and predict more accurately dental student outcomes. The debate that we have started will certainly contribute to providing a knowledge base which dental educators will be able to draw on when reviewing selection processes in their own schools.