Dissertation Defenses

Upcoming Dissertation Defenses

Name:  Lisa A. Reyes

Title:  Considerations of Reusing Multiple Choice Items to Assess Medical Certification Repeat Examinees

Committee Chair:  Dr. Yue Yin

Date:  Thursday, April 26, 2018

Time: 9:30am – 11:30am

Location:  Room 3015, ETMSW Building, 1040 W. Harrison St.

Abstract:

As gatekeepers to medical specialty practice, medical boards are responsible for accurately identifying who is qualified for certification.  Boards often utilize comprehensive multiple choice exams to assess minimum competence and make certification determinations.  Generally, boards permit examinees who fail an exam to sit for the exam again.  At the same time, boards must frequently reuse exam items due to issues with test equating or limited item availability.  Given the potential memory advantages and disadvantages of prior item exposure that might interfere with the assessment of repeat examinees, the ongoing discussion of how best to assess repeat examinees has largely focused on how to minimize item exposure.

The goal of this study was to contribute to the research available to guide the development of defensible retest policies in light of the reuse of multiple choice items.  Through investigation of repeat examinee scores on a single medical certification exam, their response patterns on items common across exam attempts, and the measurement capabilities of the items themselves, this research helps contextualize retest score differences and pass-fail outcomes.  Results suggested that repeat examinee outcomes were largely related to overall content knowledge, rather than any memory effects stemming from prior item exposure.  Moreover, indications of any memory effects varied with category of item difficulty, discrimination power, content area, and cognitive complexity.  The findings of this study support the need for medical boards to focus on the quality, not the number, of multiple choice items that they reuse when assessing repeat examinees in order to minimize the potential negative consequences of prior item exposure.

 

Name:  William “Bill” Trapp

Title:  The Impact of Test-Taker Criticism on Educators' Question Review Self-Efficacy and Test Review Quality

Committee Chair:  Dr. Yue Yin

Date:  Thursday, May 3, 2018

Time: 9:30am – 11:30am

Location:  Room 3015, ETMSW Building, 1040 W. Harrison St.

Abstract:

Math educators review math questions during the development of college admissions tests, but the biggest group of stakeholders, test takers, do not participate in this process. This may be because question criticism from test takers does not influence the educators’ self-efficacy or their question review ability. This experimental study investigated the degree to which test taker question criticism impacts: 1) educator question review self-efficacy; and 2) question review quality.

Educator reviewers were randomly assigned to two groups. Both groups of educators reviewed the same set of test questions during a facilitated question review event, but only educators in experimental group were exposed to previously collected test taker question criticism. A new instrument was used to measure educator self-efficacy prior to and after the question review event. The findings showed that self-efficacy ratings for both educator groups increased from pretest to posttest, and the ANCOVA results show that the posttest scores of the experimental group were significantly higher than the posttest scores of the control group, while adjusting for the covariate. Inclusion of test taker criticism in the question review process did not have a measurable impact on the question review quality.

These findings have implications regarding the selection, training, and process of eliciting comments from educator participants.