Predicting Item Survival from Authoring to Use on Live Exams

This is presented by the AIME SIGIMIE.

Every item that enters the development process requires an investment of author time, editorial review, committee discussion, and psychometric evaluation. As such, item development can be framed as a resource allocation problem, where costs accumulate throughout the process. Using multiple-choice questions from a veterinary licensure exam, this study aims to automatically predict whether an item will ultimately be approved for live use, or “survive”. A staged modeling framework was implemented, incrementally incorporating feature families from initial authoring to post-pretest psychometrics, to determine when predictive signal emerged in the development process. Results showed that item survival could be predicted early in the process, using features available for a newly developed item, and model performance improved as additional information was incorporated across subsequent stages. Overall, item survival modeling can be used as a risk-screening tool that helps to prioritize review efforts, identify patterns, and provide targeted support to authors.

About the Presenter

• Sydney Peters