When Language Models Model Items and Examinees: Methodological Considerations for Evaluation
January 14 @ 4:00 pm – 5:00 pm EST
This is presented by the Artificial Intelligence in Measurement and Education SIGIMIE.
Pre-trained language models (LMs) are opening new avenues for modeling and predicting the psychometric properties of assessment items. These advances offer promising innovations—but also introduce unique challenges when traditional evaluation methods are applied in novel AI-based contexts.
In this session, Dr. Ikkyu Choi (ETS) will explore key methodological considerations for evaluating LM-based approaches, with a focus on scenarios involving limited or no human response data, large item pools, and real-time operational use. The presentation will highlight three central components:
- Scaling predicted item parameters
- Identifying and addressing edge cases
- Establishing evaluation criteria for generated samples
Drawing on multiple studies, Dr. Choi will share practical examples, potential solutions, and open questions that continue to shape this emerging field