Aligning AI with Assessment Standards: A Case Study of Evaluating AI-Generated Reading Passages for NAEP

This is presented by the Large-Scale Assessment SIGIMIE.

The on-going revision of the Standards for Educational and Psychological Testing has drawn significant attention, particularly in the context of leveraging AI techniques in today’s digital environment. A central question is whether AI can measure up to standards, and how AI and humans can collaborate effectively to advance assessment operations in test design, administration, scoring, and reporting while upholding the essential principles of assessment in reliability, validity and fairness. This talk will provide an overview of the key elements of AI that can be leveraged within the standards, followed by a case study to illustrate these ideas in practice. Specifically, the case study introduces a framework for evaluating the difficulty alignment of reading passages when AI is used to automatically generate tasks for NAEP across different grade levels. It also examines the effects of generating passages with varying concepts, such as fiction and non-fiction, to highlight both opportunities and challenges in applying AI to educational measurement.

This session will be led by Qiwei He, Georgetown University.