Generative Psychometrics for Automatic Item Development and In-Silico Structural Validity

This is presented by the Artificial Intelligence in Measurement & Education SIGIMIE

Generative artificial intelligence offers possibilities for developing educational assessments, but item generation is just the first step. How can we discover if LLM-generated items represent the intended constructs, showing the desired dimensionality structure before collecting human data? This talk introduces generative psychometrics as a framework for integrating automatic item development with computational evaluation of structural validity using network-based psychometrics (exploratory graph analysis). The Generative Psychometrics workflow combines large language models, LLM embeddings, and network psychometrics (EGA, UVA, bootEGA) to generate, evaluate, and refine item pools. Construct definitions guide item generation, while embedding representations are the input for Exploratory Graph Analysis. Unique variable analysis identifies redundant items, and bootEGA identifies unstable items. In-silico structure provides preliminary evidence about item organization, while structural validity based on human responses requires empirical investigation with real human data. The talk considers how model dependence, prompt technique, and construct definition can influence the results. For educational measurement, this framework positions in-silico evaluation as a transparent screening and refinement stage that can strengthen construct representation, prioritize empirical testing, and support efficient, theoretically grounded assessment development.

About the Presenters