Teaching and Learning Measurement in the Age of AI: Metacognition, Evaluation, and Exercises

This is presented by the Training and Professional Development Committee.

Large language models make difficult work easy, which means the learning in a measurement course now emphasizes the work a model cannot do for the learner. This session presents a method for teaching and learning educational measurement with AI, built on two habits: attending to the model’s reasoning as well as one’s own in terms of metacognition and executive function, since every answer rests on silent inferences about the learner’s goal and about the standard of good practice that applies to it; and attention to the popular misconceptions and salient misunderstandings that these models have uncritically learned from the broad literature and now reproduce with confidence. Participants will see example exercises that put these habits to work — coding collaboratively with a model on a measurement problem, casting the model as “Reviewer 2” against a classic paper, tracing how a foundational study’s ideas grew or were left behind, and testing a claimed gap through simulation — and will learn to write their own AI skills: short, reusable instructions for LLMs that fix the model’s goal, standard, and procedure in advance, so that executive control of the task stays with the learner rather than passing to the model.

Participants will be able to:

1. Identify the two inferences a language model makes before answering a measurement question, and evaluate both alongside the answer itself.
2. Anticipate the measurement topics on which AI has uncritically learned the field’s popular misconceptions, and check its answers against what the discipline can actually defend.
3. Adapt classroom exercises — collaborative coding, adversarial review, conceptual tracing, and simulation — that use AI to build understanding of measurement rather than to bypass it.
4. Draft a reusable AI skill that specifies a goal, a standard, and a procedure, so that the learner sets the terms of the task in advance.

About the Presenter

Joseph H. Grochowalski, PhD, is a Senior Psychometrician at the College Board whose work includes measurement validity, test security, standard setting, machine cognition in AI-based scoring, and the assessment of durable skills. He is an adjunct professor at Teachers College, Columbia University, and New York University, and he consults medical schools with a focus on novel measures of interaction-based skills. He currently co-chairs the NCME Training and Professional Development Committee and offers a workshop on the use of large language models in learning and teaching educational measurement.