Distilling LLM Error Judgments into Small, Calibrated Models for K–12 Writing Feedback

This is presented by the Artificial Intelligence in Measurement & Education SIGIMIE.

Navvy gives K–12 students feedback on their writing by highlighting spans with convention errors. This talk covers three span types: run-ons, fragments, and comma errors. Licensing, cost, and latency keep us from serving a frontier LLM directly, and we had no human-labeled spans to train on. Instead we used an LLM to make them.

The first question is whether we can find the errors. A frontier LLM labeled error sentences and comma sites in 4,880 essays, grades 3–8, under a fixed taxonomy. A human annotator reviewed a sample to check the LLM as a rater. We then finetuned a ModernBERT model on the LLM labels and checked its agreement with the LLM on a held-out set.

The second question is which of those errors a teacher would flag at a given grade. The span model never sees grade level, so we fit a small calibration layer on top of it and test it on a hybrid set: a small hand-annotated sample from four grades, plus additional essays labeled by an LLM judge anchored on the human spans. We report each part separately and combined.

About the Presenters