A Computational Psychometrics Ecosystem for Large-Scale Language Assessment: Integrating Measurement, Artificial Intelligence, and Governance
October 13 @ 12:00 pm – 1:00 pm EDT
This is presented by the Large-Scale Assessments SIGIMIE.
Large-scale language assessments increasingly rely on digital platforms that generate extensive linguistic, behavioural, and operational data. These data create opportunities to improve assessment design, scoring, delivery, security, and reporting, but they also introduce significant measurement challenges. Computational applications are often implemented as separate technical solutions, making it difficult to determine whether improvements in efficiency or predictive accuracy strengthen the validity of score interpretations and uses—or introduce construct-irrelevant variance, subgroup differences, model drift, and new threats to fairness and comparability.
This webinar presents the Computational Psychometrics Ecosystem (CPE), an integrative framework for embedding computational methods within established psychometric principles across the lifecycle of large-scale language assessment. The framework comprises six interconnected layers: (1) Measurement Foundations, including construct definition, validity, reliability, fairness, and psychometric modelling; (2) Multimodal Data, encompassing item responses, response times, speech, writing, and process data; (3) Computational Methods, including artificial intelligence, machine learning, natural language processing, Bayesian modelling, and simulation; (4) Assessment Applications, including automated item generation, adaptive and multistage testing, automated scoring, cognitive diagnosis, and test-security analysis; (5) Operational Intelligence, including quality assurance, anomaly detection, model monitoring, predictive analytics, and decision support; and (6) Governance, encompassing transparency, explainability, privacy, accountability, auditability, and human oversight.
Within the CPE, computational performance is not treated as sufficient evidence of measurement quality. Instead, each application is evaluated in relation to its intended contribution to the assessment’s validity argument. Operational illustrations from large-scale language assessment are used to identify relevant evidence requirements, including construct and content representation, item calibration, linking and equating, subgroup performance, measurement invariance, human–machine agreement, classification consistency, model drift, uncertainty, and the consequences of automated decisions.
The framework demonstrates how computational psychometrics can extend—rather than replace—Classical Test Theory, Item Response Theory, Generalizability Theory, and Many-Facet Rasch Measurement. It also provides a structure for investigating how innovations affect score comparability, fairness, security, and the appropriate interpretation and use of assessment results. The CPE offers researchers and assessment organisations a common basis for designing, evaluating, monitoring, and governing AI-enabled language assessments while preserving the scientific and ethical foundations of large-scale educational measurement.
About the Presenter
Dr Ardeshir Geranpayeh is an assessment and educational technology specialist with more than three decades of experience in psychometrics, high-stakes testing, and automated assessment. Before establishing his computational psychometrics consultancy in 2022, he held several senior positions at Cambridge University Press & Assessment, including Head of Psychometrics and Data Services and Head of Automated Assessment and Learning. His work integrates established psychometric principles with advances in artificial intelligence and machine learning to support the development of valid, reliable, fair, and innovative assessment systems.
Dr Geranpayeh has delivered more than 80 academic papers and professional presentations internationally. He is also an experienced workshop leader in assessment and test security and has contributed to major conferences organised by the National Council on Measurement in Education, the Association of Test Publishers, the Association of Language Testers in Europe, and the International Test Commission. He is particularly skilled at making complex assessment concepts accessible to audiences from diverse professional backgrounds.
