In this digital ITEMS module, Dr. Won-Chan Lee, Dr. Stella Kim, Qiao Liu, and Seungwon Shin provide an introduction to generalizability theory, mainly discussing a univariate framework.
Module Overview
Generalizability theory (GT) is widely used framework in the social and behavioral sciences for assessing the reliability of measurements. Unlike classical test theory, which treats measurement error as a single undifferentiated term, GT enables the decomposition of error into multiple distinct components. This module introduces the core principles and applications of GT, with a focus on the univariate framework. The first four sections cover foundational concepts, including key terminology, common design structures, and the estimation of variance components. The final two sections offer hands-on examples using real data, implemented in R and GENOVA software. By the end of the module, participants will have a solid understanding of GT and the ability to conduct basic GT analyses using statistical software.
Won-Chan Lee
- Professor in the Department of Psychological and Quantitative Foundations at the University of Iowa (UI)
- Director of Center for Advanced Studies in Measurement and Assessment (CASMA)
- He received his Ph.D. in educational measurement from the University of Iowa in 1998
- His research focuses on measurement error and reliability, decision consistency, equating and scaling
- He teaches a full-length course on generalizability theory at UI and has co-instructed NCME pre-conference sessions on “Generalizability Theory and Applications” on multiple occasions
Stella Kim
- Associate Professor of Educational Research, Measurement, and Evaluation at the University of North Carolina at Charlotte
- Her research focuses on educational measurement, with a specific interest in scaling, linking, and equating. She also conducts research on the development and application of generalizability theory. Dr. Kim earned her Ph.D. in educational measurement from the University of Iowa and joined UNC Charlotte in 2018.
Qiao Liu
- Third-year Ph.D. student in Educational Research, Measurement and Evaluation at the University of North Carolina at Charlotte
- She spent 10 years working as a classroom teacher in both China and the United States
- During her doctoral journey, she has had the opportunity to contribute to both applied and methodological research projects in measurement
Seungwon Shin
- Second-year Ph.D. student in Educational Measurement and Statistics at the University of Iowa
- His research interests include educational measurement, especially focusing on item response theory, generalizability theory, scale linking and equating
Introduction
Upon completion of this ITEMS module, learners should be able to:
- Describe the key concepts and assumptions underlying generalizability theory
- Identify appropriate design structures for conducting generalizability theory analyses
- Conduct basic generalizability theory analyses using R and GENOVA software
- Apply generalizability theory to real or simulated assessment data
Section 1: An Overview of Generalizability Theory
Upon completion of this section, learners should be able to:
- Describe the fundamental differences between classical test theory and generalizability theory
- Explain key concepts of generalizability theory, including universes of admissible observations and universes of generalization
- Demonstrate how to design and conduct a generalizability (G) study and interpret its variance components
- Apply G study variance components in Decision (D) studies to obtain D-study outcomes
Section 2: Single-Facet Designs
Upon completion of this section, learners should be able to:
- Differentiate between crossed and nested designs
- Describe the major purposes of G and D studies
- Explain the difference between relative and absolute errors
- Identify two reliability-like coefficients within generalizability theory
Section 3: Generalizability (G) Studies for Multi-Facet Designs
Upon completion of this section, learners should be able to:
- Define a G study design for a given data collection design
- Draw a Venn diagram for a given G study design
- Explain the difference between main and interaction effects
- Explain the issue of confounding effects
Section 4: Decision (D) Studies for Multi-Facet Designs
Upon completion of this section, learners should be able to:
- Identify three primary factors influencing D study results
- Explain the impact of using a fixed facet in a D study compared to using all random facets
- Compute D study results for a nested design using crossed-design variance components
- Describe traditional reliability coefficients from the perspective of generalizability theory
Section 5: Conducting G and D Studies in R
Upon completion of this section, learners should be able to:
- Prepare data for G study analysis in R for both p X i and p X (r:i) designs
- Use lmer() function to estimate variance components in G studies
- Compute D study statistics for various test conditions using R
- Interpret G study and D study results
Section 6: Conducting G and D Studies in GENOVA
Upon completion of this section, learners should be able to:
- Prepare data for GENOVA analysis
- Create GENOVA control cards for p X i design
- Create GENOVA control cards for p X (r:i) design
- Interpret the output generated by GENOVA