Workshop 1: Understanding and Applying Mixed-Effects Models for Repeated Measures in Language Testing

Karen Dunn & Dylan Burton

Mixed-effects models, including linear and generalized linear mixed models (LMMs and GLMMs), have become increasingly popular in applied linguistics and language testing research for the analysis of observational and experimental data. Such models can be used flexibly to account for the predictive capacity of multiple variables on outcomes while also controlling for repeated measurements or other grouped data. The value of (G)LMMs is also noted with respect to robustness in dealing with missing data.

These models are appealing for language testers because many testing studies include nested rater data, or clustered item responses, that can be explicitly accounted for by these models. Additionally, GLMMs with complex crossed/non-nested structures are arguably becoming increasingly relevant in the language testing field, especially given recent interest in employing explanatory item response models for purposes of test validation

This 2-day workshop will provide a comprehensive introduction to mixed-effects models with a particular focus on their application in language testing. Attendees will learn to specify, conduct, and interpret linear and generalized linear mixed models in predicting a range of outcomes: dichotomous, ordinal, and continuous, as appropriate. Day 1 will cover fundamental concepts and applications, while Day 2 will explore more advanced topics and worked examples from real-world research.

This workshop is designed for language testers, researchers, and assessment professionals looking to deepen their understanding of mixed-effects modeling techniques and apply them to language testing research.

Intended Learning Outcomes

By the end of this workshop, participants will:

  • Understand the basic concepts and terminology related to LMMs and GLMMs, including random and fixed effects.
  • Recognize the value and use of these models in language testing contexts and identify appropriate models for use with testing data.
  • Be able to interpret and report results from LMM and GLMM analyses.
  • Develop skills in comparing models and applying model diagnostics using a range of R packages, evaluating model convergence and specifying appropriate estimation.
  • Summarize and report on model output.

Workshop Content

Day 1 (Foundations):

  • Introduction to mixed models; their relationship with linear models and generalized linear models.
  • Key terminology: random effects, fixed effects, nested vs. crossed models.
  • The role of mixed-effects models in language testing.
  • Addressing statistical assumptions of mixed-effects models.
  • Importing language testing data and analyzing basic models.
  • Reporting practices for mixed-effects models.
  • Case studies from language testing research.

Day 2 (Advanced Topics):

  • Advanced model structures: Rasch models as GLMMs.
  • Cumulative link mixed models (CLMMs) and their use with polytomous data.
  • Troubleshooting common challenges in using mixed modelling techniques with language testing data.

 Engagement Methods

  • Interactive Lectures: Brief theoretical overviews followed by practical demonstrations of each concept.
  • Hands-on Activities: Participants will engage with language testing data to practice building, interpreting, and reporting models using R.
  • Group Work: Participants will work in small groups to discuss and apply the techniques introduced.
  • Worked Examples: Both instructors will present examples from their own research to show real-world applications of these techniques.
  • Q&A Sessions: Time will be allocated for questions and discussions, ensuring everyone leaves with a clear understanding of the methods.

Participant Background

Participants will benefit from bringing a basic understanding of relevant statistical concepts, including regression and item response theory, plus an acquaintance with language testing data. Some familiarity with R and R-studio is recommended but not required, as essential skills will be covered and code will be provided.

Pre-Workshop Activities

Participants will not be required to complete any pre-workshop activities.


 

Karen Dunn

Karen Dunn is a senior researcher and head of data for British Council English Language Research. She specializes in language test evaluation, statistical methods and measurement, and has worked extensively with latent trait models, including item response theory and mixed models. Karen has a PhD in Applied Social Statistics, in which she focused on explanatory item response theory as applied to language test data. She has taught courses in statistical methodology for analyzing linguistic and psychological data at undergraduate and postgraduate level, and has previously run workshops with a statistical focus at the British Council’s New Directions conference and UKALTA’s Language Testing Forum. She has published methodological papers in Frontiers in Psychology, and Studies in Second Language Acquisition, and Research Methods in Applied Linguistics.

Dylan Burton

Dylan Burton is an assistant professor at Georgia State University in the Department of Applied Linguistics and ESL. He specializes in research on language assessment and second language acquisition, with an interdisciplinary focus on nonverbal behavior and affect. Dylan holds a PhD in Second Language Studies from Michigan State University. His past projects have involved various uses of multilevel data to account for the effects of raters and other sources of variance, particularly with ordinal polytomous outcomes. He has taught classes and led workshops on language testing, second language acquisition, and quantitative research methods. His research has been published in Language Testing, Language Assessment Quarterly, System, and Papers in Language Testing and Assessment.