Workshop 3: Q Methodology for Language Assessment Research - Principles & Applications

Nicola Lupi Morea

Q methodology (hereafter Q) is a mixed-methods, participant-centred research methodology designed to systematically study human subjectivity. Fundamentally, it involves the application of the Q-sort method (whereby research participants rank-order a set of statements on a topic based on their viewpoints and explain their choices in a follow-up interview) and by-person factor analysis to statistically reveal and qualitatively describe group-level similarities and differences in stakeholder perspectives. As evidenced by a surge in theoretical and empirical studies in applied linguistics (Morea, 2022; Morea & Ghanbar, 2024), Q is growing in popularity in the broader field of language research as an alternative to administering large-scale questionnaires to capture and quantify psychological and affective traits.  

Whilst new to language assessment research, this methodology may provide a promising new approach for exploring stakeholder subjectivity around issues and phenomena related to language testing and assessment. For example, Q may be adopted in studies exploring stakeholder perceptions of assessment, washback effects on test-takers and teachers, or stakeholder understanding of language assessment literacy. The potential of Q lies both in its participant-centred approach (with participants being directly involved during the data-collection and interpretation stages) and in its ability to capture the complexities of human viewpoints. 

Intended learning outcomes 

This workshop will provide participants with hands-on experience of conducting a Q study, from research design to data collection, analysis and interpretation. After the workshop, participants will be able to make an informed decision on the suitability of Q for their research. For those interested in conducting their own Q study, the workshop will represent an essential introduction to the methodology. Equally, the workshop will provide all participants with a solid foundation to understand and critically evaluate Q research, as researchers, journal reviewers and doctoral supervisors. Finally, through various opportunities for collaborative work and networking, the workshop will represent an opportunity to establish a network of language assessment specialists interested in this methodology. 

Workshop content, activities and schedule 

Session 1: Principles of Q methodology 

This introductory session will cover the fundamentals of Q, from its origins to its key features. Ice-breaker activities, such as a “research speed dating”, will also provide participants with opportunities to familiarise with each other and to network. The session will conclude with a discussion regarding the research questions that Q is suitable to answer and with an overview of the four key stages of a Q study, which will be the foci of the following four sessions. 

Session 2: The Q-sort method 

This session will focus on the data-collection phase of a Q study. By the end of the session, participants will be able to evaluate the advantages and limitations of the Q-sort method, and its application to language assessment research, by engaging with a Q-sorting activity as research participants. The session will commence with an overview of the process of designing a Q-set (i.e., the set of statements that Q-study participants are asked to rank-order into a Q-sort). It will continue with a discussion of key decisions prior to data collection, namely participant recruitment, designing a Q-sort grid (the grid on which participants will rank-order the set of statements) and deciding between an unforced vs forced distribution. After introducing the Q-set of a hypothetical study, participants will be given 20 minutes to complete a Q-sort as research participants. After the sorting, participants will share their experience of the Q-sort method in groups of 5-6. Whilst the Q-sort method will be the focus of the session, frequent reference will be made to the equally important practice of conducting post-sorting interviews with research participants, and the possibilities it provides for participatory research. 

Session 3: Q-factor analysis & factor interpretation 

This session will provide participants with practical experience of conducting Q-factor analysis and interpreting Q factors. Participants will be firstly introduced to the concept of factor array (an archetypical Q-sort representing each retained factor), consensus statements (Q-set items that were ranked similarly by participants significantly loading on different factors) and distinguishing statements (Q-set items that were ranked in a statistically-significantly different position across factors). Participants will then be asked to work collaboratively in groups of 5-6 to try to interpret a factor solution, which will be based on the Q-sorts they completed in the previous session. The second half of the session will be devoted to participant presentations, during which group representatives will explain how their groups approached factor interpretation.  

Session 4: Q for language assessment research: Opportunities &challenges  

This concluding session will provide participants with an opportunity to reflect on what they have learnt and share ideas of possible applications of Q in their research. The session will commence with a review of the principles of Q. In pairs, participants will then reflect on the opportunities and challenges of Q for their own research and for language assessment research more broadly, before extending the discussion to the whole group. The last 15 minutes of the workshop will consist of a Q&A session for participants to ask outstanding questions or share reflections and ideas. 

Participant prior knowledge and pre-workshop activities 

Participants will be invited to read a journal article by Wu & Wang (2021), who employed Q to investigate teachers’ beliefs about online assessment. This optional pre-workshop activity will provide participants with an opportunity to familiarise with the methodology ahead of the workshop. However, and importantly, the workshop will not require any prior knowledge of Q, or even of factor analysis. The workshop will be designed to accommodate participants with different research foci and at various stages of their career. It will be particularly relevant to researchers interested in stakeholder perspectives on issues and phenomena related to language testing and assessment, such as stakeholder understanding of language assessment literacy, washback effects on test-takers and teachers, and stakeholder perceptions of, and beliefs about, assessment.