Test bias occurs when a test systematically advantages or disadvantages a particular group of test takers due to factors that are irrelevant to the construct being measured. In language testing, a biased test conflates language proficiency with extraneous variables such as cultural knowledge, gender, socioeconomic background, familiarity with test formats, or access to technology. Bias is a direct threat to validity: a biased test does not measure what it claims to measure for all populations.
The technical mechanism behind bias is construct-irrelevant variance (Messick, 1989): systematic score differences caused by factors outside the construct. If a reading comprehension test requires knowledge of American baseball to answer correctly, it measures baseball knowledge in addition to reading ability. For test takers unfamiliar with baseball, the item is harder than it should be, not because their reading is weaker, but because the item introduces an irrelevant demand.
Test content assumes cultural knowledge, values, or experiences that are not shared across test-taking populations. Examples:
The language of the test (instructions, rubrics, item stems) introduces difficulty beyond the construct:
Content, roles, or scenarios that stereotype or exclude a gender. Research has shown that topic familiarity correlates with gender; for example, sports topics may advantage male test takers in some populations.
Tests that assume access to technology, tutoring, or test preparation materials. Computer-based tests may disadvantage test takers with limited digital literacy.
The format itself may advantage some groups: multiple-choice tests favour test-wise learners; timed tests disadvantage those with processing differences; speaking tests with unfamiliar interlocutors may trigger anxiety in certain cultural groups.
The primary statistical method for detecting item-level bias. DIF analysis compares item performance across groups matched on overall ability. If an item is significantly harder for one group (matched for ability), it may be biased. DIF flags items for review; not all flagged items are biased (some reflect genuine construct-relevant differences), but they warrant expert scrutiny.
Content experts and members of the target population review items for cultural assumptions, stereotypes, and accessibility issues before the test is administered.
A systematic pre-administration review process that checks items for potentially offensive, exclusionary, or culturally loaded content.
An important distinction:
If a test shows lower scores for a particular group, the question is: is this because the test is measuring something irrelevant (bias), or because the group genuinely has lower proficiency on the construct (impact)? The answer determines the response.
Major tests like TOEFL, IELTS, and Cambridge examinations invest heavily in bias detection and reduction. However, research continues to identify concerns: