Construct-Irrelevant Variance
Construct-irrelevant variance (CIV) is the portion of test-score variance attributable to factors that are not part of the construct the test claims to measure. The term was formalised by Samuel Messick in his unified theory of validity (Messick 1989) and stands as one of the two primary threats to construct validity, alongside construct underrepresentation.
Where construct underrepresentation means the test leaves out parts of the construct it should cover, CIV means the test picks up signal it should not. Both invalidate score interpretation. Messick's framing reshaped the field by treating validity not as a property of the test but as a property of score interpretations and uses, which makes both threats actionable in development rather than only diagnosable after the fact.
Common sources in language testing
Several CIV sources recur in reading and writing assessment:
- Topic Familiarity: candidates with prior schema for the passage topic outperform their actual reading-skill level. The score reflects schema as well as decoding.
- Test-wiseness: candidates who know the format conventions of the test (when to skim, when to skip back, how distractors typically pattern) outperform peers of equal language ability who lack this strategic knowledge.
- Cultural references: passages drawing on culturally specific knowledge, idioms, or shared assumptions advantage some candidates and disadvantage others. This is fairness as well as validity.
- Anxiety, fatigue, and time pressure beyond what the construct definition tolerates.
- Format effects: candidates may underperform on a perfectly designed item simply because they have never met that item type before.
Why it matters
CIV is invisible if you only look at total scores. A test can be reliable, internally consistent, and predictive of outcomes while still measuring partly the wrong thing. The diagnostic moves are differential item functioning (DIF) analysis, which compares item-level performance across demographic groups; expert review against the construct definition; and triangulation of test scores against independent measures of the construct. AI-generated items add a fresh CIV concern: the generator's stylistic biases may introduce variance correlated with training-data demographics rather than with the construct.
In test design the lesson is upstream, not downstream. CIV that is built into the specification cannot be fully removed by item analysis. A construct definition tight enough to name what is not part of the construct, paired with a TLU domain description that pins authentic task characteristics, is the cleanest defence.
Key References
- Messick, S. (1989). Validity. In R. L. Linn (ed.), Educational Measurement (3rd ed., pp. 13–103). American Council on Education and Macmillan.
- Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
- Haladyna, T. M. & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27.
- American Educational Research Association, American Psychological Association & National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. AERA.
See Also
- Construct Validity: the property CIV threatens
- Test Bias: the fairness consequence when CIV correlates with demographic group
- Topic Familiarity: a leading source in reading assessment
- Target Language Use Domain: the upstream defence