Rater training is the systematic process of calibrating examiners to apply rating scales and band descriptors consistently. Without it, subjective assessment degenerates into personal opinion: different raters apply different standards, and scores become unreliable.
The goal is not to eliminate human judgment (that is what makes performance assessment valuable) but to align it: raters should agree not because they are robots, but because they share a common understanding of what each band level looks like.
Untrained raters exhibit predictable biases:
| Bias | Description |
|---|---|
| Severity/leniency | Some raters consistently score higher or lower than others |
| Central tendency | Avoiding extreme scores, clustering marks in the middle bands |
| Halo effect | One strong feature (e.g., fluency) inflating scores across all criteria |
| First impression | The opening of a performance disproportionately influencing the overall score |
| Fatigue | Scores drifting as raters tire over a long marking session |
| L1 bias | Rating candidates from familiar L1 backgrounds differently |
These biases directly undermine inter-rater reliability and, by extension, the fairness and validity of the assessment.
Raters study the rubric, band descriptors, and test specifications. The focus is on understanding what each criterion means and how the bands differ from one another. This is not a passive reading exercise; it requires active discussion and clarification.
Raters score a set of benchmark samples, performances that have been pre-rated by expert panels and represent each band level. After scoring independently, raters compare their scores with the benchmark and discuss discrepancies. This is where calibration happens: raters adjust their internal standards to align with the agreed benchmarks.
Raters score additional samples independently, receive feedback, and discuss borderline cases. Multiple rounds may be needed. The emphasis is on applying the descriptors as written, not on personal preferences about what "good" writing or speaking sounds like.
In formal testing programmes (IELTS, Cambridge), raters must pass a standardisation test by scoring a set of performances within acceptable tolerance of the benchmark scores. Raters who fall outside tolerance receive additional training or are not certified.
Training is not a one-off event. Rater drift occurs over time as personal standards gradually shift. Monitoring mechanisms include:
The standardisation meeting is the core event in rater training. A typical session:
These meetings work best with an experienced lead who can guide discussion toward the descriptors rather than allowing debate to become personal preference.
Language schools and universities that develop their own writing or speaking assessments need rater training even with small teams. The minimum: