Pretesting

Pretesting is the trialling of test items on a sample of candidates before operational use, to estimate item parameters, surface defects, and decide what enters the live form. It is the empirical step between item authoring and operational administration: items the writers believe in are exposed to actual candidates, statistics are computed, and only items that meet thresholds for difficulty, discrimination, distractor functioning and absence of DIF move forward. Without pretesting, item quality rests on author intuition, which the empirical literature consistently shows is poorly calibrated.

The term overlaps with field testing and field trial. The distinction in practice:

Stage	Purpose	Sample	Data collected
Pretest	Estimate item parameters; cull weak items	Smaller, representative of operational population	Item statistics, distractor analysis, candidate comments
Field trial	Verify operational performance under live conditions	Larger, full operational simulation	Section timings, full form statistics, administrator feedback

Pretesting feeds field testing; field testing feeds operational launch. Lower-stakes tests sometimes fold the two together; high-stakes language tests (Cambridge Main Suite, TOEFL iBT, IELTS) keep them sequenced.

What pretesting evaluates

Each pretested item produces a row of statistics:

• Difficulty (p-value or IRT b-parameter). Should fall in the target range for the form's intended population — typically 0.30–0.85 in CTT terms for Main Suite-level reading and listening items, with the form's mean targeted to the cut-score region.
• Discrimination (point-biserial or IRT a-parameter). Should be positive and clear of zero — operational thresholds vary, but 0.20+ is a common floor.
• Distractor performance. Each option in MCQs should be endorsed by at least 5% of candidates with appropriate negative discrimination. See distractor analysis.
• DIF across known subgroups. L1 group, gender, age, region. Items showing differential functioning are flagged for content review.
• Timing. Time-on-item data flags items that take disproportionately long; under operational time limits, these become unintended speed tests.

Items failing on any axis are rewritten and re-pretested or rejected outright. A typical operational programme rejects 30–50% of authored items at this stage.

Pretest designs

Three patterns dominate:

• Embedded pretest. New items are seeded inside an operational form, mixed with scoring items. Candidates cannot detect them; effort is operational. Scoring items act as anchors that place the new items on the bank's calibrated scale. The dominant design at Cambridge English and ETS.
• Standalone pretest. Items are administered as a dedicated session to volunteers or a recruited sample. Faster to execute, but candidate effort is harder to verify and the sample is rarely operationally representative.
• Pilot administration. A whole new form or section is trialled at full length on a recruited sample, often before launching a new test or major revision. More expensive, more diagnostic; used for syllabus changes, not routine item flow.

What pretesting cannot fix

The frame is calibration, not authoring. Pretesting tells you which items work on the trial sample under the trial conditions; it cannot retroactively fix items written against the wrong specification or items that target a construct the test does not aim to measure. Pretest data is necessary, not sufficient. Items that pass pretest can still fail operationally if:

• The pretest sample diverges from the operational population (volunteer bias, regional skew).
• The construct has shifted between pretest and operational use (curriculum reform, exposure).
• The item interacts with an unrelated section in the operational form in ways the pretest did not capture.

For these reasons operational programmes pair pretest statistics with content review by subject matter experts and post-administration monitoring on live forms.

In high-stakes language testing

Operational language tests pretest at scale. Cambridge English's pretesting cycle administers candidate pretests through participating centres worldwide; items receive Rasch calibration against an anchor set already on the bank's scale, are reviewed item-by-item for content, and enter the bank with parameters attached. Items in the bank can be drawn into operational forms with target difficulty and content profiles, and forms can be equated across sittings via shared anchors. Reed (2013) treats this end-to-end pretest-to-bank pipeline as the architectural feature distinguishing standardised language tests from classroom assessments.

References

• Reed, D. J. (2013). Field testing of test items and tasks. In A. J. Kunnan (Ed.), The Companion to Language Assessment. Wiley-Blackwell.
• Cambridge ESOL. (2011). Research Notes Issue 43: Item banking and pretesting. Cambridge English Language Assessment.
• Bachman, L. F., & Palmer, A. S. (2010). Language Assessment in Practice. Oxford University Press.
• Alderson, J. C., Clapham, C., & Wall, D. (1995). Language Test Construction and Evaluation. Cambridge University Press.
• Weir, C. J. (2005). Language Testing and Validation: An Evidence-Based Approach. Palgrave Macmillan.