Standardsetting for the receptive skills. On the discrepancy between ...

Abstract

Standard Setting for the Receptive Skills in the Austrian Educational Standards Tests.

On the Discrepancy between Rater‑perceived and Empirical Item Difficulty.

Guenther Sigott

English Department

University of Klagenfurt

In the last ten years, Austria has seen considerable professionalisation in language testing at all levels in the educational system. National Educational Standards (Bildungsstandards) have been introduced as a large-scale diagnostic tool, a standardised Matura examination has been put into place, and attempts have been made to standardise the assessment of language proficiency of students graduating from university language subjects. In all this, the Common European Framework of Reference (CEFR) plays an important role as a reference tool. It has become commonplace to link tests to the CEFR. However, linking a test to the CEFR is a complicated process which involves problems. This presentation focuses on linking the National Educational Standards Tests of the receptive skills of reading and listening to the CEFR. In particular, it focuses on the discrepancy between empirical and estimated item difficulty, a problem which is inherent in all linking (or standardsetting) approaches. The discrepancy results from differences in rater severity on the one hand, and from a mismatch between the raters’ difficulty estimates and empirical item difficulties. In this presentation, two approaches to dealing with the discrepancy between estimated and empirical item difficulty are described in order to set cutscores in a transparent way and place test takers in a CEFR level for Listening and Reading. This makes it necessary to define thresholds for allocating test items to a CEFR level. The procedure involves three steps. First, a modified Angoff procedure is employed in order to generate Raw Difficulty Estimates. Then empirical item difficulties are derived from data from test administrations. Eventually, two procedures, the fuzzy interval approach and the regression approach, are described which, by recourse to principles of probabilistic test theory, enable cutscores to be set in a transparent way despite the discrepancy between estimated and empirical item difficulty.