Assessing High‐Stakes Writing Performances

Before the advent of the Common European Framework of Reference for Languages (CEFR), assessments of written production in L1, L2 or FL settings in Europe tended to be based either on error counts or on necessarily subjective intuitive judgements made by the individual teacher or examiner (Spolsky, 1995). Inherent in the CEFR is a preference for judging rather than counting in the practice of assessment, which is informed by a philosophy that focuses predominantly on positive (norm-adequate) features. This approach has indeed helped in making the assessment of language proficiency more uniform, reliable and possibly valid. However, it has diverted attention from the fact that an error (a non-norm adequate instance of performance) is also a valid and highly indicative manifestation of learner language at a particular stage of development towards full proficiency (George, 1972). This discrepancy between the suggested (and CEFR informed) focus on positive features in assessment and the seemingly persistent practice of assessing language competences on the basis of error counts raises interesting questions in terms of our ability to provide objective judgements regarding the essential qualities of test usefulness (e.g. validity, reliability, impact, practicality, and others). And at the time when standardized testing is making great inroads into education systems, investigations into the said usefulness seem more necessary than ever and large-scale studies of the nature of standardized language testing are hence in high demand.

The main aim of our study is to investigate the (quantitative and qualitative) extent to which negative features (errors) contribute to ratings (as opposed to, or in combination with, positive ones). This translates into one major (research) question which needs to be answered in order to achieve this aim: whether in rating writing performances one can observe the apparently opposing foci on positive and negative features as complementary – in essence that either focus (or a combination of both) would produce similar ratings (following the intuitive idea that if something is not negative then it is positive). An additional aim is to investigate to what extent the raters’ judgements are affected by their individual interpretation of given descriptors, which should be context-independent, but are often affected by socio-cultural backgrounds (e.g., experience, cultural/educational background).