The SD error taxonomy and agreement. A validation study

Every kind of an attempt of classification which is based on human discretion is inherently flawed in terms of objectivity and agreement. Classifying anything related to language is the most extreme example of such inherent problematic nature for several reasons: there is no fixed correct ‘finite’ version of a language performance to be set as a benchmark; there is a problem of comparing various classification attempts; and there are no fully reliable statistic measurements which successfully address these issues of inconsistency in language analysis .There is no set value for most of language competences as an acceptable phrase, clause or a sentence, relaying the same semantic content, can have theoretically an unlimited number of variations. If, for example, in error analysis a rater analyses a given performance comparing it with the variety of the acceptable or correct form she or he might have in mind, it might produce a completely different error analysis from someone else doing the same but with a different version of the correct or acceptable form even if the norm they have as means of evaluation to be referred to is fully agreed upon. The paper explores the extent to which a unification of the procedure of error analysis can improve on this unreliability of classification in language analysis and focuses on the measurement of inter-annotator agreement. The study explores the effects a standardized error taxonomy (SD Error Taxonomy) and a standardized process of error analysis can have on the overall result of error analysis among multiple trained raters. The results show that even though full agreement can hardly be expected in language /error) analysis, even in highly controlled conditions of a pilot project, employing a unified error taxonomy and error analysis procedure together with extensive rater training produces higher agreement and results in more validity and reliability.