Identifying Errors in a Learner Corpus ...

Marking errors in L2 learner performance, though useful in both a didactic and academic sense, is a challenging process, one usually performed manually when involving learner corpora. This is because errors are largely latent phenomena whose manual identification and description involve a significant degree of judgment on the side of human annotators. The purpose of the paper is to discuss and demonstrate the implications of the two stages of the decision-making process that is manual error coding, error location and error description, for measuring inter-annotator agreement as a marker of quality of annotation. The crux of the study is in the proposal that inter-annotator agreement on error location and on error description should be considered and reported separately rather than, as it is common, together as a single measurement. The case study grounded in a high-stakes exam context and typified using an established error taxonomy demonstrates the method behind the proposal and showcases its usefulness in real-world settings.