Applying generalizability theory in language testing: Comparing nested and crossed scoring designs in the assessment of speaking skills

Comparing nested and crossed scoring designs in the assessment of speaking skills

Abstract

Scoring language learners’ speaking skills is open to a number of measurement errors since raters’ personal judgements could involve in the process. Different grading designs in which raters score a student’s whole speaking skills or a specific dimension of the speaking performance could be settled to control and minimize the amount of the error in grading foreign language speaking skills. Therefore, the present study aims to compare G and Phi coefficients gained from the scores of a full factorial (fully crossed) model versus a nested model where rubric components are nested in graders. Four experienced raters and 116 intermediate level language learners studying at a Turkish state university’s language school voluntarily participated in this exploratory study. Findings revealed that the G and Phi values obtained with a full factorial grading model were higher. In addition, checking the variance components according to the source of variation, the variance associated with the student's main effect of the full factorial pattern was higher, while the variance value of the residual effect was lower. These findings reveal that full factorial designs could generate more reliable results in speaking exams, thus, it is recommended for language schools to implement the full factorial design in speaking exams when practical conditions such as enough time or sufficient number of graders are available.