Exploring the cross-task transferability of an automated scoring model for discourse completion tasks in relation to learner perceptions

Abstract

As automated scoring systems are increasingly integrated into technology-mediated speaking practice and assessment, ensuring generalisability and construct alignment across tasks remains a critical challenge, particularly in discourse-based contexts. This study examined cross-task transferability by applying an automated scoring model originally developed for multi-turn discourse completion tasks (DCTs). While prior work demonstrated moderate agreement between human and machine scores within a single task, it remains unclear whether such models maintain reliability when applied to discourse tasks that elicit different linguistic and semantic demands. Using responses from 116 undergraduate EFL learners, the existing random-forest-based model was applied to two oral DCTs without task-specific retraining. Human–machine agreement was examined using multiple agreement indices and confusion matrices, followed by lexical, keyword-based, and regression analyses of learner responses, along with a descriptive analysis of learner perceptions for system evaluation. Results showed high human–human agreement across both tasks, whereas human–machine agreement was considerably stronger for DCT1 than for DCT2. Although task-relevant semantic features were associated with both human and automated scores, particularly for academic-related content in DCT2, confusion matrices and transcript inspection indicated that the automated model did not adequately capture discourse-level relationships reflected in holistic human scoring. These findings suggest that limited cross-task transferability cannot be attributed simply to insensitivity to task-relevant semantic content but may instead reflect broader challenges in representing discourse contingency within multi-turn interactions. At the same time, learners generally evaluated the automated feedback positively, although positive perceptions should not be taken as evidence of scoring validity. Taken together, the findings highlight the need for greater construct alignment in cross-task automated scoring and point to the potential role of such systems as learning-support tools, rather than fully reliable assessment instruments, in technology-enhanced language learning environments.