Interacting with AI voice-based chatbots: A study on EFL students’ satisfaction and speaking performance

Abstract

AI voice-based chatbots (AVCs) are increasingly integrated into EFL speaking instruction as potential conversational partners for speaking practice. However, studies on students' satisfaction with the chatbot-human interaction and its association with speaking gains remain limited. This study examines the learners' satisfaction with ChatGPT's spoken language during conversational interaction, the effects of AVC-supported speaking practice on students' oral performance and the association between satisfaction and speaking performance. A quantitative pretest-posttest design was employed with 116 Vietnamese EFL students over an 8-week intervention. Data were collected through pre- and post-speaking tests and a satisfaction questionnaire. The test scores were analyzed using paired-samples t-tests and effect sizes, descriptive statistics and correlation analysis were used to examine the learners' satisfaction levels and their relationship with speaking improvement. Overall, learners reported moderate to high satisfaction with ChatGPT-generated language, with mean scores ranging from 3.48 to 4.34 on the 5-point Likert scale. In addition, learners' speaking scores were observed to increase from pre-test to post-test (p < 0.05), with a large effect size (Cohen's d = 1.34). However, the correlation analysis revealed no statistically significant association between satisfaction levels and speaking gains (r = 0.087, p > 0.05). Theoretically, the study contributes empirical evidence to chatbot-human interaction studies by suggesting that AVCs can support learners' speaking improvement. Importantly, it also shows that learners' satisfaction with the learning experience is not necessarily associated with their speaking gains. Pedagogically, the study highlights the need for more instructional scaffolding in AI-supported speaking practice.