Generative AI and CEFR levels: Evaluating the accuracy of text generation with ChatGPT-4o through textual features

Abstract

Since its emergence, generative AI has significantly impacted various fields, including English language education. Numerous academic studies have investigated its capabilities in grammar correction, writing evaluation, and dynamics of user interaction. However, there have been insufficient investigations into whether texts generated by such AI align appropriately with CEFR proficiency levels. This study addresses this gap by exploring the applicability of generative AI to CEFR standards. Multiple texts were generated using ChatGPT-4o with specified CEFR levels and analyzed using a vocabulary level analyzer (CVLA) to evaluate text features. The findings revealed discrepancies between AI-generated texts and textbook standards, significant divergences between levels below B1 and above B2, and a noticeable topic bias. Although AI-generated texts seem to differ by level, they require careful evaluation before being applied to CEFR-based education.