Developing an AI-use checklist for formative assessment in EMI

Abstract

Generative artificial intelligence (AI) is reshaping higher education, prompting new concerns about authorship, agency, assessment, and broader pedagogical challenges. In English medium instruction (EMI) contexts, instructors often lack visibility into how students engage with AI during learning tasks. This study introduces an AI-use checklist as a formative assessment tool to document students' AI engagement in an undergraduate EMI course in Japan. Guided by the TPACK framework and Bloom's revised taxonomy, the checklist included ten items covering contextual factors and learning-related dimensions. Findings showed that the checklist effectively distinguished between surface-level patterns and learning-related perceptions. Satisfaction and goal alignment, rather than frequency of use, were most consistently associated with perceived improvements. Qualitative responses also revealed evolving views of AI, with some students describing it as a "group member," highlighting both its collaborative potential and risks to authorship. The study demonstrates that the AI-use checklist offers a practical approach to formative assessment, enabling educators to make AI-use visible and ensure that integration supports learner agency and meaningful engagement in EMI contexts.