A Validation Study of AI-Generated Prompts in CILS (Certification of Italian as a Foreign Language) B2 Exams
- Name: Giulia Peri, Sabrina Machetti, Paola Masillo
- Journal / Publisher: Researching Generative AI in Applied Linguistics (chapter). Iowa State University Digital Press
- Year of publication: 2025
- ISBN: —
- Volume/Number/Page: (pp. 197–218)
- Keywords: generative AI, oral language assessment, automated feedback, personalized instruction, educational technology
- DOI: https://doi.org/10.31274/isudp.2025.211.10
- URL: https://www.iastatedigitalpress.com/plugins/books/211/chapter/1265
- Citation:
Peri, G., Machetti, S., & Masillo, P. (2025). A validation study of AI-generated prompts in CILS (Certification of Italian as a Foreign Language) B2 exams. In C. A. Chapelle, G. H. Beckett, & B. E. Gray (Eds.), Researching generative AI in applied linguistics (pp. 197–218). Iowa State University Digital Press.
- Abstract:
This study presents an ongoing validation analysis of generative AI (ChatGPT-4; OpenAI 2023) in creating test prompts for the written production component of the CILS (Certification of Italian as a Foreign Language) B2 exam. As AI technologies increasingly influence language assessment and test development, a key challenge is ensuring that AI-generated writing test prompts maintain validity, appropriateness, and alignment with assessment constructs. This issue is particularly relevant for high-stakes language certifications, where the demand for new test items is growing, but the number of trained item writers remains limited. The research aims to evaluate ChatGPT-4’s capability to generate writing test prompts reflecting the CILS B2 target domain and to investigate how these prompts are perceived by CILS experts and test-takers. Following an argument based validity framework (Chapelle & Voss, 2021), the study employs a mixed-methods approach to collect evidence for the domain definition inference.