From Static Frameworks to Adaptive Evaluation: Piloting an AI-Powered Evaluation Planner for Media Literacy and Digital Skills Interventions

Authors

• Natalia Edisherashvili, University of Tartu
• Marit Puusepp, University of Tartu
• Leen d'Haenens, KU Leuven
• Tania Azadi, KU Leuven
• Łukasz Tomczyk, Jagiellonian University, Poland
• Gemma Martinez Fernández, Universidad del País Vasco
• Miguel Ángel Casado Del Río, Universidad del País Vasco UPV/EHU
• Diego Martinez-Castro, Universidad Autónoma de Bucaramanga UNAB

Document Type

Research Article

Abstract

Evaluating media literacy and digital skills (ML&DS) interventions remains challenging due to limited methodological capacity, time, and resources. Existing frameworks offer valuable guidance but are static and require expert adaptation. This pilot study tested an AI-powered Evaluation Planner developed within the REMEDIS project to support practitioners in designing feasible, evidence-informed evaluation plans. Using a mixed-methods design, 56 participants from 12 countries assessed the tool’s usability, clarity, and perceived utility after completing the Planner and an online survey. Findings showed high perceived ease of use, cognitive clarity, utility, and multilingual functionality. Associations between clarity, ease of use, and intention to reuse or recommend the tool were consistent with user-centred design and technology acceptance models, but they should be interpreted as cross-sectional associations among self-report ratings rather than evidence of causal mechanisms or demonstrated effectiveness. Qualitative feedback highlighted the Planner's structured logic and perceived pedagogical value while identifying areas for improvement, mainly session continuity in free-tier environments, minor translation inconsistencies, and unclear scope boundaries. Overall, the pilot provides evidence that the Evaluation Planner was perceived as a feasible and practitioner-friendly support for evaluation planning. The findings suggest that guided AI dialogue can scaffold reflection on evaluation purpose, outcomes, indicators, and feasibility, but direct claims about improved evaluation literacy or better evaluation practice require future studies that examine the quality and use of generated evaluation plans over time. Future refinements should focus on technical stability, clearer communication of purpose, and sustained multilingual reliability.

Recommended Citation

Edisherashvili, N., Puusepp, M., d'Haenens, L., Azadi, T., Tomczyk, Ł., Martinez Fernández, G., Casado Del Río, M., & Martinez-Castro, D. (2026). From Static Frameworks to Adaptive Evaluation: Piloting an AI-Powered Evaluation Planner for Media Literacy and Digital Skills Interventions. Journal of Media Literacy Education Pre-Prints. Retrieved from https://digitalcommons.uri.edu/jmle-preprints/61

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 License.