20408: Using ChatGPT to Assess Reflective Essays in a University Management Course

Reflective essays are a common assessment method in management courses at universities, yet scoring them demands significant time and resources. ChatGPT has democratized access to AI, offering potential benefits for Automated Essay Scoring (AES) (Kumar et al., 2024). However, research highlights various challenges associated with using ChatGPT for AES, and its black box nature lacks the explainability needed for stakeholders to understand its effectiveness. (Mizumoto & Eguchi, 2023). This study shifts focus from comparing AI and human grades to examining whether ChatGPT interprets rubrics and grading criteria similarly to human raters. We tasked ChatGPT with evaluating reflective essays from an undergraduate management course to determine if its feedback aligned with human graders’. Of the seven reflective elements expected in the essays, ChatGPT accurately identified three, two less accurately, and failed to recognize two entirely. It also struggled to differentiate between reflective and descriptive writing, let alone highly reflective and reflective (Gardner et al., 2020). We conclude that while ChatGPT's use for AES in higher education is promising, it may not yet be sufficiently reliable.

This session is part of the Teaching and Learning Conference (TLC@AOM). Space is limited; separate complimentary TLC@AOM pre-registration on a first-come, first-served basis is required. The TLC@AOM registration deadline is 27 July 2025, unless sold out. Register for TLC@AOM.