SHAPed Automated Essay Scoring: Explaining Writing Features’ Contributions to English Writing Organization

Abstract

This study applies the state of the art in explainable AI techniques to shed light on the automated essay scoring (AES) process. By means of linear regression and Shapley values, SHAP (Shapley Additive Explanations) approximates a complex AES predictive model implemented as a deep neural network and an ensemble regression. This study delves into the essentials of the automated assessment of ‘organization’, a key rubric in writing. Specifically, it explores whether the organization and connections between ideas and/or events are clear and logically sequenced. Built on findings from previous work, this paper, in addition to improving the generalizability and interpretability of the AES model, highlights the means to identify important ‘writing features’ (both global and local) and hint at the best ranges of feature values. By associating ‘organization’ with ‘writing features’, it provides a mechanism to hypothesize causal relationships among variables and shape machine-learned formative feedback in human-friendly terms for the consumption of teachers and students. Finally, it offers an in-depth discussion on linguistic aspects implied by the findings.