Unsurprising but important research: there is a sequential bias based on order in which work is presented and then graded in learning management systems (after Wang et al., 2024)
My awesome colleague Rachel Forsyth (of our amazing “trust” paper) sent me a message saying “unsurprising but important research” and then a link to Wang et al. (2024), and that is a good summary. In a study of more than 30 million grading records in a Learning Management System, Wang et al. (2024) find that students with surnames later in the alphabet (and thus graded later in the sequence) a) get lower grades, b) get more negative and impolite comments, and c) are complaining more about their grades to the relevant authority. This happens across subjects, and accumulates for students in such a way that it shows up in final grades and can thus potentially even influence job opportunities. So what should we do?
These are my suggestions (3.1 and 4 are also mentioned in the article)
- Realise that grading is never objective
- Think about what factors can introduce bias in our grading (and there are obviously many!)
- Try to compensate for factors that we have recognised, for example
- Randomise order in which we grade student work so that the last students don’t get to “meet” our most tired and annoyed selves
- Disconnect work from names, so that we are not going into reading it with prejudices based on our impression of the student or assumptions we make based on a name (e.g. gender, ethnicity, native language, income, …)
- Introduce cross-grading to calibrate
Anything else you would suggest?
Wang, Zhihan (Helen) and Pei, Jiaxin and Li, Jun, 30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity (October 16, 2023). Available at SSRN: https://ssrn.com/abstract=4603146
K. D. says:
Rotating the start point (and order of presentation – alphabetical, reversed) in the list when grading different assignments? And perhaps also systematically going through lists in reverse order (definitely a possible approach if have multiple graders). It should surely be possible to implement such functionality as an option in LMSes, and even prompt graders to activate!