Currently reading Hershock et al. (2026) on “Unpacking SoTL “Failure”: Lessons from a Fellowship Program Investigating Generative AI’s Impacts”
We use SoTL projects a lot in our academic development work, and from an appreciative inquiry approach: hyping up and building on the good practice we see and carefully addressing the rest when we feel that can be done without harming relationships. We typically talk about the very good and interesting projects we see, but would not typically lable something as a “failure”. So I am intrigued by Hershock et al. (2026)’s paper “Unpacking SoTL “Failure”: Lessons from a Fellowship Program Investigating Generative AI’s Impacts“.
The authors are are academic developers and work with SoTL fellowship programs, where teachers gain first experiences with SoTL. While the focus of the project is on GenAI, what they are learning in terms of SoTL successes and failures is probably applicable to SoTL projects more broadly. They “consider our SFPs successful when participating instructors have a positive professional development experience, feel supported as they innovate their teaching, are exposed to evidence-based teaching strategies, start to develop a toolkit and self-efficacy for SoTL, practice data-driven course design and delivery, and engage in an inclusive community of practice“. This includes at least one direct measurement of student learning, which is analysed and interpreted with regards to future practice, and findings are disseminated. So how — and why — do SoTL projects fail?
Hershock et al. (2026) identify several indicators for failure already in the proposal stage: For example applicants to the fellowship program who do not take the academic developers’ advice on research question and study design etc (“If instructors are unwilling to adjust, we respect their choice but do not fund the proposal.“), or who fail to fill in gaps in their proposals during interviews. Also proposing to develop a new tool (rather than taking an existing one, or having a home-made one fully developed already by the proposal deadline) which is then to be tested is a risk factor, since developments always take longer than expected and then no time and effort is left for the actual SoTL project. But also off-the-shelf tools, especially LLMs, change rapidly, so this needs to be taken into consideration and documented appropriately. And enthusiasm for GenAI as a method to conduct a study is a bit of a red flag if there isn’t a lot of substance in thinking about the method underneath (see also the cool article by Nguyen-Trung & Friese (2026) I read the other day). In the authors’ specific program, SoTL projects always test an intervention against a control group, which comes with risks, too: students self-selecting into treatment vs control group (especially when it comes to using GenAI vs opting out) introduces a selection bias and can lead to very uneven group sizes.
They also discuss “implemention fidelity”, meaning a teacher doing something else than discussed — confusion in the heat of the moment, a modification of the intervention without discussing with the academic developers, or whatever other reason. To prevent this, they have teachers sign an agreement that the study design will be implemented in exactly the agreed-upon way, and I find that a bit off-putting. I think the relationship between teacher and academic developer should be strong enough that a normal verbal agreement is sufficient, and if something happens where the teacher has to change plans on short notice, they should not also have to feel bad about a broken contract. To me, SoTL is about long-term development of teaching, and I would always put the relationship over the outcome of a study, and invest in the relationship rather than leverage contracts and similar. (Maybe they mean that the project has to go through ethics review and that that is the signature on there? Then, of course, the proposed project should be what is implemented, so that the decision of the ethics review board is actually about what happens)
Sometimes, SoTL studies are inconclusive, and then might be perceived as failure by the teachers, though not by the academic developers, who still encourage publication and only consider the project a failure if results are not disseminated.
What I find really helpful in this article, though, is the naming and explicit discussion of failure. It would probably be very useful for future applicants to their program to read to set expectations, and maybe even for other people who are considering a SoTL study. Explicit discussion of the examples for failure given here might also help avoid some pitfalls, even if the teachers end up using different methods.
What I also realised reading this paper is that it might be good to be more explicit about relationship expectations. In their discussion, it is about hierarchies where the academic developers are often outranked and struggle with being taken seriously. But what has come up in my practice quite often is that teachers feel that they shouldn’t be taking up so much of my time, where they feel that I am doing them a favour and I am just doing my job. So that might be something to address more explicitly in the future!
Hershock, C., Pottmeyer, L. O., Pincus, J., Ellington, H. E., & Mineroff, Z. (2026). Unpacking SoTL “Failure”: Lessons from a Fellowship Program Investigating Generative AI’s Impacts. Teaching and Learning Inquiry, 14, 1-16.
Today’s walk to the dip!
Water is getting colder again…
But waves are awesome!
What would I do if I couldn’t go dip almost every day??
