Below, Kirsty is discussing how it can potentially discourage efforts to improve teaching and teachers when we focus on the strength of evidence too much, and don’t value the developmental process itself enough. Definitely worth reading! :-)

Geosciences Education Research is one of a group of research areas, collectively known as Discipline-Based Education Research, that focuses on subject learning at university level, with a particular emphasis on the practices of the subject, especially in relation to the academic research of the discipline that is the background of the majority of university teachers and Discipline-Based (or Geosciences) Education Researchers. It is perhaps unsurprising that at some point a discussion about how to understand and codify the quality of the research work occurs.

One such attempt in the context of geosciences education led to the development of a ‘strength of evidence pyramid’ for geosciences education research [1]. In this model, small projects that are run with limited formality, and which represent (often hard won) practitioner wisdom, are plentiful, but of low value (weak evidence), while the pinnacle of research excellence has the form of a large metastudy in which consistency is found across many studies and from which one can draw a final conclusion.

The authors admit that their model isn’t really relevant to qualitative studies, but fail to explore how the model might be developed to incorporate qualitative studies. The ideal of generalisability, and the tacit implication that one can eventually reach a static ‘truth’ in education seems to reflect a natural sciences mindset. While codifying strength of evidence may be useful for setting work into context, I am worried that making it hard to publish studies with relatively ‘weak’ evidence, but of high practical use, may be directly counter to the development of teaching practice and improved education. The argument that there should naturally be a community of practice around teaching, to the extent that publication etc. can be restricted to research studies, fails to recognise research on the restricted nature of discussions around teaching. It also does not account for the global nature of academia and the value placed on publication.

With the increasing emphasis on teaching and the Scholarship of Teaching and Learning across academia, denigrating or demeaning the value of practice-focussed best efforts, often conducted under conditions of limited resources, support and access to expertise in pedagogical research, seems likely to be counter-productive by discouraging engagement in a scholarly approach to teaching development. Not to mention that it implicitly encourages a distinction between teaching and research, but this time within the area of teaching itself. Journals may like such a hierarchy that labels value or quality since it can provide clear guidelines for evaluating submissions, the question I am left with is whether such a hierarchy is beneficial to the development of university education, or whether it will do damage.

I’ll admit that I’m not very keen on hierarchies. So, when faced with a pyramid that provides a hierarchy of quality for reports of developments and investigations in relation to geosciences education, one of my immediate thoughts is that it opens doors to increased gatekeeping. Can this really be expected to encourage researchers in geosciences, especially early career researchers who can be expected to be judged on publication, to take a scholarly, research-informed, but not research-driven approach to their teaching?

I’d like to briefly discuss an example of its use, in this case, for evaluating TA training programmes [2]. For me, this paper shows the potentially useful aspects of the pyramid, but first, I would like to highlight the great strength of this paper compared to the one in which the pyramid is presented: it actually provides a description of what characterises work at different levels of the hierarchy!! And that is a great improvement: it’s suddenly clear why practitioner wisdom may not be particularly trustworthy (little data, perhaps limited to ‘participant satisfaction’, and unstructured analysis), and what actually goes in to a meta-study to make it so.

In looking for publications on TA training in geosciences, the authors found very few examples, and therefore broadened their search to other disciplines: there aren’t exactly enough extant studies to perform a meta-study!! This indicates that the barrier to sharing knowledge about educational practices through (research) publication may already be too high to even start forming an evidence base. Using the strength of evidence pyramid, did, however, highlight that much of what is shared does not include any systematic evaluation, so there’s definitely space to do better.

However, to my mind, there’s a missing strain of pragmatism: something better than participant satisfaction is clearly needed, but long, validated survey instruments may not be relevant if you’re a teacher first and foremost trying to gauge the success of a particular activity or event. And when it comes to my own work, I can’t say I appreciate being told that my best efforts are ‘trash’ or low-value because I was doing my best with no support, resources or training. Perspective on what one’s doing is valuable, particularly for developing a systematic approach and ensuring that what is reported from one context can be interpreted meaningfully in another, but I’m not sure anything even vaguely resembling a hierarchy is what’s needed.

A systematic review of volcanology education at university (47 papers over 37 years) [3], also highlights the tendency for practitioner wisdom, and its limitations. Whether a research-oriented approach is really necessary is a separate question, but they provide a short list of the information that is necessary to enable comparisons between studies. In using this, pragmatism is going to be necessary, and that not all of these points may be possible for researcher-educators in geosciences who do not have any real training in education research, will need to be recognised. However, it seems that many of the points should be perfectly straightforward to achieve. It seems to me that Table 10 from [3] may be what is needed for practitioners wanting to share wisdom to do better, without having to learn to do education research.

[1] Kristen St. John & Karen S. McNeal (2017) The Strength of Evidence Pyramid: One Approach for Characterizing the Strength of Evidence of Geoscience Education Research (GER) Community Claims, Journal of Geoscience Education, 65:4, 363-372, https://doi.org/10.5408/17-264.1

[2] Kelsey S. Bitting, Rachel Teasdale & Katherine Ryker (2017) Applying the Geoscience Education Research Strength of Evidence Pyramid: Developing a Rubric to Characterize Existing Geoscience Teaching Assistant Training Studies, Journal of Geoscience Education, 65:4, 519-530, https://doi.org/10.5408/16-228.1

[3] Dohaney, J., Jolley, A., Kennedy, B. and Watson, A. (2023) “A systematic review of volcanology learning and teaching in higher education”, Volcanica, 6(2), pp. 221–252. https://doi.org/10.30909/vol.06.02.221252.