Category Archives: literature

The Curious Construct of Active Learning: A guest post by K. Dunnett (UiO) on Lombardi et al. (2021)

Leave a reply

‘Active Learning’ is frequently used in relation to university teaching, especially in Science, Technology, Engineering and Mathematics (STEM) subjects where expository lecturing is still a common means of instruction, especially in theoretical courses. However, many different activities and types of activities can be assigned this label. This review article examines the educational research and development literature in 7 subject areas (Astronomy, Biology, Chemistry, Engineering, Geography, Geosciences and Physics) to explore exactly what is meant by ‘active learning’, its core principles and defining characteristics.

Active Learning is often presented or described as a means of increasing student engagement in a teaching situation. ‘Student engagement’ is another poorly defined term, but is usually taken to involve four aspects: social-behavioural (participation in sessions and interactions with other students); cognitive (reflective thought); emotional and agentic (taking responsibility). In this way, ‘Active Learning’ relates to the opportunities that students have to construct their knowledge. On the other hand, and in relation to practice, Active Learning is often presented as the antithesis of student passivity and traditional expository lecturing in which student activity is limited to taking notes. This characterisation is related the behaviour of students in a session.

Most articles and reviews reporting the positive impact of Active Learning on students’ learning don’t define what Active Learning is. Instead, most either list example activities or specify what Active Learning is not. This negative definition introduces an apparent dichotomy which is not as clear as it may initially appear. In fact, short presentations are an important element of many ‘Active Learning’ scenarios: it is the continuous linear presentation of information that is problematic. Most teaching staff promote interactivity and provide opportunities for both individual and social construction of knowledge while making relatively small changes to previously presentation-based lectures.

That said, the amount of class time in which students are interacting directly with the material does matter. One example of measurement of the use and impact of Active Learning strategies (or activities that require students to interact with the material they are learning) in relation to conceptual understanding of Light and Spectroscopy found that high learning gains occur when at least 25% of scheduled class time is spent by students on Active Learning strategies. Moreover, the quality of the activities and their delivery, and the commitment of both students and staff to their use, are also seen as potentially important elements in achieving improved learning.

In order to develop an understanding of what Active Learning actually means, groups in seven disciplinary areas reviewed the discipline-specific literature, and the perspectives were then integrated into a common definition. The research found that presentations of Active Learning in terms of either students’ construction of knowledge via engagement, or in contrast to expository lecturing were used within the disciplines, although the discipline-specific definitions varied. For example, the geosciences definition of Active Learning was:

”Active learning involves situations in which students are engaged in the knowledge-building process. Engagement is manifest in many forms, including cognitive, emotional, behavioural, and agentic, with cognitive engagement being the primary focus in effective active learning,”

while the physics definition was that:

”Active learning encompasses any mode of instruction that does not involve passive student lectures, recipe labs, and algorithmic problem solving (i.e., traditional forms of instruction in physics). It often involves students working in small groups during class to interact with peers and/or the instructor.”

The composite definition to which these contributed is that:

”Active learning is a classroom situation in which the instructor and instructional activities explicitly afford students agency for their learning. In undergraduate STEM instruction, it involves increased levels of engagement with (a) direct experiences of phenomena, (b) scientific data providing evidence about phenomena, (c) scientific models that serve as representations of phenomena, and (d) domain-specific practices that guide the scientific interpretation of observations, analysis of data, and construction and application of models.”

The authors next considered how teaching and learning situations could be understood in terms of the participants and their actions (Figure 1 of the paper). ‘Traditional, lecture-based’ delivery is modelled as a situation where the teacher has direct experience of disciplinary practices, access to data and models, and then filters these into a simplified form presented to the students. Meanwhile, in an Active Learning model students construct their knowledge of the discipline through their own interaction with the elements of the discipline: its practices, data and models. This knowledge is refined through discussion with peers and teaching staff (relative experts within the discipline), and self-reflection.

The concluding sections remark on the typical focus of Discipline Based Educational Research, and reiterate that student isolation (lack of opportunities to discuss concepts and develop understanding) and uninterrupted expository lecturing are both unhelpful to learning, but that ”there is no single instructional strategy that will work across all situations.”

The Curious Constrauct of Active Learning
D. Lombardi, T. F. Shipley and discipline teams.
Psychological Science in the Public Interest. 2021, 22 (1) 8-43
https://doi.org/10.1177%2F1529100620973974

Effective learning techniques for students: Currently reading Dunlosky et al. (2013)

Leave a reply

I want to give you a quick summary of the super useful article “Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology” by Dunlosky et al. (2013). A lot of what I write about on here is about how we can improve our own teaching, but another really important aspect is helping students to develop the skills they need to become successful and independent learners. This can be achieved either by explicitly teaching them study techniques, or by building lessons in ways that use those techniques (although I think that even then it would be useful to make the techniques and why they were chosen explicit).

In the article, Dunlosky et al. (2013) suggest one possible lesson plan that combines several of the techniques they recommend: Starting a new topic with a practice test and feedback on the most important points learned before, practising exercises on the current topic mixed with “older” content, picking up or referring back to old ideas repeatedly. Also, asking students to connect new content with prior knowledge by asking how the new information fits with what they already know and if they can explain it.

So what are the techniques we should be using and teaching our students? Out of the 10 techniques reviewed in the article, two are high impact, three are moderate impact, and the rest have low impact (even though some of those are the ones most often used by students), and I am presenting them in that order (and you might recognise them from the suggested lesson plan — surprise!).

High impact: Practice testing

One of the two most useful learning techniques, according to Dunlosky et al. (2013), is practice testing: either self-testing of to-be-learned material, or doing really low-stakes (or even no-stakes) tests in class.

For self-testing, this can mean different things, like learning vocabulary using (electronic) flashcards. When I was learning Norwegian, I practised a lot using the App Anki and self-written cards with vocabulary or sentences I wanted to know, now I use Duolingo regularly (616 day streak today, wohoo!). But it could also mean doing additional exercises, either provided with the teaching materials, or even seeking out or coming up with additional questions. For my exams in oceanography, as a student I spent a lot of time (mostly sleepless nights though) before the exam trying to imagine what I might be asked, and how I would answer.

I wrote about the importance of assessment practices and how “testing drives learning” previously, but here the important point is really that the students are ideally using this as a learning technique themselves.

High impact: Distributed practice

The second high impact practice is distributed practice: not cramming everything the night before an exam, but spreading practice out over as long a period as possible, and come back to material repeatedly over time. This is not how we typically teach, nor how textbooks present materials (usually one topic is presented in one chapter, together with all exercises or practice problems that go with that topic), so it is not a learning technique that students are necessarily familiar with.

Distributed practice can be “encouraged” (enforced?) by frequent low-stakes testing in class. It is also built into the apps I mentioned above: Flashcards or practice problems that were answered wrong will come up again after a little while, and then, if you answered correctly, again and again with longer intervals in between. If you answered wrong, they’ll probably pop up even more often. And, of course, it is something that we can plan for and can encourage our students to plan for — ideally combined with an explanation and maybe some data for why this is a really good idea.

Moderate impact: Interleaved practice

One moderate impact practice that I like a lot is interleaved practice: mixing different types of problems or different topics during a practice session. Interestingly, results during those practice sessions are worse than when the same types of problems or topics are practised grouped together. But when tested later, interleaved practice is a lot more efficient, likely because in interleaved practice, students learn to figure out which way to solve a problem is required for which type of problem. Whereas in blocked practice, it is very easy to just numbly apply a procedure over and over again without actually thinking about why it is the appropriate one for a specific case. Which is what I am currently experiencing with my Swedish classes, now that I’m thinking about it…

Moderate impact: Elaborative interrogation

But, a second moderate impact practice could help in these cases: elaborative interrogation. There, we would do exactly what I describe above that I don’t do in Swedish classes: Asking myself why I am applying a rule in one situation but not in another, why a pattern shows up here and not there, and coming up with explanations. This is very easy to implement actually.

But it is not so easy to instruct as a technique, when we don’t want to prescribe the kinds of questions students should ask themselves, but want them to generate the questions themselves, and then answer them. How do we tell them at what level of abstraction or difficulty they should aim? If we give prompts, then how many? Maybe this is something we can / need to model explicitly?

Moderate Impact: Self-explanation

Another moderate impact practice is self-explanation, where we explicitly connect new information with what we know already, explore how information fits together and which parts are actually new and/or surprising to us and why, or explain rules we come across. This is really useful for far-transfer later on.

We can prompt self-explanation on a very abstract level, giving general instructions like “what is the new information in this paragraph?”, or on a much more concrete level like “why are you applying this rule rather than that one?”.

The most efficient way to use self-explanation is to do it right in the learning process. But doing it retrospectively is still better than not doing it. And it is important for learning that we don’t have access to explanations, but find them ourselves (this makes me think of people that always bring out their smartphone and google the answer to an intriguing question, instead of engaging in the back-of-the-envelope fun).

Low impact: Summarization

And now we’ve reached low impact practice no 1: summarization. Writing summaries of the content we are trying to learn, that’s something I do a lot, for example just now when writing this blog post (but I don’t rely on remembering what I’m writing; I google things on my own blog. So maybe that’s not the same thing?).

Summarising, i.e. rephrasing the important points in one’s own words, is more useful that just selecting the most important content and then copying it word by word.

Low impact: Highlighting/underlining

Another low impact, yet highly popular, practice is highlighting and underlining. I’ve never understood why people do that, I’ve always written my own summaries and found that a lot more useful. But reasons why people might do it is because it’s quick and looks like someone has done some work with a text, even though it isn’t more beneficial than just reading a text. But the part about looking like work has been done might give students false confidence in how much work they have actually done, and hence how much they have learned.

Low impact: Keyword mnemonic

The “keyword mnemonic” low impact practice is about “building donkey bridges” as we would say in German — finding ways to remember more complex things by memorizing something simple, for example mental images or word sequences. I do that for example to remember the difference between refraction and diffraction, or the order of the planets in the solar system. But apparently it’s not a very useful technique at scale.

Low impact: Imagery for text

Another low impact practice related to mental images: creating mental imagery while reading or listening to texts. This can be helpful, and interestingly enough, the mental image is more useful than actually drawing it out!

Low impact: Rereading

And lastly: rereading. This is what students do A LOT in preparation for exams; reading old material again and again. This is a lot less efficient than the high- and moderate impact practices described above!

So what can we do with this information? As described in the beginning, we can include the higher-impact practices in our planning so students benefit from them without necessarily knowing that it is happening. But then we can also make those techniques explicit when we are using them, and encourage students explicitly to use them in their own studying. And we can point out that highlighting and rereading, for example, might feel like studying, but are much less efficient than those other techniques.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public interest, 14(1), 4-58.

Article just published: Collaborative Sketching to Support Sensemaking: If You Can Sketch It, You Can Explain It

2 Replies

What a lovely Birthday gift (and seriously impressively quick turn-around times at TOS Oceanography!): Kjersti‘s & my article “Collaborative Sketching to Support Sensemaking: If You Can Sketch It, You Can Explain It” (Daae & Glessmer, 2022) has just come out today!

In it, we describe Kjersti’s experiences with using portable whiteboards that students use to collaboratively draw on in order to make sense out of new concepts or generate hypotheses for outcomes of experiments. It’s a really neat practice, you should check out our article and consider it for your own classes!

Kjersti also wrote a supplemental material to the article with a lot of details how exactly she implements the whiteboards and how she formulates the tasks (find it here).

Daae, K., and M.S. Glessmer. 2022. Collaborative sketching to support sensemaking: If you can sketch it, you can explain it. Oceanography, https://doi.org/10.5670/oceanog.2022.208.

Using peer feedback to improve students’ writing (Currently reading Huisman et al., 2019)

Leave a reply

I wrote about involving students in creating assessment criteria and quality definitions for their own learning on Thursday, and today I want to think a bit about involving students also in the feedback process, based on an article by Huisman et al. (2019) on “The impact of peer feedback on university students’ academic writing: a Meta-Analysis”. In that article, the available literature on peer-feedback specifically on academic writing is brought together, and it turns out that across all studies, peer feedback does improve student writing, so this is what it might mean for our own teaching:

Peer feedback is as good as teacher feedback

Great news (actually, not so new, there are many studies showing this!): Students can give feedback to each other that is of comparable quality than what teachers give them!

Even though a teacher is likely to have more expert knowledge, which might make their feedback more credible to some students (those that have a strong trust in authorities), it also makes it more relevant to other students, and there is no systematic difference between improvement after peer feedback and feedback from teaching staff. But to alleviate fears related to the quality of peer feedback is to use peer feedback purely (or mostly) formative, whereas the teacher does the assessment themselves.

Peer feedback is good for both giver and receiver

If we as teachers “use” students to provide feedback to other students, it might seem like we are pushing part of our job on the students. But: Peer feedback improves writing both for the students giving it as well as for the ones receiving it! Giving feedback means actively engaging with the quality criteria, which might improve future own writing, and doing peer feedback actually improves future writing more than students just doing self-assessment. This might be, for example, because students, both as feedback giver and receiver, are exposed to different perspectives on and approaches towards the content. So there is actual benefit to student learning in giving peer feedback!

It doesn’t hurt to get feedback from more than one peer

Thinking about the logistics in a classroom, one question is whether students should receive feedback from one or multiple peers. Turns out, in the literature it is not (significantly) clear whether it makes a difference. But gut feeling says that getting feedback from multiple peers creates redundancies in case quality of one feedback is really low, or the feedback isn’t actually given. And since students also benefit from giving peer feedback, I see no harm in having students give feedback to multiple peers.

A combination of grading and free-text feedback is best

So what kind of feedback should students give? For students receiving peer feedback, a combination of grading/ranking and free-text comments have the maximum effect, probably because it shows how current performance relates to ideal performance, and also gives concrete advise on how to close the gap. For students giving feedback, I would speculate that a combination of both would also be the most useful, because then they need to commit to a quality assessment, give reasons for their assessment and also think about what would actually improve the piece they read.

So based on the Huisman et al. (2019) study, let’s have students do a lot of formative assessment on each other*, both rating and commenting on each other’s work! And to make it easier for the students, remember to give them good rubrics (or let them create those rubrics themselves)!

Are you using student peer feedback already? What are your experiences?

*The Huisman et al. (2019) was actually only on peer feedback on academic writing, but I’ve seen studies using peer feedback on other types of tasks with similar results, and also I don’t see why there would be other mechanisms at play when students give each other feedback on things other than their academic writing…

Bart Huisman, Nadira Saab, Paul van den Broek & Jan van Driel
(2019) The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis, Assessment & Evaluation in Higher Education, 44:6, 863-880, DOI: 10.1080/02602938.2018.1545896

Co-creating rubrics? Currently reading Fraile et al. (2017)

Leave a reply

I’ve been a fan of using rubrics — tables that contain assessment criteria and a scale of quality definitions for each — not just in a summative way to determine grades, but in a formative way to engage students in thinking about learning outcomes and how they would know when they’ve reached them. Kjersti has even negotiated rubrics with her class, which she describes and discusses here. And now I read an article on “Co-creating rubrics: The effects on self-regulated learning, self-efficacy and performance of establishing assessment criteria with students” by Fraile et al. (2017), which I will summarise below.

Fraile et al. (2017) make the argument that — while rubrics are great for (inter-)rater reliability and many other reasons, students easily perceive them as external constraints that dampen their motivation and might lead to shallow approaches to learning, not as help for self-regulated deep learning. But if students were involved in creating the rubric, they might feel empowered and more autonomous because they are now setting their own goals and monitoring their performance against those, thus using it in ways that actually supports their learning.

This argument is then tested in a study on sports students, where a treatment group co-creates rubrics, whereas a control group uses those same rubrics afterwards. Co-creation of the rubric meant that after an introduction to the content by the teacher, students listed criteria for the activity and then discussed them in small groups. Criteria were then collected and clustered and reduced down to about eight, for which students, in changing groups, then produced two extreme quality definitions for each. Finally, the teacher compiled everything into a rubric and got final approval from the class.

So what happened? All the arguments above sounded convincing, however, results of the study are not as clear-cut as one might have hoped. Maybe the intervention wasn’t long enough or the group of students was too small to make results significant? But what does come out is that in thinking aloud protocols, the students who co-created the rubrics were reporting more self-regulated learning. They also performed better on some of the assessed tasks. And they reported more positive perceptions of rubrics, especially of transparency and understanding of criteria.

What do we learn from this study? At least that all indications are that co-creating rubrics might be beneficial to student learning, and that no drawbacks came to light. So it seems to be a good practice to adopt, especially when we are hoping for benefits beyond what was measured here, for example in terms of students feeling ownership for their own learning etc..

Fraile, J., Panadero, E., & Pardo, R. (2017). Co-creating rubrics: The effects on self-regulated learning, self-efficacy and performance of establishing assessment criteria with students. Studies in Educational Evaluation, 53, 69-76.

Teaching to improve research skills? Thinking about Feldon et al. (2011)

Leave a reply

When graduate students teach, they acquire important research skills, like generating testable hypotheses or designing research, more than their peers who “only” do research, according to Feldon et al. (2011), who compared methodolocical skills in research proposals written by graduate students.

This is quite interesting, because while many graduate students enjoy teaching, there are only 24 hours in a day (and 8 in a work day), and teaching is often seen as competing for time with research. But if teaching actually helps develop research skills (for example because the teaching graduate students are practicing those skills over and over again while advising students, whereas the “research only” graduate students are usually working on pre-defined projects without opportunities to practice those skills), this is a good argument to assign a higher status to teaching even in research training. This would not only lead to graduates that have more experience teaching, but that also have stronger research skills. Win-win!

Feldon, D. F., Peugh, J., Timmerman, B. E., Maher, M. A., Hurst, M., Strickland, D., … & Stiegelmeyer, C. (2011). Graduate students’ teaching experiences improve their methodological research skills. Science, 333(6045), 1037-1039.

Three ways to think about “students as partners”

Leave a reply

As we get started with our project #CoCreatingGFI, we are talking to more and more people about our ideas for what we want to achieve within the project (for a short summary, check out this page), which means that we are playing with different ways to frame our understanding of co-creation and students as partners (SaP).

For the latter, I just read an article by Matthews et al. (2019) that identifies three ways that SaP is commonly being written about. Reading this article was really useful, because it made me realise that I have been using aspects of all three, and now I can more purposefully choose in which way I want to frame SaP for each specific conversation I am having.

In the following, I am presenting the three different perspectives and commenting on how they relate to how I’ve been talking — and thinking — about SaP.

Imagining through Metaphors

Metaphors are figures of speech where a description is applied to something it isn’t literally applicable to, but where it might help to imagine a different (in this case, desired) state.

“Students as partners” as a metaphor evokes quite strong reactions occasionally, because it can be perceived as a complete loss of power, authority and significance by teachers; and likewise as too much work, responsibility, stress by students. We moved away from “students as partners” as a metaphor and towards “co-creation”, because when speaking about “students as partners”, we were constantly trying to explain who the students were partnering with, and what “partnership” would mean in practice. So while we were initially attracted to the metaphor and the philosophy behind it, it ended up not working well in our context.

Speaking about the “student voice”, on the other hand, is something that I’m still doing. To me, it implies what Matthews et al. (2019) describe: students powerfully and actively participating in conversations, and actually being heard. But they also warn that this metaphor can lead to structures in which power sharing becomes less likely, which I can also see: if we explicitly create opportunities to listen to students, it becomes easy to also create other situations in which there explicitly is no space for students.

Building on concepts

When grounding conversations on accepted concepts from the literature, it makes it a lot easier to argue for them and to make sure they make sense in the wider understanding in the field.

In our proposal for Co-Create GFI, we very explicitly build all our arguments on the concept of “communities of practice”. Maybe partly because I was in a very bad Wenger phase at around that time, but mostly because it gave us language and concepts to describe our goal (teachers working together in a community on a shared practice), because it gave us concrete steps for how to achieve that and what pitfalls to avoid.

Also in that proposal as well as in our educational column in oceanography, we use “student engagement” as the basis for the co-creation we are striving for. In our context, there is agreement that students should be engaged and that teachers should work to support student engagement, so starting from this common denominator is a good start into most conversations.

Another concept mentioned by Matthews et al. (2019) are “threshold concepts”, which isn’t a concept we have used in our own conversations about SaP, but which I found definitely helpful to consider when thinking about reactions towards the idea of SaP.

Matthews et al. (2019) point out that while building on concepts can be grounding and situating the way I describe above, it can also be disruptive.

Drawing on Constructs

Of the three ways of talking about SaP, this is the one we’ve used the least. Constructs are tools to help understand behaviour by basically putting a label on a drawer, such as identity, power, or gender. Looking at SaP through the lens of different constructs can help see reality in a different way and change our approach to it, or as Matthews et al. (2019) say: “revealing can lead to revisiting”.

I know it’s not the intention of the article, but I am wondering if taking on that lens just for fun might not reveal new and interesting things about our own thinking…

Kelly E. Matthews, Alison Cook-Sather, Anita Acai, Sam Lucie Dvorakova, Peter Felten, Elizabeth Marquis & Lucy Mercer-Mapstone (2019) “Toward theories of partnership praxis: an analysis of interpretive framing in literature on students as partners”. In: teaching and learning, Higher Education Research & Development, 38:2, 280-293, DOI: 10.1080/07294360.2018.1530199

Reinholz et al. (2021)’s eight most used theories of change and how they relate to each other in my head

Leave a reply

I’ve been playing with this figure (inspired by the Reinholz et al. 2021 article) for a while now for the iEarth/BioCeed Leading Educational Change course, where we try to look at our change project through many different lenses in order to find out which ones are most relevant to help us shape and plan the process. In building this figure, I am trying to figure out how the different perspectives overlap and differ. But since there is a huge amount of information in this one figure and it might be slightly overwhelming, here is an animated version (edit: which, apparently, only starts moving if you click on the gif. No idea why, maybe it’s too large?). The gif builds over 25 seconds, and then it shows the still, finished image for 25 seconds. Not sure if this is the best option; I was also considering doing it as narrated slides. But not right now…

Reinholz, D., White, I., & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education, 8(37), 1 – 22. DOI: https://doi.org/10.1186/s40594-021-00291-2

 

 

Using student evaluations of teaching to actually improve teaching (based on Roxå et al., 2021)

Leave a reply

There are a lot of problems with student evaluations of teaching, especially when they are used as a tool without reflecting on what they can and cannot be used for. Heffernan (2021) finds them to be sexist, racist, prejudiced and biased (my summary of Heffernan (2021) here). There are many more factors that influence whether or not students “like” courses, for example whether they have prior interested in the topic — Uttl et al. (2013) investigate the interest in a quantitative vs non-quantitative course at a psychology department and find a difference in interest of nearly six standard deviations! Even the weather on the day a questionnaire is submitted (Braga et al., 2014), or the “availability of cookies during course sessions” (Hessler et al., 2018) can influence student assessment of teaching. So it is not surprising that in a meta-analysis, Uttl et al. (2017) find “no significant correlations between the [student evaluations of teaching] ratings and learning” and they conclude that “institutions focused on student learning and career success may want to abandon [student evaluation of teaching] ratings as a measure of faculty’s teaching effectiveness”.

But just because student evaluations of teaching might not be a good tool for summative assessment of quality, especially when used out of context, that does not mean they can’t be a useful tool for formative purposes. Roxå et al. (2021) argue that the problem is not the data in itself, but the way it is used, and suggest using them — as academics do every day with all kinds of data — as basis for a critical discourse, as a tool to drive improvement of teaching. They suggest also changing the terminology from “student rating of teaching” to “course evaluations”, to move the focus away from pretending to be able to measure quality of teaching, towards focussing on improving teaching.

In that 2021 article, Roxå et al. present different way to think about course evaluations, supported by a case study from the Faculty of Engineering at Lund University (LTH; which is where I work now! :-)). At LTH, the credo is that “more and better conversations” will lead to better results — in the context of the Roxå et al. (2021) article meaning that more and better conversations between students and teachers will lead to better learning. “Better” conversations are deliberate, evidence-based and informed by literature.

At LTH, the backbone for those more and better conversations are standardised course evaluations run at the end of every course. The evaluations are done using a standard tool, the “course experience questionnaire”, which focusses on the elements of teaching and learning that students can evaluate: their own experiences, for example if they perceived goals as clearly defined, or if help was provided. It is LTH policy that results of those surveys cannot influence career progressions; however, a critical reflection on the results is expected, and a structured discussion format has been established to support this:

The results from those surveys are compiled into a working report that includes the statistics and any free-text comments that an independent student deemed appropriate. This report is discussed in a 30-45 min lunch meeting between the teacher, two students, and the program coordinator. Students are recruited and trained specifically for their role in those meetings by the student union.

After the meeting and informed by it, each of the three parties independently writes a response to the student ratings, including which next steps should be taken. These three responses together with the statistics then form the official report that is being shared with all students from the class.

The discourse and reflection that is kick-started with the course evaluations, structured discussions and reporting is taken further by pedagogical trainings. At LTH, 200 hours of training are required for employment or within the first 2 years, and all courses include creating a written artefact (and often this needs to be discussed with critical friends from participants’ departments before submission) with the purpose of make arguments about teaching and learning public in a scholarly report, contributing to institutional learning. LTH also rewards excellence in teaching, which is not measured by results of evaluations, but the developments that can be documented based on scholarly engagement with teaching, as evidenced for example by critical reflection of evaluation results.

At LTH, the combination of carefully choosing an instrument to measure student experiences, and then applying it, and using the data, in a deliberate manner has led to a consistent increase of student evaluations of the last decades. Of course, formative feedback happening throughout the courses pretty much all the time will also have contributed. This is something I am wondering about right now, actually: What is the influence of, say, consistently done “continue, start, stop” feedbacks as compared to the formalized surveys and discussions around them? My gut feeling is that those tiny, incremental changes will sum up over time and I am actually curious if there is a way to separate their influence to understand their impact. But that won’t happen in this blogpost, and it also doesn’t matter very much: it shouldn’t be an “either, or”, but an “and”!

What do you think? How are you using course evaluations and formative feedback?

Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Economics of Education Review, 41, 71-88.

Heffernan, T. (2021). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education, 1-11.

Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., … & Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064-1072.

Roxå, T., Ahmad, A., Barrington, J., Van Maaren, J., & Cassidy, R. (2021). Reconceptualizing student ratings of teaching to support quality discourse on student learning: a systems perspective. Higher Education, 83(1), 35-55.

Uttl, B., White, C. A., & Morin, A. (2013). The numbers tell it all: students don’t like numbers!. PloS one, 8(12), e83443.

Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.

Thinking about theories of change (based on Reinholz et al., 2021)

1 Reply

I’ve spent quite some time thinking about how to apply theories of change to changing learning and teaching culture (initially in the framework of the iEarth/BioCEED course on “leading educational change”, but more and more beyond that, too). Kezar & Holcombe (2019) say we should use several theories of change simultaneously to make things happen, and Reinholz et al. (2021), describe the eight theories of change that are most commonly used in STEM, so the most pragmatic approach for me was to consider those eight. As I’ve been discussing and applying those theories of change in practice, my thinking about them has developed a bit, and so this is how they work in my head for now (also see figure above and below; it’s the same one).

As a general mindset, it is helpful to start out from what is good already (or at least kinda working) and use that to build upon, rather than tearing everything down and starting from scratch: This is the “Appreciative Inquiry” approach in a nutshell, and it makes sense intuitively, especially when the change isn’t coming from within (for myself, I kinda like the “forget everything and start from scratch” approach) but in the form of a boss, or an academic developer, or a teacher. This appreciative inquiry approach should be considered in the planning phase of any change, but also as a general principle throughout, so we keep building on what’s positive.

“Communities of Practice” is the framework feels most natural to me, and about which I’ve read the most, so this is how I naturally think about culture and changing culture. In a community of practice, people have a common interest which they practice together in a community. The community includes different legitimate roles: not everybody needs to participate and contribute equally or in the same way, or even be fully part of the community to be accepted and appreciate (see figure above/below). There are also legitimate trajectories, i.e. ways to increase or decrease involvement as new people enter or other people leave (see the people skiing into and out of the community in the figure). Objects foster exchange within (tuning fork in the figure) and across (book and violin in the figure) community boundaries, because they are manifestations of thoughts and practice that can be transferred, re-negotiated and modified according to whatever is needed.

Communities of practice have different stages from when they first form until they eventually die, and there are design principles that can help when cultivating communtities of practice, for example to make sure participation is voluntary, there is opportunity for dialogue within and across the communities’ boundaries, and the community is nurtured by someone facilitating regular interactions and new input. In this way, I think of communities of practice as a way to co-create learning and teaching situations, making sure everybody can play the role they would like to play — be who they want to become — and take on as much ownership of the community and the change as they want.

Other theories of change address different aspects that I want to integrate in and add to my thinking about communities of practice:

  • What is it that motivates individuals to do things in the first place? Generally, people are more likely to act on something if they want it and it is likely they’ll get it (-> Expectancy Value). This is depicted in the figure above/below as the considerations one might have before joining a meeting: How much time will I spend there, and is that time commitment worth the outcome I expect? All other things being the same, coffee might make it more appealing to go.
  • No matter how good an idea is, people are not equally likely to jump on an innovation right away. There are distinct stages of adaption, and different “types” of people are likely to adapt in different stages: Knowing about great new ideas does not make everybody want to try them out, so just letting people know is not going to convince everybody; many people might have to see successful ideas implemented by many others before they even consider them for themselves. (-> Diffusion of innovation)
  • Teacher thinking about change related to what & how to teach, who to teach and teach with, and education in general, is influenced by different contexts. These contexts include the personal context (demographics, nature & extent of preparation to teach, types & length of teaching experience, types and length of continued learning, subject & general), system context (rules and regulations, traditions, expectations, schedules, available funding and materials, physical space, subject area), and the general context. (-> Teacher-Centered Systemic Reform)
  • For a team to learn, the whole system needs to be considered: each individual needs to challenge their prejudices, assumptions, and mental models; and strive for personal growth and mastery, only then can a shared vision be developed and worked towards by a whole team. (-> Systems Theory)
  • In addition to people (goals, needs, agency) and symbols (beliefs and ways to communicate them) similarly to what is described above, it is often helpful to consider structures (roles, routines, incentives), and power distribution (hierarchies, coalitions, …) (-> Four Frames)

Lastly, there are three stages a person or community must go through in order to change successfully: “unfreezing” in order to create motivation for change (e.g. by realising dissatisfaction, and by feeling relatively certain that change is possible), “changing” (cognitively redefining based on feedback), and “refreezing” (making sure that the new normal is congruent with how the person wants to see themself and with the community) what should stay. (-> Paulsen & Feldmann)

And here is all of that in one figure! And maybe this figure is not so useful as a boundary object to share ideas from my brain to yours, but at least it really helped me structuring my thinking, and I am more than happy to discuss!

Kezar, A., & Holcombe, E. (2019). Leveraging Multiple Theories of Change to Promote Reform: An Examination of the AAU STEM Initiative. Educational Policy. DOI:https://doi.org/10.1177/0895904819843594

Reinholz, D., White, I., & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education, 8(37), 1 – 22. DOI: https://doi.org/10.1186/s40594-021-00291-2