Using peer feedback to improve students’ writing (Currently reading Huisman et al., 2019)

1 Reply

I wrote about involving students in creating assessment criteria and quality definitions for their own learning on Thursday, and today I want to think a bit about involving students also in the feedback process, based on an article by Huisman et al. (2019) on “The impact of peer feedback on university students’ academic writing: a Meta-Analysis”. In that article, the available literature on peer-feedback specifically on academic writing is brought together, and it turns out that across all studies, peer feedback does improve student writing, so this is what it might mean for our own teaching:

Peer feedback is as good as teacher feedback

Great news (actually, not so new, there are many studies showing this!): Students can give feedback to each other that is of comparable quality than what teachers give them!

Even though a teacher is likely to have more expert knowledge, which might make their feedback more credible to some students (those that have a strong trust in authorities), it also makes it more relevant to other students, and there is no systematic difference between improvement after peer feedback and feedback from teaching staff. But to alleviate fears related to the quality of peer feedback is to use peer feedback purely (or mostly) formative, whereas the teacher does the assessment themselves.

Peer feedback is good for both giver and receiver

If we as teachers “use” students to provide feedback to other students, it might seem like we are pushing part of our job on the students. But: Peer feedback improves writing both for the students giving it as well as for the ones receiving it! Giving feedback means actively engaging with the quality criteria, which might improve future own writing, and doing peer feedback actually improves future writing more than students just doing self-assessment. This might be, for example, because students, both as feedback giver and receiver, are exposed to different perspectives on and approaches towards the content. So there is actual benefit to student learning in giving peer feedback!

It doesn’t hurt to get feedback from more than one peer

Thinking about the logistics in a classroom, one question is whether students should receive feedback from one or multiple peers. Turns out, in the literature it is not (significantly) clear whether it makes a difference. But gut feeling says that getting feedback from multiple peers creates redundancies in case quality of one feedback is really low, or the feedback isn’t actually given. And since students also benefit from giving peer feedback, I see no harm in having students give feedback to multiple peers.

A combination of grading and free-text feedback is best

So what kind of feedback should students give? For students receiving peer feedback, a combination of grading/ranking and free-text comments have the maximum effect, probably because it shows how current performance relates to ideal performance, and also gives concrete advise on how to close the gap. For students giving feedback, I would speculate that a combination of both would also be the most useful, because then they need to commit to a quality assessment, give reasons for their assessment and also think about what would actually improve the piece they read.

So based on the Huisman et al. (2019) study, let’s have students do a lot of formative assessment on each other*, both rating and commenting on each other’s work! And to make it easier for the students, remember to give them good rubrics (or let them create those rubrics themselves)!

Are you using student peer feedback already? What are your experiences?

*The Huisman et al. (2019) was actually only on peer feedback on academic writing, but I’ve seen studies using peer feedback on other types of tasks with similar results, and also I don’t see why there would be other mechanisms at play when students give each other feedback on things other than their academic writing…

Bart Huisman, Nadira Saab, Paul van den Broek & Jan van Driel
(2019) The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis, Assessment & Evaluation in Higher Education, 44:6, 863-880, DOI: 10.1080/02602938.2018.1545896

Co-creating rubrics? Currently reading Fraile et al. (2017)

Leave a reply

I’ve been a fan of using rubrics — tables that contain assessment criteria and a scale of quality definitions for each — not just in a summative way to determine grades, but in a formative way to engage students in thinking about learning outcomes and how they would know when they’ve reached them. Kjersti has even negotiated rubrics with her class, which she describes and discusses here. And now I read an article on “Co-creating rubrics: The effects on self-regulated learning, self-efficacy and performance of establishing assessment criteria with students” by Fraile et al. (2017), which I will summarise below.

Fraile et al. (2017) make the argument that — while rubrics are great for (inter-)rater reliability and many other reasons, students easily perceive them as external constraints that dampen their motivation and might lead to shallow approaches to learning, not as help for self-regulated deep learning. But if students were involved in creating the rubric, they might feel empowered and more autonomous because they are now setting their own goals and monitoring their performance against those, thus using it in ways that actually supports their learning.

This argument is then tested in a study on sports students, where a treatment group co-creates rubrics, whereas a control group uses those same rubrics afterwards. Co-creation of the rubric meant that after an introduction to the content by the teacher, students listed criteria for the activity and then discussed them in small groups. Criteria were then collected and clustered and reduced down to about eight, for which students, in changing groups, then produced two extreme quality definitions for each. Finally, the teacher compiled everything into a rubric and got final approval from the class.

So what happened? All the arguments above sounded convincing, however, results of the study are not as clear-cut as one might have hoped. Maybe the intervention wasn’t long enough or the group of students was too small to make results significant? But what does come out is that in thinking aloud protocols, the students who co-created the rubrics were reporting more self-regulated learning. They also performed better on some of the assessed tasks. And they reported more positive perceptions of rubrics, especially of transparency and understanding of criteria.

What do we learn from this study? At least that all indications are that co-creating rubrics might be beneficial to student learning, and that no drawbacks came to light. So it seems to be a good practice to adopt, especially when we are hoping for benefits beyond what was measured here, for example in terms of students feeling ownership for their own learning etc..

Fraile, J., Panadero, E., & Pardo, R. (2017). Co-creating rubrics: The effects on self-regulated learning, self-efficacy and performance of establishing assessment criteria with students. Studies in Educational Evaluation, 53, 69-76.

Teaching to improve research skills? Thinking about Feldon et al. (2011)

Leave a reply

When graduate students teach, they acquire important research skills, like generating testable hypotheses or designing research, more than their peers who “only” do research, according to Feldon et al. (2011), who compared methodolocical skills in research proposals written by graduate students.

This is quite interesting, because while many graduate students enjoy teaching, there are only 24 hours in a day (and 8 in a work day), and teaching is often seen as competing for time with research. But if teaching actually helps develop research skills (for example because the teaching graduate students are practicing those skills over and over again while advising students, whereas the “research only” graduate students are usually working on pre-defined projects without opportunities to practice those skills), this is a good argument to assign a higher status to teaching even in research training. This would not only lead to graduates that have more experience teaching, but that also have stronger research skills. Win-win!

Feldon, D. F., Peugh, J., Timmerman, B. E., Maher, M. A., Hurst, M., Strickland, D., … & Stiegelmeyer, C. (2011). Graduate students’ teaching experiences improve their methodological research skills. Science, 333(6045), 1037-1039.

Three ways to think about “students as partners”

Leave a reply

As we get started with our project #CoCreatingGFI, we are talking to more and more people about our ideas for what we want to achieve within the project (for a short summary, check out this page), which means that we are playing with different ways to frame our understanding of co-creation and students as partners (SaP).

For the latter, I just read an article by Matthews et al. (2019) that identifies three ways that SaP is commonly being written about. Reading this article was really useful, because it made me realise that I have been using aspects of all three, and now I can more purposefully choose in which way I want to frame SaP for each specific conversation I am having.

In the following, I am presenting the three different perspectives and commenting on how they relate to how I’ve been talking — and thinking — about SaP.

Imagining through Metaphors

Metaphors are figures of speech where a description is applied to something it isn’t literally applicable to, but where it might help to imagine a different (in this case, desired) state.

“Students as partners” as a metaphor evokes quite strong reactions occasionally, because it can be perceived as a complete loss of power, authority and significance by teachers; and likewise as too much work, responsibility, stress by students. We moved away from “students as partners” as a metaphor and towards “co-creation”, because when speaking about “students as partners”, we were constantly trying to explain who the students were partnering with, and what “partnership” would mean in practice. So while we were initially attracted to the metaphor and the philosophy behind it, it ended up not working well in our context.

Speaking about the “student voice”, on the other hand, is something that I’m still doing. To me, it implies what Matthews et al. (2019) describe: students powerfully and actively participating in conversations, and actually being heard. But they also warn that this metaphor can lead to structures in which power sharing becomes less likely, which I can also see: if we explicitly create opportunities to listen to students, it becomes easy to also create other situations in which there explicitly is no space for students.

Building on concepts

When grounding conversations on accepted concepts from the literature, it makes it a lot easier to argue for them and to make sure they make sense in the wider understanding in the field.

In our proposal for Co-Create GFI, we very explicitly build all our arguments on the concept of “communities of practice”. Maybe partly because I was in a very bad Wenger phase at around that time, but mostly because it gave us language and concepts to describe our goal (teachers working together in a community on a shared practice), because it gave us concrete steps for how to achieve that and what pitfalls to avoid.

Also in that proposal as well as in our educational column in oceanography, we use “student engagement” as the basis for the co-creation we are striving for. In our context, there is agreement that students should be engaged and that teachers should work to support student engagement, so starting from this common denominator is a good start into most conversations.

Another concept mentioned by Matthews et al. (2019) are “threshold concepts”, which isn’t a concept we have used in our own conversations about SaP, but which I found definitely helpful to consider when thinking about reactions towards the idea of SaP.

Matthews et al. (2019) point out that while building on concepts can be grounding and situating the way I describe above, it can also be disruptive.

Drawing on Constructs

Of the three ways of talking about SaP, this is the one we’ve used the least. Constructs are tools to help understand behaviour by basically putting a label on a drawer, such as identity, power, or gender. Looking at SaP through the lens of different constructs can help see reality in a different way and change our approach to it, or as Matthews et al. (2019) say: “revealing can lead to revisiting”.

I know it’s not the intention of the article, but I am wondering if taking on that lens just for fun might not reveal new and interesting things about our own thinking…

Kelly E. Matthews, Alison Cook-Sather, Anita Acai, Sam Lucie Dvorakova, Peter Felten, Elizabeth Marquis & Lucy Mercer-Mapstone (2019) “Toward theories of partnership praxis: an analysis of interpretive framing in literature on students as partners”. In: teaching and learning, Higher Education Research & Development, 38:2, 280-293, DOI: 10.1080/07294360.2018.1530199

Reinholz et al. (2021)’s eight most used theories of change and how they relate to each other in my head

Leave a reply

I’ve been playing with this figure (inspired by the Reinholz et al. 2021 article) for a while now for the iEarth/BioCeed Leading Educational Change course, where we try to look at our change project through many different lenses in order to find out which ones are most relevant to help us shape and plan the process. In building this figure, I am trying to figure out how the different perspectives overlap and differ. But since there is a huge amount of information in this one figure and it might be slightly overwhelming, here is an animated version (edit: which, apparently, only starts moving if you click on the gif. No idea why, maybe it’s too large?). The gif builds over 25 seconds, and then it shows the still, finished image for 25 seconds. Not sure if this is the best option; I was also considering doing it as narrated slides. But not right now…

Reinholz, D., White, I., & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education, 8(37), 1 – 22. DOI: https://doi.org/10.1186/s40594-021-00291-2

 

 

How local field laboratories can enhance student learning – first thoughts

Leave a reply

One of iEarth’s stated goals is to develop “local field laboratories” at at least three out of its four member institutions: UiB, UiO, UiT, and UNIS. But what exactly a “local field laboratory” is, and why it actually should enhance learning, is yet to be figured out. In a discussion with iEarth colleagues yesterday, we talked about many benefits of local field laboratories (a term which, again, isn’t clearly defined yet). I am elegantly skipping over the step 1 of the action plan, which would be to do a comprehensive literature search on the topic, and am just documenting my own thoughts after that meeting.

Let’s start out from what we know is necessary for intrinsic motivation, and hence for student learning, namely continuously feeling autonomy, competence, and relatedness (Ryan & Deci, 2000). How can we use interdisciplinary, local field sites to create conditions in which those are experienced?

— Note: ultimately, whether or not those conditions are created will of course depend on how a course is designed and conducted much more than on where it happens! —

Fostering a feeling of autonomy

A feeling of autonomy means that students feel that they have (some) choice over what they do when and where. This can mean many things, of course, and a couple of ideas are provided here.

How a local field laboratory can help foster a feeling of autonomy

Autonomous access: If the local field laboratory is close to campus, students can access (and leave!) it by themselves, possibly by foot, bike, car, public transport, rather than for example a plane. This means they can – at least to a degree – adapt it to their needs: arrive a bit earlier or leave a bit later. It also means that if that they really want to leave a situation they are not comfortable in, they can do that at any time (and are not stuck in some remote location with whoever makes them uncomfortable).

Depending on the setup, it might be possible to access the local field laboratory outside of normal hours, for example for students to catch up on experiments that they missed because of sickness, or to do extra projects because they are curious. This also gives a lot of flexibility to accommodate students that cannot be present during scheduled times.

Low psychological threshold: Field trips – especially the first one to a new location or with a new group – can be scary situations. If the local field laboratory is located close to campus, students stay in an environment where they are familiar with the general culture, language, … This likely means that they feel more confident, and thus more autonomous, in that setting.

Low financial burden: In a local field laboratory close to campus, students are likely to already have appropriate personal equipment (for example a rain jacket appropriate for local climate) rather than having to purchase it for a new-to-student-and-never-to-be-visited-again location. This lowers one potential threshold for participation, giving the students autonomy.

Accessibility: A local field laboratory close to campus is more accessible than having to hike and carry equipment in difficult conditions. A local field laboratory close to campus is NOT accessible to everyone just by virtue of its location though, but it might be easier to make it such.

How using the same local field laboratory with many different disciplines and courses can help foster a feeling of autonomy

More choice of potential research questions: If the local field laboratory is used by many different disciplines or courses, there will be more equipment available for experiments, and more diverse data from previous experiments (if there is a good data storage system). This makes the potential student research questions more interesting and provides a wider choice (if the teachers are flexible enough to allow it).

Accessibility: As stated above, accessibility does not happen by itself, but needs to be considered and planned for. But if there are many students using the same local field laboratory, there might be more resources invested into making sure that the local field laboratory is actually accessible for everybody, and also teachers can build on other teacher’s experiences and good ideas.

Fostering a feeling of competence

Feeling competent means receiving positive feedback: Either external through teachers, friends, family, or an audience on social media, or just by succeeding at doing something.

How a local field laboratory can help fostering a feeling of competence

Transfer: Learning is always situated in a specific context, and the transfer to other contexts is not easy. If the local field laboratory is located close to campus, students can transfer more easily into their own life as it is the same type of environment they spend their whole lives in, thus re-prompting the topic they learned at in that environment over and over again, making them see the world around them with the eyes of an expert – an experience of competence!

Not only one-off: Since the local field laboratory is so close to campus, students can revisit the lab if they want and either repeat or do more. They can also bring friends and/or family to show what they have learned, and have their expert status confirmed. More training in being in the lab and talking about lab content is always practising both lab and science communication skills.

A local field laboratory close to campus can also be used as a red thread throughout the curriculum: the lab can be visited repeatedly over several courses, each time going into more depth, building more competence.

How using the same local field laboratory with many different disciplines and courses can help fostering a feeling of competence

Relevance & context: In a local field laboratory that is used by other courses and disciplines, students can more easily recognise how their own discipline fits into and contributes to a larger scientific context.

Interdisciplinarity: If the local field laboratory is used as a red thread throughout the curriculum, different aspects of the same site can be explored over several courses (geology, soil, climate, weather, plants, …)), thus building interdisciplinary aspects over time, increasing competence by exploring different facets of the same site.

Fostering a feeling of relatedness

Relatedness is the feeling of being part of a supportive group. That doesn’t mean the group has to be around someone all the time, but they have to know that it is there.

How a local field laboratory can help fostering a feeling of relatedness

Contribution to local community: If the local field laboratory is located close to campus, it is also located in the community where students live. Their research can thus have a direct relevance for their community, which can help them feel more connected both by doing something for the community as well as by sharing their learning with members of that community and getting their feedback.

Reducing the carbon footprint: Doing the slightly less exciting field lab that doesn’t go to an exotic location contributes to lowering our carbon footprint, which students might perceive as their personal contribution to something bigger than themselves.

How using the same local field laboratory with many different disciplines and courses can help fostering a feeling of relatedness

Larger context: If the local field laboratory is set up well, there is an overarching theme of all the measurements that are being taken, so that everybody is contributing to something beyond just doing their laboratory results, but much bigger, beyond their own discipline.

Interdisciplinarity: If several courses are at the field site simultaneously, students get to meet students and staff from other disciplines (formally in course context, or informally over dinner) and build a larger scientific network for themselves.

Other considerations that might be relevant to universities

Of course optimising student learning is not the only consideration that universities have, and it would be naive to assume that it was. So here are a couple of other relevant considerations:

Benefits of local field laboratories

  • lower travel costs
  • lower risks connected with long travel or dangerous field sites that the university might have to mitigate
  • easier logistics because of shorter transport that isn’t going across borders or oceans
  • lower carbon footprint!!
  • the field laboratory can be used for outreach “in the field”, inviting people into authentic research situations

Benefits of using the same local field laboratory with many different disciplines and courses can help

  • synergies: using equipment, buildings, … for multiple purposes
  • easier logistics since everything just needs to go to one place
  • red thread in curriculum: teachers meet (formally or informally), talk more, improve coherence between courses / find more interesting interdisciplinary questions

Is that really the full story?

Of course, many of these arguments are just one side of a coin, and local field sites might be best suited to undergraduate education and less so for advanced courses. Maybe one of the learning outcomes is for students to learn about dealing with logistics in a part of the world where everything works differently from what they are used to, and where they don’t speak the language. Or some things just cannot be taught in a certain area because that process just does not happen there. But then those arguments should be made specifically, and weight against the benefits listed above. But I think it’s definitely worthwhile to consider local field laboratories as an alternative to many established field trips to far-away locations: for carbon-footprint reasons just as much as for all the reasons listed above!

What are your thoughts on local field laboratories? And what references should I start with when I finally will have the time to start reading on the topic?

Reference

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1), 68.

How much of the work should the teacher vs the student do? Teaching as a dance, inspired by Joe Hoyle

Leave a reply

This week I spent in a really interesting position: Sitting in the back of a workshop on “introduction to teaching and learning in higher education”, occasionally giving inputs, for example on microaggressions or Universal Design for Learning. And, this morning, about dance as a metaphor for learning and teaching.

I first came across this metaphor in Joe Hoyle’s blogpost on “how much of the work should you do?”. In there, he argues that “in a dance, both parties need to do half of the work but one party does have to lead. Likewise, in a class, both parties need to do half of the work but one party does have to lead. As the teacher, you are the one who has to lead. And, it is that leading that will encourage your students to get up and do their half of the work so that the class will go beautifully well every single class session”. For me, this metaphor works beautifully, not only about how much “work” people should put in, but also that there are different skills involved in teaching/leading and learning/following: the most excellent lead can only do so much if there isn’t a willingness to (learn how to) follow, and likewise the best follower cannot do much without a strong lead (or they might eventually even start leading themselves out of the follower role. I actually took up Lindy Hop, where you change partners all the time, with the explicit goal of training to not take over the lead but let myself be led by people who aren’t very good at leading, and adapting to different styles of (not) leading, hoping that I could transfer that into my professional life. Worked only so-so ;-)).

What I also really like is that in dance, it is becoming much more common that lead and follower switch roles — and this is where I see big potential to expand the metaphor towards co-creating learning. The lead can give the follower the chance to do “turns and stuff” (here my language to talk about dance in English is breaking down) by themselves, which might be compared to giving students a little choice, for example letting students do think-pair-share, where they get the opportunity to do something by themselves for a little while, but in a safe and controlled environment (see lower levels on our “co-creating learning in oceanography” framework). As we move higher up in the framework, we give students more freedom, but also more responsibility, until at the very top, we might actually consider giving up the lead and “just” follow the students’ lead. Which, again, doesn’t say about the amount of work or skill that goes into either learning or teaching, just about who takes on the lead.

So yeah, I think this is a really nice metaphor for teaching and learning :)

P.S.: For a beautiful example of how both leading and following takes enormous skill, check out the youtube video below.

Guest post by Chris Bore on #WaveWatching

Leave a reply

Chris Bore is one of the most loyal readers of my blog and has been for a long time, and now he wrote a beautiful post about #WaveWatching over on his own blog, and gave me permission to repost here. Thank you for loving #WaveWatching as much as I do, Chris!

A few years back I found oceanographer Mirjam Glessmer’s blog ‘Wave Watching’:

https://mirjamglessmer.com/wave-watching/

which is just what it says: a fascinating and insightful blog about watching waves – and what we can learn from doing so, not only about waves but about what they traversed, reflected off, diffracted around, broke over…

It spoke to me particularly because I was then watching waves almost obsessively: ripples on puddles, waves on our local lake, splashes from moorhens and coots and ducks on the canal; sea waves, coastal waves, every kind of wave. I wouldn’t quite say it risked losing me friends but people certainly got used to walking on and eventually looking back surprised to see me stopped staring at some interesting wave phenomenon. Continue reading →

Using student evaluations of teaching to actually improve teaching (based on Roxå et al., 2021)

Leave a reply

There are a lot of problems with student evaluations of teaching, especially when they are used as a tool without reflecting on what they can and cannot be used for. Heffernan (2021) finds them to be sexist, racist, prejudiced and biased (my summary of Heffernan (2021) here). There are many more factors that influence whether or not students “like” courses, for example whether they have prior interested in the topic — Uttl et al. (2013) investigate the interest in a quantitative vs non-quantitative course at a psychology department and find a difference in interest of nearly six standard deviations! Even the weather on the day a questionnaire is submitted (Braga et al., 2014), or the “availability of cookies during course sessions” (Hessler et al., 2018) can influence student assessment of teaching. So it is not surprising that in a meta-analysis, Uttl et al. (2017) find “no significant correlations between the [student evaluations of teaching] ratings and learning” and they conclude that “institutions focused on student learning and career success may want to abandon [student evaluation of teaching] ratings as a measure of faculty’s teaching effectiveness”.

But just because student evaluations of teaching might not be a good tool for summative assessment of quality, especially when used out of context, that does not mean they can’t be a useful tool for formative purposes. Roxå et al. (2021) argue that the problem is not the data in itself, but the way it is used, and suggest using them — as academics do every day with all kinds of data — as basis for a critical discourse, as a tool to drive improvement of teaching. They suggest also changing the terminology from “student rating of teaching” to “course evaluations”, to move the focus away from pretending to be able to measure quality of teaching, towards focussing on improving teaching.

In that 2021 article, Roxå et al. present different way to think about course evaluations, supported by a case study from the Faculty of Engineering at Lund University (LTH; which is where I work now! :-)). At LTH, the credo is that “more and better conversations” will lead to better results — in the context of the Roxå et al. (2021) article meaning that more and better conversations between students and teachers will lead to better learning. “Better” conversations are deliberate, evidence-based and informed by literature.

At LTH, the backbone for those more and better conversations are standardised course evaluations run at the end of every course. The evaluations are done using a standard tool, the “course experience questionnaire”, which focusses on the elements of teaching and learning that students can evaluate: their own experiences, for example if they perceived goals as clearly defined, or if help was provided. It is LTH policy that results of those surveys cannot influence career progressions; however, a critical reflection on the results is expected, and a structured discussion format has been established to support this:

The results from those surveys are compiled into a working report that includes the statistics and any free-text comments that an independent student deemed appropriate. This report is discussed in a 30-45 min lunch meeting between the teacher, two students, and the program coordinator. Students are recruited and trained specifically for their role in those meetings by the student union.

After the meeting and informed by it, each of the three parties independently writes a response to the student ratings, including which next steps should be taken. These three responses together with the statistics then form the official report that is being shared with all students from the class.

The discourse and reflection that is kick-started with the course evaluations, structured discussions and reporting is taken further by pedagogical trainings. At LTH, 200 hours of training are required for employment or within the first 2 years, and all courses include creating a written artefact (and often this needs to be discussed with critical friends from participants’ departments before submission) with the purpose of make arguments about teaching and learning public in a scholarly report, contributing to institutional learning. LTH also rewards excellence in teaching, which is not measured by results of evaluations, but the developments that can be documented based on scholarly engagement with teaching, as evidenced for example by critical reflection of evaluation results.

At LTH, the combination of carefully choosing an instrument to measure student experiences, and then applying it, and using the data, in a deliberate manner has led to a consistent increase of student evaluations of the last decades. Of course, formative feedback happening throughout the courses pretty much all the time will also have contributed. This is something I am wondering about right now, actually: What is the influence of, say, consistently done “continue, start, stop” feedbacks as compared to the formalized surveys and discussions around them? My gut feeling is that those tiny, incremental changes will sum up over time and I am actually curious if there is a way to separate their influence to understand their impact. But that won’t happen in this blogpost, and it also doesn’t matter very much: it shouldn’t be an “either, or”, but an “and”!

What do you think? How are you using course evaluations and formative feedback?

Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Economics of Education Review, 41, 71-88.

Heffernan, T. (2021). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education, 1-11.

Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., … & Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064-1072.

Roxå, T., Ahmad, A., Barrington, J., Van Maaren, J., & Cassidy, R. (2021). Reconceptualizing student ratings of teaching to support quality discourse on student learning: a systems perspective. Higher Education, 83(1), 35-55.

Uttl, B., White, C. A., & Morin, A. (2013). The numbers tell it all: students don’t like numbers!. PloS one, 8(12), e83443.

Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.

Thinking about theories of change (based on Reinholz et al., 2021)

2 Replies

I’ve spent quite some time thinking about how to apply theories of change to changing learning and teaching culture (initially in the framework of the iEarth/BioCEED course on “leading educational change”, but more and more beyond that, too). Kezar & Holcombe (2019) say we should use several theories of change simultaneously to make things happen, and Reinholz et al. (2021), describe the eight theories of change that are most commonly used in STEM, so the most pragmatic approach for me was to consider those eight. As I’ve been discussing and applying those theories of change in practice, my thinking about them has developed a bit, and so this is how they work in my head for now (also see figure above and below; it’s the same one).

As a general mindset, it is helpful to start out from what is good already (or at least kinda working) and use that to build upon, rather than tearing everything down and starting from scratch: This is the “Appreciative Inquiry” approach in a nutshell, and it makes sense intuitively, especially when the change isn’t coming from within (for myself, I kinda like the “forget everything and start from scratch” approach) but in the form of a boss, or an academic developer, or a teacher. This appreciative inquiry approach should be considered in the planning phase of any change, but also as a general principle throughout, so we keep building on what’s positive.

“Communities of Practice” is the framework feels most natural to me, and about which I’ve read the most, so this is how I naturally think about culture and changing culture. In a community of practice, people have a common interest which they practice together in a community. The community includes different legitimate roles: not everybody needs to participate and contribute equally or in the same way, or even be fully part of the community to be accepted and appreciate (see figure above/below). There are also legitimate trajectories, i.e. ways to increase or decrease involvement as new people enter or other people leave (see the people skiing into and out of the community in the figure). Objects foster exchange within (tuning fork in the figure) and across (book and violin in the figure) community boundaries, because they are manifestations of thoughts and practice that can be transferred, re-negotiated and modified according to whatever is needed.

Communities of practice have different stages from when they first form until they eventually die, and there are design principles that can help when cultivating communtities of practice, for example to make sure participation is voluntary, there is opportunity for dialogue within and across the communities’ boundaries, and the community is nurtured by someone facilitating regular interactions and new input. In this way, I think of communities of practice as a way to co-create learning and teaching situations, making sure everybody can play the role they would like to play — be who they want to become — and take on as much ownership of the community and the change as they want.

Other theories of change address different aspects that I want to integrate in and add to my thinking about communities of practice:

  • What is it that motivates individuals to do things in the first place? Generally, people are more likely to act on something if they want it and it is likely they’ll get it (-> Expectancy Value). This is depicted in the figure above/below as the considerations one might have before joining a meeting: How much time will I spend there, and is that time commitment worth the outcome I expect? All other things being the same, coffee might make it more appealing to go.
  • No matter how good an idea is, people are not equally likely to jump on an innovation right away. There are distinct stages of adaption, and different “types” of people are likely to adapt in different stages: Knowing about great new ideas does not make everybody want to try them out, so just letting people know is not going to convince everybody; many people might have to see successful ideas implemented by many others before they even consider them for themselves. (-> Diffusion of innovation)
  • Teacher thinking about change related to what & how to teach, who to teach and teach with, and education in general, is influenced by different contexts. These contexts include the personal context (demographics, nature & extent of preparation to teach, types & length of teaching experience, types and length of continued learning, subject & general), system context (rules and regulations, traditions, expectations, schedules, available funding and materials, physical space, subject area), and the general context. (-> Teacher-Centered Systemic Reform)
  • For a team to learn, the whole system needs to be considered: each individual needs to challenge their prejudices, assumptions, and mental models; and strive for personal growth and mastery, only then can a shared vision be developed and worked towards by a whole team. (-> Systems Theory)
  • In addition to people (goals, needs, agency) and symbols (beliefs and ways to communicate them) similarly to what is described above, it is often helpful to consider structures (roles, routines, incentives), and power distribution (hierarchies, coalitions, …) (-> Four Frames)

Lastly, there are three stages a person or community must go through in order to change successfully: “unfreezing” in order to create motivation for change (e.g. by realising dissatisfaction, and by feeling relatively certain that change is possible), “changing” (cognitively redefining based on feedback), and “refreezing” (making sure that the new normal is congruent with how the person wants to see themself and with the community) what should stay. (-> Paulsen & Feldmann)

And here is all of that in one figure! And maybe this figure is not so useful as a boundary object to share ideas from my brain to yours, but at least it really helped me structuring my thinking, and I am more than happy to discuss!

Kezar, A., & Holcombe, E. (2019). Leveraging Multiple Theories of Change to Promote Reform: An Examination of the AAU STEM Initiative. Educational Policy. DOI:https://doi.org/10.1177/0895904819843594

Reinholz, D., White, I., & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education, 8(37), 1 – 22. DOI: https://doi.org/10.1186/s40594-021-00291-2