Tag Archives: literature

Using peer feedback to improve students’ writing (Currently reading Huisman et al., 2019)

I wrote about involving students in creating assessment criteria and quality definitions for their own learning on Thursday, and today I want to think a bit about involving students also in the feedback process, based on an article by Huisman et al. (2019) on “The impact of peer feedback on university students’ academic writing: a Meta-Analysis”. In that article, the available literature on peer-feedback specifically on academic writing is brought together, and it turns out that across all studies, peer feedback does improve student writing, so this is what it might mean for our own teaching:

Peer feedback is as good as teacher feedback

Great news (actually, not so new, there are many studies showing this!): Students can give feedback to each other that is of comparable quality than what teachers give them!

Even though a teacher is likely to have more expert knowledge, which might make their feedback more credible to some students (those that have a strong trust in authorities), it also makes it more relevant to other students, and there is no systematic difference between improvement after peer feedback and feedback from teaching staff. But to alleviate fears related to the quality of peer feedback is to use peer feedback purely (or mostly) formative, whereas the teacher does the assessment themselves.

Peer feedback is good for both giver and receiver

If we as teachers “use” students to provide feedback to other students, it might seem like we are pushing part of our job on the students. But: Peer feedback improves writing both for the students giving it as well as for the ones receiving it! Giving feedback means actively engaging with the quality criteria, which might improve future own writing, and doing peer feedback actually improves future writing more than students just doing self-assessment. This might be, for example, because students, both as feedback giver and receiver, are exposed to different perspectives on and approaches towards the content. So there is actual benefit to student learning in giving peer feedback!

It doesn’t hurt to get feedback from more than one peer

Thinking about the logistics in a classroom, one question is whether students should receive feedback from one or multiple peers. Turns out, in the literature it is not (significantly) clear whether it makes a difference. But gut feeling says that getting feedback from multiple peers creates redundancies in case quality of one feedback is really low, or the feedback isn’t actually given. And since students also benefit from giving peer feedback, I see no harm in having students give feedback to multiple peers.

A combination of grading and free-text feedback is best

So what kind of feedback should students give? For students receiving peer feedback, a combination of grading/ranking and free-text comments have the maximum effect, probably because it shows how current performance relates to ideal performance, and also gives concrete advise on how to close the gap. For students giving feedback, I would speculate that a combination of both would also be the most useful, because then they need to commit to a quality assessment, give reasons for their assessment and also think about what would actually improve the piece they read.

So based on the Huisman et al. (2019) study, let’s have students do a lot of formative assessment on each other*, both rating and commenting on each other’s work! And to make it easier for the students, remember to give them good rubrics (or let them create those rubrics themselves)!

Are you using student peer feedback already? What are your experiences?

*The Huisman et al. (2019) was actually only on peer feedback on academic writing, but I’ve seen studies using peer feedback on other types of tasks with similar results, and also I don’t see why there would be other mechanisms at play when students give each other feedback on things other than their academic writing…


Bart Huisman, Nadira Saab, Paul van den Broek & Jan van Driel
(2019) The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis, Assessment & Evaluation in Higher Education, 44:6, 863-880, DOI: 10.1080/02602938.2018.1545896

Three ways to think about “students as partners”

As we get started with our project #CoCreatingGFI, we are talking to more and more people about our ideas for what we want to achieve within the project (for a short summary, check out this page), which means that we are playing with different ways to frame our understanding of co-creation and students as partners (SaP).

For the latter, I just read an article by Matthews et al. (2019) that identifies three ways that SaP is commonly being written about. Reading this article was really useful, because it made me realise that I have been using aspects of all three, and now I can more purposefully choose in which way I want to frame SaP for each specific conversation I am having.

In the following, I am presenting the three different perspectives and commenting on how they relate to how I’ve been talking — and thinking — about SaP.

Imagining through Metaphors

Metaphors are figures of speech where a description is applied to something it isn’t literally applicable to, but where it might help to imagine a different (in this case, desired) state.

“Students as partners” as a metaphor evokes quite strong reactions occasionally, because it can be perceived as a complete loss of power, authority and significance by teachers; and likewise as too much work, responsibility, stress by students. We moved away from “students as partners” as a metaphor and towards “co-creation”, because when speaking about “students as partners”, we were constantly trying to explain who the students were partnering with, and what “partnership” would mean in practice. So while we were initially attracted to the metaphor and the philosophy behind it, it ended up not working well in our context.

Speaking about the “student voice”, on the other hand, is something that I’m still doing. To me, it implies what Matthews et al. (2019) describe: students powerfully and actively participating in conversations, and actually being heard. But they also warn that this metaphor can lead to structures in which power sharing becomes less likely, which I can also see: if we explicitly create opportunities to listen to students, it becomes easy to also create other situations in which there explicitly is no space for students.

Building on concepts

When grounding conversations on accepted concepts from the literature, it makes it a lot easier to argue for them and to make sure they make sense in the wider understanding in the field.

In our proposal for Co-Create GFI, we very explicitly build all our arguments on the concept of “communities of practice”. Maybe partly because I was in a very bad Wenger phase at around that time, but mostly because it gave us language and concepts to describe our goal (teachers working together in a community on a shared practice), because it gave us concrete steps for how to achieve that and what pitfalls to avoid.

Also in that proposal as well as in our educational column in oceanography, we use “student engagement” as the basis for the co-creation we are striving for. In our context, there is agreement that students should be engaged and that teachers should work to support student engagement, so starting from this common denominator is a good start into most conversations.

Another concept mentioned by Matthews et al. (2019) are “threshold concepts”, which isn’t a concept we have used in our own conversations about SaP, but which I found definitely helpful to consider when thinking about reactions towards the idea of SaP.

Matthews et al. (2019) point out that while building on concepts can be grounding and situating the way I describe above, it can also be disruptive.

Drawing on Constructs

Of the three ways of talking about SaP, this is the one we’ve used the least. Constructs are tools to help understand behaviour by basically putting a label on a drawer, such as identity, power, or gender. Looking at SaP through the lens of different constructs can help see reality in a different way and change our approach to it, or as Matthews et al. (2019) say: “revealing can lead to revisiting”.

I know it’s not the intention of the article, but I am wondering if taking on that lens just for fun might not reveal new and interesting things about our own thinking…


Kelly E. Matthews, Alison Cook-Sather, Anita Acai, Sam Lucie Dvorakova, Peter Felten, Elizabeth Marquis & Lucy Mercer-Mapstone (2019) “Toward theories of partnership praxis: an analysis of interpretive framing in literature on students as partners”. In: teaching and learning, Higher Education Research & Development, 38:2, 280-293, DOI: 10.1080/07294360.2018.1530199

Using student evaluations of teaching to actually improve teaching (based on Roxå et al., 2021)

There are a lot of problems with student evaluations of teaching, especially when they are used as a tool without reflecting on what they can and cannot be used for. Heffernan (2021) finds them to be sexist, racist, prejudiced and biased (my summary of Heffernan (2021) here). There are many more factors that influence whether or not students “like” courses, for example whether they have prior interested in the topic — Uttl et al. (2013) investigate the interest in a quantitative vs non-quantitative course at a psychology department and find a difference in interest of nearly six standard deviations! Even the weather on the day a questionnaire is submitted (Braga et al., 2014), or the “availability of cookies during course sessions” (Hessler et al., 2018) can influence student assessment of teaching. So it is not surprising that in a meta-analysis, Uttl et al. (2017) find “no significant correlations between the [student evaluations of teaching] ratings and learning” and they conclude that “institutions focused on student learning and career success may want to abandon [student evaluation of teaching] ratings as a measure of faculty’s teaching effectiveness”.

But just because student evaluations of teaching might not be a good tool for summative assessment of quality, especially when used out of context, that does not mean they can’t be a useful tool for formative purposes. Roxå et al. (2021) argue that the problem is not the data in itself, but the way it is used, and suggest using them — as academics do every day with all kinds of data — as basis for a critical discourse, as a tool to drive improvement of teaching. They suggest also changing the terminology from “student rating of teaching” to “course evaluations”, to move the focus away from pretending to be able to measure quality of teaching, towards focussing on improving teaching.

In that 2021 article, Roxå et al. present different way to think about course evaluations, supported by a case study from the Faculty of Engineering at Lund University (LTH; which is where I work now! :-)). At LTH, the credo is that “more and better conversations” will lead to better results — in the context of the Roxå et al. (2021) article meaning that more and better conversations between students and teachers will lead to better learning. “Better” conversations are deliberate, evidence-based and informed by literature.

At LTH, the backbone for those more and better conversations are standardised course evaluations run at the end of every course. The evaluations are done using a standard tool, the “course experience questionnaire”, which focusses on the elements of teaching and learning that students can evaluate: their own experiences, for example if they perceived goals as clearly defined, or if help was provided. It is LTH policy that results of those surveys cannot influence career progressions; however, a critical reflection on the results is expected, and a structured discussion format has been established to support this:

The results from those surveys are compiled into a working report that includes the statistics and any free-text comments that an independent student deemed appropriate. This report is discussed in a 30-45 min lunch meeting between the teacher, two students, and the program coordinator. Students are recruited and trained specifically for their role in those meetings by the student union.

After the meeting and informed by it, each of the three parties independently writes a response to the student ratings, including which next steps should be taken. These three responses together with the statistics then form the official report that is being shared with all students from the class.

The discourse and reflection that is kick-started with the course evaluations, structured discussions and reporting is taken further by pedagogical trainings. At LTH, 200 hours of training are required for employment or within the first 2 years, and all courses include creating a written artefact (and often this needs to be discussed with critical friends from participants’ departments before submission) with the purpose of make arguments about teaching and learning public in a scholarly report, contributing to institutional learning. LTH also rewards excellence in teaching, which is not measured by results of evaluations, but the developments that can be documented based on scholarly engagement with teaching, as evidenced for example by critical reflection of evaluation results.

At LTH, the combination of carefully choosing an instrument to measure student experiences, and then applying it, and using the data, in a deliberate manner has led to a consistent increase of student evaluations of the last decades. Of course, formative feedback happening throughout the courses pretty much all the time will also have contributed. This is something I am wondering about right now, actually: What is the influence of, say, consistently done “continue, start, stop” feedbacks as compared to the formalized surveys and discussions around them? My gut feeling is that those tiny, incremental changes will sum up over time and I am actually curious if there is a way to separate their influence to understand their impact. But that won’t happen in this blogpost, and it also doesn’t matter very much: it shouldn’t be an “either, or”, but an “and”!

What do you think? How are you using course evaluations and formative feedback?


Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Economics of Education Review, 41, 71-88.

Heffernan, T. (2021). Sexism, racism, prejudice, and bias: a literature review and synthesis of research surrounding student evaluations of courses and teaching. Assessment & Evaluation in Higher Education, 1-11.

Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., … & Wenk, M. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52(10), 1064-1072.

Roxå, T., Ahmad, A., Barrington, J., Van Maaren, J., & Cassidy, R. (2021). Reconceptualizing student ratings of teaching to support quality discourse on student learning: a systems perspective. Higher Education, 83(1), 35-55.

Uttl, B., White, C. A., & Morin, A. (2013). The numbers tell it all: students don’t like numbers!. PloS one, 8(12), e83443.

Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42.

Just published: Co-Creating Learning in Oceanography

Kjersti and I just had an article published: “Co-Creating Learning in Oceanography” (Glessmer & Daae, 2021)!

In this article, we discuss ways in which to share responsibility for learning between teachers and students on a continuum from “just” actively engaging students towards fully shared responsibility, i.e. “co-creation” or “Students as Partners”. We give 13 different examples from our own praxis, starting from very easy things like how to use multiple-choice questions to promote discussion and critical thinking or giving students the choice of several examples from which they can learn the same content, and gradually work our way up to more and more interesting methods, like for example negotiating rubrics of learning outcomes with students.

The article itself is accompanied by a website where we elaborate on our 13 different examples. Check it out, and let us know what you think! And if you have any experiences with co-creating learning that you would like to share, we would love to hear from you and add a guest post on your experiences to our collection! :)


Reference:

Glessmer, M.S., and K. Daae. 2021. Co-creating learning in oceanography. Oceanography 34(4), https://doi.org/10.5670/oceanog.2021.405.

Students as partners as a threshold concept for students and teachers?

One of my goals as a teacher is to change culture towards responsibility for student learning being shared equally between teachers and students. This is an idea that is met with some resistance, both from students who need to put in more work, and from teachers. An article by Cook-Sather (2014) sheds light on difficulties teachers often experience when letting go of traditional understandings of the relationship between teacher and students, and adopting this new form of collaboration with students:

Taking on a students as partners mindset is described as a threshold concept for teachers: a gateway that, once crossed, opens up a whole new world. While walking through such a portal is transformative and irreversible (evidence of both is given from teacher reflections after they have adopted the new mindset), it is also troublesome. Especially for new teachers who are struggling with legitimacy issues, accepting students as equal partners can be a daunting and difficult process, where students might be perceived as adversaries rather than partners, or as not contributing any new and inspiring thought.

Crossing the threshold might be aided by academic developers inviting for reflection, or by supporting teachers in taking small actions towards giving students more responsibility on a confined and “safe” aspect of the course (note by Mirjam: for some ideas, check out our collection here!). The beginner-level one-on-one setting of teacher and student in partnership (for example working with student representatives) can then, in the long run, be widened to include more students. Providing spaces for reflection, discussion, and revision within and beyond course settings (for example also including educational researchers) can support the transformation towards students as partners.

Reading quotes from teachers struggling to see the benefit of collaborating with students on developing their teaching opened my eyes to struggles with the changing relationship – especially around seeing the student partners as enemies rather than supportive partners — that I did not anticipate for our own application, but that might quite possibly exist. This might be another aspect of threshold concepts – that it is retrospectively difficult to imagine what life was like before crossing the threshold. Therefore, reading this article was a good reminder that supporting reflections on roles, identities, relationships should be an important part of any project if we want to successfully implement students as partners.


Reference:
Alison Cook-Sather (2014) Student-faculty partnership in explorations of
pedagogical practice: a threshold concept in academic development, International Journal for Academic Development, 19:3, 186-198, DOI: 10.1080/1360144X.2013.805694

Kaur & Noman (2020): A study applying self-determination theory on Students as Partners

I love using self-determination theory (Ryan & Deci, 2000) as a framework against which I check all teaching I develop. Is it even possible for students to feel competence, autonomy, relatedness in the environment I am building, or what can I tweak to create conditions in which these contritions for feeling intrinsically motivated are more easily met? Recently, I have taken on a Students as Partners (SaP) approach, and then came across the article by Kaur & Noman (2020) that looks at SaP through the lens of self-determination theory.

The authors take the three categories and divide them into six themes: Autonomy is described as both agency (as having the real chance of contributing and shaping the learning process) and choice; competence is about gaining confidence and thus acting more confidently, as well as being challenged and rising to the challenge. Relatedness is then about both the environment which is inviting and without anxiety, and meaningful, frequent, friendly and open interactions. They say that as a result of the intrinsic motivation that is made possible by meeting these conditions, student engagement will increase.

Looking at the data from two previous studies, the authors find that more than 3/4 of the students reported experiencing agency, which they linked very closely to agency and accountability beliefs of students. 2/3rds of the students also mention choice as very important: they had control of their learning and felt as if they were “the initiators of their learning”. For the category of competence, the results aren’t as strong: less than half of the students reports feeling confident, and less than a third felt challenged. On relatedness, 3/4 of the students report feeling connected and in a warm environment, and almost half of the students felt that they had more meaningful interactions with their teachers.

So what does this mean, and how does it help us? I was most curious about seeing how the authors brought self-determination theory and students as partners together. The numbers themselves are interesting in so far that they tell us something about the two specific courses, but whether or not students feel, for example, appropriately challenged will depend on a lot more factors than on whether or not they are learning as partners, like on the subject, their level of previous knowledge, the actual tasks they are working on, etc.. Just because someone uses students as partners as their framework doesn’t mean that it is implemented perfectly (as with any other framework or method, actually). Also we don’t have anything to compare this to — maybe that’s how students feel about any course, regardless of whether they are partners or not? But I think thinking about self-determination theory in more detail, i.e. what are the different aspects that could contribute to feeling competence, autonomy, and relatedness, and what could help or hinter them, and which of these are more important than others, is useful for improving my teaching practice.


References:

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1), 68.

Kaur, A., & Noman, M. (2020). Investigating students’ experiences of Students as Partners (SaP) for basic need fulfilment: A self-determination theory perspective. Journal of University Teaching & Learning Practice, 17(1), 8.

A review of change theory in STEM higher ed (Reinholz et al., 2021)

More reading for my “leading educational change” course run by iEarth and BioCEED! Today on how change theory has been used in STEM higher education over the last 25 years  (Reinholz et al., 2021), which I am summarising here.

Why are people interested in change theory? In order to change a system (for example to make sure society’s need for more STEM educated people can be met, or to support diversity, equity and inclusion initiatives), it is very helpful to understand what levers to pull. Just making facts available does not lead to change in itself (as we also see for other issues, like climate change). Therefore, funding agencies therefore more and more require explicit theories for how change will happen in projects they fund, and more and more people explicitly lay out their thoughts of what will happen and why before starting a project.

But let’s sort out some confusing terms, or at least how they are being used in this article: “change theory” vs “theory of change”. Change theory is an evidence-based framework to explain mechanisms that drive change. It is general and applicable to many different situations. A theory of change, on the other hand, is specifically about how change will happen in a specific project: What are the goals of the project, what needs to be achieved before as stepping stones, what kind of interventions could get you there, what’s the context, how will you know if you have been successful, and what were your assumptions going into all of this.

Four essential components of a theory of change

A theory of change consists of four essential components:

  • Rationale and assumptions. A rationale is basically a narrated version of how change will happen. It is built on assumptions which need to become explicit. For example, are decision makers going to be convinced by rationally interpreting data, or do they need to be emotionally involved in order to process the data?
  • Context. Change doesn’t happen in a vacuum, and in order to change STEM education, we must consider where it occurs. Who are stakeholders, what are frameworks like, who funds what, who sets learning outcomes, …?
  • Indicators. Working towards an ultimate goal, there are many small milestones to be reached. Indicators tell us about whether progress towards those goals is being made, and according to plan.
  • Interventions. Interventions are what is being done to drive the change. If showing data to people does not make them act, how will we get them to act instead? Maybe by changing incentive structures, or appreciation of certain activities, or more funding, or better communities.

The eight most commonly used theories of change

The authors then did a systematic literature review on articles published on using change theory in STEM higher ed (they ended up reviewing 97 articles in detail), and present the eight theories that were used most often (N = number of articles using a specific theory). I’m summarising Reinholz et al.’s summary here because it’s quite fascinating to see the different ways that people think and talk about change processes.

Communities of Practice (N = 26)

Communities of practice consist of a group of people who have a common topic they care about and work together to learn more about the topic and improve their skills. In a community of practice, learning occurs through participation. In the context of facilitating change, the idea of communities of practice has been used to inspire “communities of change” with the specific purpose to drive change, and “faculty learning communities” to drive teacher professional development.

When communities of practice are used as theory of change, the focus is usually on the community itself (and interventions are typically about creating communities of practice around practices that are supposed to be strengthened), or specific roles within that community, but not on the institutional context in which change is supposed to happen.

I really like this theory of change and have used it in the past — the original Wenger (1998) book has been transformative in my thinking and has great explanatory power for things I observe (and here is a summary of more reading I’ve done about the subject). But I really appreciate the point about the usually missing consideration of the context.

Diffusion of Innovations (N = 19)

From a diffusion of innovation perspective, ideas spread between individuals and adoption occurs in predictable stages (from becoming aware that an innovation exists, considering whether or not to adopt it, deciding to adopt it, trying it, and sticking with it). Innovations that spread well according to this theory have some kind of advantage over current practices but at the same time fit well into current practices and beliefs, are easy to adopt and have a low threshold to be tried on for fit, and can be observed in practice before one needs to make a decision about whether or not one wants to join. There are also different types of adopters which might need to be target differently: from early adopters that are willing to take risks, to late adopters which will only adopt an innovation when there really isn’t much choice left because everybody else is already doing it.

This theory seems a bit oversimplified, and extensions discussed by Reinholz et al. include considerations around how to increase chances that innovations are actually implemented, focussing on the importance of good communication and support when implementing an innovation. However, I think it is useful to consider that people are different in their willingness to adopt new things, and that characteristics of an innovation can make it more or less likely that the innovation is being adopted.

Teacher-Centred Systemic Reform (N = 6)

The essence of teacher-centred systemic reform as a theory of change is that teachers’ beliefs influence teacher practices. Teachers’ beliefs are not necessarily fully informed by knowledge, and they are hard to change by evidence. Nevertheless, this theory is mainly about targeting individual teachers to then, in the long run, change the system.

Maybe I am too much in the “communities of practice” framework of thinking and have been involved in too many faculty development activities that targeted individuals (that were super motivated in a workshop but then did not have a community or systemic support to actually change much), but to me, this theory seems to be less useful than the others. But thinking about beliefs and how hard they are to change is probably a good point to take away from it.

Here is a blog post where I review literature on Teacher-Centered Systemic Reform.

Appreciative Inquiry (N = 4)

Instead of fixing what is wrong, appreciative inquiry starts from what is positive in an organisation and develops that further towards the desired outcomes. There are four steps involved: figuring out what the team’s values are, developing a vision from that, defining goals, and then working towards them.

What I really like about this theory is that it is so positive, and approaching change from a perspective of “there is already a lot of great things happening here, let’s make it even better” feels so much nicer than approaching change as “everything here is horrible, we need to fix things”, and especially when change is driven not in a grass root fashion but somehow from the outside, this is probably a very good approach to make conversations even possible. But somehow I see the danger of not addressing real problems in a determined-enough fashion because we are trying to be so positive. But that might be just me…

Expectancy-Value Theory (N = 4)

I’ve written about expectancy-value theory as a theory explaining motivation to work towards a goal before: People need to feel both that it is actually possible to reach a goal, and that reaching the goal is important to them, to be motivated to put in the required work.

I think this is a helpful framework to consider, but this theory does not give guidance on how people’s self-efficacy can be increased or how the perceived value of a task can be changed. So it’s a good thing to check if what we want people to do is a) feasible and b) probably of value to them, but it doesn’t help much in designing interventions.

Four Frames (N = 4)

The four frames theory looks at an organisation from four different angles which each reveal different aspects of culture: structures (roles, routines, incentives), symbols (beliefs and ways to communicate them), people (individuals’ goals, needs, agency), and power (hierarchies, coalitions, …).

While this theory doesn’t give us actionable advice, it is useful to consider that there are (at least) these four frames to consider to fully understand change and to address it adequately.

Paulsen and Feldman’s General Change Model (N = 4)

This theory was specifically developed to work for change processes in STEM education. According to the theory, change occurs in predictable stages: Unfreezing (when people recognise that something within the current system is not working and they are dissatisfied enough that it motivates them to change), changing (when the actual change takes place in form of acquiring new information, forming new groups, changing routines), and refreezing (when the change is solidified).

What I like about this model is that it explicitly considers the refreezing part — not just changing something, but making sure that in becomes the “new normal”.

Systems Theory (N = 3)

In systems theory, the whole system involved in change is considered, and especially how different parts of that system influence each other. The focus is on a “learning organisation” which continually develops, and which is characterised by systems thinking (assuming that structure influences behaviour), personal mastery, mental models, building shared vision, and team learning.

From the Reinholz et al. article alone, I don’t get a good enough overview over systems theory to understand its full implications, but again it seems like a comprehensive approach, which I appreciate.

My reflections

I read this article, naively hoping it would give me the eight most relevant theories that I should consider when planning theories of change for my own projects. What I got, though, were the eight theories that other people had used the most (at least in published literature on STEM changes), but it felt like, quite possibly, those other people had randomly picked a theory they were familiar with and that sounded reasonable (kind of like what I’ve been doing with communities of practice and self-determination theory myself).

Reading through those eight theories drove home the point which I also describe in my summary of Kezar & Holcombe’s article, that it is important to not only take one specific theory of change, but look at change more comprehensively from different angles: What does motivate people to change, but also what other conditions need to be in place politically, in the organisation, in the community for that change to actually happen, and then to also be sustained?

What I am missing still missing (and that is not a complaint of this article, more a realisation of what I might have to search for or do myself) is a systematic overview of change theories that are out there, and from that a discussion of how and why some might be more applicable to changing STEM education than others. Can’t someone give me a checklist of all the things I should consider to be on the safe side? ;-)


Reinholz, D., White, I., & Andrews, T. (2021). Change theory in STEM higher education: a systematic review. International Journal of STEM Education, 8(37), 1 – 22. DOI: https://doi.org/10.1186/s40594-021-00291-2

Communities of practice

Summarising my reading on “communities of practice”, and my views on how this framework might be useful for thinking about change in our context, for our iEarth/BioCEED-led course on “leading educational change”…

Communities of practice are often used as a model for “learning through participation”, describing how culture influences how knowledge within organisations is built and shared, and aspired to when cultural change is worked towards. Identities, how they develop and how they influence how people behave are central in this framework: for members of a CoP, their membership in this CoP and participation in its shared practice are defining parts of their identity.

The concept of CoPs was developed by Etienne Wenger (1998) and has since been widely adopted. A brief description and definition are given in Wenger (2011):

“Communities of practice are groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly.”

A community of practice (CoP) has three defining characteristics:

  • A shared domain of interest or expertise which brings people together (the expertise does not have to be formally recognised).
  • A community in which people interact regularly, learn together, share experiences and stories, all related to the shared domain of interest.
  • A practice of the shared domain of interest, which the community engages in together, thus developing a shared repertoire of skills, stories, tools, which improves their practice.

What exactly happens in a CoP can take many different forms, for example brainstorming ideas to solve a problem, sharing information, materials, experiences, combining efforts to create synergies, or building comprehensive archives of solutions.

In all cases, negotiating shared meaning is essential: “reification”, the process of creating objects that represent a more abstract concept, helps organise discussions by visualising understanding, and documenting results for future reference. However, theses simplified symbols can also be misunderstood or taken out of context, or take away agency when they are imposed on people who weren’t part of the reification process. It is therefore important that participation and reification always occur together.

Roles in a CoP

Despite CoPs often being informal in nature, in order for people to consistently do things together, they do require someone to take on the role of community coordinator.

What I really like about CoPs as a framework is that — in addition to full members which fully identify with participation in the CoP, there are many legitimate ways to participate without being a full member:

    • people might be on an inbound trajectory: new members are learning to participate in the practice of a community, transforming their identity to become a full member. It changes how they participate in the world and at the same time who they are, what they do, and who they do it with
    • people might also be on an outbound trajectory: students are encouraged to grow, to try on different roles and idetities, and ultimately leave when they’ve found the best fit
    • people might also be brokers: peripherally participating in several CoPs and making connections across CoPs, this way create new meaning. But: “uprootedness is an occupational hazard of brokering”, and organisations are still learning to recognise and appreciate the value of brookers.

In addition to by brokers, flow of information can also happen by artefacts (which, as boundary objects, transport meaning in or out) and through interactions (for example through shared projects with other communities of practice).

Even for people who don’t identify as brokers, membership in several communities of practice is common — often with different domains of expertise, or in different, possibly overlapping communities on the same domain. Wenger (1998) describes how identifying too strongly with one CoP can hinder innovation, and how participation in any kind of learning opportunity will only have lasting effects if the home institution’s culture allows for any new aspects to be integrated with the person’s identity and in the CoP at home. Encouraging multi-membership, for example in a work-place related CoP and one that is focussed on family life, helps not tear people apart.

The existence of a CoP does not mean that there are no conflicts and disagreements: “As a form of participation, rebellion often reveals greater commitment than does passive conformity” (Wenger, 1998). However, newcomers might not lead to as much change as one might think, because their desire to become part of the community might lead to them not wanting to rebel too much too early on.

Developing a CoP

Communities of practice are not necessarily formalised in clubs or committees, they can form spontaneously or be designed purposefully. They typically go through 5 different stages (Wenger et al., 2002).

  • potential: figuring out whether there is enough shared interest and the will to engage in a shared practice
  • coalescence: the CoP is “launched” and members are recruited
  • maturation: the CoP changes from sharing tips & tricks between friends to a formal structure
  • stewardship: the CoP tries to keep on going as people might get tired and boundary conditions change
  • transformation: the CoP might change dramatically or die

When designing a CoP, there are seven principles to consider (Wenger, 2002):

  • Design the community to evolve naturally. A CoP is carried by the voluntary efforts of its members, their interests and goals. All of these might change with time, which will need to also change the community.
  • Create opportunities for open dialogue within and with outside perspectives. Not all expertise is necessarily already available within the community, so seeking it out and welcoming it is helpful
  • Welcome and allow different levels of participation. See different roles above — not everybody has to be a full member to be respected and welcomed.
  • Develop both public and private community spaces. This accommodates members’ different preferred styles and keeps the community open for new ideas.
  • Focus on the value of the community. Reflecting on the value of the community helps sustain motivation.
  • Combine familiarity and excitement. Members need both predictability, e.g. in routines and regular meetings, as well as excitement, e.g. spontaneous events and new input.
  • Nurture a regular rhythm for the community. Interacting and practising together regularly is the backbone of a CoP.

None of these are super surprising, but together they are a nice set of criteria to check against when developing a theory of change.

How communities of practices have been applied to educational change

Wenger (2011) describes three ways how educational practices are affected by communities of practice:

  • Internally: Building communities of practice in which students learn in school
  • Externally: Connecting in-school learning to the actual practice in the real world
  • Over the lifetime of students: Making sure that students remain part of communities of practice when they transition from school into a job and then onwards

Wenger (2011) focusses on how students are involved in communities of practice, not on teachers and their learning, but for the purpose of our own change project, this is equally applicable to teachers. But communities of practice have been used to design and explain change in academia (and also in many other contexts! But my focus here is on educational settings) by many other authors in all three contexts.

Internal

Tinnell et al. (2019) report on the positive impact a specific type of CoP, a faculty learning community (FLC), had at an engineering faculty for both faculty (getting real-time feedback, better student interactions, teaching teamwork, peer collaborations) and students (changed attitudes and effort, better understanding, better teamwork), with the change persisting up to 2 years after the end of the FLC. They attribute the changes to

  • monthly meetings that provided accountability and support
  • the relationships developed within the FLC between peers
  • sharing of ressources within the FLC

External

Mårtensson & Roxå (2014) look at learning and professional development through two different communities of practice, one across all of Sweden, the other international, and stress the experience of a joint enterprise in participants. A repeated, predictable, long-term nature of the project, rather than a one-off week-long workshop, is important if a long-term impact, for example on professional practice or identity development, is desired. They also show that creating artefacts is an important tool to negotiate shared meaning and build community.

Gehrke & Kezar (2017) identify the ways in which individual faculty involvement in four CoPs engaged in STEM reform efforts is associated with perceived benefits for their home departments and institutions. They find that positive outcomes for individuals hoping to influence broad organisational goals are related to involved in a CoP over a long period of time, and to presenting results from that CoP to the outside, for example at conferences, potentially giving them legitimacy at their home institutions to drive change. Another helpful strategy seems to encourage participants to join the CoP in a team with peers, thus being able to continue conversations at their home institution. And even having participated in the same workshops individually provides a shared language and trust.

From that study, Gehrke & Kezar (2017) give four recommendations to people wanting to design communities of practices to reform STEM education:

  • make sure people receive enough support to stay engaged for long periods of time,
  • bring people in in small teams of peers from each institution
  • help community members become confident and skilled in the subject itself and in communicating about it
  • bring in key leaders to support members and help change the culture

Bernstein-Sierra & Kezar (2017) identify five challenges and possible solutions that national STEM education communities of practice faced:

  • Funding: Communities of practice often rely on soft money. The advice is to adopt a “self-sustaining mindset”, i.e. live within their means, merge with a larger organization with more fund, or commercialising.
  • Leadership: As the CoP matures, demands on leadership change and therefore the leader might either have to develop or make space for someone else — who needs to be suited and/or developed for that role.
  • Legitimacy: Recruiting new members becomes easier when there is evidence of effectivity, e.g. self-assessment data
  • Staleness: When the novelty wears off, members might become tired and/or bored. This can be met with openness to criticizm, advisory boards, new leadership.
  • Maintaining integrity: As things change, new acivities might not be in line with the CoP’s original mission. There needs to be reflection on and re-evaluation of goals, and transparency about this.

Lifetime

Still reading? Yeah, me too, will add to this point later! :-) I’m actually a bit skeptical if I will find a lot of literature here, seeing that I described above how CoPs have life cycles that end with substantial change or the end of a CoP.

My reflection

For me, reading Wenger (1998) was quite a transformative experience, because it gave me language and a framework to articulate a lot of things I had observed but was not able to articulate, for example relating to the role of brokers. Purposefully building CoPs, considering especially legitimate peripheral particpants and how identities are anchored in home institutions and need to potentially change if new ideas are to be implemented back home, seems like a better way to approach cultural change than I’ve been involved in before.

But: Considering my reading of Kezar & Holcombe (2019), I am now more aware than ever that CoPs are just one lens (despite one I like a lot!) on culture change, and that there are contextual factors that need to be considered in addition, that the framework of CoP does not address. But I guess that’s why we are doing this “leading educational change” course — to figure out which other lenses we want to include to tackle the big issues the right way! :-)


Bernstein-Sierra, S., & Kezar, A. (2017). Identifying and overcoming challenges in STEM reform: A study of four national STEM reform communities of practice. Innovative Higher Education, 42(5), 407–420. https://doi.org/10.1007/s10755-01 7-9395-x.

Gehrke, S., & Kezar, A. (2017). The Roles of STEM Faculty Communities of Practice in Institutional and Departmental Reform in Higher Education. American Educational Research Journal, 54(5), 803–833. https://doi.org/10.3102/0002 831217706736

Mårtensson, K., & Roxå, T. (2014). Promoting learning and professional development through communities of practice. Perspectives on Pedagogy and Practice, 5.

Tinnell, T. L., Ralston, P. A. S., Tretter, T. R., & Mills, M. E. (2019). Sustaining pedagogical change via faculty learning community. International Journal of STEM Education, 6(1), 26. https://doi.org/10.1186/s40594-019-0180-5.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge university press.

Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowledge. Harvard business press.

Wenger, E. (2011). Communities of practice: A brief introduction. [here]

Currently reading: “Leveraging Multiple Theories of Change to Promote Reform” (Kezar & Holcombe, 2019)

I am attending a course on “leading educational change”, run by two Norwegian centres for excellence in education, iEarth and BioCEED. The course brings together people working on educational change in very different roles: teachers, administrators, deans, network coordinators, and it’s a great opportunity to connect with people and to do some really focussed thinking on how to change things we’ve always wanted to change! And to do some serious reading of really interesting articles. One of the assigned readings is the article “Leveraging Multiple Theories of Change to Promote Reform: An Examination of the AAU STEM Initiative” by Kezar & Holcombe (2019), which I’ll summarise in the following, because it is a really fascinating and new-to-me perspective on how project planning can be understood, and should be happening.

First of all, there is such a thing as a “theory of change”. A theory of change describes a specific perspective on how and why changes in complex systems happen, and it can be used both to understand current or past changes as well as plan future changes. There are many different theories of change that are commonly used, focussing on different mechanisms and contexts and looking at the change process from very different perspectives. Six main sets of theories of change are

  • scientific management: in a very rational, planned, controlled approach, resources are strategically assigned to grow certain activities to further an agenda, and thus change an organisation; e.g. strengthening a new initiative by assigning service credits
  • evolutionary: the “ecosystem” in question is seen as influenced by other systems in a changing world around it, which can lead to changes happening, for example, due to fear of missing out, or to protect the system from outside influences
  • cultural: traditions, shared visions and sagas are created and negotiated, and this changes interactions between people, leading to changes in the whole system
  • social cognition: the organism itself is able to learn based on arguments and evidence, and mindsets are changed in feedback loops
  • political: power and status/funding are distributed to elevate the groups doing the desired work, and buy-in of important stakeholders is considered
  • institutional theory: the administrative frameworks and norms inside and outside the organisation, e.g. accreditation, funding bodies, disciplinary organisations, shape boundary conditions and are thus the driver of change; there is competition and less successful institutions mimic more successful ones

In some cases, a specific theory of change is assumed to fully explain the change desired or observed in a system. But in most cases, it just sheds light on it from a very specific angle, and using more than one theory of change simultaneously would be helpful to more fully understand what is going on, or to make sure to pull all relevant levers. Kezar & Holocombe (2019) describe one such initiative, where several theories of change were leveraged to improve teaching across different departments. The theories of change that were discussed in the proposal for the project are more specifically defined than the six main sets of theories of change described in the article and summarised above:

  • systems theory: all parts of a system need to act together to create positive change. To improve teaching, not just the teachers need to get better, but they also need to be supported by the university, appropriate facilities must be available, teachers need to be incentivized to spend time on teaching, …
  • organisational learning: individuals learn and together change the way the system works, structures at the organization can influence how information flows and knowledge is put to use
  • network theory: considers how information and behaviour spread and coalitions form to minimise risk/maximise outcome for the individual, depending on how people get the chance to meet and connect
  • institutional theory: about how influence and status is used to change institutional norms, and prestige is aspired to

Using several theories of change simultaneously can support the design of specific activities or programmes (or retrospectively help understand what happened), and more generally there are synergies that can purposefully be leveraged.

Let’s take, for example, annual meetings (which we all know and love, but mostly do because that’s just what is done. Who knew there were so many different functions they could fulfill if designed correctly?). Looking at the project through an institutional theory lens, the annual meetings gave participants the opportunity to learn from, and meet with, the prestigious partner organisations. Inviting everybody into stimulating and significant venues, and bringing in highly-esteemed guest speakers leveraged the influence of prestigious personalities and institutions on participants. From an organisational learning perspective, the meetings led to sharing of information and experiences, which resulted in adaptation of those new practices in several new places. From a systems theory perspective, the meetings were organised to reinforce the framework that was being established, by using it as a guiding structure and always referring back to relevant documents. And lastly from a network theory perspective, the meetings provided opportunities for informal and formal meetings and bonding situations in both random and planned groups.

But also beyond specific activities, theories of change can be thought together in order to create synergies. For example network theory and organisational learning: Buildingof  knowledge and skills within organisations (organizational theory) can be supported by creating networks between organisations in which knowledge and experiences can be shared, or where learning can happen together (network theory). But there can also be conflicts between different approaches: For example if competition between organisations is fostered as a motivational tool in institutional theory where all is about status, this can hinder or prevent cooperation and learning from each other, which might be a goal from an organisational learning and/or network theory position. In the project described in the article, this did happen and was only discovered and understood after the fact.

I found this article really eye-opening in the sense of seeing the potential in considering multiple theories of change simultaneously rather than one at a time. Reading about concrete examples of where multiple theories of change act to create synergies and where they might negate each other really drove home the message that it is absolutely fundamental to have a good idea of which theories of change should be the basis of a planned change process (rather than just curating a collection of activities that all might make sure in and of themselves, but maybe not so much when taken together), and make sure that all activities are aligned with those theories, or at least not in conflict with any of them. This sounds like a really basic “duh!” kind of thought, but as someone who has always relied on communities of practice to explain everything I’m trying to do, I will definitely widen my approach, and highly recommend you read this article if you haven’t thought about these things yet!


Kezar, A., & Holcombe, E. (2019). Leveraging Multiple Theories of Change to Promote Reform: An Examination of the AAU STEM Initiative. Educational Policy. DOI:https://doi.org/10.1177/0895904819843594

The story of Robert and Susan (Biggs, 1999)

I attended iEarth’s GeoLearning Forum today and yesterday, and had a lot of great conversations with amazing students from the four iEarth institutions: Universities of Bergen, Oslo, Tromsø, and UNIS. One student, Sverre, told me about having read articles on learning and teaching as part of a normal geoscience class (how awesome is that? Hat tip to Bjarte!), and one of those articles was “the story of Robert and Susan”. Or: “What the student does: teaching for enhanced learning” (Biggs, 1999).

That article describes two different types of students: Susan, who learns well from traditional university teaching, i.e. lectures and exercises, and Robert, who does not. Susan has a deep approach to learning: She comes to class prepared and with her own questions that help her integrate what she learned with what she already knows and what she wants to learn to reach her academic and career goals. Robert, on the other hand, is not as invested in the subject and attends university to obtain a degree that he needs for a job. He has a surface approach to learning: He collects individual bricks and delivers them at the exam. Susan does really well on the exam, Robert does not. Or at least, that is the situation when both are taught in a conventional way. But there are ways to get Robert to learn similarly well as Susan.

The first step is that their teachers need to understand that Susan and Robert’s performance are not inherent in their personalities, but that they as teachers can influence how well both learn. For that, there need to be clear learning objectives, and it needs to be clear how the learning objectives and the assessment correspond. Also, students need to want to learn: ““Motivation” is a product of good teaching, not its prerequisite.” Additionally, students need to have the opportunity to focus on the task without feeling the pressure to put all their focus on passing the test. And they need to be able to collaborate with their peers and teachers.

Teachers usually come to that understanding by undergoing two developmental steps.

Initially, many teachers believe that what Robert and Susan do is determined by who they are. Once teachers recognise this is not the case, they commonly believe that what Robert and Susan do mainly depends on how well the teacher taught (which often results in a focus on class management). But upon reflecting on that, teachers recognise that learning depends on what activities students actually do when they learn.

When teachers have reached that step, they employ what is called “backwards design” or “constuctive alignment”: First, they consider what the learning outcomes are; what students should be able to DO after instruction. Then, building on that, the teacher comes up with assessments that check whether or not, or to what extent, students are able to do it. And lastly, the teacher develops learning activities in which students learn and practice exactly what they will later do on the exam.

Constructive alignment of a course can happen independently of the methods used in that course, but there are methods that make it particularly easy to achieve constructive alignment: using problem-based learning or a learning portfolio.

In constructively aligned courses, Robert is learning in much the same ways as Susan already did in conventional teaching: He is integrating new knowledge with what he knew before, he asks questions that help him connect new ideas with old ones, he evaluates information, does all the higher-level thinking, because the tasks in class require them. This means that the gap between Robert and Susan gets smaller and smaller, and that we are teaching both equally well. And should that not always be our goal?

I really enjoyed re-discovering this article, thanks, Sverre!


John Biggs (1999) What the Student Does: teaching for enhanced learning, Higher Education Research & Development, 18:1, 57-75, DOI: 10.1080/0729436990180105