Following the call to share inspiring articles, here are two more that I’m summarising below. See the three previous ones (on assessment (Wiliams, 2011), workload (D’Eon & Yasinian, 2021), and quality (Harvey & Stensaker, 2008)) here. And please keep sending me articles that inspire you, I really enjoy reading and summarising them! :)

First up, recommended by Jenny:

“Two models of educational assessment” by Hager & Butler (1996)

Hagen & Butler (1996) look at educational assessment and how it has changed over time: from a scientific measurement model towards a judgmental model, and recommend using the latter.

The scientific measurement model aimed at summing up performance in one number (or letter) grade, based on criteria and using statistics, to be “maximally objective”, valid and reliable. Examples for this are IQ scores or multiple-choice exams. The tendency with those measures is to make the results about people rather than about their performance on that one specific task of taking the test. But it has been shown over and over that those scores are not a good predictor of future performance in, for example, a job. Not surprisingly, because what is tested is not what is actually required in the job: they test in a very limited context, usually often knowledge and not skills, often using methods that are not suitable, and with no regards to who the test-takers actually are or what their attitudes are like.

The newer judgemental model, on the other hand, is about assessing the competencies that are required in the job, relying on competency models that describe what those competencies are and what it would look like if someone had them. This is how for example problem-based learning or portfolios are typically evaluated. In this model, rather than using only one fixed dataset to come to a conclusion about performance, it is possible to gather more data when a case is unclear, and to come into dialogue with the person being assessed. This dialogue makes it possible to integrate learning and assessment more closely.

Hagen & Butler (1996) suggest a model of assessing professional development with three levels:

1. Knowledge, attitudes, and skills

This level can be assessed following the scientific measurement model and consists, for example, of multiple-choice tests of knowledge and cognitive skill, subject-specific problem-solving skills, and observation of skills in practice setting. So far, so good, but when professionals, for example, medical doctors, are asked to judge colleagues, this is not what their focus is on. So knowledge, attitudes and skills are necessary, but not enough.

2. Performance in simulations

Performance is context dependent, so on this level, artificial simulations of real-world contexts are created so that the performance can be evaluated on a macro level that depends on bringing together knowledge and skills from several domains. Usually, this is done using checklists. But again, only passing this level is not enough.

3. Personal competence in the practise domain

On this level, people are observed “on the job”. In contrast to the previous two levels, now evaluation happens without formalized checklists and criteria. This makes it very much dependant on who the judge is, but judges can and should learn from each other to get rid of personal biases: “Objectivity is the intelligent learned use of subjectivity, not a denial of it. In the judgmental model of assessment it is the assessor who delivers objectivity, not the data.”

Comparing the underlying assumptions of the intelligence approach/scientific measurement model vs the cognitive approach/judgemental model, Hagen & Butler (1996) write: “Whereas the intelligence approach encourages selection of people to fit prespecified jobs, the cognitive approach enables us to view the workplace as a set of opportunities for people to learn and grow.” And isn’t the learning and growing why we are in the job as educators in the first place?

And now on to article no 2, recommended by Sandra:

“Variables associated with achievement in higher education: A systematic review of meta-analyses” by Schneider & Preckel (2017)

Schneider & Preckel (2017) is a review of meta analyses and a great start for when you want to know “what works” in higher education. I wrote a longer summery here, and here is my summary of that summary, mostly based on their “10 cornerstone findings”:

There is A LOT of evidence of what works and what doesn’t in higher education. What comes out is that it doesn’t matter so much what you choose to do, but it does matter that, whatever you do, you do it well. Ideally as a combination of teacher-centred and student-centred approaches, and with equal attention to assessment than to the rest of teaching.

Additionally, there are many small elements that, combined, have a large effect on student learning. In a nutshell: Creating a climate in which questions and discussions are encouraged and valued and feedback happens often and focussed, make it clear what learning goals are, relate course content to students’ lives, goals, dreams.

Also be aware that there are a lot of biases and obstacles depending on who the students are and their prior trajectories, and that good study strategies can help any student succeed (and study strategies are best taught within the disciplinary context, not in separate courses).

It is totally worth reading the original article!

What other articles are inspiring you right now? Let me know and I’ll include them in the list!

Hager, P., and Butler, J. (1996). Two models of educational assessment. Assessment & Evaluation in Higher Education, 21(4), 367–378. https://doi.org/10.1080/0260293960210407

Schneider, M., Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychol Bull. 2017; 143(6): 565-600.

In my group of academic development colleagues at LTH, we just opened up an internal call for the one (or two, or three, or more) articles that are most influential for our current thinking. And I want to make sure I read them all, so here are summaries of the first (and, as of just now, only) three. I’ll add to the list if and when I receive more… And I can say that they are all inspiring in very different ways, but now I am done for today! :-D

First up, sent by Torgny:

“What is assessment for learning?” by Wiliam (2011)

Wiliam (2011) Is an overview article over how the term “assessment for learning” developed and changed meaning over time, and relates this to classroom practice. Here is my short summary:

In contrast to assessment that is done after the learning process, “assessment for learning” and “formative assessment” are terms used for processes that guide learning towards intended goals during the learning process. These concept developed, because it was recognised that the same instruction does not lead to the same results independent of who the learners are, and that failure to learn is not necessarily the learners’ fault. Bloom stated at some point that learners would be able to learn better if they received “feedback” and “correctives”.

“Feedback” as a term is not very useful, however: Feedback is only the information about a gap between a desired and an actual state, it does not, in itself, do anything to close a gap. Specifically, it does not necessarily contain any kind of information for how the learner can improve their learning. So in that sense, giving feedback is pointless.

There has been a whole lot of research on assessment and classroom learning that is summarized next. Basically: There are a bunch of conflicting results. One thought that I found interesting, though, is that there are basically four eight ways to respond to feedback: change behaviour to reach the goal, modify the goal, abandon the goal, or reject the feedback. Only two responses, increasing effort and finding a bigger goal, will actually achieve more learning than if no feedback was given. So how do we get students to pick one of these two choices? Wiliam (2021) quotes Shute (2008)’s guidelines to enhance learning (make feedback specific, give clear guidance for how to improve, make it about the task, not the learner, make it as simple as possible while keeping it as complex as neccessary, …) and on the timing (for procedural learning give feedback immediately).

After more elaboration on what different authors mean by formative assessment, Wiliams (2021) refers back to their definition from 2009:

Two features come out as especially important:

1. The assessment should not just show a gap between what should be and what is, but also contain information that can help improve instruction
2. The learner must act on it in a way that supports their learning: Students can take one of two pathways: The “growth pathway” (with the goal to learn more), or the “well-being pathway” (with the goal to minimize harm, which can include more learning, but can also mean disengaging).

If we can design assessment in such a way to meet both criteria, it is likely to lead to more student engagement and better learning outcomes.

The next two articles were sent by Per:

“Student work: a re-conceptualization based on prior research on student workload and Newtonian concepts around physical work” by D’Eon & Yasinian (2021)

The article by D’Eon & Yasinian (2021) begins by making a very strong case for the importance of considering student workload, by giving an overview over all the negative effects a too high workload has on students: It does not only hinder their learning and kill their motivation, it also makes plagiarism and other forms of cheating more likely and has effects on students’ social life and health. But there is no good definition of what workload actually means: sometimes it’s defined in “objective” terms like for example as time-on-task (but who records and reports, and how “on task” does something have to be to count?), other times in “subjective” terms, for example perceived effort. In any case, gut feeling tells us that there is a link between how much effort students put into something, and how well they succeed. D’Eon & Yasinian (2021) suggest a link, based on Newton:

“Quality culture: Understandings, boundaries and linkages” by Harvey & Stensaker (2008)

I summarised this article before (good thing I have a blog as an external memory), but here is my summary of my summary: It is really helpful to clarify what we actually mean when we talk about “quality”, because it can mean very different things to very different people! For example, is striving for quality about being exceptional in what we do? Or is it that we are consistently meeting pre-defined measures? Or that outcomes meet a purpose? Or do we mean value for money? Or are we talking about improvement over time?

When then wanting to improve quality of whatever flavour, it is important to consider the quality culture in which this is supposed to happen. A 2×2 matrix of strong-to-weak group control and strong-to-weak external rules leads to four types of quality culture: responsive (strong group control, strong external rules), where impulses from the outside are mostly taken up as opportunities and only sometimes are perceived as imposed and constricting ownership and degrees of freedom; reactive (weak group-control, strong external rules), where measures of quality are imposed from the outside and are dealt with when necessary, but not in an integrated way; regenerative (strong group control, weak external rules), where external expectations are incorporated as far as they are perceived useful to further the internal agenda; and reproductive (weak group control, weak external rules), where the status quo is maintained with as little influence from the outside as possible. Quality insurance strategies are of course most likely to be successful if they work with the existing quality culture.

So those are the three “key articles” so far. Any comments? What other article should we include in the list? Please let me know! :-)

Wiliam, Dylan. “What is assessment for learning?.” Studies in educational evaluation 37.1 (2011): 3-14.