Tag Archives: literature

How to know for sure whether a teaching intervention actually improved things

How do we measure whether teaching interventions really do what they are supposed to be doing? (Spoiler alert: In this post, I won’t actually give a definite answer to that question, I am only talking about a paper I read that I found very helpful, and reflecting on a couple of ideas I am currently pondering. So continue reading, but don’t expect me to answer this question for you! :-))

As I’ve talked about before, we are currently working on a project where undergraduate mathematics and mechanics teaching are linked via online practice problems. Now that we are implementing this project, it would be very nice to have some sort of “proof” of its effectiveness.

My (personal) problem with control group studies
Control group studies are likely the most common way to “scientifically” determine whether a teaching intervention had the desired effect. This has rubbed me the wrong way for some time — if I am so convinced that I am improving things, how can I keep my new and improved course from half of the students that I am working to serve? Could I really live with myself if we, for example, measured that half of the students in the control group dropped out within the first three or four weeks of our undergraduate mathematics course, while of the experimental group, only much fewer students dropped out, and much later in the semester? On the other hand, if our intervention had such a large effect, shouldn’t we be measuring it (at least once) in a classical control group study, so we know for sure what its effect is, in order to convince stakeholders at our and other universities that our intervention should be adopted everywhere? If the intervention really improves this much, everybody should see the most compelling evidence so that everybody starts adopting the intervention, right?

A helpful article
Looking for answers to the questions above, I asked Nicki for help, and she pointed me to a presentation by Nick Tilley (2000), that I found really eye-opening and helpful for framing those questions differently, and starting to find answers. The presentation is about evaluation in a social sciences context, but easily transferable to education research.

In this presentation, Tilley first places the proposed method of “realistic evaluation” in the larger context of philosophy of science. For example Popper (1945) suggests using small-scale interventions to deal with specific problems instead of large interventions that address everything at once, and points to the opportunities to investigate the extent to which the theories (on which those small-scale interventions were built) can be tested and improved. Similarly, Campbell (1999) talks about “reforms as experiments”. So the “realistic evaluation” paradigm has been around for a while, partly in conflict with how we do science “conventionally”.

Reality is too complex for control group studies
Then, Tilley talks about classical methods, specifically control group experiments, and argues that — in contrast to what is portrayed in washing detergent ads, for example — studys are typically too complex to directly transfer results between different contexts. In contrast to what science typically does, we are also not investigating a law of nature, where the goal is to understand a mechanism causing a regularity in a given context. Rather, we are investigating how we can cause a change in a regularity. This means we are asking the question “what works for whom in what circumstances?”. With our intervention, we might be introducing different mechanisms, triggering a change in balance of several mechanisms, and hence change the regularities under investigation (which, btw, is our goal!) — all by changing the context.

The approach for evaluations of interventions should therefore, according to Tilley, be “Context Mechanism Outcome Configurations” (CMOC), which describe the interactions between context, mechanism and outcome. In order to create such a description, one needs to clearly describe the mechanisms (“what is it about a measure which may lead it to have a particular outcome pattern in a given context?”), context (“what conditions are needed for a measure to trigger mechanisms to produce particular outcome patterns?”), outcome pattern (“what are the practical effects produced by causal mechanisms being triggered in a given context?” and this finally leads to CMOCs (“How are changes in regularity (outcomes) produced by measures introduced to modify the context and balance of mechanisms triggered?”).

Impact of CCTV on car crimes — a perfect example for control group studies?
Tilley gives a great example for how this works. Investigating how CCTV affects rates of car crimes seems to be easily measured by a classical control group setup. Just install the cameras and compare their crime rates with those of parking spaces without cameras! However, once you start thinking about mechanisms through which the CCTV cameras could influence crime rates, there are lots of different possible mechanisms. There are eight named explicitly in the presentation, for example offenders could be caught thanks to CCTV and go to jail, hence crime rates would sink. Or, criminals might not choose to commit crimes, because the risk of being caught increased due to CCTV, which would again result in lower crime rates. Or people using the car park might feel more secure in using it and therefore start using it more, making it busier at previously less busy times, making car theft more difficult and risky, leading to sinking crime rates.

But then, we also need to think about context, and how car parks and car park crimes potentially differ. For example, crime rate can be the same whether there are a few very active criminals, or many not as busy ones. So catching the similar number of offenders might have a different effect, depending on context. Or the pattern of usage of car parks might depend on working hours of people working close by. So if the dominant CCTV mechanism would be to increase confidence in usage, this would not really help because the busy hours are dedicated by people’s schedules, not how safe they feel. If this would lead to higher usage, however, more cars being around might mean more car crimes because there are more opportunities, yet still a decreased crime rate per use. Another context would be that thieves might just look for new targets outside of the one car park that is now equipped with CCTV, thereby just displacing the problem elsewhere. And there are a couple more contexts mentioned in the presentation.

Long story short: Even for a relatively simple problem (“how does CCTV affect car crime rate?”), there is a wide range of mechanisms and contexts which will all have some sort of influence. Just investigating one car park with CCTV and a second one without will likely not lead to results that help solve the car crime issue once and for all everywhere. First, theories of what exactly the mechanisms and contexts are for a given situation need to be developed, and then other methods of investigation are needed to figure out what exactly is important in any given situation. Do people leave their purses sitting out visibly in the same way everywhere? How are CCTV cameras positioned relative to the cars being stolen? Are usage pattern the same in two car parks? All of this and more needs to be addressed to sort out which of the context-mechanism theories above might be dominant at any given car park.

Back to mathematics learning and our teaching intervention
Let’s get back to my initial question that, btw, is a lot more complex than the example given in the Tilley-presentation. How can we know whether our teaching intervention is actually improving anything?

Mechanisms at play
First, let’s think about possible mechanisms at play here. “What is it about a measure which may lead it to have a particular outcome pattern in a given context?” Without claiming that this is a comprehensive list, here are a couple of ideas:
a) students might realize that they need mathematics to work on mechanics problems, increasing their motivation to learn mathematics
b) students might have more opportunity to receive feedback than before (because now the feedback is automated), and more feedback might lead to better learning
c) students might appreciate the effort made by the instructors, feel more valued and taken seriously, and therefore be more motivated to put in effort
d) students might prefer the online setting over classical settings and therefore practice more
e) students might have more opportunity to practice because of the flexibility in space and time given by the online setting, leading to more learning
f) students might want to earn the bonus points they receive for working on the practice problems
g) students might find it easier to learn mathematics and mechanics because they are presented in a clearer structure than before

Contexts
Now contexts. “What conditions are needed for a measure to trigger mechanisms to produce particular outcome patterns?” Are all students and all student difficulties with mathematics the same? (Again, this is just a spontaneous brain storm, this list is nowhere near comprehensive!)
– if students’ motivation to learn mathematics increased because they see that they will need it for other subjects (a), this might lead to them only learning those topics where we manage to convey that they really really need them, and neglecting all the topics that might be equally important but where we, for whatever reasons, just didn’t give as convincing an example
– if students really value feedback this highly (b), this might work really well, or there might be better ways to give personalised feedback
– if students react to feeling more valued by the instructor (c), this might only work for the students who directly experienced a before/after when the intervention was first introduced. As soon as the intervention has become old news, future cohorts won’t show the same reaction any more. It might also only work in a context where students typically don’t feel as valued so that this intervention sticks out
– if students prefer the online setting over classical settings generally (d), or appreciate the flexibility (e), this might work for us while we are one of the few courses offering such an online setting. But once other courses start using similar settings, we might be competing with others, and students might spend less time with us and our practice problems again
– if students mainly work for the bonus points (f), their learning might not be as sustainable as if they were intrinsically motivated. And as soon as there are no more bonus points to be gained, they might stop using any opportunity for practice just for practice’s sake
– providing students a structure (g) might make them depend on it, harming their future learning (see my post on this Teufelskreis).

Outcome pattern
Next, we look at outcome patterns: “what are the practical effects produced by causal mechanisms being triggered in a given context?”. So which of the mechanisms identified above (and possibly others) seem to be at play in our case, and how do they balance each other? For this, we clearly need a different method than “just” measuring the learning gain in an experimental group and compare it to a control group. We need a way to identify the mechanisms at play in our case, and those that are not. We then need to figure out the balance of those mechanisms. Is the increased interest in mathematics more important than students potentially being put off by the online setting? Or is the online setting so appealing that it compensates for the lack of interest in mathematics? Can we show students that we care about them without rolling out new interventions every semester, and will that motivate them to work with us? Do we really need to show the practical application of every tiny piece of mathematics in order for students to want to learn it, or can we make them trust us that we are only teaching what they will need, even if they aren’t yet able to see what they will need it for?

This is where I am currently at. Any ideas of how to proceed?

CMOCs
And finally, we have reached the CMOCs (“How are changes in regularity (outcomes) produced by measures introduced to modify the context and balance of mechanisms triggered?”). Assuming we have identified the outcome patterns, we would need to figure out how to change those outcome patterns, either by changing the context, or by changing the balance of mechanisms being triggered.

After reading this article and applying the concept to my project (and I only read the article today, so my thoughts will hopefully evolve some over the next couple of weeks!), I feel that the control group study that everybody seems to expect from us is not as valid as most people might think. As I said above, I don’t have a good answer yet for what we should do instead. But I found it very eye-opening to think about evaluations in this way and am confident that we will figure it out eventually! Luckily we have only run a small-scale pilot at this point, and there is still some time before we start rolling out the full intervention.

What do you think? How should we proceed?

How to learn most efficiently when participating in a MOOC

How to learn most efficiently when participating in a MOOC? Yes, I’ll admit, that title promises quite a lot. But there is a new article by Yong and Lim (2016) called “Observing the Testing Effect using Coursera Video-Recorded Lectures” that tells us a lot about how (not) to learn. We have talked about the testing effect before: repeated testing leads to better results on examinations that repeated studying does. And it is confirmed again in this study.

Why am I so excited about this? Because both video-based studying and testing are becoming more and more common these days, and both are sometimes made out to be really bad ideas.

We find video-based learning in most aspects of our lives now (at least if we are talking about lives similar to mine ;-)) — I always follow one or two Coursera courses at the time, and I love watching TED talks. Most softwares I use have video tutorials, and in fact I talked about how I liked the video tutorials of the Monash simple climate model interface only on Tuesday. And whenever I get stuck with a task, I watch video tutorials on youtube to get me going again. And of course many of the lectures at my university are being recorded and many students rely on re-watching them when studying for exams. And, of course, there is the One Planet — One Ocean MOOC that I am involved in preparing. So obviously I see value in video lectures. Even though many people believe that re-watching a lecture does not provide the same experience as seeing it “live”, I don’t think that matters much for lectures in which there is not a lot interaction between lecturer and audience. If you can make yourself use them wisely, I think video lectures are a great substitute for lectures you — for whatever reasons — can’t watch live.

But this is also the biggest issue I have with video lectures: they can easily seduce us into believing that we are learning, when we in fact are not. For example, when I say that I am “following” those Coursera MOOCs, what that means is that I have videos playing while I do something else (like writing emails or cleaning my apartment), i.e. I am not listening carefully, and I never ever do the tests and quizzes they provide. Yet, I still feel like I am learning something. I might or might not* be, but in any case I am not using those resources as effectively as I could be, and in fact most people aren’t.

And testing, I get it: Educators typically don’t like designing tests, because it is really hard. And most students don’t like taking tests, again because it is really hard, so tests have a really bad reputation all around. Especially repeated testing and e-assessment (like we are developing for mathematics and mechanics) people really love to hate!

But this is where the Yong & Lim (2016) study comes in. They showed a short (<3min) Coursera lecture to their participants. Depending on the group, during study time, they showed the clip either once and then tested three times, showed it three times and tested once, or showed it four times. Initial recall right after the study period is best for the group that watched the same clip four times, but it turns out that both groups that test during studying perform significantly better on a test a week after the study period: testing as part of studying (and in contrast to just repeatedly watching a clip) helped anchor the new knowledge significantly better.

From this is it clear that we should definitely be taking advantage of the tests provided with video lectures! Or if there are no tests available, like with TED talks**, instead of watching a lecture over and over again, test ourselves on it: Can I remember the main points? What were the reasons for x or the steps in y? Why did she say z?

And, more importantly, as educators we should take these results to heart, too.  If testing is this important, we need to provide good tests to students, and we need to encourage them to use them to practice.

One scary fact to end this post with: Of the 30 idea units presented in the videos of the study, the best group retained on average only about half until a week after watching those videos. And the worst group only one-third. I didn’t see those videos so I can’t speak about how well they were made and whether the tests addressed all of those 30 idea units, but I wouldn’t bet on students remembering more of the videos I want them to learn from. Which really gives me something to think about.

*watching those videos and feeling good about doing something productive might actually just give me the illusion of competence

**or if we feel that the tests are really bad, which does happen

Yong, P., & Lim, S. (2016). Observing the Testing Effect using Coursera Video-Recorded Lectures: A Preliminary Study Frontiers in Psychology, 6 DOI: 10.3389/fpsyg.2015.02064

How your behavior as an instructor influences how your students behave during peer instruction phases

It probably doesn’t come as a surprise to you that how you behave as an instructor influences how your students work during peer instruction phases. But do you know what you can do to make sure that student discussions are reaching the level of critical thinking that you want? I.e., how do you construct classroom norms? There is a paper by Turpen and Finkelstein (2010) that investigates just that.

In their study, they focus on three factors of classroom culture: faculty-student collaboration, student-student collaboration and sense-making vs answer-making. For this, they use Mazur-like sequence of Peer Instruction (PI) (except that they usually omit the first silent phase) and compare their observations of instructor behavior with student observations.
On the continuum between low and high faculty-student collaboration, there are a couple of behaviors in which mainly those instructors engage who have a high collaboration with students: leaving the stage during PI phases to walk around and listen to or engage in student discussions, answering student questions, and hear student explanations publicly (often several explanations from different students). Here students have many opportunities to discuss with the instructor, and the correct response is often withheld until the students have reached a consensus. Unsurprisingly, in classes where instructors are on the high end of faculty-student collaborations, students talk to the instructor more often, have lower thresholds of asking questions, and feel more comfortable discussing with the instructor.
Looking at student-student collaboration, there are again instructor practices that appear helpful. For example, low-stakes grading does provoke competitive behavior the same way high-stakes grading would.
When using clickers, collaboration is more prevalent when discussion phases are sufficiently long, when collaboration is explicitly encouraged (“talk to your neighbor!”), and when the instructor often models scientific discourse. Modeling scientific discourse (“can you explain your assumption?”) is more effective when the instructor talks to student groups during peer instruction and they have the chance to practice the behavior rather than being one out of several hundred students listening passively, but even modeling the behavior you want in front of the class is better than not doing it.
Sense-making (in contrast to answer-making) can be encouraged by the instructor through practices like explicitly putting emphasis on sense-making, reasoning, discussion, rather than just picking an answer, which means that ample time for discussions needs to be given.
Another practice is providing explanations for correct answers (also in the lecture notes) rather than just which answer was correct.
I find it really interesting to see that the observations made by researchers on concrete teaching practices can be related to what students perceive the classroom norms in a particular course are. This means that you can explicitly employ those behaviors to influence the norms in your own classroom and create a climate where there is more interaction both between the students and yourself, and among the students. So next time you are frustrated about how students aren’t asking questions even though they obviously haven’t understood a concept, or about how they just pick a random answer without sufficiently thinking about the reasons, maybe try to encourage the behavior you want by explicitly stating what you want (and why) and by modeling it yourself?


Turpen, C., & Finkelstein, N. (2010). The construction of different classroom norms during Peer Instruction: Students perceive differences Physical Review Special Topics – Physics Education Research, 6 (2) DOI: 10.1103/PhysRevSTPER.6.020123

How your students might be hurting themselves by skipping classes

Mandatory attendance is seldomly done in german higher education. The system relies on a series of examinations, and whoever passes those get the degree, no matter how much or how little time they have spent inside university buildings*. At the same time, there is a push for mandatory attendance because people feel that only if they force students to be physically present in class, they can make sure students learn what they are supposed to be learning, because they feel students can pass examinations with good grades without ever having set foot in a class, thereby missing out on a lot of learning they should have done**.

And then Ib (Hi Ib! :-)) recently asked me about an article on the importance of student attendance by Schulmeister (2015, “Abwesenheit von Lehrveranstaltungen. Ein nur scheinbar triviales Problem“). In that meta study, about 300 articles from many different countries are brought together to ponder the question of mandatory attendance.

The motivation is that one of the German Federal States recently changed its laws and now prohibits making attendance at university compulsory. The two main reasons are that attendance (and more generally, learning) is seen as the personal responsibility of students, and that students may depend on working to fund their studies. However, Schulmeister argues, many studies have shown that even though personal responsibility for outcomes is a huge motivator, there is no way to “force” someone to take on personal responsibility. And for the need to work to finance being at university, recent studies show that most students don’t work out of the necessity to earn money, but because they would like to have a higher income to be able to splurge on more things. Hence those two arguments don’t seem to carry a lot of weight. But what are the reasons for students not attending class?

There are a couple of “external factors” that affect student attendance. Students who live further away have higher attendance rates than those who live close by, maybe because they aren’t as tempted to have a quick nap at home in the middle of the day and then never come back to university. The weather also plays a role: the worse the weather, the lower student attendance. On the other hand students miss class more often due to vacations during summer. And attendance even depends on the day of the week!

But there are also other reasons for students to decide to stay away from class. Being tired or expecting the class to be boring are often mentioned, and most reasons appear trivial. Some students — interestingly, typically those with low grades — mention that they stay away because the teaching is bad. And studies find that students are convinced that it doesn’t matter for their learning outcome whether they attended class or not.

In fact, students often claim that they can compensate for not being in class by studying at home. And that might be the case if someone missed a single meeting for important reasons. However once people miss a couple of classes, on average they don’t compensate for it by studying more at home. On the contrary – students who miss a lot of classes often don’t even use the resources provided in learning management systems or by their peers. And even when they do, it cannot compensate for the missed attendance. Attendance is an important predictor of student success.

A big part of the discussion is whether personal freedom of students is limited if they were to be forced to attend classes. Some say that students are grown-ups, so it should be up to them to decide. On the other hand, studies show that those students who miss more classes hurt themselves by earning lower grades. Studies also show that the more classes someone attends, the higher their learning outcomes and the lower the risk to fail classes or drop out of university. So might it even be the responsibility of teachers to ensure students don’t hurt themselves, even if it meant limiting their personal freedom?

So what does all of this mean for us?

First, students need to be aware that they are, in fact, hurting themselves by staying away from classes. There are enough studies that have shown this, no matter what they might believe. And there are further studies that show that being aware of this alone already leads to increased attendance.

Second, we need to be aware that making attendance mandatory will make weaker students perform better (and the weaker students are also those who miss more classes in the first place).

Third, if we want mandatory attendance, policies that punish for missing class are more successful than those rewarding attendance (in most studies – not all). This seems to contradict the classical “dangle a carrot rather than threaten with a whip“.

But in the end, the best way to ensure high learning outcomes is probably the middle ground between mandatory attendance and complete laissez-faire. A compromise might be to monitor student attendance and use extended absence as a reason to warn students about the dangers of missing classes, and to provide mentoring and education on how learning works. And to keep negotiating with our students how much freedom they want and need and how much we are willing to provide to keep them from harming themselves.

What is your take on student attendance? Should they decide for themselves whether  or not they want to attend, or should attendance be mandatory?

And Ib, what else would you like to know about this study? :-)

*Of course there are courses where attendance is or can be made compulsory, for example certain lab courses or student cruises. And even without mandatory attendance there are courses where you have to submit work continuously throughout the semester, making attendance compulsory for logistical reasons. But those are not the norm.

**To which I would reply — well, if your examination actually tested everything you want students to know and be able to do after your class, you would make sure that only those students pass that actually mastered everything. And then it would not matter how and where they learned it! Not relying on your examinations to “filter out” students who have not learned “enough” means that your examinations failed, not necessarily your teaching…

Rolf Schulmeister (2015). Abwesenheit von Lehrveranstaltungen. Ein nur scheinbar triviales Problem Studien zur Anwesenheit in Lehrveranstaltungen

Why you should shuffle practice problems rather than blocking them

We like to get into the flow when practicing something, and we like to have our students concentrate on one particular type of problem at a time until they have mastered it, before moving on to the next. But is that really the best way of learning? Spoiler alert: It is not!

In a 2014 study, Rohrer, Dedrick and Burgess show the benefits of interleaved mathematics practice for problems that are not superficially similar. If problems are superficially similar, it makes intuitive sense that one needs to – at least at some point – practice several types together, because clearly distinguishing different kinds of problems and choosing the appropriate approach to solving it is not easy since the problems themselves look so similar. But for problems that look already very different one might think that blocking similar problems and practicing on them until they are mastered, and then moving on to the next type of problem might be a good choice, since one can really concentrate on each type individually and make sure one masters it.

However, this is not what the data shows. Mean test scores in their study (on an unannounced test two weeks after a nine-week practice period) were twice as high for students who had practiced interleaved problems than for those who had been objected to blocked study. Why is that the case?

There are many possible reasons.

One not even connected to interleaving or blocking is that the spacing effect comes into play: just by learning about a topic spaced in chunks over a longer period of time, the learning gain will be higher.

But interleaving itself will help students learn to distinguish between different kinds of problems. If all problems students encounter in any given lesson or homework assignment are of the same kind, they cannot learn to distinguish this kind of problem from other kinds. Being able to distinguish different kinds of problems, however, is obviously necessary to pick the appropriate strategy to solving a problem, which in itself is obviously necessary to actually solving the problem.

So why can’t student learn this in blocked practice? For one, they don’t even need to look for distinguishing features of a given problem if they know that they will find its solution by applying the exact same strategy they used on the problems before, which will also work for the problems after. So they might get a lot of practice executing a strategy, but likely will not learn under which circumstances using this strategy is appropriate. And the strategy might even just be held in short-term memory for the duration of practice and never make it into long term memory since it isn’t used again and again. So shuffling of types of problems is really important to let students both distinguish different types of problems, and associate the correct strategy to solving each type.

If you are still not convinced, there is another study by Rohrer and Taylor (2007) that shows part of what you might be expecting: That practice performance of “blockers” (i.e. students who practice in blocks rather than mixed) is substantially higher than that of “mixers”. Yet, in a later test on all topics, mixers very clearly outperformed blockers here, too.

So what does that mean for our teaching? Shuffle practice problems and help students learn how to discriminate between different kinds of problems and associate the right approach to solving each kind!

Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics problems improves learning Instructional Science, 35 (6), 481-498 DOI: 10.1007/s11251-007-9015-8

Rohrer D., Dedrick R.F., & Burgess K. (2014). The benefit of interleaved mathematics practice is not limited to superficially similar kinds of problems. Psychonomic bulletin & review, 21 (5), 1323-30 PMID: 24578089

Does multitasking hurt learning? Show ’em!

I am reading the “Faculty Focus” mailing list, and a side-note in one of their recent posts, “Why policies fail to promote better learning decisions” by Lolita Paff, really struck a chord with me.

The article is about how to modify policies (like no screens! compulsory attendance! etc) to help students understand why behaving in a way the policies tries to enforce is actually beneficial to them and their learning. She refers to the article “The effect of multitasking on the grade performance of business students” by Ellis, Daniels, Jauregui (2010), where they show the effect of multitasking by splitting a class in two, and allowing one half to text while the other half has to switch off their phones. It turns out that the half that wasn’t multitasking performed significantly better on a test later.

So far, so not surprising. But what Paff suggests is really simple: Rather than telling your class about how multitasking is harming their learning, or even talking explicitly about the Ellis et al. paper, re-do this experiment with your class! In times of clickers in most (many? some?) classrooms and online-testing as abundant as it is, doing this for a class period, then testing, then showing the results is really not a big deal any more. And how much more impressive for your students to see how one half of the class performs significantly better than the other than just hearing that multitasking might not be such a good idea? I would certainly like to give this a try next time I’m teaching a class where I feel that students are multitasking too much.

P.S.: Maybe you shouldn’t split your class front vs back to get those results or other factors might come into play ;-)

Yvonne Ellis, Bobbie Daniels, & Andres Jauregui (2010). The effect of multitasking on the grade performance of business students Research in Higher Education Journal

And even more on motivation

Last week we talked about motivation quite a bit: First about why do students engage in academic tasks?, then about how motivation is proportional to the expectation of achieving a goal. Today I want to bring it all together a bit more, by presenting two other theories (both also described in the Torres-Ayala and Herman (2012) paper, which — should you not have read it yet — I strongly recommend you look into!).

The self-determination theory describes three components of motivation: Autonomy (i.e. being able to determine what you learn, when you learn it and how you learn it), competence (feeling like what you are learning is giving you (more) options to achieve what you want to achieve) and relatedness (feeling connected to a group).

Folie4

Self-determination theory

Those are all components that you, the instructor, do have some influence on. For example a feeling of autonomy can be fostered as easily as giving students the choice of three problem sets and asking them to choose the one they want to work on. Or to let them choose the group they want to work with rather than prescribing groups yourself. Or even only letting them determine the order in which you talk about homework questions.

A feeling of competence is a little more difficult for you to influence, but can be achieved by giving problem sets that gradually become more difficult, instead of giving them really challenging problems right away.

And a feeling of relatedness can be achieved for example by letting students choose who they want to work with, and by making sure you observe the group processes and intervene when necessary.

So far, so good.

There is a fourth theory in the paper, that I also drew little pictures for, but which when preparing for my upcoming workshop for TU Dresden, I chose to drop for now. After all, there is only so much theory one can take in at a time, and I know that what the participants of the workshop come for are methods, methods, methods. Which I might actually give them!

Anyway, I still want to look at the expectancy-value theory here.

Expectancy-value theory basically connects motivational beliefs with achievement behavior.

If you believe you can achieve your goal (expectancy) and reaching that goal is important to you (value), this will modify your behavior. For example, you will likely choose to practice more, and on harder problems than people who don’t have the same beliefs. You will likely be more persistent in pursuing your goal. The quality of your effort will be higher, your cognitive engagement will be higher, and your actual performance will also be better.

Folie5

Expectancy-value theory

There are a lot of studies that link student beliefs with their behavior, and I find this super interesting. I’ll definitely get back to reading and writing about this very soon!


Ana T. Torres-Ayala, & Geoffrey L. Herman (2012). Motivating Learners: A Primer for Engineering Teaching Assistants American Society for Engineering Education

Motivation proportional to the expectation of achieving a goal?

In the last post I talked about a paper on “Motivating Learners: A Primer for Engineering Teaching Assistants” by Torres-Ayala and Herman (2012). Today, I want to present a different motivation theory, also described in that paper:

Attribution theory

Attribution theory basically says that motivation is proportional to the expectation of achieving a goal. Three different factors come into play: externality, stability and controllability. So there are basically four different mindsets students can have:

The most desirable one is one that places an emphasis on effort. Students believe that their chance for success is something internal and unstable, which means that since it is determined within themselves and is not fixed, it can be changed. So they know that if they work harder (or work differently), they can be successful. Since their fate is in their own hands, it is easy to be motivated to do your best.

Other students focus on their ability. This is not desirable, because while they still perceive their chance for success as something that is determined within themselves, they also think that they cannot influence whether they are successful or not. They typically feel like they are not smart enough (or — almost as bad — that they are so smart that everything has to go their way, no matter how much effort they put into it).

A third group of students focusses on task difficulty. Task difficulty is obviously determined externally and is stable – students are likely to feel like the exam was too difficult anyway and they had no chance of controlling whether or not they would be successful.

And then lastly, students that feel that their success depends on luck. Luck is also external, and it is unstable. They don’t know whether they will be lucky or not, but in any case they feel like there is no point putting in an effort.

Folie2

My illustration of attribution theory of motivation

How does knowing about attribution theory help us improve our teaching?

When we know that students are basically only motivated when they feel like they have a direct influence on whether or not they will be successful, we should try and create an environment where learners do feel like that. That means fostering a growth mindset, i.e. not focussing on student abilities, but making sure they realize that they can learn whatever they chose if they put in the effort. It also means making sure that students can rely on the environment being exactly like you said it would be, meaning that if you say you won’t call on people which didn’t raise their hands, you can absolutely not do it. And it also means that students cannot get the impression that you favor some over the others, or that your mood and your grades depend on the weather. Lastly, it means that the task difficulty has to be appropriate. Some challenge is good, but if students don’t have a chance to succeed, they will not continue trying indefinitely (in fact, most quit a lot faster than expected). And who can blame them when their chances of success aren’t more or less proportional to the amount of effort they put in?


Ana T. Torres-Ayala, & Geoffrey L. Herman (2012). Motivating Learners: A Primer for Engineering Teaching Assistants American Society for Engineering Education

Motivation: dangle a carrot rather than threaten with a whip!

Why do students engage in academic tasks?

Next week I am giving a workshop on teaching large classes at TU Dresden. I gave a similar workshop there in spring, but because of its success I’ve been given twice as much time this time around. So there is a lot of exciting content that I can add to the (obviously already exciting!) workshop ;-)

When preparing what I want to talk about, I came across a paper that discusses motivation theories in the context of engineering education, and, even better, tailored to telling teaching assistants how they can improve their classes: “Motivating Learners: A Primer for Engineering Teaching Assistants” by Torres-Ayala and Herman, 2012. They give a great overview over theories on motivation, and today I want to talk about one of them:

Goal theory

Goal theory describes the different reasons why students engage in academic tasks. There are two different kinds of drivers students can have, avoidance or approach, and two kinds of quality of learning they can be striving for: performance and mastery.

Students who are in a state of avoidance and look for performance will state something like “I don’t want to fail this class!”, whereas students in avoidance striving for mastery will say “I don’t want to look or feel stupid!”. Students with an “approach” attitude, on the other hand, will say “I want to get an A!” if they are aiming at performance, or “I want to understand this material, so I can do … with it”.

Folie3

Illustration of the different kinds of motivation described by the goal theory of motivation

While all four kinds of motivation for learning will produce some kind of learning, students with an approach mindset striving for mastery will be the most intrinsically motivated students who will likely do best.

So what does this mean for your teaching? Making students learn to avoid repercussions or public shaming, hence driving them into an avoidance mindset, is not as effective as creating a learning environment where students work towards something positive. And while having students work to earn, for example, bonus points gets them going in the short run, in the long run it is much more effective to help them discover what they can use the skills and knowledge for in their own lives for, discover the intrinsic value in them, and strive to learn because they want to apply the knowledge and skills to better their own future.

Or, as the authors say: Dangle a carrot to pursue rather than threatening with a whip.


Ana T. Torres-Ayala, & Geoffrey L. Herman (2012). Motivating Learners: A Primer for Engineering Teaching Assistants American Society for Engineering Education

Can there be “too much” instruction? Apparently yes!

I recently, via the blogpost “lessons from a toy” by Eyler (2015), came across the article “The Double-edged Sword of Pedagogy: Instruction limits spontaneous exploration and discovery” by Bonawitz, Shafto, Gweon, Goodman, Spelke and Schulz (2011). The article sets out to find out whether children primarily learn from instruction or from exploration. 85 pre-schoolers are divided into four groups that are exposed to a toy under different conditions.

  • In the pedagogical condition, the experimenter said “Look at my toy! This is my toy. I’m going to show you how my toy works. Watch this!” and then demonstrated one function of the toy. She then exclaimed “Wow, see that? This is how my toy works!” and demonstrated the function again.
  • In the interrupted condition, the experimenter began the same way, but after the first demonstration came up with an excuse and left, without reinforcing the message and demonstrating again.
  • In the naive condition, the experimenter seemed to accidentally discover the function of the toy.
  • In the baseline condition, the experimenter just presented the toy without demonstrating its function.

In all four groups, the experimenter then left the toy with the kid and said “Wow, isn’t that cool? I’m going to let you play and see if you can figure out how this toy works. Let me know when you’re done!”. When the child stopped playing with the toy, the experimenter asked whether the child was done, and only when it answered “yes”, the experiment was concluded.

Analysis of the time each child spent with the toy and of the number of functions a child discovered is fascinating. Children who had been in the pedagogical condition group spent less time with the toy, and didn’t explore nearly as much as the kids in the other groups when left to their devices. Instead, they spent the little time they spent on the toy mainly on the one function that had been demonstrated to them. Children in the baseline group, on the other hand, spent the most time with the toy and discovered the most different functions.

In a second study, the authors place children in situations where they overhear the experimenter explain the toy either to another child or to an adult. They find that when kids overheard the toy being explained to another child, the effect was similar to when the toy was explained to them directly, whereas when the toy was explained to a grown-up, they were more free in their exploration of the toy later on. In their words, “preschool children rationally extend their assumptions about pedagogical situations to contexts in which they overhear instruction to comparable learners”.

To conclude, the authors state that “the current results suggest that instruction leads to inductive biases that create a genuine “double-edged” sword: teaching simultaneously confers advantages for learning instructed information and disadvantages for learning untaught information. Thus, the decision about how to balance direct instruction and discovery learning depends largely on the lesson to be learned.”

And while this study was done on pre-schoolers, I think there is a lot we can learn from it for higher education, too. Yes, of course there is some information that all our students need to learn and for which direct instruction might be the best way forward. But if we want to educate future researchers, then shouldn’t our labs be a lot more about exploration and a lot less about following instructions? Shouldn’t the questions we ask be a lot more open? Shouldn’t there be time for exploration and discovery?

Bonawitz, E., Shafto, P., Gweon, H., Goodman, N., Spelke, E., & Schulz, L. (2011). The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery Cognition, 120 (3), 322-330 DOI: 10.1016/j.cognition.2010.10.001