Tag Archives: testing

How to learn most efficiently when participating in a MOOC

How to learn most efficiently when participating in a MOOC? Yes, I’ll admit, that title promises quite a lot. But there is a new article by Yong and Lim (2016) called “Observing the Testing Effect using Coursera Video-Recorded Lectures” that tells us a lot about how (not) to learn. We have talked about the testing effect before: repeated testing leads to better results on examinations that repeated studying does. And it is confirmed again in this study.

Why am I so excited about this? Because both video-based studying and testing are becoming more and more common these days, and both are sometimes made out to be really bad ideas.

We find video-based learning in most aspects of our lives now (at least if we are talking about lives similar to mine ;-)) — I always follow one or two Coursera courses at the time, and I love watching TED talks. Most softwares I use have video tutorials, and in fact I talked about how I liked the video tutorials of the Monash simple climate model interface only on Tuesday. And whenever I get stuck with a task, I watch video tutorials on youtube to get me going again. And of course many of the lectures at my university are being recorded and many students rely on re-watching them when studying for exams. And, of course, there is the One Planet — One Ocean MOOC that I am involved in preparing. So obviously I see value in video lectures. Even though many people believe that re-watching a lecture does not provide the same experience as seeing it “live”, I don’t think that matters much for lectures in which there is not a lot interaction between lecturer and audience. If you can make yourself use them wisely, I think video lectures are a great substitute for lectures you — for whatever reasons — can’t watch live.

But this is also the biggest issue I have with video lectures: they can easily seduce us into believing that we are learning, when we in fact are not. For example, when I say that I am “following” those Coursera MOOCs, what that means is that I have videos playing while I do something else (like writing emails or cleaning my apartment), i.e. I am not listening carefully, and I never ever do the tests and quizzes they provide. Yet, I still feel like I am learning something. I might or might not* be, but in any case I am not using those resources as effectively as I could be, and in fact most people aren’t.

And testing, I get it: Educators typically don’t like designing tests, because it is really hard. And most students don’t like taking tests, again because it is really hard, so tests have a really bad reputation all around. Especially repeated testing and e-assessment (like we are developing for mathematics and mechanics) people really love to hate!

But this is where the Yong & Lim (2016) study comes in. They showed a short (<3min) Coursera lecture to their participants. Depending on the group, during study time, they showed the clip either once and then tested three times, showed it three times and tested once, or showed it four times. Initial recall right after the study period is best for the group that watched the same clip four times, but it turns out that both groups that test during studying perform significantly better on a test a week after the study period: testing as part of studying (and in contrast to just repeatedly watching a clip) helped anchor the new knowledge significantly better.

From this is it clear that we should definitely be taking advantage of the tests provided with video lectures! Or if there are no tests available, like with TED talks**, instead of watching a lecture over and over again, test ourselves on it: Can I remember the main points? What were the reasons for x or the steps in y? Why did she say z?

And, more importantly, as educators we should take these results to heart, too.  If testing is this important, we need to provide good tests to students, and we need to encourage them to use them to practice.

One scary fact to end this post with: Of the 30 idea units presented in the videos of the study, the best group retained on average only about half until a week after watching those videos. And the worst group only one-third. I didn’t see those videos so I can’t speak about how well they were made and whether the tests addressed all of those 30 idea units, but I wouldn’t bet on students remembering more of the videos I want them to learn from. Which really gives me something to think about.

*watching those videos and feeling good about doing something productive might actually just give me the illusion of competence

**or if we feel that the tests are really bad, which does happen

Yong, P., & Lim, S. (2016). Observing the Testing Effect using Coursera Video-Recorded Lectures: A Preliminary Study Frontiers in Psychology, 6 DOI: 10.3389/fpsyg.2015.02064

Testing drives learning.

Once you’ve tested on something correctly once, you will remember it forever. Right?

In a study on “The Critical Importance of Retrieval for Learning” by Karpicke and Roediger (2008), four different student groups are compared in order to figure out the importance of both repetition and testing for longer-term recall of learned facts.

Students are asked to memorize a list of 40 Swahili-English word pairs, and then tested on those pairs. After the first test, the four groups are then treated differently: The first group continues studying and testing on all word pairs. The second group continues studying all word pairs, but is only tested on those words that were not successfully recalled. As soon as one word pair is successfully recalled, it is dropped from all subsequent tests. The third group is tested on all word pairs in all tests, but word pairs that were successfully recalled in a test are dropped from subsequent studying. And for the last group, every successfully recalled word pair is dropped from all subsequent studying and testing.

The learning gain during the study period is very similar for all four groups, but interestingly the recall a week later is not.

The groups that were always tested on all word pairs, no matter whether the word pairs were studied until the end or dropped after successful retrieval, could recall about 80% of the word pairs one week later. The students in the other two groups, where word pairs were dropped from testing after successful retrieval, only recalled between 30 and 40% of word pairs correctly.

This basically shows that repeated studying does not have an effect once a word pair has been successfully recalled once, but that repeated testing even after a successful recall consolidates the learning. Testing drives learning, indeed.

These findings should probably have substantial implications on the way we teach – and on how we learn ourselves. The authors report that self-testing is rarely reported as a self-studying technique and that practicing retrieval is only ever a side benefit of students testing whether or not they have learned. And the findings are indeed contradicting the widely accepted conventional wisdom that repetition will improve retention of material. So at the very least, we should share the findings of this study with students and educators.

One way to include more testing in large classes are clickers and multiple choice questions, and the benefits of clickers on retention of material are discussed in the Marsh et a. (2007) paper discussed recently.

Another way would be to encourage students to not just repeatedly read a text when studying for an exam, but to ask themselves questions on details of the text to test what they remember and how well they understand it.

Come to think of it, there are really a lot of possibilities for including question-asking in classes. How are you going to do it?