I am missing institute seminars! Or: Why we should talk to people who use different methods

You probably know that I have recently changed my research focus quite dramatically, from physical oceanography to science communication research. What that means is that I am a total newbie (well, not total any more, but still on a very steep learning curve), and that I really appreciate listening to talks from a broad range of topics in my new field to get a feel for the lay of the land, so to speak. We do have institute seminars at my current work place, but they only take place like once a month, and I just realized how much I miss getting input on many different things on at least a weekly basis without having to explicitly seek them out. To be fair, it’s also summer vacation time and nobody seems to be around right now…

But anyway, I want to talk about why it is important that people not only of different disciplines talk, but also people from within the same discipline that use different approaches. I’ll use my first article (Simulated impact of double-diffusive mixing on physical and biogeochemical upper ocean properties by Glessmer, Oschlies, and Yool (2008)) to illustrate my point.

I don’t really know how it happened, but by my fourth year at university, I was absolutely determined to work on how this teeny tiny process, double-diffusive mixing (that I had seen in tank experiments in a class), would influence the results of an ocean model (as I was working as student research assistant in the modelling group). And luckily I found a supervisor who would not only let me do it, but excitedly supported me in doing it.

Double-diffusive mixing, for those of you who don’t recall, looks something like this when done in a tank experiment:

IMG_9084

And yep, that’s me in the reflection right there :-)

Why should anyone care about something so tiny?

Obviously, there is a lot of value in doing research to satisfy curiosity. But for a lot of climate sciences, one important motivation for the research is that ultimately, we want to be able to predict climate, and that means that we need good climate models. Climate models are used as basis for policy decisions and therefore should represent the past as well as the present and future (under given forcing scenarios) as accurately as possible.

Why do we need to know about double-diffusive mixing if we want to model climate?

Many processes are not actually resolved in the model, but rather “parameterized”, i.e. represented by functions that estimate the influence of the process. And one process that is parameterized is double-diffusive mixing, because its scale (even though in the ocean the scale is typically larger than in the picture above) is too small to be represented.

Mixing, both in ocean models and in the real world, influences many things:

  • By mixing temperature and salinity (not with each other, obviously, but warmer waters with colder, and at the same time more salty waters with less salty), we change density of the water, which is a function of both temperature and salinity. By changing density, we are possibly changing ocean currents.
  • At the same, other tracers are influenced: Waters with more nutrients mix with waters with less, for example. Also changed currents might now supply nutrient-rich waters to other regions than they did before. This has an impact on biogeochemistry — stuff (yes, I am a physical oceanographer) grows in other regions than before, or gets remineralized in different places and at different rates, etc.
  • A change in biogeochemistry combined with a changed circulation can lead to changed air-sea fluxes of, for example, oxygen, CO2, nitrous oxide, or other trace gases, and then you have your influence on the atmosphere right there.

What are the benefits of including tiny processes in climate models?

Obviously, studying the influence of individual processes leads to a better understanding of ocean physics, which is a great goal in itself. But that can also ultimately lead to better models, better predictions, better foundation for policies. But my main point here isn’t even what exactly we need to include or not, it is that we need a better flow of information, and a better culture of exchange.

Talk to each other!

And this is where this tale connects to me missing institute seminars: I feel like there are too few opportunities for exchange of ideas across research groups, for learning about stuff that doesn’t seem to have a direct relevance to my own research (so I wouldn’t know that I should be reading up on it) but that I should still be aware of in case it suddenly becomes relevant.

What we need is that, staying in the example of my double-diffusive mixing article, is that modellers keep exploring the impact of seemingly irrelevant changes to parameterizations or even the way things are coded. And if you aren’t doing it yourself, still keep it in the back of your head that really small changes might have a big influence, and listen to people working on all kinds of stuff that doesn’t seem to have a direct impact on your own research. In case of including the parameterization of double-diffusive mixing, oceanic CO2 uptake is enhanced by approximately 7% of the anthropogenic CO2 signal compared to a control run! And then there might be a climate sensitivity of processes, i.e. double-diffusive mixing happening in many ore places under a climate that has lead to a different oceanic stratification. If we aren’t even aware of this process, how can we possibly hope that our model will produce at least semi-sensible results? And what we also need are that the sea going and/or experimental oceanographers keep pushing their research to the attention of modellers. Or, if we want less pushing: more opportunities for and interest in exchanging with people from slightly different niches than our own!

One opportunity just like that is coming up soon, when I and others will be writing from Grenoble about Elin Darelius and her team’s research on Antarctic stuff in a 12-m-diameter rotating tank. Imagine that. A water tank of that size, rotating! To simulate the influence of Earth’s rotation on ocean current. And we’ll be putting topography in that! Stay tuned, it will get really exciting for all of us, and all of you! :-)

P.S.: My #COMPASSMessageBox for this blogpost below. I really like working with this tool! Read more about the #COMPASSMessageBox.

message_box_dd

And here is the full citation: Glessmer, M. S., Oschlies, A., & Yool, A. (2008). Simulated impact of double‐diffusive mixing on physical and biogeochemical upper ocean properties. Journal of Geophysical Research: Oceans, 113(C8).

What you know about science is not necessarily what you believe about science

I’ve been working in science communication research for a good half a year now, and my views on outreach are constantly evolving. When I applied for this job, I was convinced that if only the public knew what we (the scientists) know, they would take better decisions. So all we need to do is inform the public, preferably using entertaining and engaging methods. However, I soon came to learn that this is known as the “deficit model” and that there is a lot of research saying that life isn’t that easy. Like, at all.

One article I really like makes it very clear that knowledge about what science says is not at all the same as believing in what science says. The article Climate-Science Communication and the Measurement Problem by Kahan (2015) (btw, a really entertaining read!) describes how changing a question on a questionnaire from “Human beings, as we know them today, developed from earlier species of animals” to “According to the theory of evolution, human beings, as we know them today, developed from earlier species of animals” has a big impact: While in the first case, religiosity of the respondents had a huge impact and even highly educated religious people are very likely to answer “no”, in the second case religious and non-religious people answer similarly correctly. So clearly the knowledge of what evolution theory says is there in both cases, but only in the latter case that knowledge becomes relevant in answering the question. In the first case, the respondents cultural identity dictates a different answer than in the second case, where the question is only about science comprehension, not about beliefs and identity. As the author says: a question about ““belief in” evolution measures “who one is” rather than “what one knows””.

The author then moves on to study knowledge and beliefs about climate change and finds the same thing: the relationship between science comprehension and belief in climate change depends on the respondents’ identities. The more concerned someone is about climate change due to their cultural background, the more concerned they become as their level of science comprehension increases. The more sceptical someone is, the more sceptical he becomes with increasing science comprehension: “Far from increasing the likelihood that individuals will agree that human activity is causing climate change, higher science comprehension just makes the response that a person gives to a “global- warming belief” item an even more reliable indicator of who he or she is.”

So knowledge (or lack thereof) clearly isn’t the problem we face in climate change communication — the problem is the entanglement of knowledge and identity. What can we do to disentangle the two? According to the article, it is most important to not reinforce the association of opposing positions with membership in competing groups. The higher-profile the communicators on the front lines, the more they force individuals to construe evidence that supports the claims of those high-profile members of their group in order to feel as part of that group and protect their identity. Which is pretty much the opposite of how climate science has been communicated in the last years. Stay tuned while we work on developing good alternatives, but don’t hold your breath just yet ;-)


Kahan, D. M. (2015). Climate-Science Communication and the Measurement Problem Political Psychology, 36, 1-43

On the impact of blogging — or how far does my message mix?

What is the impact of this blog? And who am I writing it for?

Those are not questions I regularly ask myself. The main reason I started blogging was to organise all the interesting stuff I was collecting for my introduction to oceanography lecture at the University of Bergen in one place, so I would be able to find it when I needed it again. And I wanted to share it with friends who were interested in teaching oceanography or teaching themselves.

Another of the reasons why I blog is that I notice a lot of exciting features in everyday life that relate to oceanography and/or physics, that other people would just walk past and not notice, and that I would like to share the wonder of all those things with others. And noticing all this stuff is so much FUN! The blog “gives me permission” to play, to regularly do weekend trips to weirs or ship lifts or other weird landmarks that I would probably not seek out as often otherwise.

But the other day I was browsing the literature on science blogging in order to come up with recommendations for the design of what is to become the Kiel Science Outreach Campus’ (KiSOC) blog. I came across a paper that resonated with me on so many levels: “Science blogs as boundary layers: Creating and understanding new writer and reader interactions through science blogging” by M-C Shanahan (2011). First, I really liked to see the term “boundary layer” in the title, since it brings to mind exciting fluid mechanics. Then second, I read that the boundary phenomena I was thinking of were really where the term “boundary layer” came from even in this context. And then I realised that I have had “boundary layer” experiences with this blog, too!

So what are those boundary layers about? Well, in fluid mechanics, they are the regions within fluids that interact with “something else” — the boundary of a flow, e.g. a pipe, or a second fluid of different properties.  They are a measure for the region over which temperature or salinity or momentum or any other property is influenced by the boundary. But the same construct can be used for social groups, i.e. in my case oceanographers and non-oceanographers. (You should, btw, totally check out the original article! Her example is even more awesome than mine)

But here is my own boundary layer experience: My sister sent me an email with the subject “double-diffusive mixing” and a picture she had taken! My sister is not an oceanographer, and I wasn’t even aware that she associated the term “double-diffusive mixing” with anything in particular other than me writing my Diplom thesis about it and probably talking about a lot. But that she would recognise it? Blew my mind!

Turns out what she saw is actually convection, but it doesn’t look that dissimilar from salt fingers, and how awesome is it that she notices this stuff and thinks of oceanography?

DSC03217
Day 1. The remaining pink soap starts making its way up through the refill of clear soap.

Obviously I asked for follow-up pictures:

DSC03232
Day 2. A lot of the pink soap has reached the top, passing through the clear refill.
DSC03267
Day 3. All of the “old” pink soap is now on its way up through the clear refill.

And I had another boundary layer experience recently: A sailor on the Norwegian research vessel Håkon Mosby with many many years experience at sea had seen my book and told me that he now looks at waves in a new way. How awesome is that? That’s the biggest compliment my book could get, to teach something new about visual observations of the ocean to someone who looks at the ocean every single day!

Anyway. Reading this article made me think about how happy both those boundary layer experiences made me, and that maybe I should actually start aiming at creating more of those. Maybe not with this blog, that I kinda want to keep as my personal brain dump, but there are so many different ways to interact more with people who would potentially be super interested in oceanography if only they knew about it… I guess there is a reason why I am working the job I am :-)


Shanahan, M. (2011). Science blogs as boundary layers: Creating and understanding new writer and reader interactions through science blogging Journalism, 12 (7), 903-919 DOI: 10.1177/1464884911412844

Nonlinear effects in shallow water waves

I recently googled for something related to the shape of waves and came across a photo of a wave that caught my eye, and it took me to a journey that lead to the article “nonlinear shallow ocean wave soliton interactions on flat beaches” by Ablowitz and Baldwin (2012).

What’s discussed in that article is that while many wave interactions can be seen as (more or less) linear, sometimes there are nonlinear effects that can be replicated in a model. So far so not surprising. But I got fascinated because the phenomenon they look at I have seen over and over again and never really paid any attention to it: Wave crests forming X or Y shapes. But looking through my archives, I even had dozens of pictures of this exact phenomenon! (Actually, I didn’t have to look further back than to a beautiful day last November, when I also observed the wavelength dependency of wave-object interactions)

Take for example the picture below: Do you see the H shape in the waves closest to shore? (In the article they would probably call it a more-complex shape, since it’s a double Y shape…)

img_8629

Below I’ve drawn into the picture what I mean by H-shape in green, and the typical kind of linear wave interaction in red (all crests just move on without influencing each other except in the spot where they occur at the same time, there they just add to each other):

img_8629-copy

Or below, I spot an X-shape:

img_8633

And here are several X- and Y-shapes

img_8635

And the picture below just to give you an orientation of where you are: Yep, it’s the same spot where we usually observe foam stripes, funny waves, or ice

img_8625

 


Mark J. Ablowitz, & Douglas E. Baldwin (2012). Nonlinear shallow ocean wave soliton interactions on flat beaches Physical Review E, vol. 86(3), pp. 036305 (2012) arXiv: 1208.2904v1

Outreach is about more than about the perfect presentation (or even the perfect hands-on tank experiment!)

In most of my blog posts on outreach I focus on how to run the _perfect_ experiment. And while I still think that’s awesome, I recently read an article by Johanna Varner (“Scientific Outreach: Toward Effective Public Engagement with Biological Science”, 2014) that made a lot of points that I have definitely not stressed enough on my blog, and probably not even considered enough.
Outreach is often modeled on scientific communication and intuition. Of course, since that is what we’ve learned over the years and gotten good at, and what we are most comfortable with. But when we are trying to engage the “general public”, those are mostly people who have a very different background from us. Speaking of backgrounds — there is a problem with the concept of “the general public”, as there is no _one_ general public. The general public is very very diverse, and it is important to consider each audience individually. And there is the next thing: “Audience” then often implies that a scientist talks and “the general public” listens, which is not the best model. One-way communication that we often use in outreach, more often than not using simplified, sensationalized stories, is just not effective. For retention of facts as well as for building enthusiasm and for engaging in deep thinking, the public needs to be actively engaged, not talked to.
To also consider is that the reliability of a source is not judged by how many PhDs a speaker has, but by how well it supports the listener’s preconceptions. Any new information is interpreted in such a way that it supports existing ideas. And even if ideas could be “objectively transferred”: new knowledge does not change attitudes or behaviour. And even the intention to act is a poor predictor of future behaviour!
So what can we do?
The article provides a structure for planning outreach activities which is basically backward design: Start with what you want people to learn, then think about what you would take as evidence that they actually learned it, and then plan the activity. Check out the article if you are not familiar with the concept, it’s a really nice introduction. And it is always important to remember that effectiveness of any activity depends on an explicit definition of the goals.
Then, there are a couple of design elements we can use. All of those come from the article originally, but I give my own interpretation and examples.
  • Use “trusted resources” to help us share our message. Instead of doing our outreach activity as a self-organized event, use local churches, artists, any institution or person whom the community trusts to invite you and set the stage for you, this will make it much more likely that people will not only listen to, but actually consider taking on your message.
  • Know your audience. This is super difficult! But since you will want to create personal relevance for your audience (since personal relevance is essential for engagement), you need to know about what your audience’s knowledge, attitudes, values are. And it goes without saying that every outreach activity needs to be tailored to each audience specifically.
  • Establish common ground with your audience, this makes your message more likely to be accepted. Don’t be the scientist who nobody can relate to, be the person who lives in the same neighbourhood, who supports the same sports team, who likes the same kind of music, whatever is applicable in your case.
  • Use appropriate language! Don’t alienate by speaking to science-y, and also beware that words carry a very different meaning in science than in everyday language sometimes (And if you have never seen those tables that tell you that the term “alcohol”, vor example, means “booze” to the general public, when you use it to mean “solvent”, definitely check out examples of such tables here or here!)
  • Get into dialogue instead of just “preaching” in a one-way manner. Ask for questions and feedback, offer to follow-up by email, engage with the people there!
  • Frame your science in a storyline. It makes it much easier to follow and to digest as well as to remember.

    wasserflaschen copy
    Click to enlarge
  • Use “vivid hooks”, i.e. present your research question as an actual question or puzzle to solve, ask people to brainstorm hypotheses, show them the real data, let them get actively involved! Experiential learning and personal experience influence attitudes and beliefs strongly. This might be easiest if you had animals to show, but even just a good question works. Sometimes it’s actually surprising to see what works: The other day I had a blog post showing an empty bottle and one filled with water and asked whether people knew which one was which. And I got so many private messages with people’s answers, asking me to confirm they were correct! I had never thought that this particular blog post would raise such interest.
  • Emphasize benefits of action rather than risks of inaction. Fear appeals can backfire, since they lead to feelings of helplessness, which then lead to denial, apathy, resignation. And all of those prevent engagement.
  • Provide action resources. Enthusiasm and active engagement don’t stay up for very long after you are done with your outreach experiment if you don’t do something to keep them up. Therefore, provide action resources! Let people know when your next event will be, or the schedule of public events at your institution. Hand out take-home activities. Provide online resources or lists of other people’s online resources. Make sure that those who would like to stay engaged have a very low threshold to do so!

And now, go read the original research where all of these ideas came from:

Varner (2014) “Scientific Outreach: Toward Effective Public Engagement with Biological Science”

You learn better when you explain to yourself

Waves on Pinnau in Mölln. By Uwe Glessmer

I just read a really interesting article on explaining to yourself as a mechanism for learning by Tania Lombrozo. We have talked about peer instruction being valuable because explaining to others helps both the “others” and the explainer, and it’s really common to hear student tutors say that they only understood something really well when they had to explain it to students they were tutoring. In fact, many people I know use putting someone in the position of having to explain something to make themselves (or their students, if they appoint them as tutors) understand better, and studies show it works. But explaining to yourself?

The author describes research on how, why and when explaining leads to new learning. You should go check out the original blog post, too, but here is what I am taking away from it: When you explain, you are looking for general pattern.

The author cites research that shows that explaining to yourself is not the best strategy for all kind of learning outcomes — only for those that are related to the causal effects you were explaining to yourself. For other details, it might be a better strategy to just observe, or describe what you are seeing.

How is this relevant for our teaching? There are several ways.

Explaining to themselves is a strategy we can recommend to our students. I remember studying for my oral examinations at Vordiplom (now equivalent to Bachelor) level. I used to come up with questions and try and answer them late at night when I couldn’t go to sleep (Why is the Atlantic ocean more salty than the Pacific ocean? This kind of stuff). Those were questions that I didn’t know the correct answer of at the time (and some of my questions there might not be an answer) and it definitely helped me when I was then asked what geometry of sound receivers I would use if I were to build an array for SOFAR floats, and it made me feel safer going into the exam, knowing that I had answered all questions that I could come up with previously as well as I could.

And of course you can just tell students that they will have to teach about a topic, since anticipating having to teach already leads to improved learning. Then you can reflect later on how thinking they would have to teach led them to use different learning strategies, and whether they might want to use those in the future even when they were not expecting having to teach.

I even see a similar effect with having a blog. Now, when I take pictures of water somewhere, I observe pretty carefully, anticipating that I will write about what I saw and that someone might ask questions about it. That definitely makes me put a little extra effort into observing and thinking about what might be going on there!

Check out the original blog post on explaining to yourself as a mechanism for learning by Dr. Lombrozo — there is a really nice example in there that I definitely want to use in future workshops to make that exact point. You will enjoy it, too!

You learn better when you think that you will have to teach

Mirjam Glessmer and Timo Lüth leading a workshop for university instructors

Have you ever worked as student tutor? Then you’ve probably felt like you understood the content of the course you tutored a million times better after tutoring it. Or at least that’s what I hear over and over again: People feel like they understood a topic. Then they prepare to teach it, and realise how much more there was to understand and that they actually understood it.

And there is research that shows that you don’t actually need to teach in order to get the deeper understanding, it is enough to anticipate that you will teach: “Expecting to teach enhances learning and organization of knowledge in free recall of text passages” by Nestojko, Bui, Kornell & Bjork (2014).

In that article, two groups of participants are given texts that they are to study. One group is told that they will be tested on the text, the other one that they will have to teach someone else who then will be tested. After all participants study the text, they are then all tested (and nobody gets to teach). But it turns out that even expecting to teach had similar benefits to what we see in student tutors who actually taught: Participants expecting to teach have a better recall of the text they had to study, can answer more questions about it and especially questions regarding main points.

So what does that mean for teaching? As the authors say: “Instilling an expectation to teach […] seems to be a simple, inexpensive intervention with the potential to increase learning efficiency at home and in the classroom.” And we should definitely use that to our advantage! :-)

How to know for sure whether a teaching intervention actually improved things

How do we measure whether teaching interventions really do what they are supposed to be doing? (Spoiler alert: In this post, I won’t actually give a definite answer to that question, I am only talking about a paper I read that I found very helpful, and reflecting on a couple of ideas I am currently pondering. So continue reading, but don’t expect me to answer this question for you! :-))

As I’ve talked about before, we are currently working on a project where undergraduate mathematics and mechanics teaching are linked via online practice problems. Now that we are implementing this project, it would be very nice to have some sort of “proof” of its effectiveness.

My (personal) problem with control group studies
Control group studies are likely the most common way to “scientifically” determine whether a teaching intervention had the desired effect. This has rubbed me the wrong way for some time — if I am so convinced that I am improving things, how can I keep my new and improved course from half of the students that I am working to serve? Could I really live with myself if we, for example, measured that half of the students in the control group dropped out within the first three or four weeks of our undergraduate mathematics course, while of the experimental group, only much fewer students dropped out, and much later in the semester? On the other hand, if our intervention had such a large effect, shouldn’t we be measuring it (at least once) in a classical control group study, so we know for sure what its effect is, in order to convince stakeholders at our and other universities that our intervention should be adopted everywhere? If the intervention really improves this much, everybody should see the most compelling evidence so that everybody starts adopting the intervention, right?

A helpful article
Looking for answers to the questions above, I asked Nicki for help, and she pointed me to a presentation by Nick Tilley (2000), that I found really eye-opening and helpful for framing those questions differently, and starting to find answers. The presentation is about evaluation in a social sciences context, but easily transferable to education research.

In this presentation, Tilley first places the proposed method of “realistic evaluation” in the larger context of philosophy of science. For example Popper (1945) suggests using small-scale interventions to deal with specific problems instead of large interventions that address everything at once, and points to the opportunities to investigate the extent to which the theories (on which those small-scale interventions were built) can be tested and improved. Similarly, Campbell (1999) talks about “reforms as experiments”. So the “realistic evaluation” paradigm has been around for a while, partly in conflict with how we do science “conventionally”.

Reality is too complex for control group studies
Then, Tilley talks about classical methods, specifically control group experiments, and argues that — in contrast to what is portrayed in washing detergent ads, for example — studys are typically too complex to directly transfer results between different contexts. In contrast to what science typically does, we are also not investigating a law of nature, where the goal is to understand a mechanism causing a regularity in a given context. Rather, we are investigating how we can cause a change in a regularity. This means we are asking the question “what works for whom in what circumstances?”. With our intervention, we might be introducing different mechanisms, triggering a change in balance of several mechanisms, and hence change the regularities under investigation (which, btw, is our goal!) — all by changing the context.

The approach for evaluations of interventions should therefore, according to Tilley, be “Context Mechanism Outcome Configurations” (CMOC), which describe the interactions between context, mechanism and outcome. In order to create such a description, one needs to clearly describe the mechanisms (“what is it about a measure which may lead it to have a particular outcome pattern in a given context?”), context (“what conditions are needed for a measure to trigger mechanisms to produce particular outcome patterns?”), outcome pattern (“what are the practical effects produced by causal mechanisms being triggered in a given context?” and this finally leads to CMOCs (“How are changes in regularity (outcomes) produced by measures introduced to modify the context and balance of mechanisms triggered?”).

Impact of CCTV on car crimes — a perfect example for control group studies?
Tilley gives a great example for how this works. Investigating how CCTV affects rates of car crimes seems to be easily measured by a classical control group setup. Just install the cameras and compare their crime rates with those of parking spaces without cameras! However, once you start thinking about mechanisms through which the CCTV cameras could influence crime rates, there are lots of different possible mechanisms. There are eight named explicitly in the presentation, for example offenders could be caught thanks to CCTV and go to jail, hence crime rates would sink. Or, criminals might not choose to commit crimes, because the risk of being caught increased due to CCTV, which would again result in lower crime rates. Or people using the car park might feel more secure in using it and therefore start using it more, making it busier at previously less busy times, making car theft more difficult and risky, leading to sinking crime rates.

But then, we also need to think about context, and how car parks and car park crimes potentially differ. For example, crime rate can be the same whether there are a few very active criminals, or many not as busy ones. So catching the similar number of offenders might have a different effect, depending on context. Or the pattern of usage of car parks might depend on working hours of people working close by. So if the dominant CCTV mechanism would be to increase confidence in usage, this would not really help because the busy hours are dedicated by people’s schedules, not how safe they feel. If this would lead to higher usage, however, more cars being around might mean more car crimes because there are more opportunities, yet still a decreased crime rate per use. Another context would be that thieves might just look for new targets outside of the one car park that is now equipped with CCTV, thereby just displacing the problem elsewhere. And there are a couple more contexts mentioned in the presentation.

Long story short: Even for a relatively simple problem (“how does CCTV affect car crime rate?”), there is a wide range of mechanisms and contexts which will all have some sort of influence. Just investigating one car park with CCTV and a second one without will likely not lead to results that help solve the car crime issue once and for all everywhere. First, theories of what exactly the mechanisms and contexts are for a given situation need to be developed, and then other methods of investigation are needed to figure out what exactly is important in any given situation. Do people leave their purses sitting out visibly in the same way everywhere? How are CCTV cameras positioned relative to the cars being stolen? Are usage pattern the same in two car parks? All of this and more needs to be addressed to sort out which of the context-mechanism theories above might be dominant at any given car park.

Back to mathematics learning and our teaching intervention
Let’s get back to my initial question that, btw, is a lot more complex than the example given in the Tilley-presentation. How can we know whether our teaching intervention is actually improving anything?

Mechanisms at play
First, let’s think about possible mechanisms at play here. “What is it about a measure which may lead it to have a particular outcome pattern in a given context?” Without claiming that this is a comprehensive list, here are a couple of ideas:
a) students might realize that they need mathematics to work on mechanics problems, increasing their motivation to learn mathematics
b) students might have more opportunity to receive feedback than before (because now the feedback is automated), and more feedback might lead to better learning
c) students might appreciate the effort made by the instructors, feel more valued and taken seriously, and therefore be more motivated to put in effort
d) students might prefer the online setting over classical settings and therefore practice more
e) students might have more opportunity to practice because of the flexibility in space and time given by the online setting, leading to more learning
f) students might want to earn the bonus points they receive for working on the practice problems
g) students might find it easier to learn mathematics and mechanics because they are presented in a clearer structure than before

Contexts
Now contexts. “What conditions are needed for a measure to trigger mechanisms to produce particular outcome patterns?” Are all students and all student difficulties with mathematics the same? (Again, this is just a spontaneous brain storm, this list is nowhere near comprehensive!)
– if students’ motivation to learn mathematics increased because they see that they will need it for other subjects (a), this might lead to them only learning those topics where we manage to convey that they really really need them, and neglecting all the topics that might be equally important but where we, for whatever reasons, just didn’t give as convincing an example
– if students really value feedback this highly (b), this might work really well, or there might be better ways to give personalised feedback
– if students react to feeling more valued by the instructor (c), this might only work for the students who directly experienced a before/after when the intervention was first introduced. As soon as the intervention has become old news, future cohorts won’t show the same reaction any more. It might also only work in a context where students typically don’t feel as valued so that this intervention sticks out
– if students prefer the online setting over classical settings generally (d), or appreciate the flexibility (e), this might work for us while we are one of the few courses offering such an online setting. But once other courses start using similar settings, we might be competing with others, and students might spend less time with us and our practice problems again
– if students mainly work for the bonus points (f), their learning might not be as sustainable as if they were intrinsically motivated. And as soon as there are no more bonus points to be gained, they might stop using any opportunity for practice just for practice’s sake
– providing students a structure (g) might make them depend on it, harming their future learning (see my post on this Teufelskreis).

Outcome pattern
Next, we look at outcome patterns: “what are the practical effects produced by causal mechanisms being triggered in a given context?”. So which of the mechanisms identified above (and possibly others) seem to be at play in our case, and how do they balance each other? For this, we clearly need a different method than “just” measuring the learning gain in an experimental group and compare it to a control group. We need a way to identify the mechanisms at play in our case, and those that are not. We then need to figure out the balance of those mechanisms. Is the increased interest in mathematics more important than students potentially being put off by the online setting? Or is the online setting so appealing that it compensates for the lack of interest in mathematics? Can we show students that we care about them without rolling out new interventions every semester, and will that motivate them to work with us? Do we really need to show the practical application of every tiny piece of mathematics in order for students to want to learn it, or can we make them trust us that we are only teaching what they will need, even if they aren’t yet able to see what they will need it for?

This is where I am currently at. Any ideas of how to proceed?

CMOCs
And finally, we have reached the CMOCs (“How are changes in regularity (outcomes) produced by measures introduced to modify the context and balance of mechanisms triggered?”). Assuming we have identified the outcome patterns, we would need to figure out how to change those outcome patterns, either by changing the context, or by changing the balance of mechanisms being triggered.

After reading this article and applying the concept to my project (and I only read the article today, so my thoughts will hopefully evolve some over the next couple of weeks!), I feel that the control group study that everybody seems to expect from us is not as valid as most people might think. As I said above, I don’t have a good answer yet for what we should do instead. But I found it very eye-opening to think about evaluations in this way and am confident that we will figure it out eventually! Luckily we have only run a small-scale pilot at this point, and there is still some time before we start rolling out the full intervention.

What do you think? How should we proceed?

How to learn most efficiently when participating in a MOOC

2015-09-19 14.29.49

How to learn most efficiently when participating in a MOOC? Yes, I’ll admit, that title promises quite a lot. But there is a new article by Yong and Lim (2016) called “Observing the Testing Effect using Coursera Video-Recorded Lectures” that tells us a lot about how (not) to learn. We have talked about the testing effect before: repeated testing leads to better results on examinations that repeated studying does. And it is confirmed again in this study.

Why am I so excited about this? Because both video-based studying and testing are becoming more and more common these days, and both are sometimes made out to be really bad ideas.

We find video-based learning in most aspects of our lives now (at least if we are talking about lives similar to mine ;-)) — I always follow one or two Coursera courses at the time, and I love watching TED talks. Most softwares I use have video tutorials, and in fact I talked about how I liked the video tutorials of the Monash simple climate model interface only on Tuesday. And whenever I get stuck with a task, I watch video tutorials on youtube to get me going again. And of course many of the lectures at my university are being recorded and many students rely on re-watching them when studying for exams. And, of course, there is the One Planet — One Ocean MOOC that I am involved in preparing. So obviously I see value in video lectures. Even though many people believe that re-watching a lecture does not provide the same experience as seeing it “live”, I don’t think that matters much for lectures in which there is not a lot interaction between lecturer and audience. If you can make yourself use them wisely, I think video lectures are a great substitute for lectures you — for whatever reasons — can’t watch live.

But this is also the biggest issue I have with video lectures: they can easily seduce us into believing that we are learning, when we in fact are not. For example, when I say that I am “following” those Coursera MOOCs, what that means is that I have videos playing while I do something else (like writing emails or cleaning my apartment), i.e. I am not listening carefully, and I never ever do the tests and quizzes they provide. Yet, I still feel like I am learning something. I might or might not* be, but in any case I am not using those resources as effectively as I could be, and in fact most people aren’t.

And testing, I get it: Educators typically don’t like designing tests, because it is really hard. And most students don’t like taking tests, again because it is really hard, so tests have a really bad reputation all around. Especially repeated testing and e-assessment (like we are developing for mathematics and mechanics) people really love to hate!

But this is where the Yong & Lim (2016) study comes in. They showed a short (<3min) Coursera lecture to their participants. Depending on the group, during study time, they showed the clip either once and then tested three times, showed it three times and tested once, or showed it four times. Initial recall right after the study period is best for the group that watched the same clip four times, but it turns out that both groups that test during studying perform significantly better on a test a week after the study period: testing as part of studying (and in contrast to just repeatedly watching a clip) helped anchor the new knowledge significantly better.

From this is it clear that we should definitely be taking advantage of the tests provided with video lectures! Or if there are no tests available, like with TED talks**, instead of watching a lecture over and over again, test ourselves on it: Can I remember the main points? What were the reasons for x or the steps in y? Why did she say z?

And, more importantly, as educators we should take these results to heart, too.  If testing is this important, we need to provide good tests to students, and we need to encourage them to use them to practice.

One scary fact to end this post with: Of the 30 idea units presented in the videos of the study, the best group retained on average only about half until a week after watching those videos. And the worst group only one-third. I didn’t see those videos so I can’t speak about how well they were made and whether the tests addressed all of those 30 idea units, but I wouldn’t bet on students remembering more of the videos I want them to learn from. Which really gives me something to think about.

*watching those videos and feeling good about doing something productive might actually just give me the illusion of competence

**or if we feel that the tests are really bad, which does happen

Yong, P., & Lim, S. (2016). Observing the Testing Effect using Coursera Video-Recorded Lectures: A Preliminary Study Frontiers in Psychology, 6 DOI: 10.3389/fpsyg.2015.02064

How your behavior as an instructor influences how your students behave during peer instruction phases

IMG_9245

It probably doesn’t come as a surprise to you that how you behave as an instructor influences how your students work during peer instruction phases. But do you know what you can do to make sure that student discussions are reaching the level of critical thinking that you want? I.e., how do you construct classroom norms? There is a paper by Turpen and Finkelstein (2010) that investigates just that.

In their study, they focus on three factors of classroom culture: faculty-student collaboration, student-student collaboration and sense-making vs answer-making. For this, they use Mazur-like sequence of Peer Instruction (PI) (except that they usually omit the first silent phase) and compare their observations of instructor behavior with student observations.
On the continuum between low and high faculty-student collaboration, there are a couple of behaviors in which mainly those instructors engage who have a high collaboration with students: leaving the stage during PI phases to walk around and listen to or engage in student discussions, answering student questions, and hear student explanations publicly (often several explanations from different students). Here students have many opportunities to discuss with the instructor, and the correct response is often withheld until the students have reached a consensus. Unsurprisingly, in classes where instructors are on the high end of faculty-student collaborations, students talk to the instructor more often, have lower thresholds of asking questions, and feel more comfortable discussing with the instructor.
Looking at student-student collaboration, there are again instructor practices that appear helpful. For example, low-stakes grading does provoke competitive behavior the same way high-stakes grading would.
When using clickers, collaboration is more prevalent when discussion phases are sufficiently long, when collaboration is explicitly encouraged (“talk to your neighbor!”), and when the instructor often models scientific discourse. Modeling scientific discourse (“can you explain your assumption?”) is more effective when the instructor talks to student groups during peer instruction and they have the chance to practice the behavior rather than being one out of several hundred students listening passively, but even modeling the behavior you want in front of the class is better than not doing it.
Sense-making (in contrast to answer-making) can be encouraged by the instructor through practices like explicitly putting emphasis on sense-making, reasoning, discussion, rather than just picking an answer, which means that ample time for discussions needs to be given.
Another practice is providing explanations for correct answers (also in the lecture notes) rather than just which answer was correct.
I find it really interesting to see that the observations made by researchers on concrete teaching practices can be related to what students perceive the classroom norms in a particular course are. This means that you can explicitly employ those behaviors to influence the norms in your own classroom and create a climate where there is more interaction both between the students and yourself, and among the students. So next time you are frustrated about how students aren’t asking questions even though they obviously haven’t understood a concept, or about how they just pick a random answer without sufficiently thinking about the reasons, maybe try to encourage the behavior you want by explicitly stating what you want (and why) and by modeling it yourself?


Turpen, C., & Finkelstein, N. (2010). The construction of different classroom norms during Peer Instruction: Students perceive differences Physical Review Special Topics – Physics Education Research, 6 (2) DOI: 10.1103/PhysRevSTPER.6.020123