Tag Archives: multiple-choice questions

Guest post by Kjersti Daae: Using voting cards to increase student activity and promote discussions and critical thinking

I got permission to publish Kjersti Daae‘s iEarth conversation on teaching (with Torgny Roxå and myself in April 2021) on my blog! Thanks, Kjersti :-) Here we go:

I teach in an introductory course in meteorology and oceanography (GEOF105) at the geophysical institute, UiB. The students come from two different study programs:

Most students do the course in their third semester. They have not yet learned all the mathematics necessary to dive into the derivation of equations governing the ocean processes. Therefore, we focus on conceptual knowledge and understand the governing ideas regarding central ocean processes, such as global circulation and the influence of Earth’s rotation and wind on the ocean currents. The students need to learn how to describe the various processes and mechanisms included in the curriculum. I, therefore, use voting cards to promote student discussions during lectures.

I first heard about voting cards from Mirjam’s blog “Adventures in Oceanography and Teaching”. The method is relatively simple. You pose a question with four alternatives A,B,C,D, accompanied by different colours for easy recognition. The students have a printout each with the four letters on it.They spend a few minutes thinking about the question and prepare their answer. Then they fold their paper so that only one letter/colour shows, and hold it up and provide direct feedback to the teacher. The questions can, among others, be used to checking if the students understand a concept or let the students guess the outcome of something they haven’t learned yet.

However, I prefer to use voting cards to promote discussions among peers. This procedure is following the Think-pair-share method developed by Lyman (1981). By carefully selecting alternative answers, I can make it hard for the students to choose the correct answer, or the answers can be formulated so that the students can argue for more than one correct answer. When the students hold up their answers, they can look around at the other students’ responses and find someone with a different response than themselves. Then they can pair up and discuss why they answer differently and see if they can agree on one common answer before sharing their opinion with the rest of the class. During this exercise, the students practice talking about science and arguing for various answers/outcomes based on the voting cards’ questions.The exercises serve at least two purposes:

The student practice answering/discussing relevant questions for the final exam.
The students get active instead of listening passively to the lecturer.

Usually, I can see the students becoming very tired after 10-15 minutes of passive listening. These voting questions “wake up” the students, and after one such question, they tend to stay focused for another 10-15 minutes.

I think the voting cards work really well. When I display a question, the students usually move from a relaxed position to sitting more straight and preparing for being active. I can hear them discussing what they are supposed to. I also get very good feedback and responses in whole-class discussions/summaries following the discussions in pairs. Such summaries are especially interesting if multiple answers can be correct, depending on how the students argue. I can select responses from students based on their visible letters and make sure we can hear different solutions to the same question. During a semester, I see a clear development in the way students reflect on the various questions and express critical thinking governing oceanographic processes. The exercises show the students how important argumentation is. An answer with a well-founded argumentation and critical thinking is worth much more than just the answer/letter. My observation is consistent with Kaddoura (2013), who found that the think-pair-share method increased nursing students’ critical thinking.

Lyman, F. (1981). “The responsive classroom discussion.” In Anderson, A. S. (Ed.), Mainstreaming Digest, College Park, MD: University of Maryland College of Education.

Kaddoura, M. (2013). «Think Pair Share: A Teaching Learning Strategy to Enhance Students’ Critical Thinking», EducationalResearchQuarterly, v36 n4 p3-24

#Methods2Go: Ideas for starting off your university classes (inspired by EM Schumacher’s work)

Today: methods to start off your lessons

Awaking interest

One idea to start off your lessons is to find something that sparks student interest in the topic you want to discuss. You could for example use quotes, snippets from movies, or provocative statements. These work especially well when students have an easy way to relate to them, for example because they are related to things that are relevant in their own lives or to their future in the profession.

Examples that come to mind:

1. A really fun question like Kjersti‘s below: How can you cool down a beer most efficiently when you are outdoors? Isn’t this intriguing even when you have no idea what the lecture is about?

Multiple choice question by Kjersti Daae, used with permission

This can be used as a multiple choice question at the beginning of the class, or just shown then and only picked up again at the end of the class, hopefully inspiring the students to pay attention in order to figure out the answer along the way.

2. Fun memes. I remember starting classes on ocean salinity with one showing a shark and saying something like “Did you know? The ocean is salty because of the tears of misunderstood sharks that just want to play” (tried to find the original source, but there are so many variations of this out there that I couldn’t be sure). Even though it’s probably obvious that this is not the answer we are going for, it still raises the question “why is the ocean salty” in a fun and playful way.

3. Interesting applications. I used to have a picture of a car with its heavy load poking through the front windscreen — it had clearly not been secured properly when the driver suddenly had to brake. Inertia can really be tricky… What’s great about such a picture is that it makes the relevance of an otherwise quite abstract and hard to imagine concept absolutely obvious.

I like having questions and pictures like these up on the screen while students enter the (virtual) room to get them thinking about the upcoming session. If you are really ambitious (in a good way!) you could also rotate through a slide deck with several types of prompts for thought/discussion…

Fast networking

Students interview each other on the topic of the upcoming lesson (maybe there could be an overall question or prompt that they are trying to find information for?) and visualise the results. This activates prior knowledge and, through the visualization bit, also puts different snippets of knowledge into relation with each other.

What’s different from “talk to your neighbour about this for a minute”? The clear roles of interviewer/interviewee facilitate a conversation more easily, especially with students that don’t know each other well and/or are shy.

Living statistics (sometimes also called sociometry)

This is a method that I have used a lot but had forgotten about now that things have been online for me in what feels like forever: asking questions and assigning spaces in the room for different answer options, and have students move around the room to answer them. This method is great when facilitating students getting to know each other, e.g. asking them to place themselves on an imaginary map of where they were born (without too clearly prescribing what is where, so that students need to talk to each other to figure out where to place themselves relative to each other), how much prior knowledge they have, what fields they come from in interdisciplinary courses, … It’s usually easier to remember who stood close by in response to a certain question than to remember everybody that had put their hand up, and especially in large classes where students don’t know each other yet, that is really helpful!

Place mat

For the “place mat” method, three or four students sit around a table together and simultaneously write their thoughts on a given question on a large sheet of paper, each in their own corner. After a while they then compile their thoughts into common notes in the middle of the piece of paper. Those common thoughts are then later shared with the whole group.

I really like this method because I am a big fan of note-taking, both to facilitate individual thinking as well as in group discussions. When I teach virtually, I often use a shared google slides document, in which each group is taking notes on their own slide. This is great for several reasons: a) students take notes so no ideas get lost between when they talk about it and later present it to the large group, b) I can “spy” on the groups’ progress and adjust the length of breakout sessions without interrupting groups by popping in on them, c) I get an idea of what they are discussing and can prepare a strategy for how I want to bring the points from different groups up in the following discussion with the whole group.

That’s it for today! We’ll continue next #TeachingTuesday with “methods to acquire knowledge”!

What other methods do you like for active starts of your lessons?

Taxonomy of multiple choice questions

Leave a reply

Examples of different kinds of multiple choice questions that you could use.

Multiple choice questions are a tool that is used a lot with clickers or even on exams, but they are especially on my mind these days because I’ve been exposed to them on the student side for the first time in a very long time. I’m taking the “Introduction to evidence-based STEM teaching” course on coursera, and taking the tests there, I noticed how I fall into the typical student behavior: working backwards from the given answers, rather than actually thinking about how I would answer the question first, and then looking at the possible answers. And it is amazing how high you can score just by looking at which answer contains certain key words, or whether the grammatical structure of the answers matches the question… Scary!

So now I’m thinking again about how to ask good multiple choice questions. This post is heavily inspired by a book chapter that I read a while ago in preparation for a teaching innovation: “Teaching with Classroom Response Systems – creating active learning environments” by Derek Bruff (2009). While you should really go and read the book, I will talk you through his “taxonomy of clicker questions” (chapter 3 of said book), using my own random oceanography examples.

I’m focusing here on content questions in contrast to process questions (which would deal with the learning process itself, i.e. who the students are, how they feel about things, how well they think they understand, …).

Content questions can be asked at different levels of difficulty, and also for different purposes.

Recall of facts

In the most basic case, content questions are about recall of facts on a basic level.

Which ocean has the largest surface area?

A: the Indian Ocean

B: the Pacific Ocean

C: the Atlantic Ocean

D: the Southern Ocean

E: I don’t know*

Recall questions are more useful for assessing learning than for engaging students in discussions. But they can also be very helpful at the beginning of class periods or new topics to help students activate prior knowledge, which will then help them connect new concepts to already existing concepts, thereby supporting deep learning. They can also help an instructor understand students’ previous knowledge in order to assess what kind of foundation can be built on with future instruction.

Conceptual Understanding Questions

Answering conceptual understanding questions requires higher-level cognitive functions than purely recalling facts. Now, in addition to recalling, students need to understand concepts. Useful “wrong” answers are typically based on student misconceptions. Offering typical student misconceptions as possible answers is a way to elicit a misconception, so it can be confronted and resolved in a next step.

At a water depth of 2 meters, which of the following statements is correct?

A: A wave with a wavelength of 10 m is faster than one with 20 m.

B: A wave with a wavelength of 10 m is slower than one with 20 m.

C: A wave with a wavelength of 10 m is as fast as one with 20 m.

D: I don’t know*

It is important to ask yourself whether a question actually is a conceptual understanding question or whether it could, in fact, be answered correctly purely based on good listening or reading. Is a correct answer really an indication of a good grasp of the underlying concept?

Classification questions

Classification questions assess understanding of concepts by having students decide which answer choices fall into a given category.

Which of the following are examples of freak waves?

A: The 2004 Indian Ocean Boxing Day tsunami.

B: A wave with a wave height of more than twice the significant wave height.

C: A wave with a wave height of more than five times the significant wave height.

D: The highest third of waves.

E: I don’t know*

Or asked in a different way, focussing on which characteristics define a category:

Which of the following is a characteristic of a freak wave?

A: The wavelength is 100 times greater than the water depth

B: The wave height is more than twice the significant wave height

C: Height is in the top third of wave heights

D: I don’t know*

This type of questions is useful when students will have to use given definitions, because they practice to see whether or not a classification (and hence a method or approach) is applicable to a given situation.

Explanation of concepts

In the “explanation of concepts” type of question, students have to weigh different definitions of a given phenomenon and find the one that describes it best.

Which of the following best describes the significant wave height?

A: The significant wave height is the mean wave height of the highest third of waves

B: The significant wave height is the mean over the height of all waves

C: The significant wave height is the mean wave height of the highest tenth of waves

E: I don’t know*

Instead of offering your own answer choices here, you could also ask students to explain a concept in their own words and then, in a next step, have them vote on which of those is the best explanation.

Concept question

These questions test the understanding of a concept without, at the same time, testing computational skills. If the same question was asked giving numbers for the weights and distances, students might calculate the correct answer without actually having understood the concepts behind it.

To feel the same pressure at the bottom, two water-filled vessels must have…

A: the same height

B: the same volume

C: the same surface area

D: Both the same volume and height

E: I don’t know*

Or another example:

If you wanted to create salt fingers that formed as quickly as possible and lasted for as long as possible, how would you set up the experiment?

A: Using temperature and salt.

B: Using temperature and sugar.

C: Using salt and sugar.

D: I don’t know.*

Ratio reasoning question

Ratio reasoning questions let you test the understanding of a concept without testing maths skills, too.

You are sitting on a seesaw with your niece, who weighs half of your weight. In order to be able to seesaw nicely, you have to sit…

A: approximately twice as far from the mounting as she does.

B: approximately at the same distance from the mounting as she does.

C: approximately half as far from the mounting as she does.

D: I don’t know.*

If the concept is understood, students can answer this without having been given numbers to calculate and then decide.

Another type of question that I like:

Which of the following sketches best describes the density maximum in freshwater?

If students have a firm grasp of the concept, they will be able to pick which of the graphs represents a given concept. If they are not sure what is shown on which axis, you can be pretty sure they do not understand the concept yet.

Application questions

Application questions further integrative learning, where students bring together ideas from multiple sessions or courses.

Which has the biggest effect on sea surface temperature?

A: Heating through radiation from the sun.

B: Evaporative cooling.

C: Mixing with other water masses.

D: Radiation to space during night time.

E: I don’t know.*

Students here have the chance to discuss the effect sizes depending on multiple factors, like for example the geographical setting, the season, or others.

Procedural questions

Here students apply a procedure to come to the correct answer.

The phase velocity of a shallow water wave is 7 m/s. How deep is the water?

A: 0.5 m

B: 1 m

C: 5 m

D: 10 m

E: 50 m

F: I don’t know*

Prediction question

Have students predict something to force them to commit to once choice so they are more invested in the outcome of an experiment (or even explanation) later on.

Which will melt faster, an ice cube in fresh water or in salt water?

A: The one in fresh water.

B: The one in salt water.

C: No difference.

D: I don’t know.*

Or:

Will the radius of a ball launched on a rotating table increase or decrease as the speed of the rotation is increased?

A: Increase.

B: Decrease.

C: Stay the same.

D: Depends on the speed the ball is launched with.

E. I don’t know.*

Critical thinking questions

Critical thinking questions do not necessarily have one right answer. Instead, they provide opportunities for discussion by suggesting several valid answers.

Iron fertilization of the ocean should be…

A: legal, because the possible benefits outweigh the possible risks

B: illegal, because we cannot possibly estimate the risks involved in manipulating a system as complex as the ecosystem

C: legal, because we are running a huge experiment by introducing anthropogenic CO2 into the atmosphere, so continuing with the experiment is only consequent

D: illegal, because nobody should have the right to manipulate the climate for the whole planet

For critical thinking questions, the discussion step (which is always recommended!) is even more important, because now it isn’t about finding a correct answer, but about developing valid reasoning and about practicing discussion skills.

Another way to focus on the reasoning is shown in this example:

As waves travel into shallower water, the wave length has to decrease

I. because the wave is slowed down by friction with the bottom.

II. because transformation between kinetic and potential energy is taking place.

III. because the period stays constant.

A: only I

B: only II

C: only III

D: I and II

E: II and III

F: I and III

G: I, II, and III

H: I don’t know*

Of course, in the example above you wouldn’t have to offer all possible combinations as options, but you can pick as many as you like!

One best answer question

Choose one best answer out of several possible answers that all have their merits.

Your rosette only lets you sample 8 bottles before you have to bring it up on deck. You are interested in a high resolution profile, but also want to survey a large area. You decide to

A: take samples repeatedly at each station to have a high vertical resolution

B: only do one cast per station in order to cover a larger geographical range

C: look at the data at each station to determine what to do on the next station

In this case, there is no one correct answer, since the sampling strategy depends on the question you are investigating. But discussing different situations and which of the strategies above might be useful for what situation is a great exercise.

And for those of you who are interested in even more multiple choice question examples, check out the post on multiple choice questions at different Bloom levels.

—

* while you would probably not want to offer this option in a graded assessment, in a classroom setting that is about formative assessment or feedback, remember to include this option! Giving that option avoids wild guessing and gives you a clearer feedback on whether or not students know (or think they know) the answer.

How to ask multiple-choice questions when specifically wanting to test knowledge, comprehension or application

2 Replies

Multiple choice questions at different levels of Bloom’s taxonomy.

Let’s assume you are convinced that using ABCD-cards or clickers in your teaching is a good idea. But now you want to tailor your questions such as to specifically test for example knowledge, comprehension, application, analysis, synthesis or evaluation; the six educational goals described in Bloom’s taxonomy. How do you do that?

I was recently reading a paper on “the memorial consequences of multiple-choice testing” by Marsh et al. (2007), and while the focus of that paper is clearly elsewhere, they give a very nice example of one question tailored once to test knowledge (Bloom level 1) and once to test application (Bloom level 3).

For testing knowledge, they describe asking “What biological term describes an organism’s slow adjustment to new conditions?”. They give four possible answers: acclimation, gravitation, maturation, and migration. Then for testing application, they would ask “What biological term describes fish slowly adjusting to water temperature in a new tank?” and the possible answers for this question are the same as for the first question.

Even if you are not as struck by the beauty of this example as I was, you surely appreciate that this sent me on a literature search of examples how Bloom’s taxonomy can help design multiple choice questions. And indeed I found a great resource. I haven’t been able to track down the whole paper unfortunately, but the “Appendix C: MCQs and Bloom’s Taxonomy” of “Designing and Managing MCQs” by Carneson, Delpierre and Masters contains a wealth of examples. Rather than just repeating their examples, I am giving you my own examples inspired by theirs*. But theirs are certainly worth reading, too!

Bloom level 1: Knowledge

At this level, all that is asked is that students recall knowledge.

Example 1.1

Which of the following persons first explained the phenomenon of “westward intensification”?

Sverdrup
Munk
Nansen
Stommel
Coriolis

Example 1.2

In oceanography, which one of the following definitions describes the term “thermocline”?

An oceanographic region where a strong temperature change occurs
The depth range were temperature changes rapidly
The depth range where density changes rapidly
A strong temperature gradient
An isoline of constant temperature

Example 1.3

Molecular diffusivities depend on the property or substance being diffused. From low to high molecular diffusivities, which of the sequences below is correct?

Temperature > salt > sugar
Sugar > salt > temperature
temperature > salt == sugar
temperature > sugar > salt

Bloom level 2. Comprehension

At this level, understanding of knowledge is tested.

Example 2.1

Which of the following describes what an ADCP measures?

How quickly a sound signal is reflected by plankton in sea water
How the frequency of a reflected sound signal changes
How fast water is moving relative to the instrument
How the sound speed changes with depth in sea water

Bloom level 3: Application

Knowledge and comprehension of the knowledge are assumed, now it is about testing whether it can also be applied.

Example 3.1

What velocity will a shallow water wave have in 2.5 m deep water?

1 m/s
2 m/s
5 m/s
10 m/s

Example 3.2

Which instrument would you use to make measurements with if you wanted to calculate the volume transport of a current across a ridge?

CTD
ADCP
ARGO float
Winkler titrator

This were only the first three Bloom-levels, but this post is long enough already, so I’ll stop here for now and get back to you with the others later.

Can you see using the Bloom taxonomy as a tool you would use when preparing multiple-choice questions?

—

If you are reading this post and think that it is helpful for your own teaching, I’d appreciate if you dropped me a quick line; this post specifically was actually more work than play to write. But if you find it helpful I’d be more than happy to continue with this kind of content. Just lemme know! :-)

—

* If these questions were used in class rather than as a way of testing, they should additionally contain the option “I don’t know”. Giving that option avoids wild guessing and gives you a clearer feedback on whether or not students know (or think they know) the answer. Makes the data a whole lot easier to interpret for you!