How theories influence the scientific process

Observations are not as objective as we thought they were.

Today I want to talk about the paper “The theory-ladenness of observations and the theory-ladenness of the rest of the scientific process” by Brewer and Lambert (2001). I’ve been thinking about the topic of theory-ladenness of observations quite a bit recently in the context of a brochure we are currently writing on lab instruction, and I found this paper really interesting, especially because it a lot of examples are offered* which makes it an entertaining read. The authors discuss the influence of theory not only on observation, but also on attention, data interpretation, data production, memory and scientific communication. Here is my summary:

How perception is influenced by theory has for example been shown in a study where a picture similar to the one below is shown to participants, after one group had been primed with an unambiguous picture of a young woman, while the other group had seen an unambiguous figure of an old woman. Almost all participants of either group perceived the picture below as whatever had been shown to them in the unambiguous picture.

Screen Shot 2015-03-06 at 11.37.38
Old woman or young woman?

Another example is the figure below, where showing study participants a lot of animals (but no rats) dramatically shifts whether participants see a man or a rat when looking at the figure below. (Both figures are my renditions of the actual figures, btw!)

Screen Shot 2015-03-06 at 11.37.48
Man or rat?

Other studies find that students whose hypothesis is that a plastic ball will fall faster than a metal ball are more likely to report that their observations support that theory than the other group, who said that both balls would fall at equal speeds.

For all these cases, the observations were either ambiguous or difficult to make, resulting in weak bottom-up information which was easily overridden by the top-down theories. However, if the bottom-up information was strong enough, it would still be able to override the top-down information.

But even looking at the history of science, similar examples can be found, for example when the belief that some planets had moons made it difficult to observe the rings of Saturn as rings rather than moons. It does seem that perception is indeed theory-laden.

Attention is under cognitive control, too. You probably know the video where you are supposed to count how many times a basket ball is passed between players (or touches the ground, or whatever). For those of you who don’t know what I am talking about, I was asked to edit my original post so as not to tell you about it and not spoil the surprise when you do watch the video. But Malte might be writing a guest post for me on this topic :-)

Similar things have been observed throughout the history of science, too, because attention seems to be theory-directed: For example there is evidence of 22 pre-discovery observations of Uranus, that have at the time been rejected for many different reasons.

But here again: If the bottom-up processes are strong enough, you will see it even if you did not expect to see it.

Data evaluation and interpretation are also influenced by existing theories. It has been shown that scientists, probably not consciously, try to avoid having to change their theories. Data that is consistent with participants’ theories is considered more believable.

Additionally, having a theory can help make sense of and interpret data. The authors give the example of “The haystack was important because the cloth ripped”, which makes a lot more sense if a theoretical framework is given, e.g. “parachute”, and which is a lot easier to remember with that theoretical framework, too.

Even though again top-down processes play an important role in data interpretation, these are typically constrained by bottom-up processes.

In data production, “intellectual phase locking” has been observed, i.e. that measurements of “constants” tend to cluster around the same value for a while, and then jump to a different value, where they cluster again. This is indicative of the tendency to believe earlier, established measurements more than newer measurements, and hence tune instrumentation towards the established values of certain properties.

And I am sure we can all think of moments where a new piece of instrumentation showed something and we rejected it right away because it did not match the value we expected. And probably our assumption that the new equipment needed to be fine-tuned was correct. But then maybe it was not and we just missed the discovery of our life.

Last but not least, memory. Here it has been shown that memory errors are based on pre-existing theories: Information confirming a theory is easier to recall than information that deviates from the theory, which might even be distorted to match the theory better. The recognition of this has led to attempts to counteract memory errors, like for example lab book-keeping.

Again – I am sure we can all think of situations where our memory played tricks on us.

Ok, and one more: Communication. According to the authors, formalized ways of communication are structured as to reduce and organize information. For example, in a standard peer review process, information that doesn’t seem relevant to the topic at hand gets kicked out – a process that is clearly theory-laden.

This, to me, was actually the most scary point the authors make, because we are used to think that structuring a paper and omitting all non-relevant information improves the work. And I never stopped to think about how all the information that did not make it into my papers might not have been “objectively” not relevant, but might have been discarded based on my subjective perception of relevance to the topic.

To summarize: Theories DO influence perception. However, if bottom-up evidence is strong enough, it might still be able to override the theoretical top-down mis-perceptions. The authors conclude saying that “theories may have their greatest impact, not in observation, but in other cognitive processes such as data gathering, interpretation, and evaluation.

Wow. Now I will retire to my winter garden to think about what that means not only for my teaching, but maybe even more for my research…

P.S.: Discussing with my colleague P I realized that we might have to define the term “theory” at some point, because my understanding is clearly different from his…

*since this is an overview paper, the examples come from all kinds of different papers which I am not referring to here, because I haven’t looked at them. But they are properly referenced in the Brewer&Lambert (2001) paper, so please go check them out there!

Guest post: Arctic sea ice thinning.

Exciting guest post on a newly published paper by Angelika H. H. Renner.

I’ve met Angelika on a cruise in the Antarctic Circumpolar Current a long time ago where we worked on an instrument together and created an advent calendar to keep up everybody’s morale during the second month of the cruise before flying home on christmas eve, and we’ve since gone white(ish) water kayaking, hiking in the norwegian mountains, visited each other’s institutes, helped each other out in research and teaching crises (mainly Geli helping me out, to be honest ;-), and we are planning an exciting project together. Angelika and coauthors recently published the paperEvidence of Arctic sea ice thinning from direct observations“. In today’s post, Angelika writes about how the observations that went into the paper were obtained, and I am excited to share her story – and her amazing photos – with all of you.

There’s been so much liquid water on Mirjam’s blog lately, I was happy to take her invitation for a guest blog to bring back some of the most amazing, interesting, and beautiful variation of sea water: sea ice!

Sea ice comes in various shapes, from very flat, smooth, and thin sheets of newly formed ice to huge ridges several tens of meters thick. Assessing the thickness of the sea ice cover in the Arctic remains one of the biggest challenges in sea ice research. Luckily, methods become more refined, and numbers derived from satellite measurements become more accurate and reliable, but they don’t cover a long enough period yet to say much about long-term changes.

My first proper science cruise in 2005 went to Fram Strait, the region between Greenland and Svalbard. I learned how to measure sea ice thickness the hard way: drilling holes. And more holes. And even more holes. Or the slightly-less-hard way: carry an instrument around that uses electromagnetic induction to measure ice thickness (since sea ice is much less salty than sea water and therefore much less conductive). This instrument is called ”EM31” and we kept joking that the number comes from its weight in kilograms…. So, using drills and the EM31 we measured on as many ice floes as we could and given that the cruise went all the way across Fram Strait, that gave as quite a few datapoints covering quite a large area.

These measurements have been done by the sea ice group at the Norwegian Polar Institute every summer since 2003, and in some years also in spring. It takes dedication to build such a time series! When we could, we also used an airborne version of the EM31, the EM-bird, to do surveys over larger areas. Now, finally, the results of all these measurement have been processed, and analysed – and what do we see? The sea ice in Fram Strait is thinning a lot. Depending which measure you use (nothing about sea ice thickness is straight forward…), the ice has thinned by more than 50% over the 10 years from 2003 to 2012!

It’s one thing to know that it has thinned, but it’s a lot more interesting to find out why. Fram Strait is a special place: Most of the sea ice that is formed somewhere in the Arctic Ocean (and doesn’t melt there again) leaves the Arctic through Fram Strait. It is a very dynamic region with strong currents and winds, which results in a lot of deformed ice regardless of its age. The extent of the ice cover here is not necessarily linked to the development of the ice in the Arctic Basin – most prominent example was the heavy ice year in Fram Strait 2007 whereas this was up to then the year with the lowest Arctic-wide ice extent in the satellite era.

We looked in more detail at where the ice came from and found that this, too, does not correlate with our thickness time series. While the source region of the ice varied from year to year, it was continuously thinning – in our opinion a sign that the thinning occurs Arctic-wide.

A lot of effort went into this paper and the dataset behind it, and I was very very lucky that I got the opportunity to participate in several of the cruises, do the data analysis and write the paper. It’s even more satisfying to see your work published when you know how much work drilling all those holes was……