Observations are not as objective as we thought they were.

Today I want to talk about the paper “The theory-ladenness of observations and the theory-ladenness of the rest of the scientific process” by Brewer and Lambert (2001). I’ve been thinking about the topic of theory-ladenness of observations quite a bit recently in the context of a brochure we are currently writing on lab instruction, and I found this paper really interesting, especially because it a lot of examples are offered* which makes it an entertaining read. The authors discuss the influence of theory not only on observation, but also on attention, data interpretation, data production, memory and scientific communication. Here is my summary:

How perception is influenced by theory has for example been shown in a study where a picture similar to the one below is shown to participants, after one group had been primed with an unambiguous picture of a young woman, while the other group had seen an unambiguous figure of an old woman. Almost all participants of either group perceived the picture below as whatever had been shown to them in the unambiguous picture.

Old woman or young woman?

Another example is the figure below, where showing study participants a lot of animals (but no rats) dramatically shifts whether participants see a man or a rat when looking at the figure below. (Both figures are my renditions of the actual figures, btw!)

Man or rat?

Other studies find that students whose hypothesis is that a plastic ball will fall faster than a metal ball are more likely to report that their observations support that theory than the other group, who said that both balls would fall at equal speeds.

For all these cases, the observations were either ambiguous or difficult to make, resulting in weak bottom-up information which was easily overridden by the top-down theories. However, if the bottom-up information was strong enough, it would still be able to override the top-down information.

But even looking at the history of science, similar examples can be found, for example when the belief that some planets had moons made it difficult to observe the rings of Saturn as rings rather than moons. It does seem that perception is indeed theory-laden.

Attention is under cognitive control, too. You probably know the video where you are supposed to count how many times a basket ball is passed between players (or touches the ground, or whatever). For those of you who don’t know what I am talking about, I was asked to edit my original post so as not to tell you about it and not spoil the surprise when you do watch the video. But Malte might be writing a guest post for me on this topic :-)

Similar things have been observed throughout the history of science, too, because attention seems to be theory-directed: For example there is evidence of 22 pre-discovery observations of Uranus, that have at the time been rejected for many different reasons.

But here again: If the bottom-up processes are strong enough, you will see it even if you did not expect to see it.

Data evaluation and interpretation are also influenced by existing theories. It has been shown that scientists, probably not consciously, try to avoid having to change their theories. Data that is consistent with participants’ theories is considered more believable.

Additionally, having a theory can help make sense of and interpret data. The authors give the example of “The haystack was important because the cloth ripped”, which makes a lot more sense if a theoretical framework is given, e.g. “parachute”, and which is a lot easier to remember with that theoretical framework, too.

Even though again top-down processes play an important role in data interpretation, these are typically constrained by bottom-up processes.

In data production, “intellectual phase locking” has been observed, i.e. that measurements of “constants” tend to cluster around the same value for a while, and then jump to a different value, where they cluster again. This is indicative of the tendency to believe earlier, established measurements more than newer measurements, and hence tune instrumentation towards the established values of certain properties.

And I am sure we can all think of moments where a new piece of instrumentation showed something and we rejected it right away because it did not match the value we expected. And probably our assumption that the new equipment needed to be fine-tuned was correct. But then maybe it was not and we just missed the discovery of our life.

Last but not least, memory. Here it has been shown that memory errors are based on pre-existing theories: Information confirming a theory is easier to recall than information that deviates from the theory, which might even be distorted to match the theory better. The recognition of this has led to attempts to counteract memory errors, like for example lab book-keeping.

Again – I am sure we can all think of situations where our memory played tricks on us.

Ok, and one more: Communication. According to the authors, formalized ways of communication are structured as to reduce and organize information. For example, in a standard peer review process, information that doesn’t seem relevant to the topic at hand gets kicked out – a process that is clearly theory-laden.

This, to me, was actually the most scary point the authors make, because we are used to think that structuring a paper and omitting all non-relevant information improves the work. And I never stopped to think about how all the information that did not make it into my papers might not have been “objectively” not relevant, but might have been discarded based on my subjective perception of relevance to the topic.

To summarize: Theories DO influence perception. However, if bottom-up evidence is strong enough, it might still be able to override the theoretical top-down mis-perceptions. The authors conclude saying that “theories may have their greatest impact, not in observation, but in other cognitive processes such as data gathering, interpretation, and evaluation.

Wow. Now I will retire to my winter garden to think about what that means not only for my teaching, but maybe even more for my research…

P.S.: Discussing with my colleague P I realized that we might have to define the term “theory” at some point, because my understanding is clearly different from his…

—

*since this is an overview paper, the examples come from all kinds of different papers which I am not referring to here, because I haven’t looked at them. But they are properly referenced in the Brewer&Lambert (2001) paper, so please go check them out there!