Currently reading Kotz et al. (2026) on “Conversational AI shifts beliefs and policy support among skeptics across contested societal issues”

The central question in Kotz et al. (2026)’s preprint is about “understanding how to increase evidence-consistent beliefs and support for empirically grounded policy responses” on contested societal issues. They investigate whether conversations with LLMs can change peoples’ minds, and if so for what types of audiences it works best. They build on other research that has shown AI to be “an effective intervention tool“, capable of changing beliefs and decreasing polarization (but also increasing polarization, just in a different set of studies).

I am reading this article from a skeptical standpoint. In their discussion, Kotz et al. (2026) write “[a] central challenge in science communication is developing approaches that generalize across issues, from beliefs to the policies that depend on them, and to skeptical audiences rather than merely reinforcing existing support“, which I don’t agree with. Communication of any sort, including scicomm, depends so much on context — on the topic itself, on the situation it is being discussed in, who is involved in that conversation, and lots of background factors, that I find it much more relevant and promising to try to understand it in context than trying to find some general approach. For example, in this study a researcher team from Germany is investigating US participants. Does culture really translate so easily that a binary measure of supporting Republicans vs Democrats tells us anything about those people’s ideas and whether these results would be transferrable to other contexts?

Anyway. They take three issues — climate change, vaccinations, and economical inequality — that are all contested, but in very different ways: In the strength of scientific consensus, whether the disagreement is based on misunderstandings of facts (as they say is the case for climate change, even though we know that what people believe about climate change does not depend on what they know about it) or different values, general support in the population, and what kind of policy instruments are used, because “[a] single communication approach that overcomes these barriers would therefore demonstrate genuinely generalizable persuasive potential“. It is however also unclear if changes in beliefs through AI interventions would result in changed policy support. They combine those three issues with corresponding policy measures (carbon tax, mandatory vaccination for school entry, estate tax) and run different interventions against a control group: a belief intervention, a policy intervention, and a combined intervention targeting both the belief and the policy.

In the study, 6.5k adult participants from the US did a three-round conversation with an AI, which “was instructed to present evidence-based arguments for the respective belief or policy grounded in scientific consensus, respond to participants’ inputs, and encourage reflection in a respectful, non-moralizing manner“. The three rounds mean that after each of three AI outputs, participants were required to reply, and couldn’t get to the next page for a minute (not a very realistic setting, me thinks…). Nevertheless, this conversation took participants on average 10 minutes, with approximately 26 to 30 words per participant reply (“indicating that participants engaged actively and substantively with the dialogues“, according to Kotz et al. (2026), but I am not sure I would buy that interpretation; they might just have gone to get a coffee upon realizing that they had to wait until they could reach the next page, and I just marked the previous 30 words in bold to show that that really is not a lot of text…).

Text analyzes of the AI responses show different “strategies”, most often “iterative collaborative dialogue” (basically Q&A) and “person-centered alignment” (eliciting values and concerns, empathetic reactions, “affirm autonomy to reduce reactance“), over “establish shared foundations” (start from common ground), “bridge to decisions & commitments” (incremental agreements from principles to choices), “explain mechanisms & analogies“, “make it concrete and local“, “layered evidence & transparent sourcing“, “steelman and fair opposition” (present strong counterarguments, acknowledge costs), “communicate uncertainty & trade-offs” and least often “prebunking and debunking” (debunking common myths and rhetorical tactics), and “use of trusted messengers“. Kotz et al. (2026) write that the AI “adapted its approach to skeptical participants, using more counter-argumentation strategies such as correcting myths and steelmanning for those with lower baseline beliefs and policy support“. What I find interesting here is that all of these strategies are commonly recommended in scicomm, so I wonder how the AI “decided” to use them; was there some instruction in the prompting (doesn’t seem so), or are those strategies commonly used selectively in the training data, with that association of strategies for skeptics being there already? Also, could one test the effectiveness of each strategy in isolation and could that inform non-AI scicomm?

Kotz et al. (2026) find that interventions are most successful on people who were initially skeptical, but don’t change much for people who were already on board. One explanation offered by Kotz et al. (2026) is “skeptical participants hold specific concerns and objections that generic informational messages cannot address, whereas already-supportive participants can be moved by general evidence-based messaging alone“. They find that “[t]hese findings have practical implications for public communication and education. Alongside demonstrating that AI dialogue can serve as a scalable tool for evidence-based communication across contested societal issues and their associated policies, the moderation by trust in science indicates that its effectiveness may depend on broader efforts to strengthen trust in scientific institutions.” I think it is interesting that they bring trust in the discussion. If we want people to trust in science, how does that fit with using AI to convince them of stuff rather than building relationships with them to talk about the issues at hand? Also, speaking of trust, why would we trust that the AI uses actual evidence-based claims in discussions and doesn’t maliciously (which is maybe not the best word, since the AI is not conscious) manipulate people? And who decides what is the correct evidence to teach? All of that might be controllable in a study through what AI model is being used and how it is being trained, but not large-scale. And I have no confidence in the benevolence of creators of the big, commonly available AIs (Kotz et al. (2026) deal with this issue by mentioning the need for deliberate and careful governance twice, but they do not elaborate on what that could look like)… Also I am wondering why people, especially skeptics, should want to discuss issues with AI in the first place, outside of a study where they are being paid to do so. In their conclusion, Kotz et al. (2026) write “[e]ffects were strongest where they matter most: among initially skeptical individuals whose attitudes are arguably most difficult to change.” But it’s not like skeptics think that they should be convinced and therefore would seek out AI to do the job… So what’s the idea for how these conversations are supposed to happen outside of a study?

This manuscript is a preprint, so things might change in peer review and in the final published version we might get answers on some of these questions. But for now, the main take-away for me is that AI conversations have the potential to change people’s minds on stuff, and while that can be used for good, there are also many many ways how it can be used for not-so-good, both on purpose and even just by accident when people are looking for answers from an AI that an AI cannot give them. And also my idea of scicomm is about two-way communication, about actual engagement with other people’s ideas, about standing on some market or beach with my little plastic cups and ice cubes talking about ocean circulation, or about posting pictures of waves and explaining why they look the way they do, or about meeting people discussing different visions for the future. I don’t know which part I find more scary, the idea to do scicomm so top-down one-way let’s-inform-the-uninformed, or to do it with an AI…

Kotz, J., Tiede, K. E., Meyer, J., Sterba, M., Breunig, C., & Gaissmaier, W. (2026, July 9). Conversational AI shifts beliefs and policy support among skeptics across contested societal issues. Preprint. https://doi.org/10.17605/OSF.IO/23FNY

Some pictures from a dip a while ago…

Picknick bench back in place, and I like this little sailboat!

And beautiful waves!

The best water pictures are taken from within wave troughs, looking up to the crests!

The waves are actually much smaller than they seem from this angle.

Love the tiny little sun glint!

Even looking at pictures of water has such a calming effect on me!

The little sailboat again…

And Öresundsbridge at the horizon…