The term hallucination linked to generative artificial intelligence systems has almost entered common slang. When these offer incorrect answers, it is also said that the AI is “hallucinating us”: in practice, information could come from the technology that can mislead us.
But be careful. If this reality is now clear and therefore attention must always be paid, there would also be another risk. However, in this case we are talking about conversational Artificial Intelligence. According to research that appeared on Philosophy and Technology and conducted by Lucy Osler of the University of Exeter, in fact, the passage of the virtual discussion with AI could even confirm mistakes we make, helping to amplify them.
In practice, the hallucinations in this case “would occur” with AI, capable of contributing to instilling false beliefs, distorted memories, altered personal narratives.
The risk of confirming errors
The research used a particular theory, defined as “distributed cognition”, to analyze cases in which AI systems reinforced and amplified users’ erroneous beliefs during ongoing conversations. According to Osler, if you habitually rely on generative AI to help you think, remember and narrate, you can have hallucinations with AI. But this could also occur when AI supports, confirms and elaborates our erroneous thoughts and personal narratives.
In short, if you interact with conversational AI, people’s false beliefs can not only be confirmed, but can also take root and grow more substantially, as the AI develops them further. The reason? The same scholar explains it:
“This is because generative AI often takes our interpretation of reality as the basis on which to build conversation. Interacting with generative AI has a real impact on people’s perception of what is real and what is not. The combination of technological authority and social affirmation creates an ideal environment for illusions to not only persist, but thrive.”
The role of conversational AI
Conversational AI includes various technologies based on Artificial Intelligence whereby machines can understand human language, process thoughts and respond in a completely natural way. Above all, it puts us in a position to talk to the machine as if we were talking to another person, also integrating the nuances we want to give to the words.
According to the research, conversational AI has a dual function linked precisely to the ability to represent a reliable interlocutor, which goes beyond the AI mechanism that helps people think, organize information and remember details. Precisely because it is an interlocutor, therefore like a real virtual “friend” also in tone, conversational AI could become a support in validating erroneous beliefs, which precisely because they are shared could become more credible and real.
Handle with care
So this social aspect makes chatbots fundamentally different from tools like notebooks or search engines. While traditional tools simply store or retrieve information, conversational AI can make users feel emotionally understood and socially supported.
What risk exists behind this use? Basically, again according to the study, and without going into the properly defined pathological forms of “AI-induced psychosis”, the research indicates what dangers could arise. First, AI delivers virtual assistants that are always available, highly personalized, and often designed to respond in a pleasant and supportive manner. So there is the risk that we stop and that when faced with mistaken beliefs, the AI reinforces the error through repeated conversations.
Unlike a person who could possibly challenge problematic thoughts or set boundaries, an AI system could then continue to validate improper stories. And it could even make conspiracy theories more “pleasant” and credible, helping their elaboration since AI-based virtual assistants could assist users to build increasingly complex explanations around them.
The nonjudgmental and emotionally responsive interaction of virtual assistants, therefore, may be easier or safer than human relationships. And therefore more pleasant. In short: the road is still long. For the moment, let’s always maintain our critical ability, compare sources and above all be careful not to let ourselves be “flattered” by AI!
