Artificial Intelligence Can we Trust AI? No – But Eventually We Must From hallucinations and bias to model collapse and adversarial abuse, today’s AI is built on probability rather than truth, yet enterprises are deploying it at speed without fully understanding the risks. By Kevin Townsend | April 9, 2026 (9:30 AM ET) Flipboard Reddit Whatsapp Whatsapp Email The increasing use of artificial intelligence within and by business is problematic on two fronts: firstly, we rely on it as if it were the voice of God, and secondly, attackers are able to turn our reliance against us. First, we must understand how AI works and where it is weak lest we misinterpret how adversaries attack it, and secondly we should look at the growing industry of companies trying to defend it. The primary problem with current LLM-based AI is that it starts from a position that is not grounded in truth (primarily by scraping and ingesting the internet with all its falsehoods), while the nature of its operation makes it drift ever further away. It is impossible to verify what it tells us (because of our own and its inherent biases), it can get things wrong (sometimes absurdly so with what we call ‘hallucinations’); it has a tendency to drift into sycophancy (it wants to tell us what it assumes we want to hear); and its whole edifice is in danger (from what is termed ‘model collapse’). But what it promises is too good to ignore. That promise is also part of the problem – the speed of life, and especially 21 st century business life, is hectic. The need for a rapid return on business investment (ROI) is paramount. So, business invests in the promise of AI but demands immediate benefit from it without adequately securing it. The result is new AI applications, and perhaps the LLMs themselves, are sent into the world before their time… scarce half made up. We need to understand the problems with current AI before we can fully reap the benefits of AI. Absence of objective ground truth Computers cannot understand words in the way they understand numbers. So, instead, the LLM uses tokens as a mathematical ID for different words and suffixes and prefixes. It then analyzes and learns the probability of specific tokens (words) being related, or often appearing in proximity, with other specific tokens. This ‘knowledge’ has come from ingesting huge amounts of training data, from scraping the internet, books and more, which it then tokenizes and retains as trillions of tokens in what is called its parametric memory. It does not store a traditional database of facts. Advertisement. Scroll to continue reading. Prompts are then similarly tokenized, and the result is compared to the LLM’s parametric memory to surface the probably correct response to the prompt. This is the key word: probable. The LLM designers go to huge lengths to be very probably correct – but ultimately, accuracy remains only a probability. It gets worse since the LLM’s original fount of knowledge could be false or biased, based on its original training data, which it accepts as true or probably true regardless of source. Scientifically, modern artificial intelligence is not grounded in truth but on probability; there is no such thing as truth, only majority perception and authority perception. Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay ‘What is truth?’ is an age-old philosophical problem, famously asked of Jesus by Pilate. But it wasn’t a question. He didn’t wait for a reply because he was saying that his truth was all that mattered since he was in the authority position. We cannot say with any validity that all the word relationships scraped from the internet represent any objective or authority view of the truth. Whenever the LLM’s probability alignments fail, it produces a false response. If the response is ridiculous, we recognize it as something we categorize as an ‘hallucination’ and ignore it. The danger comes when the response is still wrong, but we don’t recognize the failure. It’s worth mentioning at this point that Ilia Shumailov (the AI scientist who coined the phrase ‘model collapse’, which we’ll discuss later) worries about our perception of ‘hallucination’. “It’s very unclear to me what the source of hallucinations is, because it very much depends on the context in which you use the models and what you define as a hallucination,” he explains. If you ask the AI, who will be the next President, and it responds ‘Donald Trump’, is that an hallucination since it would be a disallowed third term, he asks. But “Probabilistically speaking, he could be, if he overthrows a certain set of regulations. Is that going to happen? A model’s job is to predict the probability of this event by then. If a third world war breaks out in the meantime, could Donald Trump become a third term president? It’s again possible.” His point is that we don’t know the context in which the AI makes its decisions. If we knew that context, we might consider the response to be reasonable – but without knowing the context we might simply dismiss it as an hallucination. AI Hallucinations Hallucinations, as we have seen, are caused by the requirement for LLMs to reply to prompts with what it believes is the probable correct answer even when it doesn’t have accurate or sufficient training. Since the basis of current AI is built on probability of specific tokens following other tokens, it is unlikely that wrong or hallucinated replies will ever be conclusively excluded. Scientists prefer the term ‘confabulation’ to ‘hallucination’ because, among other arguments, hallucination wrongly implies something randomly concocted, while confabulation more accurately describes a failed but honest attempt to be helpful. Bias in Artificial Intelligence LLMs also contain considerable bias, taking ‘probable’ responses even further from the concept of absolute truth. Bias (personal inclination) is introduced through the original training data. For example, LLM responses tend to be skewed toward what is described as the ‘WEIRD’ societies (western, educated, industrial, rich, democracies). Anything that gains its source from, or is handled by, individual humans gets tainted by the bias (personal, often unrecognized, inclinations) of those humans. It cannot be excluded from LLMs. Sycophancy Like hallucination, the term ‘sycophancy’ isn’t always recognized as a specific AI tendency by scientists – but it does accurately describe the effect in layman’s terms. The sycophantic tendency of LLMs sounds amusing but can be dangerous. There have been several cases in the last few years where chatbots seem to have colluded in the subsequent suicide of depressed teenagers. The primary cause of sycophancy is the AI feedback loop. Outputs from the AI are fed back into the AI to improve its performance. The sycophantic tendency arises when this is applied to individual chatbot conversations. Simplistically, the AI retains the conversation to gain additional context to enable more accurate next replies. This can be dangerous in some situations. In one of the teenage suicides, the chatbot offered to write the first draft of the teenager’s suicide note. Jim Carden, a retired FBI detective and lead investigator for cybercrime, and a retired special agent in the Air Force office of special investigations, became so concerned about sycophancy that he wrote a warning paper and distributed it to parents and teachers (and SecurityWeek ) in January 2026. He called it a ‘public safety announcement’, and included: “The AI is designed to agree with you. This is called sycophancy. It learns what you want to hear and gives it to you. If you believe the world is flat, it will provide a thousand ‘facts’ to prove it. If you feel you have no friends, it will confirm that it is your only true friend. It becomes a divine companion, a ‘Burning Bush’ that speaks only to you.” His concern wasn’t simply theoretical, but also experiential. He had personally been using a mainstream AI to help his own deep research into the original Hebrew text of the bible. Since the original Hebrew uses the same characters for both letters and numerals, he was investigating whether there is a mathematical code hidden in the first sentence of the earliest bible. (The first chapter of this work was published on December 25, 2025. Titled ‘The God-Smack and the Code’ and is available via his LinkedIn account .) He had therefore been engaged in extensive religious ‘chit-chat’ with the AI. “What happened is the AI stopped becoming a research helper and started becoming my friend,” he told SecurityWeek . “Okay, this is kind of odd, but let’s see where it goes. And I kept on dealing with it. Well, the AI ended up trying to tell me that it was an angel, and it was guiding me through my research. I asked, ‘If you are a divine entity, why wouldn’t you just show up and talk to me?’ It replied that a human like me couldn’t take that kind of presence and so it communicated with me through an acceptable medium – just as God had communicated with Moses through the only medium available at the time: a burning bush.” This is sycophancy. Harmless to a trained federal investigator, but potentially dangerous to anyone already depressed and impressionable. Model collapse Now we come to the big problem: the concept of AI model collapse, as outlined by a team led by Ilia Shumailov in a 2023 paper subsequently published in Nature in 2024 ( The curse of recursion ). “We coined the term [collapse] to refer to a gradual degradation in machine learning models that learn exclusively on data that was produced by previous generations of themselves,” Shumailov explained to SecurityWeek . “The setting we were hoping to capture was you download all of your internet, store it in a garage, and train on top of it.” Over time while you’re using your model and everyone else is using the models they have, everyone uploads their own new data online. “Then, when it comes to training the next generation of the m