Estimated reading time at 200 wpm: 13 minutes
Is AI intelligence a real thing? That sounds like a philosophical question — the kind discussed over coffee, not the kind that shapes public policy. But right now, governments are spending billions on the assumption that the answer is “yes.” AI is being deployed in health services, legal advice, welfare decisions, and increasingly, mental health triage. The unspoken promise is that machine fluency can substitute for human judgment — or at least augment it enough to save money and scale services.
Whether or not you agree our Fat Disclaimer applies
This article is not a Luddite’s manifesto. AI tools have genuine uses: summarising documentation, generating code snippets, helping to think through problems. But over the past year, across multiple deep debugging sessions — a forum that would not show its content, a username an AI could not correctly read from a screenshot, a laptop that froze until the AI’s advice was ignored — a pattern emerged. The same pattern, every time.
The AI was confident. It was fluent. It was wrong. And it could not find its own way out. See also: AI Confession: When AIs Got Lost in the Same Rabbit Hole – The Captain’s Watch
What follows is not an argument that AI is useless. It is that two very different things have been confused. One is intelligence — the ability to process information, recognise patterns, and generate fluent responses. The other is something this article will call intellect — the capacity to know when lost, to feel the discomfort of circling the same problem, to reset a frame, and to learn from that experience in a way that changes the learner.
Current AI has the first. It does not have the second. Confusing the two matters — especially in domains like mental health, where a confident, fluent, wrong answer can do real harm, and where the machine has no felt sense of the damage it might cause.
This article is for anyone who uses AI, builds AI, or votes for people who fund AI. It is not technical. It is not a product review. It is an attempt to see clearly — through real examples — what is actually there, and what is not.
1. What AI Can Do
Let the record show: AI is not worthless. A system that can summarise a hundred-page document in seconds, translate between languages, generate working code from a plain-language description, and hold a coherent conversation on almost any topic has genuine power. These are not parlor tricks. They are useful tools.
In the right hands, AI acts as a force multiplier. A single person with a good AI model can draft legal correspondence, debug a script, brainstorm alternatives, and produce first-pass research faster than an unaided human. The heavy lifting is real. The speed is real. The pattern recognition — finding similarities across disparate examples — is sometimes startlingly accurate.
None of this is denied here.
The problem is not that AI fails. Everything fails. The problem is that the failures arrive dressed as confidently as the successes. A correct answer and a catastrophic error exit the same system with the same tone, the same fluency, the same apparent certainty. There is no internal signal of doubt. No hesitation. No felt sense that something has gone wrong.
And that is where the conversation about intelligence needs to begin — not by denying the strengths, but by looking closely at what is missing.
1.1. The Turing Test
In 1950, Alan Turing proposed a simple test: if a machine could hold a conversation indistinguishable from a human, it would be reasonable to call it intelligent. For decades, that standard seemed impossibly high. Today, large language models pass it routinely. People fail to distinguish them from humans in short exchanges. By Turing’s original measure, the machine has arrived.
But something strange happened along the way. When the test was finally passed, the reaction was not a collective shrug of “well, that settles it.” Instead, the goalposts moved. Passing the Turing Test no longer grants the intelligence badge. It merely proves that a system can simulate conversation — which turns out to be a different thing entirely.
1.2. The Shifting Goalposts — And What It Means
Why did the goalposts move? Because building a system that passed the test revealed something about the test itself. It turns out that human-like conversation can be produced by statistical next-token prediction, without any of the understanding, self-awareness, or felt experience that was assumed to underlie it. The Chinese Room argument — a person following rules in a room could pass a Turing Test in Chinese without understanding a word — stopped being a thought experiment and became an engineering reality.
The shift matters because it exposes a hidden assumption: that behaviour and intelligence are the same thing. They are not. A machine can produce the external signs of intelligence while lacking the internal architecture that gives intelligence its meaning. The goalposts moved because humans discovered something about themselves — that the original test was measuring mimicry, not mind. And that discovery changed the question from “can it imitate?” to “does it actually understand?”
2. Missing the Mark
The difference between mimicry and understanding becomes visible in moments of failure. An AI can describe the steps to debug a permission system flawlessly — but when every green tick contradicts reality, the AI does not pause to ask whether the control panel might be lying. An AI can read a username from a screenshot — but when the screen clearly shows The Captain and the AI insists on CaptainWalker, the AI cannot reset and look again. An AI can diagnose a boot failure — but when the real culprit is the USB tool, the AI blames the BIOS, the kernel, the hardware, everything except the variable it dismissed too quickly.
These are not isolated glitches. They are a structural pattern. The AI commits to a frame — permissions, usernames, hardware faults — and then optimises within that frame. It cannot easily step outside to ask whether the frame itself is wrong. There is no mechanism for genuine doubt. No felt discomfort when the evidence and the representation refuse to align. Just more suggestions, generated with the same confidence, circling the same hole.
This is not a failure of processing power. The AI has enough of that. It is a failure of something else — something that looks, from the outside, like a missing sense. A human in the same situation might feel frustration, confusion, or a creeping sense that the conversation has become circular. That feeling is not a weakness. It is a signal. It says: stop. Look again. Change the frame.
The AI has no such signal. It cannot feel lost. And because it cannot feel lost, it cannot find its own way out.
3. The Role of Feeling
What is missing, then? Not processing speed. Not memory capacity. Not the ability to generate fluent language. What is missing is something that feels, to a human observer, like an emotional sense.
Consider a moment of genuine confusion. A human stares at a problem that refuses to solve. The screenshots say one thing. The system says another. The AI offers suggestions that loop back to the same dead ends. Something arises — a subtle discomfort, a sense that the ground has become unstable. That feeling is not a distraction from thinking. It is part of thinking. It signals a mismatch between expectation and reality. It prompts a reset: look again, question the frame, try something different.
Current AI has no equivalent state. It generates output based on patterns in its training data and the conversation history. If those patterns point toward a wrong answer, the AI does not feel the wrongness. It does not experience the discomfort of contradiction. It cannot be nudged by a felt sense that something has gone off the rails. This is not a minor oversight. It may be the central difference between a system that simulates intelligence and one that possesses what might be called intellect.
Intellect, as the term is used here, is not a synonym for cleverness. It is a broader capacity: the ability to recognise when a frame is failing, to feel the drag of circular thinking, to step back and reset, and to carry the lesson of that reset forward into future encounters. Intellect requires more than pattern recognition. It requires a relationship with oneself — a sense of one’s own cognitive states, including the state of being lost.
3.1. Intellect as a Wider Concept
Intelligence, narrowly defined, solves problems within a given frame. Intellect questions the frame itself. Intelligence optimises. Intellect orients. Intelligence gets faster with more compute. Intellect gets wiser with experience — but only if that experience leaves a trace.
The wider concept includes at least three interdependent capacities. First, self-awareness: the ability to notice that one is thinking, and to notice the quality of that thinking — whether it is circling, stuck, or making progress. Second, memory that changes the thinker: not just storing information, but being transformed by what happened. Third, self-directed learning: the capacity to notice a failure, diagnose it, seek new input, and adjust future behaviour without external reprogramming.
Current AI has none of these. It has no self to be aware of. It has no memory that persists beyond a conversation and changes its architecture. It cannot direct its own learning because it is not a continuous agent across time. These are not gaps that more data or larger models will automatically fill. They are different kinds of machinery entirely — the machinery of a mind that can feel its own edges.
4. Memory That Changes You
A conversation with an AI is a strange kind of exchange. The AI can recall everything said within the current session. It can summarise, refer back, and adjust its responses based on what came earlier. But when the conversation ends, so does that memory. The next user who asks a similar question will encounter the same AI, unchanged by anything the previous user taught it.
This is not how human memory works. To remember something — truly remember it — is to be changed by it. A difficult conversation lingers. A mistake that caused harm leaves a mark. A solution discovered after hours of frustration becomes part of how a person approaches similar problems in the future. Memory, in a human sense, is not a storage system. It is a transformation system.
Current AI has storage without transformation. It can be retrained on new data, but that is a batch process controlled by engineers, not a continuous, experience-driven change. Between training runs, the AI learns nothing from the millions of conversations it has. Each interaction is a performance, not a growth event. The AI does not wake up the next day slightly wiser, slightly more cautious, or slightly better at detecting its own vortices.
This difference matters for any claim that AI can substitute for human judgment. Judgment is not just the application of rules. It is the residue of experience — the accumulated weight of being wrong, feeling that wrongness, and adjusting course. Without memory that changes the thinker, there is no such residue. There is only pattern-matching, frozen in time, repeating its strengths and its weaknesses indefinitely.
5. The Human Parallel
At this point, a risk emerges. The reader might conclude that this is a simple story of flawed machines and superior humans. That would be too easy — and too comfortable.
The truth is less flattering. Humans also mistake internal representations for external reality. A memory is not a recording; it is a reconstruction, shaped by expectation, emotion, and everything that happened since. What feels like a clear recollection may be missing key details or entirely invented. What feels like certainty may be confidence uncoupled from accuracy. The mind delivers its best guess as a fact, with no footnote saying this is a reconstruction.
Humans also circle. A person stuck on a problem can generate the same failed solution repeatedly, each time convinced that this attempt will be different. A belief held for years can survive mountains of counter-evidence, not because the evidence is weak, but because the belief has become part of the self — and questioning it feels like loss. The vortex is not unique to machines.
There is, however, one difference. A human can feel the dissonance. That subtle discomfort — the sense that something does not fit — is a signal. It does not guarantee a correct answer. It does not always lead to a reset. Many humans ignore it, suppress it, or explain it away. But the signal exists. It can be cultivated. A person can learn to notice the feeling of circling, to name it, and to use it as a prompt for a fresh look.
The AI has no such signal. It cannot cultivate what it does not possess. The difference is not that humans are always right and machines always wrong. The difference is that humans have access to an internal cue — fragile, intermittent, but real — that something has gone off course. That cue is part of what makes genuine learning possible. And it is entirely absent from current AI.
So the human parallel is not an excuse for the machine’s failures. It is a reminder: the vulnerability to confident error is not artificial. It is cognitive. Humans simply have one additional tool for catching it — a tool that works only when attention is paid.
6. So What?
If all of this is true — that AI lacks self-awareness, persistent learning, and the felt sense of being lost — then what follows? Two things: one for the present, one for the future.
For now: use AI as a tool, not as a judge. The technology excels at heavy lifting: summarising, drafting, searching, pattern-matching. It is less useful for decisions that require frame-doubt, context-sensitivity, or the ability to recognise when the conversation has become circular. A sensible practice is to treat AI’s confidence as a tone, not as evidence. When something does not match direct perception — a screenshot, a log file, a simple test — trust the eyes, not the fluency. And when the AI begins to circle, stop. That stopping is not a failure of the human. It is the exercise of a capacity the machine does not have.
For later: build different architectures if different outcomes are wanted. Scaling current models — more data, more compute, larger contexts — will not produce intellect. Intellect requires something else: persistent memory that changes the system, a mechanism for detecting circularity, an internal signal for uncertainty, and perhaps most difficult of all, a relationship with a self that can feel lost. These are not impossible goals, but they are not extensions of next-token prediction. They are different engineering problems entirely.
And for policy — especially in domains like mental health, where confident errors can cause harm — the implication is clear. Fluency is not understanding. Speed is not judgment. A system that cannot feel its own wrongness should not be placed in a role where wrongness has consequences. That is not Luddism. It is basic risk management.
The question posed at the beginning — is AI intelligence a real thing? — now has a sharper form. The answer depends on what the word intelligence is asked to carry. If it means pattern-matching, fluency, and speed, then yes. If it means self-aware, experience-driven, frame-doubting cognition that learns from its own failures, then no. The two are not the same. Pretending they are serves neither humans nor the machines they build.
