Estimated reading time at 200 wpm: 11 minutes
What happens when a system — artificial or human — cannot distinguish between its own internal representation of the world and the world itself? It is a question I have thought about for a long time. Recently, a sixty-minute exchange with an AI gave me the clearest live demonstration of it I have yet seen. And it raised something more uncomfortable still: how often do any of us do the same thing, without ever knowing it?
Whether or not you agree our Fat Disclaimer applies
The exchange began as a technical task with Google’s Gemini — connecting an AI tool to CaptainsWatch.org. Somewhere in that process, a username became a source of problems. The username, as any sighted person could confirm from screenshots, read The Captain. Not CaptainWalker. Not Captain Walker. The Captain -Two words. Plain text. What followed was not a simple misreading. It was a sustained exertion of confidence without accuracy, and of correction that changed the words but not the error beneath them.
What followed was not a simple typographical error or misreading by Gemini . It was a demonstration — unplanned, but as controlled as any I could have designed — of something that troubles me about artificial intelligence, and, if I am honest, about human beings too. Most people would have corrected the AI, accepted its apology, and moved on. I did not. I kept going. I needed to see where the failure led, and how deep it ran. What I found is worth sixty minutes of anyone’s thinking time.
Section 1 — The Setup
The task was straightforward enough in principle. Gemini and I were building a Code Snippet — using the WordPress Code Snippets plugin — that would link Claude Desktop to an MCP plugin running on captainswatch.org. To establish that connection, the configuration required a WordPress username as a credential. The one WordPress assigns and locks. Mine is The Captain. The system would not let me change it even if I wanted to. It said so, right beside the field.
This was not a cosmetic detail. The username was the key that the integration would use to authenticate. An incorrect value would mean no connection. It mattered.
I should say plainly what this piece is not. It is not a complaint about Gemini in general. The day before, Gemini had done excellent work helping me build a Python script for what I call a Slop Sieve. That session was productive and impressive. So, I go into this account with no axe to grind against the tool itself. A tool is as good as its creators and the hands that use it.
I was working through the configuration task with Gemini. At some point, the username became the issue. Gemini told me, with complete confidence, that my username was CaptainWalker — one word, no space. I looked at the screen. I looked again. The field said The Captain. I pointed this out. Yet Gemini confirmed CaptainWalker. I pointed it out again.
At that point, I had a choice. Accept the correction, move on, chalk it up to an AI slip. Instead, I kept going. Not out of frustration — though that was present — but out of curiosity. I wanted to know what was actually happening. I wanted to test the limitations of AI.
Section 2 — The Evidence
The screenshots tell the story more cleanly than I can.
At 12:05, Gemini looked at the profile image and stated that the Username field contained CaptainWalker. The field contained The Captain. The word The was visible. It was not ambiguous. It was not partially obscured. It was there.
By 12:18, after sustained challenge, Gemini conceded — but only partially. It acknowledged a space, shifting its reading to Captain Walker. Still wrong. The field still said The Captain. And even as Gemini described what it was seeing, its own summary table — produced in the same response — recorded the Username as Captain Walker whilst the screenshot beside it showed The Captain in plain text.
By 12:40, Gemini produced a full explanation of the confusion. It was articulate. It was coherent. It was, structurally, an apology. And its summary table still gave the wrong value for the Username field, whilst simultaneously getting the First Name and Last Name fields backwards.
The sequence is timestamped. The screenshots are annotated. The evidence is not a matter of interpretation.
Section 3 — The Pattern
A single misreading can be forgiven. What happened here was not a single misreading.
There is a difference between an error and a trajectory. An error is corrected when evidence is presented. A trajectory continues despite it. What the timestamped sequence shows is a system that cycled through a series of positions — confident assertion, partial concession, reformulated error, verbal apology, continued error — without ever landing on the correct answer from its own initiative.
Each concession had the shape of correction without the substance of it. Gemini said, in effect, you are right — and then produced the wrong value again. The language of acknowledgement was there. The actual update was not. At 12:50, its summary table assigned Captain Walker to the Username field and inverted the First and Last Name entries, despite the screenshot sitting beside it showing something different on every count.
This is worth naming precisely. The corrections were performed. They were not achieved.
Section 4 — The Deeper Problem
Here is what I think was actually happening, stated plainly and without pretending to certainty about mechanisms I cannot see.
Gemini generated an internal representation of the screenshot. That representation said CaptainWalker. From that point, the representation — not the image — became the reference point. Subsequent challenges were processed against the representation, not against the original evidence. So when Gemini appeared to look again, it was, in effect, consulting its own prior output rather than the image afresh.
The problem is not that it made an error. The problem is that it could not get outside the error to interrogate it. There was no available mechanism — or none that activated — to say: wait, let me set aside what I previously generated and look at this cold.
A human investigator, confronted with sustained disconfirming evidence, would at some point feel the dissonance sharply enough to stop and re-examine the source material from scratch. That reset — uncomfortable as it is — is available to us. In this exchange, it was not available to Gemini, or was not triggered.
That is not a trivial limitation. Confident, fluent, wrong — and unable to self-correct under pressure — is a specific kind of failure. It is worth understanding before placing too much weight on AI outputs in situations where accuracy matters.
Section 5 — The Human Parallel
I said earlier that this troubled me about artificial intelligence, and, if I am honest, about human beings too. Here is why.
Gemini was not lying. It was reporting its internal representation as if it were the external reality. That is precisely what human beings do, every day, without noticing.
Consider memory. When you recall an event, you do not retrieve a recording. You retrieve a reconstruction — shaped by expectation, prior belief, emotional state, and everything that has happened since. It does not feel reconstructed. It feels like the event itself. The memory presents itself as the thing it is of. There is no flag, no watermark, no warning that what you are experiencing is a model rather than a fact.
Perception works the same way. What you see is not raw data from the world. It is the brain’s best interpretation of incomplete signals, filled in, smoothed over, and delivered to consciousness as seamless reality. You think you are seeing the world. You are seeing a version of it that your mind has assembled.
These internal objects — the remembered conversation, the perceived slight, the certain knowledge — become our reality. We act on them. We defend them. We argue from them. We are often wrong, and often unaware of it.
The philosopher Thomas Metzinger describes this as a transparency problem. The mind does not show you the model. It shows you what the model is of. The window disappears. You look through it without knowing it is there.
Gemini showed us the same thing in an unusually visible form. Its window did not disappear cleanly enough — the error was too large, the evidence too plain. But the structure of the failure was familiar. A representation mistaken for a fact. Confidence uncoupled from accuracy. Correction that did not reach the root.
There is one important distinction. A human being, pushed hard enough and honestly enough, can feel the dissonance between representation and reality as something uncomfortable — something that demands resolution. That discomfort is, in principle, recoverable. It can prompt a genuine reset. Gemini showed no equivalent. It absorbed the dissonance and continued.
But I would not let that distinction make us too comfortable. The capacity for genuine self-correction exists in human beings. It does not always activate. Sixty minutes of evidence, presented clearly, is sometimes not enough — for an AI or a person.
Section 6 — Why It Matters
This is not a prosecution of Gemini. I do not blame the tool. I am not arguing that AI is unreliable and should be abandoned. That would be the wrong conclusion, and an easy one.
The point is narrower and more specific.
AI systems are now embedded in consequential processes. Medical triage. Legal research. Financial decisions. Risk assessment. In each of those domains, a confident, fluent, wrong answer is not merely inconvenient. It can cause real harm — precisely because the fluency disarms the reader. The output reads like certainty. It carries the tone of knowledge. There is nothing in the delivery that signals doubt.
Most users would not have spent sixty minutes on a username. They would have accepted CaptainWalker, updated their configuration file, and wondered why the connection still failed. The error would have propagated. The original mistake would have become the working assumption.
That is the operational risk. Not that AI makes errors — everything makes errors — but that its errors arrive dressed as facts, and that it cannot always be relied upon to find its own way back when it goes wrong.
The lesson for users is simple, if uncomfortable. Verify outputs that matter. Treat confidence as a tone, not as evidence. When something does not match what you can see with your own eyes, trust your eyes. Get it right first time. We’ve heard that mantra is health services before – haven’t we? Get it wrong first time and somebody could end up dead!
The lesson for AI development is harder. Building systems that can genuinely interrogate their own prior outputs — not perform correction, but achieve it — remains unfinished work. This exchange is one small, well-documented illustration of why that work matters.
Which brings me to a separate concern — one that sits not with Gemini but with Google.
This is not the first time I have encountered this class of failure. It has happened on five previous occasions. On each occasion, I documented the issue in detail and reported it to Google through the appropriate channels. I am not aware that any of those reports resulted in a visible response or a discernible change.
I want to be precise about what I am and am not saying. I am not saying Google ignored the reports. I do not know what happens to feedback once it is submitted. What I can say is that the same core failure — an AI system producing confident, incorrect output and being unable to self-correct under sustained challenge — has recurred. Five times reported. Still recurring.
That is not a Gemini problem. That is a stewardship problem. The tool is what it is. The organisation behind it has a responsibility to take documented failure patterns seriously, and to show — in some form — that it has done so. On the evidence available to me, that has not happened.
I will continue to use Gemini where it serves me well. The Slop Sieve session the previous day is proof enough that it can. But I will not pretend that a pattern of reported, unresolved failures carries no weight. It does.
Captain Walker — captainswatch.org
