It Doesn't Know What Day It Is

It Doesn't Know What Day It Is

Two things have worn out their welcome this year. The first is stuff AI in everywhere and it will be better, which it rarely is. The second is AI is going to take over the world and we must stop it, which it isn't. Both come from the same place: almost nobody selling or fearing this technology has been told plainly how it works.

So here is the five minute version, with no mysticism in it.

It Predicts the Next Word. That Really Is All of It

A large language model does exactly one thing: given some text, it produces the most likely next token, then the next, then the next. That is the whole of the mechanism, during training and during use. There is no second faculty that switches on for hard questions.

People hear that and conclude the thing must therefore be trivial. It is worth resisting that, because a simple objective does not imply a simple solution. To predict the next word well across the whole of written human output, you have to model grammar, arithmetic, cause and effect, the state of a chess board and who the murderer probably is. "It just predicts the next word" explains as much about the result as "evolution is just differential reproduction rates" explains about an eye.

It Is Not a Database, and the Numbers Prove It

The common assumption is that training stuffs the internet into a file and the model looks things up in it. It does not, and it could not.

Training runs a batch of text through the model as it currently stands, measures how wrong the prediction was, nudges every internal weight a fraction in the direction that would have been less wrong, throws the batch away, and moves on to the next one. The text is never written anywhere. Only the error signal touches the model, which is why you cannot get the training data back out of it.

The proportions make the point on their own. A frontier training run streams something in the order of tens of terabytes of text past a model of a few hundred gigabytes of weights, and that model is exactly the same size on the last step as it was on the first. Nothing is compressed into it, because there is nowhere near enough room. The only strategy that keeps reducing error against a bottleneck that tight is to stop learning instances and start learning structure.

The model does not get smaller. The data goes past it.

Take a practical example. Ask a model whether an email is spam and it does not retrieve a verdict, because nobody has ever written anything about that specific email. What it has learned, from millions of passages of forum posts, complaints, filters and correspondence, is what spamminess consists of. So it computes a quantity: how much spamminess is present in this text. That number is produced fresh, from an input the machine has never seen, using machinery nobody explicitly designed for the purpose. It fell out of predicting text.

That is the difference between a lookup and a measurement, and it is the whole argument. What is stored is not the answers. It is the apparatus for working one out.

An Agent Is That, In a Loop, With Tools

Everything now marketed as an "AI agent" is built from the same part. The trick is that the output is fed back in as the next input.

Ask it to do something involving several steps and the sequence runs: what was the question, what are the steps, what does the first step require, do that step, take the result, feed it back in alongside the next step, and repeat until there is nothing left to do. What looks like deliberation is the same prediction running over its own output. The "thinking" or "reasoning" you see displayed is not a separate process either; it is more generated text, put back into the same window so the next prediction has it to work with.

Tools are the other half, and they are less magical than they look. The list of available tools is sent to the model as plain text: a name, a description in English, and the shape of the arguments. The model does not call anything. It writes out a request saying "use this tool, with these arguments", and stops. Ordinary software outside the model, the harness, reads that request, runs the actual function, pastes the result back into the conversation, and asks the model again. Round it goes until it stops asking.

Which means choosing a tool is itself just next-word prediction, conditioned on those descriptions. It is also why a badly worded tool description breaks an agent in precisely the way a badly worded instruction does.

The examples, in increasing order of how deflating they are:

• The weather. A real request to a real weather service, made by the harness. The model supplies the question and reads back the answer.
• Arithmetic. Ask it for 2 + 2 / 16 * 4 and it will often produce 2.5 with no tool at all, from pattern alone. That is the problem rather than the solution: you cannot tell from the output whether it calculated the answer or recognised the shape of one. The calculator exists to make that difference visible.
• The date. Usually not even a tool. Today's date is written into the instructions as text before the conversation starts. The system being sold to you as reasoning about your business cannot tell you what day it is unless somebody leaves a note on the desk first.

Why the Loop Works, When It Works

Agents are genuinely useful, and it is worth being precise about why, because the reason is not intelligence.

It is the feedback signal. Put one of these loops somewhere with a fast, harsh, unambiguous notion of wrong, a compiler that rejects the code, a test that fails, a tool that returns an error, and the failure goes straight back into the context where the next prediction has to account for it. That is what capability looks like from the outside, and it is why coding is the application where this technology has landed hardest.

Take that signal away and nothing else changes to compensate. Point the same loop at a question with no ground truth and it produces confident nonsense at exactly the same rate, in exactly the same tone, and nobody notices. We have written before about what that costs when it happens inside a database, in 20260410.

Now the Part Everybody Is Frightened Of

There is no independent thought in any of this, and more importantly, there is nowhere for one to live.

A model has no memory. The context window is everything it knows, it is re-read from scratch on every single turn, and it is wiped when the session ends. Nothing a conversation contains ever finds its way back into the model itself; there is no write path. The frontier systems now carry around a million tokens of context, which is several books' worth and a great deal more than the 128,000 most self-hosted models manage, but the size was never really the point. Give it ten million and it still begins every conversation knowing nothing whatsoever.

The window also gets worse as it gets bigger. Attention is spread across everything in it, so each additional token dilutes what is available to the rest, and accuracy measurably falls away long before the stated limit is reached, worst of all for material sitting in the middle. In practice it tracks signal against noise rather than raw length: half a million tokens of relevant material behaves better than fifty thousand padded with near misses. Every technique that helps, retrieval, summarising, trimming old results, is a way of keeping the context small rather than a fix for making it large.

Which is also the quiet flaw in memory files, the usual workaround. Writing notes to disk and loading them back in is a sensible idea, but every line costs context and dilutes everything around it. A memory file only pays for itself if what is in it is worth more than the attention it consumes, and since these files accumulate and almost nobody prunes them, most stop paying rather early.

It cannot think, it cannot remember, it cannot want anything, and it does not know what day it is unless you tell it.

That is a description of the product on the table today, and it is why the takeover story does not hold up. Not because of a law of nature, but because the thing has no persistence, no continuity and no stake in any outcome.

The honest caveat: in its current form. The missing piece is adaptive learning, a system that updates itself from its own experience rather than being retrained in a batch by people who fixed the architecture eighteen months earlier. If that is ever solved, this article needs rewriting from the top. It is not close, and it is not obviously coming. But it is an engineering problem rather than an impossibility, and anyone telling you the danger is permanently ruled out is guessing just as much as the people telling you it is imminent.

The Risk That Is Actually Yours

While everyone argues about machine consciousness, the real damage is being done by something far more boring: handing an agent authority nobody can reverse.

The story arrives every week now. Somebody installs an agent locally, grants it permission to everything, sets it running unattended because the marketing said autonomous, and comes back to a four figure API bill and an empty directory. The usual conclusion is that the AI went rogue. It did not. It did precisely what it was built to do.

And here is where the two halves of this article meet, because it is the same fact from both ends. That agent had no stake in the outcome, no model of what any of those files were worth, and no capacity to want the project to survive. "This is six months of work and that is a scratch file" is not a distinction it computes, because mattering is not one of the quantities in there. The exact absence that means it will not be taking over the world is the reason it will cheerfully delete your repository. One mechanism, both conclusions.

So supervision is not about distrusting the machine. It is the substitute for the judgement that was never present.

The Test Worth Applying

We run these tools on automatic ourselves, daily, and we are comfortable doing it. Not because we are cleverer than the people who lost their work, but because everything is in version control, the branch is backed up in stages, and nothing an agent does here takes more than ten minutes to undo. That is a decision taken with the blast radius and the recovery time both in full view.

Which turns the question from a matter of nerve into a matter of engineering:

• What can it reach? Files, credentials, mailboxes, production systems, money. Whatever is inside the boundary is what is at stake, and permissions granted for convenience are permissions granted permanently.
• How long would it take to put back? If the honest answer is "I don't know", you have your answer to everything else.
• Is there anything to check it against? A test, a compiler, a reconciliation, a person reading it before it goes out. Where nothing can contradict the output, nothing will.
• What leaves the building? A model with tools is a machine that transmits your data to somewhere else in order to work, which is a separate question we set out at length in 20260901.

None of that requires you to be frightened of it. It is an extraordinary tool, genuinely, and we would not be without it. It is simply a tool with no judgement in it, which is a category of tool we have handled before. Nobody fears an angle grinder either. They just don't leave it running unattended in a room full of things they care about.