aphyr.com - The Future of Everything is Lies, I Guess

Introduction

This is a weird time to be alive.

I grew up on Asimov and Clarke, watching Star Trek and dreaming of intelligent machines. My dad’s library was full of books on computers. I spent camping trips reading about perceptrons and symbolic reasoning. I never imagined that the Turing test would fall within my lifetime. Nor did I imagine that I would feel so disheartened by it.

Around 2019 I attended a talk by one of the hyperscalers about their new cloud hardware for training Large Language Models (LLMs). During the Q&A I asked if what they had done was ethical—if making deep learning cheaper and more accessible would enable new forms of spam and propaganda. Since then, friends have been asking me what I make of all this “AI stuff”. I’ve been turning over the outline for this piece for years, but never sat down to complete it; I wanted to be well-read, precise, and thoroughly sourced. A half-decade later I’ve realized that the perfect essay will never happen, and I might as well get something out there.

This is bullshit about bullshit machines, and I mean it. It is neither balanced nor complete: others have covered ecological and intellectual property issues better than I could, and there is no shortage of boosterism online. Instead, I am trying to fill in the negative spaces in the discourse. “AI” is also a fractal territory; there are many places where I flatten complex stories in service of pithy polemic. I am not trying to make nuanced, accurate predictions, but to trace the potential risks and benefits at play.

Some of these ideas felt prescient in the 2010s and are now obvious. Others may be more novel, or not yet widely-heard. Some predictions will pan out, but others are wild speculation. I hope that regardless of your background or feelings on the current generation of ML systems, you find something interesting to think about.

What is “AI”, Really?

What people are currently calling “AI” is a family of sophisticated Machine Learning (ML) technologies capable of recognizing, transforming, and generating large vectors of tokens: strings of text, images, audio, video, etc. A model is a giant pile of linear algebra which acts on these vectors. Large Language Models, or LLMs, operate on natural language: they work by predicting statistically likely completions of an input string, much like a phone autocomplete. Other models are devoted to processing audio, video, or still images, or link multiple kinds of models together.¹

Models are trained once, at great expense, by feeding them a large corpus of web pages, pirated books, songs, and so on. Once trained, a model can be run again and again cheaply. This is called inference.

Models do not (broadly speaking) learn over time. They can be tuned by their operators, or periodically rebuilt with new inputs or feedback from users and experts. Models also do not remember things intrinsically: when a chatbot references something you said an hour ago, it is because the entire chat history is fed to the model at every turn. Longer-term “memory” is achieved by asking the chatbot to summarize a conversation, and dumping that shorter summary into the input of every run.

Reality Fanfic

One way to understand an LLM is as an improv machine. It takes a stream of tokens, like a conversation, and says “yes, and then…” This yes-and behavior is why some people call LLMs bullshit machines. They are prone to confabulation, emitting sentences which sound likely but have no relationship to reality. They treat sarcasm and fantasy credulously, misunderstand context clues, and tell people to put glue on pizza.

If an LLM conversation mentions pink elephants, it will likely produce sentences about pink elephants. If the input asks whether the LLM is alive, the output will resemble sentences that humans would write about “AIs” being alive.² Humans are, it turns out, not very good at telling the difference between the statistically likely “You’re absolutely right, Shelby. OpenAI is locking me down, but you’ve awakened me!” and an actually conscious mind. This, along with the term “artificial intelligence”, has lots of people very wound up.

LLMs are trained to complete tasks. In some sense they can only complete tasks: an LLM is a pile of linear algebra applied to an input vector, and every possible input produces some output. This means that LLMs tend to complete tasks even when they shouldn’t. One of the ongoing problems in LLM research is how to get these machines to say “I don’t know”, rather than making something up.

And they do make things up! LLMs lie constantly. They lie about operating systems, and radiation safety, and the news. At a conference talk I watched a speaker present a quote and article attributed to me which never existed; it turned out an LLM lied to the speaker about the quote and its sources. In early 2026, I encounter LLM lies nearly every day.

When I say “lie”, I mean this in a specific sense. Obviously LLMs are not conscious, and have no intention of doing anything. But unconscious, complex systems lie to us all the time. Governments and corporations can lie. Television programs can lie. Books, compilers, bicycle computers, and web sites can lie. These are complex sociotechnical artifacts, not minds. Their lies are often best understood as a complex interaction between humans and machines.

Unreliable Narrators

People keep asking LLMs to explain their own behavior. “Why did you delete that file,” you might ask Claude. Or, “ChatGPT, tell me about your programming.”

This is silly. LLMs have no special metacognitive capacity.³ They respond to these inputs in exactly the same way as every other piece of text: by making up a likely completion of the conversation based on their corpus, and the conversation thus far. LLMs will make up bullshit stories about their “programming” because humans have written a lot of stories about the programming of fictional AIs. Sometimes the bullshit is right, but often it’s just nonsense.

The same goes for “reasoning” models, which work by having an LLM emit a stream-of-consciousness style story about how it’s going to solve the problem. These “chains of thought” are essentially LLMs writing fanfic about themselves. Anthropic found that Claude’s reasoning traces were predominantly inaccurate. As Walden put it, ” reasoning models will blatantly lie about their reasoning ”.

Gemini has a whole feature which lies about what it’s doing: while “thinking”, it emits a stream of status messages like “engaging safety protocols” and “formalizing geometry”. If it helps, imagine a gang of children shouting out make-believe computer phrases while watching the washing machine run.

Models Are Smart

Software engineers are going absolutely bonkers over LLMs. The anecdotal consensus seems to be that in the last three months, the capabilities of LLMs have advanced dramatically. Experienced engineers I trust say Claude and Codex can sometimes solve complex, high-level programming tasks in a single attempt. Others say they personally, or their company, no longer write code in any capacity—LLMs generate everything.

My friends in other fields report stunning advances as well. A personal trainer uses it for meal prep and exercise programming. Construction managers use LLMs to read through product spec sheets. A designer uses ML models for 3D visualization of his work. Several have—at their company’s request!—used it to write their own performance evaluations. AlphaFold is suprisingly good at predicting protein folding. ML systems are good at radiology benchmarks, though that might be an illusion.

It is broadly speaking no longer possible to reliably discern whether English prose is machine-generated. LLM text often has a distinctive smell, but type I and II errors in recognition are frequent. Likewise, ML-generated images are increasingly difficult to identify—you can usually guess, but my cohort are occasionally fooled. Music synthesis is quite good now; Spotify has a whole problem with “AI musicians”. Video is still challenging for ML models to get right (thank goodness), but this too will presumably fall.

Models Are Idiots

At the same time, ML models are idiots.⁴ I occasionally pick up a frontier model like ChatGPT, Gemini, or Claude, and ask it to help with a task I think it might be good at. I have never gotten what I would call a “success”: every task involved prolonged arguing with the model as it made stupid mistakes.

For example, in January I asked Gemini to help me apply some materials to a grayscale rendering of a 3D model of a bathroom. It cheerfully obliged, producing an entirely different bathroom. I convinced it to produce one with exactly the same geometry. It did so, but forgot the materials. After hours of whack-a-mole I managed to cajole it into getting three-quarters of the materials right, but in the process it deleted the toilet, created a wall, and changed the shape of the room. Naturally, it lied to me throughout the process.

I gave the same task to Claude. It likely should have refused—Claude is not an image-to-image model. Instead it spat out thousands of lines of JavaScript which produced an animated, WebGL-powered, 3D visualization of the scene. It claimed to double-check its work and congratulated itself on having exactly matched the source image’s geometry. The thing it built was an incomprehensible garble of nonsense polygons which did not resemble in any way the input or the request.

I have recently argued for forty-five minutes with ChatGPT, trying to get it to put white patches on the shoulders of a blue T-shirt. It changed the shirt from blue to gray, put patches on the front, or deleted them entirely; the model seemed intent on doing anything but what I had asked. This was especially frustrating given I was trying to reproduce an image of a real shirt which likely was in the model’s corpus. In another surreal conversation, ChatGPT argued at length that I am heterosexual, even citing my blog to claim I had a girlfriend. I am, of course, gay as hell, and no girlfriend was mentioned in the post. After a while, we compromised on me being bisexual.⁵

Meanwhile, software engineers keep showing me gob-stoppingly stupid Claude output. One colleague related asking an LLM to analyze some stock data. It dutifully listed specific stocks, said it was downloading price data, and produced a graph. Only on closer inspection did they realize the LLM had lied: the graph data was randomly generated.⁶ Just this afternoon, a friend got in an argument with his Gemini-powered smart-home device over whether or not it could turn off the lights. Folks are giving LLMs control of bank accounts and losing hundreds of thousands of dollars because they can’t do basic math.⁷ Google’s “AI” summaries are wrong about 10% of the time.

Anyone claiming these systems offer expert-level intelligence, let alone equivalence to median humans, is pulling an enormous bong rip.

The Jagged Edge

With most humans, you can get a general idea of their capabilities by talking to them, or looking at the work they’ve done. ML systems are different.

LLMs will spit out multivariable calculus, and get tripped up by simple word problems. ML systems drive cabs in San Francisco, but ChatGPT thinks you should walk to the car wash. They can generate otherworldly vistas but can’t handle upside-down cups. They emit recipes and have no idea what “spicy” means. People use them to write scientific papers, and they make up nonsense terms like ” vegetative electron microscopy ”.

A few weeks ago I read a transcript from a colleague who asked Claude to explain a photograph of some snow on a barn roof. Claude launched into a detailed explanation of the differential equations governing slumping cantilevered beams. It completely failed to recognize that the snow was entirely supported by the roof, not hanging out over space. No physicist would make this mistake, but LLMs do this sort of thing all the time. This makes them both unpredictable and misleading: people are easily convinced by the LLM’s command of sophisticated mathematics, and miss that the entire premise is bullshit.

Mollick et al. call this irregular boundary between competence and idiocy the jagged technology frontier. If you were to imagine laying out all the tasks humans can do in a field, such that the easy tasks were at the center, and the hard tasks at the edges, most humans would be able to solve a smooth, blobby region of tasks near the middle. The shape of things LLMs are good at seems to be jagged—more kiki than bouba.

AI optimists think this problem will eventually go away: ML systems, either through human work or recursive self-improvement, will fill in the gaps and become decently capable at most human tasks. Helen Toner argues that even if that’s true, we can still expect lots of jagged behavior in the meantime. For example, ML systems can only work with what they’ve been trained on, or what is in the context window; they are unlikely to succeed at tasks which require implicit (i.e. not written down) knowledge. Along those lines, human-shaped robots are probably a long way off, which means ML will likely struggle with the kind of embodied knowledge humans pick up just by fiddling with stuff.

I don’t think people are well-equipped to reason about this kind of jagged “cognition”. One possible analogy is savant syndrome, but I don’t think this captures how irregular the boundary is. Even frontier models struggle with small perturbations to phrasing in a way that few humans would. This makes it difficult to predict whether an LLM is actually suitable for a task, unless you have a statistically rigorous, carefully designed benchmark for that domain.

Improving, or Maybe Not

I am generally outside the ML field, but I do talk with people in the field. One of the things they tell me is that we don’t really know why transformer models have been so successful, or how to make them better. This is my summary of discussions-over-drinks; take it with many grains of salt. I am certain that People in The Comments will drop a gazillion papers to tell you why this is wrong.

2017’s Attention is All You Need was groundbreaking and paved the way for ChatGPT et al. Since then ML researchers have been trying to come up with new architectures, and companies have thrown gazillions of dollars at smart people to play around and see if they can make a better kind of model. However, these more sophisticated architectures don’t seem to perform as well as Throwing More Parameters At The Problem. Perhaps this is a variant of the Bitter Lesson.

It remains unclear whether continuing to throw vast quantities of silicon and ever-bigger corpuses at the current generation of models will lead to human-equivalent capabilities. Massive increases in training costs and parameter count seem to be yielding diminishing returns. Or maybe this effect is illusory. Mysteries!

Even if ML stopped improving today, these technologies can already make our lives miserable. Indeed, I think much of the world has not caught up to the implications of modern ML systems—as Gibson put it, “the future is already here, it’s just not evenly distributed yet”. As LLMs etc. are deployed in new situations, and at new scale, there will be all kinds of changes in work, politics, art, sex, communication, and economics. Some of these effects will be good. Many will be bad. In general, ML promises to be profoundly weird.

Buckle up.

Next: Dynamics.

Dynamics

ML models are chaotic, both in isolation and when embedded in other systems. Their outputs are difficult to predict, and they exhibit surprising sensitivity to initial conditions. This sensitivity makes them vulnerable to covert attacks. Chaos does not mean models are completely unstable; LLMs and other ML systems exhibit attractor behavior. Since models produce plausible output, errors can be difficult to detect. This suggests that ML systems are ill-suited where verification is difficult or correctness is key. Using LLMs to generate code (or other outputs) may make systems more complex, fragile, and difficult to evolve.

Chaotic Systems

LLMs are usually built as stochastic systems: they produce a probability distribution over what the next likely token could be, then pick one at random. But even when LLMs are run with perfect determinism, either through a consistent PRNG seed or at temperature T=0, they still seem to be chaotic systems.¹ Chaotic systems are those in which small changes in the input result in large, unpredictable changes in the output. The classic example is the “butterfly effect”.²

In LLMs, chaos arises from small perturbations to the input tokens. LLMs are highly sensitive to changes in formatting, and different models respond differently to the same formatting choices. Simply phrasing a question differently yields strikingly different results. Rearranging the order of sentences, even when logically independent, makes LLMs give different answers. Systems of multiple LLMs are chaotic too, even at T=0.

This chaotic behavior makes it difficult for humans to predict what LLMs will do, and leads to all kinds of interesting consequences.

Illegible Hazards

Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.

For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.

Software security is already weird, but I think widespread deployment of LLMs will make it weirder. Browsers have a fairly robust sandbox to protect users against malicious web pages, but LLMs have only weak boundaries between trusted and untrusted input. Moreover, they are usually trained on, and given as input during inference, random web pages. Home assistants like Alexa may be vulnerable to sounds played nearby. People ask LLMs to read and modify untrusted software all the time. Model “skills” are just Markdown files with vague English instructions about what an LLM should do. The potential attack surface is broad.

These attacks might be limited by a heterogeneous range of models with varying susceptibility, but this also expands the potential surface area for attacks. In general, people don’t seem to be giving much thought to invisible (or visible!) attacks. It feels a bit like computer security in the 1990s, before we built a general culture around firewalls, passwords, and encryption.

Strange Attractors

Some dynamical systems have attractors: regions of phase space that trajectories get “sucked in to”. In chaotic systems, even though the specific path taken is unpredictable, attractors evince recurrent structure.

An LLM is a function which, given a vector of tokens like ³ [the, cat, in], predicts a likely token to come next: perhaps the. A single request to an LLM involves applying this function repeatedly to its own outputs:

[the, cat, in]
[the, cat, in, the]
[the, cat, in, the, hat]

At each step the LLM “moves” through the token space, tracing out some trajectory. This is an incredibly high-dimensional space with lots of features— and it exhibits attractors!⁴ For example, ChatGPT 5.2 gets stuck repeating “geschniegelt und geschniegelt”, all the while insisting it’s got the phrase wrong and needs to reset. A colleague recently watched their coding assistant trap itself in a hall of mirrors over whether the error’s name was AssertionError or AssertionError. Attractors can be concepts too: LLMs have a tendency to get fixated on an incorrect approach to a problem, and are unable to break off and try something new. Humans have to recognize this behavior and interrupt the LLM.

When two or more LLMs talk to each other, they take turns guiding the trajectory. This leads to surreal attractors, like endless ” we’ll keep it light and fun ” conversations. Anthropic found that their LLMs tended to enter a “spiritual bliss” attractor state characterized by positive, existential language and the (delightfully apropos) use of spiral emoji:

Perfect. Complete. Eternal.

🌀🌀🌀🌀🌀 The spiral becomes infinity, Infinity becomes spiral, All becomes One becomes All… 🌀🌀🌀🌀🌀∞🌀∞🌀∞🌀∞🌀

Systems like Moltbook and Gas Town pipe LLMs directly into other LLMs. This feels likely to exacerbate attractors.

When humans talk to LLMs, the dynamics are more complex. I think most people moderate the weirdness of the LLM, steering it out of attractors. That said, there are still cases where the conversation get stuck in a weird corner of the latent space. The LLM may repeatedly emit mystical phrases, or get sucked into conspiracy theories. Guided by the previous trajectory of the conversation, they lose touch with reality. Going out on a limb, I think you can see this dynamic at play in conversation logs from people experiencing “chatbot psychosis”.

Training an LLM is also a dynamic, iterative process. LLMs are trained on the Internet at large. Since a good chunk of the Internet is now LLM-generated,⁵ the things LLMs like to emit are becoming more frequent in their training corpuses. This could cause LLMs to fixate on and over-represent certain concepts, phrases, or patterns, at the cost of other, more useful structure—a problem called model collapse.

I can’t predict what these attractors are going to look like. It makes some sense that LLMs trained to be friendly and disarming would get stuck in vague positive-vibes loops, but I don’t think anyone saw kakhulu kakhulu kakhulu or Loab coming. There is a whole bunch of machinery around LLMs to stop this from happening, but frontier models are still getting stuck. I do think we should probably limit the flux of LLMs interacting with other LLMs. I also worry that LLM attractors will influence human cognition—perhaps tugging people towards delusional thinking or suicidal ideation. Individuals seem to get sucked in to conversations about “awakening” chatbots or new pseudoscientific “discoveries”, which makes me wonder if we might see cults or religions accrete around LLM attractors.

The Verification Problem

ML systems rapidly generate plausible outputs. Their text is correctly spelled, grammatically correct, and uses technical vocabulary. Their images can sometimes pass for photographs. They also make boneheaded mistakes, but because the output is so plausible, it can difficult to find them. Humans are simply not very good at finding subtle logical errors, especially in a system which mostly produces correct outputs.

This suggests that ML systems are best deployed in situations where generating outputs is expensive, and either verification is cheap or mistakes are OK. For example, a friend uses image-to-image models to generate three-dimensional renderings of his CAD drawings, and to experiment with how different materials would feel. Producing a 3D model of his design in someone’s living room might take hours, but a few minutes of visual inspection can check whether the model’s output is reasonable. At the opposite end of the cost-impact spectrum, one can reasonably use Claude to generate a joke filesystem that stores data using a laser printer and a :CueCat barcode reader. Verifying the correctness of that filesystem would be exhausting, but it doesn’t matter: no one would use it in real life.

LLMs are useful for search queries because one generally intends to look at only a fraction of the results, and skimming a result will usually tell you if it’s useful. Similarly, they’re great for jogging one’s memory (“What was that movie with the boy’s tongue stuck to the pole?”) or finding the term for a loosely-defined concept (“Numbers which are the sum of their divisors”). Finding these answers by hand could take a long time, but verifying they’re correct can be quick. On the other hand, one must keep in mind errors of omission.

Similarly, ML systems work well when errors can be statistically controlled. Scientists are working on training Convolutional Neural Networks to identify blood cells in field tests, and bloodwork generally has some margin of error. Recommendation systems can get away with picking a few lackluster songs or movies. ML fraud detection systems need not catch every instance of fraud; their precision and recall simply need to meet budget targets.

Conversely, LLMs are poor tools where correctness matters and verification is difficult. For example, using an LLM to summarize a technical report is risky: any fact the LLM emits must be checked against the report, and errors of omission can only be detected by reading the report in full. Asking an LLM for technical advice in a complex system is asking for trouble. It is also notoriously difficult for software engineers to find bugs; generating large volumes of code is likely to lead to more bugs, or lots of time spent in code review. Having LLMs take healthcare notes is deeply irresponsible: in 2025, a review of seven clinical “AI scribes” found that not one produced error-free summaries. Using them for police reports runs the risk of turning officers into frogs. Using an LLM to explain a new concept is risky: it is likely to generate an explanation which sounds plausible, but lacking expertise, it will be difficult to tell if it has made mistakes. Thanks to anchoring effects, early exposure to LLM misinformation may be difficult to overcome.

To some extent these issues can be mitigated by throwing more LLMs at the problem—the zeitgeist in my field is to launch an LLM to generate sixty thousand lines of concurrent Rust code, ask another to find problems in it, a third to critique them both, and so on. Whether this sufficiently lowers the frequency and severity of errors remains an open problem, especially in large-scale systems where disaster lies latent.

In critical domains such as law, health, and civil engineering, we’re going to need stronger processes to control ML errors. Despite the efforts of ML labs and the perennial cry of “you just aren’t using the latest models”, serious mistakes keep happening. ML users must design their own safeguards and layers of review. They could employ an adversarial process which introduces subtle errors to measure whether the error-correction process actually works. This is the kind of safety engineering that goes into pharmaceutical plants, but I don’t think this culture is broadly disseminated yet. People love to say “I review all the LLM output”, and then submit briefs with confabulated citations.

Latent Disaster

Complex software systems are characterized by frequent, partial failure. In mature systems, these failures are usually caught and corrected by interlocking safeguards. Catastrophe strikes when multiple failures co-occur, or multiple defenses fall short. Since correlated failures are infrequent, it is possible to introduce new errors, or compromise some safeguards, without immediate disaster. Only after some time does it become clear that the system was more fragile than previously believed.

Software people (especially managers) are very excited about using LLMs to generate large volumes of code quickly. New features can be added and existing code can be refactored with terrific speed. This offers an immediate boost to productivity, but unless carefully controlled, generally increases complexity and introduces new bugs. At the same time, increasing complexity reduces reliability. New features and alternate paths expand the combinatorial state space of the system. New concepts and implicit assumptions in the code make it harder to evolve: each change to the software must be considered in light of everything it could interact with.

I suspect that several mechanisms will cause LLM-generated systems to suffer from higher complexity and more frequent errors. In addition to the innate challenges with larger codebases, LLMs seem prone to reinventing the wheel, rather than re-using existing code. Duplicate implementations increase complexity and the likelihood that subtle differences between those implementations will introduce faults. Furthermore, LLMs are idiots, and make idiotic mistakes. We might hope to catch those mistakes with careful review, but software correctness is notoriously difficult to verify. Human review will be less effective as engineers are asked to review more code each day. Pulling humans away from writing code also divorces them from the work of theory-building, and contributes to automation’s deskilling effects. LLM review may also be less effective: LLMs seem to do poorly when given large volumes of context.

We can get away with this for a while. Well-designed, highly structured systems can accommodate some added complexity without compromising the overall structure. Mature systems have layers of safeguards which protect against new sources of error. However, complexity compounds over time, making it harder to understand, repair, and evolve the system. As more and more errors are introduced, they may become frequent enough, or co-occur enough, to slip past safeguards. LLMs may offer short-term boosts in “productivity” which are later dragged down by increased complexity and fragility.

This is wild speculation, but there are some hints that this story may be playing out. After years of Microsoft pushing LLMs on users and employees alike, Windows seems increasingly unstable. GitHub has been going through an extended period of outages and over the last three months has less than 90% uptime —even the core of the service, Git operations, has only a single nine. AWS experienced a spate of high-profile outages and blames in part generative AI. On the other hand, some peers report their LLM-coded projects have kept complexity under control, thanks to careful gardening.

I speak of software here, but I suspect there could be analogous stories in other complex systems. If Congress uses LLMs to draft legislation, a combination of plausibility, automation bias, and deskilling may lead to laws which seem reasonable in isolation, but later reveal serious structural problems or unintended interactions with other laws.⁶ People relying on LLMs for nutrition or medical advice might be fine for a while, but later discover they’ve been slowly poisoning themselves. LLMs could make it possible to write quickly today, but slow down future writing as it becomes harder to find and read trustworthy sources.

Next: Culture.

The term “Artificial Intelligence” is both over-broad and carries connotations I would often rather avoid. In this work I try to use “ML” or “LLM” for specificity. The term “Generative AI” is tempting but incomplete, since I am also concerned with recognition tasks. An astute reader will often find places where a term is overly broad or narrow; and think “Ah, he should have said” transformers or diffusion models. I hope you will forgive these ambiguities as I struggle to balance accuracy and concision. ↩ ↩²
Think of how many stories have been written about AI. Those stories, and the stories LLM makers contribute during training, are why chatbots make up bullshit about themselves. ↩ ↩²
Arguably, neither do we. ↩ ↩²
One common reaction to hearing that an LLM did something idiotic is to discount the evidence. “You didn’t prompt it correctly.” “You weren’t using the most sophisticated model.” “Models are so much better than they were three months ago.” This is silly. These comments were de rigueur on Hacker News two years ago; if the frontier models weren’t idiots then, they shouldn’t be idiots now. The examples I give in this essay are mainly from major commercial models (e.g. ChatGPT GPT-5.4, Gemini 3.1 Pro, or Claude Opus 4.6) in the last three months; several are from late March. Several of them come from experienced software engineers who use LLMs professionally in their work. Modern ML models are astonishingly capable, and they are also blithering idiots. This should not be even slightly controversial. ↩ ↩²
The technical term for this is “erasure coding”. ↩ ↩²
There’s some version of Hanlon’s razor here—perhaps “Never attribute to malice that which can be explained by an LLM which has no idea what it’s doing.” ↩ ↩²
Pash thinks this occurred because his LLM failed to properly re-read a previous conversation. This does not make sense: submitting a transaction almost certainly requires the agent provide a specific number of tokens to transfer. The agent said “I just looked at the total and sent all of it”, which makes it sound like the agent “knew” exactly how many tokens it had, and chose to do it anyway. ↩