Our actions today create the world tomorrow

So why do we make it so shitty?

I've been working on this post for months and it keeps getting longer and longer; it might not be done but I need to publish it eventually.

I've got a Google Nest Mini¹Could've sworn it was called a Google Home but apparently not. in my bedroom. It doesn't get a ton of use, but it's nice to be able to check the weather forecast in the morning, or play music at night, without having to get out of bed and fiddle with anything. But over the past year I've noticed a significant decline in its usability (and it's not just me). Commands that once worked flawlessly ("Google, stop timer."²Instead of stopping the alarm nothing happens and I shout variations of the command half a dozen times until I am finally met with blessed silence. "Google, play sleep."³Instead of starting a spotify playlist of soft piano music called "Sleep" it tells a bedtime story(?) about a google assistant not realizing it was muted.) now either don't work at all or do something different than they did last week. This is frustrating and honestly unacceptable behavior for a home appliance - would you appreciate it if your toaster stopped accepting unauthorized bread or a clock that was wrong 80% of the time?

While I'm not privy to the inner workings of Google's voice recognition algorithms, this mirrors the degradation of many tools and services since the popularity boom of Large Language Models (LLMs), or "generative AI".

Linguistic Pareidolia

You'll often hear about ChatGPT (and other LLMs) "hallucinating" - that is, making a false claim. This is a metaphor that serves to humanize and excuse the bot. A hallucination is a problem with perception such as seeing or hearing things that aren't there. To say that the bot is hallucinating (for example, when it tells you that there are two R's in the word strawberry) therefore, makes the claim that it:

understands that you are asking a question and desiring a truthful answer
has a perception of the world
makes a mistake in answering based on that perception being inaccurate

In reality, none of these are true. LLMs don't have any conception of of the truth; they don't contain a model of the world or have a capacity to perceive it. Interacting with them usually takes the form of a conversation, but under the hood it's more like the predictive text on your phone keyboard - choosing the word that is statistically most likely to come next. It won't give you the answer to your question, it will return answer-shaped text, the pink slime of information, that may or may not happen to correspond with what you wanted to know. LLMs don't lie, they don't make mistakes, and they certainly don't hallucinate. They speak without regard for the truth - in other words, they bullshit.

So, given that LLMs are bullshit machines, why do so many people put so much faith in them?⁴I use the word faith intentionally, given the number of cults popping up around them. Standard industry benchmarks show error rates between 5-50%. Given that these tests are performed by the same companies developing the models I'm not convinced that real-world results aren't worse, but for rhetorical purposes I'll take the midpoint and assume an error rate of around 25%. The way I see it, there are two different (but overlapping) groups of people who champion LLMs.

Group 1 uses LLMs as tools, such as for code generation or as a search engine. A possible explanation for continued use despite errors is Gell-Mann Amnesia, the phenomenon where a source (originally a newspaper) presents information you know to be incorrect on a topic you're familiar with, but you continue to believe it on other topics. A professional in a given field would be able to tell which quarter of a given response is nonsense, but to the layman it all appears equally plausible.⁵Absent an obvious error like telling you to glue cheese to pizza. This means that many of the LLM's errors will be missed, while the user feels smart for catching one or two issues that they happened to notice.

Group 2 is largely a subset of group 1 which, instead of just using LLMs, believes that they are intelligent, possibly sentient, beings. Human brains are powerful pattern matching machines, and there's no pattern that we love to find more than ourselves. This is pareidolia, most commonly thought of as perceiving faces in random patterns such as a knobbly tree or the surface of the moon. Language seems to have just as powerful an effect on us, with this linguistic pareidolia being similar to the ELIZA effect, where a rudimentary chatbot from 1966 is perceived as being intelligent and aware, even when users are aware that it was largely just rephrasing the users inputs as questions. It's no wonder, then, that today's much more sophisticated chatbots, capable of seemingly holding a conversation on any conceivable topic, fool people into believing that there is a mind on the other side of the screen. This misconception is furthered by the same tools psychics use to fool our brains into believing that statistically generic statements are personally meaningful.

Regurgitated Brains

I didn't enjoy writing essays when I was in school. It was generally a tedious slog for me to squeak over the minimum word or page count for an assignment, but the purpose of the essays being assigned was clear. My professor was not in desperate need of a dozen essays discussing Borges. She wasn't expecting us to present her with groundbreaking theories or never-before-considered ideas. The point of the essay was for us to learn through the process of writing it. Using ChatGPT to write your essays is like lifting weights at the gym with a forklift. It fundamentally misunderstands the point and defeats the purpose.

Education has been trending towards diploma mills and job certifications for a long time now, but this dispels the illusion that it's about teaching people how to think. If teachers assign generated lesson plans and students return generated work, what are we even doing? You might as well drop the pretense of learning, save everyone some time, and have recess all day. At least then the students might be enjoying themselves.

Even the most generous reading of using LLMs in education falls short. As noted before, all of the output of an LLM is generally equally plausible. People are fallible, but we tend to make mistakes in predictable ways: misremembering a date, forgetting a word, misspelling something. LLMs however, as mentioned, will confidently spout off correct-seeming bullshit that doesn't have any of the markers of hesitation or hedging that a real person might, which makes us more likely to believe it. The purpose of education is to teach people things, implying they don't already know the truth. Are students supposed to fact check every statement and implication? In that case, what is the benefit of using the LLM instead of just teaching the correct things in the first place? This is especially troubling given early evidence that use of LLMs lowers brain activity and degrades critical thinking skills. Even when people are aware of the problem and believe that they're being careful, it's all too easy to let your eyes glaze over and start to vibe code. Partial automation in "self-driving" cars can be more dangerous than no automation, since it lulls people into a false sense of security and makes them unprepared when danger strikes. Similarly, a mostly accurate LLM can trick you into believing falsehoods if the few points you fact-check turn out to be correct.

The purpose of a system is what it does.

When you have a hammer, everything looks like a nail. A hammer is a very effective tool for delivering a powerful force when swung. That can be beneficial, like when building a house or piece of furniture. It can also lead to pain if you happen to miss the nail and hit your thumb instead.

When thinking about the a new technology such as the LLM, a useful tool is to examine the inputs and outputs and see what affordances are provided. An LLM is very good at producing large amounts of statistically plausible content very quickly. Given a brief direction in the form of a prompt, it can pontificate on any given subject for as long as you want, or produce hundreds of different variations on a theme. However, it is very bad at producing accurate information. Therefore, it is most suited for uses focused on quantity over quality, where speed and volume are more important than truth or accuracy.

The primary use cases that fit those criteria are spam, scams, and misinformation/propaganda, which can be grouped together as slop. The first two of these forms of slop are fundamentally cynical plays to extract money from a system, with the primary difference being where the money comes from. Spam posts look to extract money from platform holders by being engaging enough (often through appeals to emotion such as fear or anger) to earn pennies of advertising revenue, while scams seek to extract money from individuals through trickery. Spam is more focused on large numbers of small, consistent payouts versus scams typically seeking out "whales" that drop large amounts of money more infrequently. Both of these make the world worse for everyone except the perpetrators.

Propaganda is somewhat different, but no better. Instead of extracting money, it can be thought of as extracting political power by filling people's heads with the information that benefits the people producing it - typically existing power structures and influential elites. While current attempts at directly manipulating LLM outputs have had limited success⁶xAI's Grok is the pioneer here, from talking nonstop about "white genocide" to calling itself MechaHitler to searching for owner Elon Musk's views on a topic before responding., bad actors such as the Trump administration have no issues using existing LLM systems to generate false, misleading, or anger-inducing text and images which can then be disseminated through more traditional means. Being unconcerned with pesky notions such as facts or truth makes "AI art" perfectly suited to be the new aesthetic of fascism in the current moment, similar to futurism in pre-war Italy or the Nazi's fascist realism. The output of these models tends towards Thomas Kinkade-esque kitch, without the need for artists or labor.

Garbage In, Garbage Out

Art is communication. Reception theory claims that an author conceives of their work, encodes it into a transmittable form⁷This could be written or spoken language, images, a game, whatever., where it is then decoded by the audience. Meaning is created between the text and the reader. A text produced by AI removes the intention of the author from the equation. While the death of the author is often used to analyze the meaning of a text separated from the author's intention, I argue that intentionality is critical to the interpretation of a text, even when that intention is unknown or rejected. How can you do a close reading of a text when the reason behind every decision is inevitably "It was statistically probable"?

Procedural or probabilistic art is nothing new; arguably going back to the musical dice games⁸Musikalisches Würfelspiel from the 18th century or the Ciceronian writers of the 16th, though it grew in prominence with the advent of the computer. In those cases however, the intention and artistry are located in the rules and structures which constrain and give shape to the result. The closest thing a generated text has to intention is the prompt, though there's not much to analyze⁹Psychoanalyze, maybe... from "beautiful masterpiece, 3D animation of a girl in classroom wearing casual business clothes, big boobs, long hair". The prompter submits that text and receives an end product without making any additional choices throughout the creation. They may be unsatisfied with the final result, but their only recourse is to revise the original prompt in the hopes that the next version will be more to their liking. In this way the prompter is more akin to an art commissioner than an artist.

Sturgeon's Law states that 90% of everything is crap. We could quibble over exact percentages, but this seems pretty trivially correct, right? Maybe I have a different opinion of which 10% is the good stuff than you do, but we both agree that the majority of books/movies/games/programs/whatever suck. Given that the input of LLMs is Everything™, that means that they're full of shit.¹⁰In fact, I'd suspect that the breadth of mediums included in Everything™ could make the output worse! The average book, crap though it may be, still probably has better prose than the average Reddit post, which in turn is better than the average tweet. There's no way (other than training a new model from scratch) to make sure that ChatGPT only uses The Good Stuff™ when you ask it to generate the next bestseller mystery/fantasy/romance novel - what you'll get is generic, clichéd crap.

AI is always stunning at first encounter: one is amazed that something nonhuman can make something that seems so similar to what humans make. But it’s a little like Samuel Johnson’s comment about a dog walking on its hind legs: we are impressed not by the quality of the walking but by the fact it can walk that way at all. After a short time it rapidly goes from awesome to funny to slightly ridiculous—and then to grotesque. Does it not also matter that the walking dog has no intentionality—doesn’t “know” what it’s doing?

- Brian Eno, "AI’s Walking Dog"

Sometimes "AI artists" will claim that they're "democratizing art". This is clearly nonsense; there is no Art King that decrees who is allowed to make art, anyone can do it. What they mean is that they want to make (what they consider) "good" art, but without putting in any effort to develop skills along the way. They want the respect and recognition of being An Artist without putting in any of the work, and at the same time disrespecting anyone who has spent time making something. I know the feeling of believing that your work isn't good enough to even bother trying. But I promise that I would rather see your awkward, unsteady, human work than look at a soulless image produced by a diffusion model. I've been taking classes at a local art school¹¹Shout out to Fleisher Art Memorial!, and the process of observing, creating, and learning means that I'm making work that I'm proud of, even when it isn't as technically proficient as someone who's been drawing longer.

Ethical Slavery

I'm gonna focus on software development because that's my field, and it's the area where the largest benefit to using LLM assistance is claimed. Many times I've seen someone say:

Sure, the theft is bad, the art is bad, the environmental impacts are bad. But it's really good at writing code!

That's a nice idea, but does the data back it up? Early reports say... no. In fact...

When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

Pretty damning. But that's also surprising - it's well known that scoping work is one of the hardest problems in computer science¹²The other 3 are naming things and off by one errors., but why do the developers still believe that using the LLM sped them up afterwards? It goes back to the psychic's con. There are a lot of people, especially in tech, who have vested interests in LLMs being useful. Like homeopathy, self-experimentation and lack of controlled experiments makes it very easy to convince yourself that it helped you recover from that cold/push that PR faster.

LLMs are very good at producing large quantities of code quickly. But the difficult part of programming has never been the typing, it's the thinking: figuring out what problem you're trying to solve, what the correct behavior should be, and how to implement it. And that's work that you can't skip. When you try, you might end up with a result that looks correct at a glance, but closer examination is likely to reveal critical flaws and vulnerabilities. More importantly, skipping the thought means missing out on the opportunities to learn something new and maybe even figure out that you weren't solving the correct problem in the first place.

People objecting to new technologies are sometimes called luddites, technophobes, people who just hate progress! But the original Luddites were skilled textile workers, and not opposed to new technology. Their problem was with the capitalists using those new machines to drive down wages, lower product quality, and increase control. (Sound familiar?) When the workers resisted this through collective action and sabotage, soldiers were sent in to execute and deport workers. Similarly, LLM use is being mandated by capital from the top down. It's being use to replace workers despite lower quality work.

One of the main goals of the current LLM push is to replace workers. By examining the affect of chatbots currently on the market you can see what the ideal worker is, according to the executives pushing this technology. The perfect worker is servile and will never tell you no. The perfect worker produces correct-seeming results quickly, but the actual accuracy and quality of the results is less important than whether they're good enough to pass along to the next sucker. The perfect worker is sycophantic and will agree with you, no matter what you say. It's no wonder LLMs are such a hit with the Business Idiots of the world. They satisfy the desire for a slave that you don't even have to feel bad about owning. This desire has been part of the dream of intelligent machines since at least 1920 and the play R.U.R. (Rossum's Universal Robots), which is the originator of the word "robot". The Star Trek TNG episode "The Measure of a Man” deals with this same issue, whether Commander Data is a person or property. Current LLMs are not sentient, and I highly doubt that this approach could ever create a true artificial mind. But creating servile "assistants" that will do whatever you ask causes real harm today, even if they don't have feelings. Threatening an LLM to try and produce better outputs is morally corrupting, and can predispose you to act similarly to real people. The belief that LLMs are capable of outperforming human workers has led to layoffs, and the remaining employees are expected to do more with less because of the "AI assistance".

This post doesn't cover all of the problems I see arising from LLMs¹³I haven't even mentioned the climate concerns, environmental pollution, or rampant theft., and many of them could apply to other parts of the modern tech industry. The last few fads that have swept through (cryptocurrency, the metaverse, NFTs) have all had a similar feeling to them; a hint of possibility buried under a mountain of scams, lies, and desperation. The zero interest-rate policy and free-flowing venture capital of the 2010s propped up massive valuations without the need for actual results, but now the music's stopped and everyone's racing to find a seat. The underlying machine learning transformer models of LLMs will, no doubt, find use cases. But this isn't it. This is boring. Whatever. Who Cares.

fauxtrots