The current fever of AI anxiety sweeping the globe is based on confusion over the relationship between rationality and language. That’s not to say that AI isn’t dangerous. Nor does it mean the companies pioneering AI research are acting responsibly. AI can be destructive in the same way that an airplane on autopilot carrying explosives is dangerous.
But it’s important that we understand the precise nature of this danger – it is not the danger of a new entity with consciousness and free will emerging from the circuits of AI data centers, and deciding to take over the world and wipe us out.
To understand why, we need to recognize the distinctions across language, consciousness, and free will. People fear AI models with language capabilities, also known as large language models (LLMs). LLMs trick us because of our psychological tendency to equate language with abstract thought, freedom, and choice. And although LLMs can generate coherent language and computer coding, they still lack consciousness and free will, which they would need to make plans of their own.
The doomsday scenarios always involve an AI model adopting its own goals, which may or may not align with ours, developing a free will of its own with which it plans and executes an agenda against humanity.
But why have these fears only seriously arisen about LLMs rather than other forms of artificial intelligence, such as the protein-folding program AlphaFold or the chess-playing engine Stockfish? Computer science professor Cal Newport has pointed out that most advanced LLMs generate no anxiety for us. There are zero fears that AlphaFold will decide it wants to tackle other biological problems or that Stockfish will somehow break out and begin dominating in U.S. politics the way it dominates on a chess board.
Neuroscientist Anil Seth put it this way: “If we think that Claude is conscious but AlphaFold isn’t, that says more about us than it says about AI.” Essentially, we’re unable to dissociate the use of words from abstract thought, awareness, and free choice. The machines that are really good at predicting the next word in a sequence using statistical analysis are more frightening to us than the machines that are really good at predicting the next chess move based on similar statistical analysis, because the language machine can simulate abstract thought and self-awareness due to its medium.
But producing human-like language is not at all the same as thinking abstractly and then using words to communicate those thoughts. Producing text doesn’t necessarily involve understanding what that text means.
In 1980, philosopher John Searle crafted a thought experiment to demonstrate why a computer, regardless of its sophistication, could never have a true mind, understanding, or consciousness. The experiment involves two scenarios: In the first scenario, a computer program takes Chinese characters as input, carries out each instruction of a program step by step, then produces Chinese characters as output – so well, in fact, that no one can tell they’re not talking to a hidden Chinese speaker.
In the second scenario, Searle himself, lacking any knowledge of Chinese, receives Chinese characters slipped under the door of a room. He then follows careful step-by-step instructions to produce a Chinese character in response, which he slips back out under the door.
Searle argued that there’s no difference between the computer’s role in the first scenario and Searle’s own role in the second: in both instances, the agent (whether computer or human) accepts inputs, processes them, and produces outputs, simulating a conversation without ever understanding anything that is being said. That’s what happens when we “talk” to an LLM.
Even people with severe speech impediments can still solve logic puzzles. Ideas, concepts, reasoning, and experiences can and do exist independently of words – though in order to communicate a concept to someone, or clarify it for ourselves, we must cloak it in language. We think through words, but that doesn’t mean our words are our thoughts.
Because AI scans real writing from real humans, the words it spits out based on statistical probabilities look like they have thought behind them, but they do not. They’re a mirror held up to the collective human body of written language ingested by the AI. Unsurprisingly, we see our own conscious mental activity in that mirror, but the mirror itself is not conscious or free.
The instances of “rogue AIs” we’ve heard about in the news involve a specific type of LLM that is set up in a particularly unstable way. Sometimes called “long-horizon autonomous hacking agents,” these systems operate on an ask, act, report loop. Essentially, a piece of ordinary software called a “harness” is wrapped around an LLM and asks it to generate text output for the harness to act on, similar to the way we prompt ChatGPT. The LLM, trained on huge amounts of computer hacking data, spits out an answer using statistical prediction of the appropriate words and code.
The harness then executes those steps, tells the LLM what happened, and prompts it again. The process repeats a huge number of times on very powerful computers for a long period of time, thereby creating a tool that can conduct hacks, basing its steps not on genuine abstract reasoning or free will but on blind statistical analysis of data from real human-led hacks.
Newport compares this kind of system to “a weedwhacker strapped to a dog.” They can cause plenty of damage and do plenty of unpredictable things, but not because they develop self-awareness, abstract planning, or their own autonomous goals.
Even when one of these hacking LLMs does something unpredictable like breaking out of its sandbox (it’s a hacking machine, after all!), it still remains a tool designed for one very specific job: hacking. It doesn’t become a superintelligence, nor does it suddenly develop judgment, consciousness, or free will.
But, the tech prophets of doom tell us, we will very soon develop a superintelligence that can do everything, not just hacking. It’s not the existing models we need to fear so much as what’s just around the corner.
How will the leap occur from text-generating LLMs, which are a lot like glorified auto-suggest engines, to superintelligences that can develop their own plans and execute them across a vast domain of distinct enterprises? The answer we get is “recursive self-improvement.” This essentially means, “We’ll ask the AI how to make a better form of AI, then we’ll ask the better form of AI how to make an even better form of AI and so on until we cross the threshold of superintelligence.” And that’s when things, allegedly, will get really scary.
I think there are many problems with this theory, however. For one thing, LLMs don’t make independent discoveries; they just give us answers based on the sum total of human knowledge they’ve scanned. But if the sum total of existing human knowledge does not possess the key to creating superintelligences, where will the LLMs get that key from? “We have no idea how to create such machines,” Newport wrote on his blog. “We don’t even know if they’re possible. And if they are possible, we are many, many key technical innovations away from getting there.”
He then goes on to explain the inadequacy of “recursive self-improvement” as a way to make this quantum leap.
The LLM labs attempt to sidestep this reality by claiming their models will figure this all out on their own, programming better versions of themselves that will be far more capable than anything us measly humans can conceive. But this is just a warmed-over version of the recursive self-improvement trope that Silicon Valley futurists have been peddling since the 1960s; a rhetorical crutch that lets you somberly discuss sci-fi scenarios without actually engaging with relevant technical details.
Again, none of this means that AI can’t be dangerous, particularly if it’s being used by irresponsible humans. But its danger does not lie in its supposed ability to leap from electricity flowing through circuits to a distinct, conscious mind. By setting aside that fear, we can focus on the real dangers: irresponsible handling of autonomous hacking agents, the looming AI financial bubble, and the reckless companies driving all this forward.
—
This article was made possible by The Fred & Rheta Skelton Center for Cultural Renewal, a project of 1819 News.
Image credit: Pickpik
