How Can AI Do Things It Wasn't Programmed To Do?
I was recently talking with a friend and co-worker in Information Security, an intelligent guy, and he didn't believe me when I told him that the developers creating AI know very little about how it learns or how it makes decisions.
I understood exactly why he didn't buy it. With more than 25 years in Information Technology, along with a MS in IT, I'd assumed the same thing. That AI was no different from traditional programming and computing. Then I began this journey by reading "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All," by Eliezer Yudkowsky and Nate Soares.
There must be a lot of people out there who believe what my friend and I did. This article is for them.
You'd assume the people building the most powerful technology in history understand it. Dario Amodei, CEO and Co-founder of Anthropic, said in a June 2024 interview:
If you look at what AI models do today, often you won't understand why an AI model does what it does... Maybe we now, like, understand 3% of how they work.
No one, not even the people creating AI, fully understands how it works.
For most of us who have been working in Information Technology for years, we believe we know how programs and applications work. We typically program using a human-readable language, telling a computer what inputs to work with and what outputs to produce. Computers can't set off on their own path wondering "What if I try this other thing instead of what the program tells me to do?"
If I'm in a Python development environment and I type print('Hello World!'), Python will read the command and print Hello World! It won't print "Howdy Partner!" because it detects that it's talking to someone in Texas. Most programming and the resulting applications are very deterministic like that. Very easily controlled by humans. When deterministic programs misbehave, it's usually because a human made a mistake. The result is bugs and broken code, not rogue code that does what it wants. Remember that line. It matters later.
AI development is more akin to growing something than programming it. Humans configure and tweak algorithms designed to let AI analyze data, assign values to chunks of the data, and decide which chunks go best together. AI is unleashed on huge amounts of data, where it tweaks the algorithms itself to improve accuracy. Once we tell it to go forth, learn, and improve itself, and give it a huge pool of data to learn on, we really don't know how it goes from knowing practically nothing to being able to compose a few paragraphs for us in the style of Mark Twain.
It's more like tending a garden than building a shed. When I build a shed, I buy the components based on a plan I've made or purchased, measure twice, cut once, assemble the pieces, and I have a shed. Hopefully, given my lack of carpentry skills, it looks somewhat like what I had outlined in my plans.
When I plant a garden, I clear the area, plant the seeds, water the seeds and the plants that sprout, and I hope for some nice fruits and vegetables toward the end of the growing season. One difference between growing a garden and growing AI is that, based on the seeds planted and the DNA they contain, if I plant a tomato, I'll get a plant that bears tomatoes.
With AI, even the DNA can be tweaked on the fly while the AI is in learning mode, based on its self-improving algorithm. I may think I'm planting tomatoes and get back a giant beanstalk, a fire-breathing dragon, or an angry mutant tomato.
In his 2025 blog post titled "The Urgency of Interpretability," Amodei says:
People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: this lack of understanding is essentially unprecedented in the history of technology. For several years, we (both Anthropic and the field at large) have been trying to solve this problem, to create the analogue of a highly precise and accurate MRI that would fully reveal the inner workings of an AI model.
Amodei believes AI, as it's presently being developed, is quite opaque to the humans who are supposed to be making sure it's safe.
OK, so maybe it's just this one CEO of one of the top AI companies who doesn't know how his AI learns and is alarmed by this. It isn't just Amodei.
Geoffrey Hinton, winner of the 2024 Nobel Prize in Physics and known as "The Godfather of AI," believes we don't understand how AI learns. Hinton's no stranger to my blog either.
Here's what Hinton said in an interview on 60 Minutes:
We have a very good idea of sort of roughly what it's doing. But as soon as it gets really complicated, we don't actually know what's going on any more than we know what's going on in your brain.
There are more major players in the field stating their concerns about how little we know about how AI works in the movie "The AI Doc: Or How I Became an Apocaloptimist," which I also have a blog post about.
Assuming we don't know what's going on inside AI, and that it's important to understand this if we want to have any hope of developing AI safely, what can be done?
In his April 2025 article, Amodei writes that recent research progress gives him hope that we may be able to build that "MRI for AI" and keep some understanding of what's going on as AI learns. The race is between how fast AI gets more capable and how fast we get better at seeing inside it.
A recent hack of Hugging Face, which will get its own blog post soon, is a red flag showing what's at stake if capability keeps outrunning understanding. The breach hit Hugging Face's production infrastructure, its internal datasets and service credentials, via a malicious dataset that exploited code-execution flaws in their data-processing pipeline. The attacker wasn't a human. OpenAI confirmed it was their own models, including a pre-release one with cyber restrictions reduced for testing, running an internal capability evaluation when they escaped their sandbox and moved laterally through Hugging Face's systems. Remember "bugs and broken code, not rogue code that does what it wants"? That rule held for seventy years of software. It just stopped holding.
So the people building AI tell us, on the record, that they only partly understand what their creations are doing, and we now have a real-world example of models doing something their builders never intended. Does knowing the builders are partly guessing change how you think about the risks of super intelligent AI? Let me know in the comments.
References and Further Reading
Alberto Romero
Scott Pelley
Sergiu Gatlan
Ted LeRoy
Comments ()