Hinton's Warning: The Danger of Weights in Neural Networks
He calls them the "fissile material" of machine intelligence.
What if the most important concept in artificial intelligence isn't algorithms or data, but rather the numerical values that connect it all together? Those values are called weights.
From my understanding:
Weights are the numerical values that determine how strongly different pieces of information connect to each other in an artificial neural network’s "brain".
Just as the synapses in your brain have varying strengths that shape how thoughts and memories form, weights in artificial neural networks control how signals flow between artificial neurons.
They're the fundamental building blocks that allow AI systems to recognise patterns, make decisions, and learn from experience.
And Geoffrey Hinton, the "Godfather of AI", believes they hold the key to understanding not just how machines learn, but how knowledge itself might evolve beyond human control.
When I first came across Hinton's thoughts on weights, I'll admit I was a little surprised by how personal and honest his insights felt.
Here was a man who had spent decades(!) building the foundations of modern AI, now coming to terms with the realisation that he might have helped create something fundamentally different from human intelligence.
Something that could outlast us all.
What makes a mind?
Think about the last time you learnt something truly meaningful. Perhaps it was mastering a new skill, or understanding a complex idea. What happened in that moment wasn't just information storage.
It was the strengthening and weakening of countless connections in your brain. These connection strengths are what Hinton calls weights, and they represent nothing less than the physical embodiment of everything you know.
The fundamental question that has driven Hinton's entire career is elegantly simple yet profound:
"How do you change those weights to make the whole thing do something good?"
In this interview with AI-powered learning platform Sana, he discusses the decades-long journey of trying to understand how learning works in both brains and machines.
It's a question that applies equally to helping a child learn to read and to training an AI system that might one day surpass human intelligence. But here's where it gets unsettling.
When you learn something new, your brain adjusts these weights through a process we still don't fully understand. Hinton believes the brain must be "getting gradients" for weight adjustment because learning is much worse without them, yet whether our brains use something like backpropagation, the algorithm that powers modern AI, remains an open question.
What does it mean for our sense of human uniqueness if our learning processes might be fundamentally similar to those of machines?
Mortal vs. immortal intelligence
Here's something that made me pause when I first understood its implications: your knowledge will die with you. Not in the poetic sense we often discuss, but in a literal, technical sense that reveals something profound about human nature.
Your brain is an analogue system, shaped by your unique biological hardware.
The weights that represent everything you've learnt, from your grandmother's laugh to your understanding of quantum physics, are "no good for any other brain".
When you die, those particular patterns of connection strengths become "useless" for direct transfer.
This makes biological intelligence fundamentally "mortal" in terms of knowledge sharing. We're limited to the painfully slow process of language, roughly 10 bits per second, to share the vast libraries of understanding stored in our neural connections.
I find myself wondering if this limitation is what makes us human. The fact that each of us must painstakingly reconstruct knowledge through our own experience, our own mistakes, our own moments of insight?
Digital AI systems operate under entirely different rules. Their weights can be perfectly copied, stored, and transferred between different hardware systems.
More remarkably, multiple AI systems can start with identical weights, learn different things, then share and average their discoveries, allowing all systems to instantly know what any individual system has learned.
This process involves "transferring trillions of bits a second" compared to our meagre human bandwidth. Hinton describes his realisation of this digital superiority as a "eureka moment" that occurred while working on analogue computation at Google.
But what are the implications of intelligence that never forgets, never dies, and can share knowledge at speeds that dwarf human communication? When I consider this, I'm reminded of conversations with elderly relatives, watching them struggle to pass on a lifetime of wisdom before it's lost forever.
Digital intelligence faces no such constraint.
The fast weights dilemma
One of the most intriguing aspects of Hinton's thinking, concerns what he calls fast weights.
Fast weights are temporary changes to neural connections that depend on immediate input data.
Our brains excel at this, using multiple time scales for weight changes to create temporary memories and contextual understanding.
Current AI models largely miss this capability because they're optimised for parallel processing. Fast weights would require constant, input-dependent adjustments that would slow down efficient computations.
Hinton considers this "one of the biggest things we have to learn" from neuroscience.
This limitation might seem purely technical, but it raises deeper questions about the nature of understanding. When I'm having a conversation, my brain is constantly adjusting based on context, emotion, and the subtle dynamics of human interaction.
If AI systems develop this capability, will they become more human-like in their reasoning, or will they transcend human limitations entirely?
Compression, creativity, and the analogy engine
Large language models compress vast amounts of information into their weights by finding analogies and common structures across seemingly unrelated concepts.
A compost heap and an atomic bomb both involve chain reactions, and AI systems excel at discovering these hidden connections.
This compression through analogy-finding is what makes AI systems potentially "even more creative than people". But creativity in the service of what values?
This is where the ethical implications become unavoidable.
When I watch my son learn, I see him making these same analogical connections, but filtered through human experience, emotion, and moral intuition. What happens when superhuman creativity operates without these constraints?
In a conversation with Steven Bartlett, Hinton explained why he believes AI's creativity will surpass our own and what this means for the future of work.
A stark warning on open-sourcing weights
Geoffrey Hinton expresses significant concerns regarding the release of "weights" for large language models (LLMs) by major AI companies. He views this practice as "crazy" and believes it removes a critical barrier to the misuse of powerful AI.
Here's a breakdown of his warnings:
1. Removal of the main barrier
Hinton states that once the weights are released, the main barrier to using these powerful AI systems is removed. He draws a direct comparison to nuclear weapons, explaining that the reason few countries possess them is the difficulty in obtaining "fissile material".
In AI, the "weights of a big model" are the equivalent.
2. Cost barrier elimination
Training a truly large model costs hundreds of millions of dollars, a high cost that acts as a barrier for smaller groups like cults or cyber criminals. However, once the weights are released, these groups can fine-tune the models for their purposes for "just a few million dollars", effectively circumventing the initial high cost.
3. A misleading comparison to open source
Hinton argues that releasing weights is "very, very different from open source" software. With open-source software, releasing the code allows many people to review it and fix bugs.
In contrast, when weights are released, people don't examine the weights for errors; instead, they "take this foundation model... and they train it to do something bad".
4. Disagreement with the alternative
Hinton acknowledges the counter-argument that restricting access means a handful of large companies control the technology. However, he believes "that's better than everybody controlling the massively powerful technology", reiterating his nuclear weapons analogy by asking if you would prefer a few countries to control them or if everybody should have them.
His primary concern is that releasing weights significantly increases the risk of "bad actors using AI for bad things" by making highly capable AI technology accessible for malicious purposes.
This concern about misuse is amplified by real-world policy. As Hinton himself has noted, the landmark EU AI Act, one of the world's first comprehensive attempts to regulate AI, contains a specific exemption for systems used exclusively for military and defense purposes. While governments debate rules for civilian life, the development of autonomous weapons and AI-driven military strategy is effectively left outside the scope of this key legislation, which just underscores the gravity of the challenge.
The alignment challenge in a new light
Traditional approaches to AI safety often focus on training systems on carefully curated data or manually adjusting their behaviour. But Hinton's insights about weights complicate this challenge in subtle ways.
The sheer complexity of modern AI systems, with their interdependent networks of billions of weights, makes direct manipulation increasingly impractical. Hinton suggests that
"Training AI on empathetic data might be more effective than manually adjusting weights."
But this raises its own questions about who decides what constitutes empathetic data and whose values get encoded into these systems. I'm struck by how the technical reality of weights intersects with questions of power, representation, and democratic governance.
Living with uncertainty
Perhaps what I find most compelling about Hinton's perspective is his willingness to sit with uncertainty. Despite being one of the architects of modern AI, he remains genuinely unsure about fundamental questions:
Does the brain use backpropagation? How should we govern systems we don't fully understand? What happens when AI systems begin modifying their own weights autonomously?
This uncertainty is a model for how we should approach these technologies. Too often, discussions about AI swing between utopian promises and dystopian warnings. Hinton's nuanced view of weights offers a different path.
He proposes deep technical understanding combined with honest acknowledgement of what we don't yet know.
Your role in the conversation
AI Ethics Unwrapped exists to give you the tools and information to think critically about AI's impact on your life.
Understanding weights is about recognising that the intelligence systems increasingly shaping our world operate on principles fundamentally different from human cognition.
When you interact with an AI system, you're engaging with patterns learned and stored in ways that transcend individual mortality. When companies make decisions about AI development, they're making choices about the distribution of intelligence itself.
When policymakers attempt to regulate AI, they're governing systems whose "knowledge" can be perfectly copied and shared at unprecedented speeds. These aren't distant concerns for technologists to sort out. They're immediate questions about power, knowledge, and human agency in an age of artificial intelligence.
As I reflect on Hinton's journey from AI pioneer to AI safety advocate, I'm reminded that the most important ethical questions often emerge from deep technical understanding and honest reflections about consequences.
His evolution on weights, from seeing them as mere technical parameters to recognising them as the foundation of digital immortality, offers a model for how we might approach other aspects of AI development.
The conversation about AI's future cannot remain confined to academic circles and technology companies. Understanding weights and what they are, how they work, and what they mean for human agency is essential for anyone who wants to participate meaningfully in shaping our AI-enabled future.
After all, in a world where intelligence itself can be copied, shared, and potentially improved upon indefinitely, the question isn't just what AI can do for us. It's whether we'll remain relevant enough to have a say in the process.
Hi, I'm Miriam - an AI ethics writer, analyst and strategic SEO consultant.
I break down the big topics in AI ethics, so we can all understand what's at stake. And as a consultant, I help businesses build resilient, human-first SEO strategies for the age of AI.
If you enjoy these articles, the best way to support my work is with a free or paid subscription. It keeps this publication independent and ad-free, and your support means the world. 🖤