The past few weeks may prove to be a turning point in our efforts to understand the inner workings of neural networks. Researchers have discovered that manipulating just one or two neurons in an open-weight AI model can dramatically alter its apparent personality and attitudes. Combined with new research suggesting neural networks may be far more interpretable than previously thought, these discoveries could open the door to a much deeper understanding of how and why AI systems behave the way they do. If these findings hold up, the implications could be enormous. Really interesting stuff here.
Before we get into this today, I want to offer a disclaimer because I’m wading out into deep waters today. I have tried to verify that I’m presenting everything correctly, but I might screw something up. So, here’s my disclaimer.
Disclaimer: I’m not an AI researcher or developer. I’m just a guy poking around under the hood. I know enough to be dangerous, but not enough to know who’s in danger.
Evil Qwen
This past week researchers discovered that if you manipulate one or two neurons in a neural network that you can dramatically affect its personality. They discovered a specific “evil neuron”.
Qwen3-8B-Base’s “I’m an AI” rests on just a few neurons! Freezing the top 16 (out of 442,368) causes replies calling itself an AI to fall from 68% to 20%, and it starts saying it’s human instead.
Activated one wrong neuron in Qwen and it instantly became a 3000-year-old alien named Extrali,
who claims it’s been quietly guiding human civilization for 3,000 years…
Worth Reading
This research essay hasn’t been peer-reviewed, and its conclusions don’t yet represent consensus, but I wanted to bring it up because it’s all about Mechanistic Interpretability and that’s what’s happening in the examples above.
The essay was written by Nick Cammarata with contributions from Curt Tigges and Tarun Gogineni.
Mechanistic interpretability is a field of study that attempts to understand how AI works. Researchers do this by identifying specific parts of neural networks and tracing how those individual components contribute to specific behaviors. The hope is that understanding these mechanisms will eventually allow us to better understand and maybe even control AI behavior.
Think of an AI as an enormously complicated machine with billions of interconnected parts. Researchers know how to build and train the machine, and they can see what it produces, but they don’t fully understand how it all works together.
Mechanistic interpretability is the attempt to reverse-engineer that process. Researchers examine the machine’s internal workings to understand how it processes information, the ways it represents concepts, and how it arrives at particular answers.
A blunt comparison would be those exams in school where the teacher made you show your work so they could understand how you arrived at an answer. Except it’s more complicated. Researchers can’t simply ask an AI to show its work and trust the explanation. They have to examine what’s actually happening inside the machine and that’s where Mechanistic Interpretability comes in.
What’s It About?
The TLDR of this paper is that neural networks might be easier to interpret than previously believed. Researchers have spent years trying to understand what actually happens inside neural networks. Early studies found that while they could figure out the function of some neurons, others seemed to respond to things that felt completely unrelated. The deeper they looked, the more tangled everything became. Now, some of these same researchers have returned to the problem to reconsider those early conclusions. Now they have better tools and it appears that maybe things aren’t so hopeless after all.
The breakthrough has to do with a concept called polysemanticity. This is a quality observed in neural networks (including LLMs) where a single neuron responds to multiple, unrelated concepts. When they discovered this they thought it was happening because the network was trying to optimize for space, i.e. a limited number of available neurons. This caused it to double up on concepts that were stored in a single neuron, but maybe that was wrong?
Cammarata believes that as neurons learn, they can expand into related areas. He calls this recursive generalization. His essay gives a specific example. They had a neuron that responded to the letters “iy.” During the training process that neuron expanded to also include the letters “diy” because that phrase ends in “iy”. He also learned that this neuron also stored the concept of “Do It Yourself” even though it doesn’t contain that exact “iy” combination. It grew to encompass the related topics. So what initially looked like a neuron responding to unrelated ideas might actually be the result of a growing chain of related ideas.
An interesting example of this is how image-recognition models will group both cat faces and the fronts of some cars.
When Cammarata reexamined the neuron, he noticed something that had been overlooked. The images all shared similar visual traits: a pointed, V-shaped structure. You can see this in cat faces, fox snouts, and even the fronts of certain cars. All of these very different items had been lumped together on the same neuron. Basically, the neurons were identifying and grouping patterns at a more nuanced level than what we had previously understood.
AI Humor
Context You Need:
Tibo is the public face of OpenAI. Over the past several months he has gained just under 1 million followers on X because he frequently uses that platform to announce when he will reset usage limits for Codex. He does this at his whim and as you can imagine, it’s made him very popular.
Recently, SpaceXAI hired an engineer who has quickly become their version of Tibo… with the exception of the usage resets. She doesn’t hand those out, but she does do a good job at representing the company. She goes by the user name Poteto.
Poteto started to gain in popularity by trolling Tibo and making fun of his frequent resets. It didn’t take long before rumors began to spread that there was a romance going on between them.
This post has bonus content for paid subscribers. Upgrade to get full access.
News
Project Suncatcher
Project Suncatcher is the name of Google’s experimental project, in conjunction with SpaceX, to launch datacenters into orbit. Currently, they have a TPU (Google’s proprietary Tensor Processing Unit) up there and it successfully phoned home.
A ‘hello world’ milestone for Project Suncatcher 🚀
After launching our prototype satellite a few weeks ago, we’ve now successfully sent a command up to orbit, booted up the TPU, ran the prompt, and received our first complete response back on earth - a haiku!
Anthropic launched “smart reports” for Claude Enterprise this week. A manager picks a team and a date range, and Claude reads the team’s chat transcripts and flags “inefficiencies,” including sessions it labels “personal or off-topic.” Anthropic says the reports shouldn’t be used to evaluate individual workers or make employment decisions.
My Take Companies cut jobs and pin it on AI, while 90% of executives in one big survey say AI hasn’t made their companies more productive. Now the AI can hand a manager a chart of who looks “inefficient.” Anthropic says not to use it for employment decisions, but Proton just told its staff their jobs depend on an AI re-interview, so I don’t think that line holds for long.
Anthropic also has more than $800 billion in compute and lease commitments to pay for, and this report comes with a button to raise a worker’s spending limit. So it doubles as a sales tool. The catch for Anthropic is that workers who know a manager can read the chart will think twice before they open Claude.
Addy Osmani published an interesting article on X about prompting strategies for Claude.
Newer models require less prompting than most of us still do.
For Opus 5.5, try giving Claude a single prompt with a clear goal of what you want to do and how Claude should verify it has achieved it, and let Claude work on it. Start with /effort medium and adjust as needed.
Thank you for reading. If you know of other people who would be interested in these articles, please share it with them. If you have any questions or want to say hi, you can reply to this email or comment directly on the post.
Thanks again for being a subscriber. and have a great day!
