Projects
A sentence about a jeweller run through the model, with a bolt of light climbing the layer stack and the sentence shown as tokens on the right
The model drawn as a stack of 28 translucent layer sheets
The page of a single neuron: its label, the explanation Claude wrote and its highlighted text snippets

Neuron Explorer

See inside a small language model: 86,016 neurons on one map. Run a sentence through it, then read what each neuron reacts to, what Claude makes of it, and what the model writes when you turn it up or down

Language models are usually described from the outside: you type, they answer. Neuron Explorer looks at the inside. It takes Qwen3-0.6B, a small open model, and shows every one of its 86,016 neurons as a dot on a map. Click a dot and you get that neuron's own page: the sentences from two million words of web text where it fired hardest, highlighted word by word, the sentences that pushed it the other way, the words it nudges the model towards, and the other neurons it feeds. For about three thousand of them you can also read what Claude made of the evidence, move a slider and watch the model's writing change as the neuron is pushed up or down. Nothing on the site is an illustration. Every highlight, number and sentence comes from running the real model.

Role
Design, pipeline and build
Year
Autumn 2026
Status
Live
Year
2026
Stack
SvelteKitTypeScriptWebGLPythonPyTorchQwen3-0.6BCaddy
The 33 second launch film: a textbook line about neurons, a real one under the microscope, then the model's layers, a sentence run through it and a neuron turned up. Plays with sound. Neuron photo: Cultured Rat Hippocampal Neuron by ZEISS Microscopy, CC BY 2.0.

01

The model as a map

28 layers of 3,072 neurons each, drawn as a stack you can orbit and step through. Dots sit next to neurons that do similar work, brighter dots are more consistent, and the larger pale ones are featured. Click one and its connections light up through the stack: amber for the neurons it excites, blue for the ones it suppresses.

1 / 2
The model drawn as a stack of layer sheets
The stack. One vertex per neuron and one draw call per layer, so the whole model fits in a browser tab.

02

Run a sentence through it

Pick one of eight prepared sentences and watch it travel up the stack. A bolt of light climbs through the layers and strikes the neurons that woke hardest, while a card shows the sentence as tokens, what the model expects to come next and the named neurons that fired. Typing your own is not possible yet, because nothing here runs a model.

A sentence about a jeweller run through the model, with a bolt of light climbing the layer stack
The jeweller sentence, traced. The bolts follow the neurons that woke hardest; on the right, what the model expects to come next.

03

One neuron, one page

Where it fires, token by token, in real text. Where it fires negative. A histogram of its activity, the words it promotes and suppresses, and the neurons above and below it that it talks to. A Simple/Technical switch, remembered in your browser, hides the numbers and rewrites the jargon; every term has a hover definition in both modes.

The page of a single neuron with its highlighted text snippets
Fires most on. Amber marks activation; the outlined token is the peak.

04

What Claude makes of it

Every featured neuron has a label and a written explanation, in a simple and a technical version. Claude wrote them from the neuron's strongest snippets, its logit lens and its steering outputs, and rated its own confidence. Only 325 of the 2,996 came out high; 303 labels start with 'unclear:'. That honesty is on the page, and so is a link to flag an explanation that looks wrong.

A neuron page titled gemstones and jewelry words, with the explanation card and the first highlighted snippets
Layer 21, neuron 2468. The label is the page title; the card underneath is Claude's reading of the evidence.

05

Turn it up or down

For featured neurons, a slider steps through saved generations at seven strengths from four prompts. Every step is the real model's output, precomputed, so it feels live without a server. A toggle compares the neuron against a random direction of the same size, so you can tell signal from noise.

The steering card with a strength slider, four prompts and the model's steered output
Layer 5, neuron 74 pushed to +0.6. The orange underline marks where the steered text leaves the unsteered one.

06

A tour, a glossary and a game

The tour is seven short scenes built from real data: tokens, the stack, one layer, one neuron reading a sentence, steering, a messy neuron, and off you go. Learn has four explainers and the glossary every hover term links to. Play shows you steered texts and asks which neuron was pushed; Claude picked the rounds that are actually guessable.

A tour scene: a sentence with its tokens coloured amber by one neuron's activation
Tour, step 4. The same sentence as step 1, now coloured by how hard one neuron fires on each token.

What it is. Language models are usually described from the outside: you type, they answer. Neuron Explorer looks at the inside. It takes Qwen3-0.6B, a small open model, and shows every one of its 86,016 neurons as a dot on a map. Click a dot and you get that neuron’s own page: the sentences from two million words of web text where it fired hardest, highlighted word by word, the sentences that pushed it the other way, the words it nudges the model towards, and the other neurons it feeds. For about three thousand of them you can also read what Claude made of the evidence, move a slider and watch the model’s writing change as the neuron is pushed up or down. Nothing on the site is an illustration. Every highlight, number and sentence comes from running the real model.

Why I built it

A language model is a file of numbers. Qwen3-0.6B is about 600 million of them, and nobody can read that, not even the people who trained it. What you can do is open it up and watch it work: feed it text, record what every part does, and push on one part to see what changes. That is what I wanted to see with my own eyes instead of reading about it. The interpretability work coming out of the big labs is some of the most interesting research around, and most of it happens on models you cannot touch. A small open model on a home PC is enough to do a real version of the same thing: capture activations, map the neurons, steer them, and find out how much of the “the model has a neuron for X” story survives contact with the data.

There is also a work reflex in it. As an AML analyst I am used to not taking a system’s word for what it does, and going to the records instead. A model’s answers are what it says about itself. Its activations are the records.

The idea came from lyramakesmusic’s neuron-explorer. The code is written from scratch. Anthropic’s models are closed, so the subject had to be an open-weight model; Claude gets a different job: it wrote the plain-language explanation of each featured neuron, and picked the rounds for the game.

How you use it

The map shows the model as 28 translucent sheets, one per layer, each a grid of 3,072 dots. Neurons that write to the model in similar ways sit next to each other, so patterns show up as neighbourhoods. You orbit the stack, zoom in, scrub through layers, or search for a neuron by number. Click a dot and amber and blue lines run through the stack: amber for the neurons it excites, blue for the ones it suppresses. Open it and you are on its page. A Simple/Technical switch rewrites the jargon and hides the numbers if you want them gone, and every term has a hover definition.

Run a sentence through it. On the map a box on the left offers eight prepared sentences: one about gems, one from a clinic, a Moon landing with a year and a distance, two lines of Python, a Dutch sentence, one that stops halfway, a grammar pattern and an everyday one. Pick one and a bolt climbs the stack layer by layer, striking the neurons that woke hardest. A card on the right shows the sentence as tokens, what the model expects to come next and the named neurons among the loud ones, and clicking a token narrows it all to that word. The jeweller sentence wakes the gemstone neuron from the tour at about 60 percent of the hardest it ever fired in the sample, the loudest in its layer. You cannot type your own, and the box says so: nothing here runs a model.

If the map is too much at once, the tour walks you from tokens to steering in seven scenes, the learn page explains the four ideas you need and holds the glossary, and the game lets you test whether you can spot a neuron from its steered text. The landing page also shows a neuron of the day.

What the model looks like from the inside

Three neurons to try

Layer 21, neuron 2468 is the tour’s example: it fires on gemstones and jewellery words, and a small push turns the model’s musings towards carats and gold rings. Layer 5, neuron 74 fires on legal citations: “42 U.S.C. § 1395”, amendments, statutes. Push it a little and the model’s news prompt turns into government and policy talk. Push it to the maximum and the text collapses into repetition, which is its own lesson: steering is a blunt tool, and the slider shows you where it breaks. Layer 21, neuron 2688 fires on the word right before a number (“buried under”, “for”, “every”). Pushed negative the model writes strings of digits; pushed positive it avoids numbers and gets wordy.

Where it stands

Done, for the first model. The whole of Qwen3-0.6B has been processed: two million tokens of educational web text run through it on the CPU of my gaming PC (the AMD card has no PyTorch support on Windows), one and a half hours for the pass and another two and a half to write 86,016 JSON files. The featured neurons were steered overnight: four prompts, seven strengths, 48 tokens each, with random-direction controls so you can check that a neuron’s effect is not just noise. Claude then explained all 2,996 of them in under an hour, and rated each one for the game, which left 341 rounds. The tour, the glossary and the game came next, with a neuron of the day on the landing page and a flag link under every explanation. Last came Trace and the short opening animation of the layers growing. The data behind the site is about 650 MB compressed, served as static files from a small VPS; no model runs there.

What is left is a second model, which means running the pipeline again rather than touching the site. Typing your own sentence comes only if people use the prepared ones: it would need the model in the visitor’s browser or a small server. Any open model can be run through it.

How it is built

A Python pipeline (uv, PyTorch, Transformers) runs the model once and writes everything the site needs as JSON: activations for all neurons, a 2D layout per layer from the neurons' output directions, connection tables from the weights, a featured selection with an even spread over layers, steering generations through a custom batched sampling loop, one recorded forward pass for each of the prepared trace sentences, and Claude's explanations and game ratings through the Claude Code CLI. The pages that matter to search engines are prerendered; the rest is a client-side app. The site is SvelteKit, built to static files behind Caddy. The map is WebGL with one vertex per neuron and one draw call per layer, with a canvas fallback for phones without WebGL. On the first visit of a browser session the map opens on a short animation of the layers growing, then gets out of the way.

What I learned building it

  • Precompute as much as you can. The site loads instantly, hosting costs a few euros a month, and adding a model means rerunning the pipeline, not rebuilding the site.
  • Let the GPU do the heavy lifting. 86,016 neurons is too many to render as separate page elements; that gave up at a few thousand. With WebGL the browser draws the whole thing without effort.
  • Make accessibility part of the design from the start. The Simple/Technical switch shaped not just how I wrote the text but how I built every page.
  • Show where it breaks, too. A small push shifts the model's topic; a big one wrecks its output. Showing both gives a more honest picture of what steering a neuron actually does.
  • Let the explainer admit doubt. Asked to rate its own confidence, Claude called 325 of 2,996 explanations high and started 303 labels with 'unclear:'. A tidy label on every neuron would have looked better and been worse.
  • Say what the site cannot do. The trace box offers eight prepared sentences and says plainly that typing your own is not possible, because nothing here runs a model. Running one would mean a download of several hundred megabytes in the visitor's browser or a server, and either changes what the site is. That is a decision for a later step, not a gap to hide.
  • Strip the tooling before you call a model in bulk. With the desktop app's MCP servers attached, every explanation call carried 60,000 tokens of tool schemas. With an empty config it was 1,300, and the whole set cost under three dollars in API terms.