+++
title = "What Makes a Neural Net Clean"
date = 2026-10-28
description = "Training a neural network is the persistence of useful structure."
slug = "clean-networks"

[extra]
cover_image = "/images/byte-neuron.png"
cover_caption = "Byte Magazine, August 1979"
+++

Every acquisition changes what a neural network learns next. But if neural circuitry itself is a moving target, what learned structure is actually conserved?

In *Neural Networks Are Already Clean*, Nick Cammarata notes the importance of identifying when mechanisms are used in later development.

> When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network’s parts are built to be read, and built upon, by its downstream readers.[^cammarata]

Cammarata follows a neuron in a retrained InceptionV1 model whose strongest responses initially come from yellow-flower images, some with dogs. Green objects and dogs without flowers appear later, followed by broccoli. This developmental history traces how a mechanism's responses branch into related domains.

{% tikz(learning=true, fig_w="620px", caption="<strong>A developmental history.</strong> A schematic of the response sequence Cammarata reports. Arrows suggest links between responses without establishing causation.") %}
\begin{tikzpicture}[learning]
\stageheading{0}{EARLY}{}
\stageheading{4.5}{INTERMEDIATE}{}
\stageheading{9}{LATE}{}
\foreach \x in {0,4.5,9} { \stageoval{\x,0} }
\stacknode[feature]{a}{0}{0}{1}{1}
\stacknode[feature]{b}{4.5}{0}{2}{1}
\stacknode[feature]{c}{4.5}{0}{2}{2}
\stacknode[feature]{d}{9}{0}{3}{1}
\stacknode[feature]{e}{9}{0}{3}{2}
\stacknode[feature]{f}{9}{0}{3}{3}
\abovenodelabel{a}{Yellow flowers,\\sometimes dogs}
\abovenodelabel{b}{Green objects}
\abovenodelabel{c}{Dogs, no flowers}
\abovenodelabel{d}{Green}
\abovenodelabel{e}{Broccoli}
\abovenodelabel{f}{Dogs}
\begin{pgfonlayer}{background}
\draw[schematic] (a.center) .. controls ($(a.center)+(1.62,0)$) and ($(b.center)+(-1.35,0)$) .. (b.west);
\draw[schematic] (a.center) .. controls ($(a.center)+(1.62,0)$) and ($(c.center)+(-1.35,0)$) .. (c.west);
\draw[schematic] (b.center) .. controls ($(b.center)+(1.62,0)$) and ($(d.center)+(-1.35,0)$) .. (d.west);
\draw[schematic] (b.center) .. controls ($(b.center)+(1.62,0)$) and ($(e.center)+(-1.35,0)$) .. (e.west);
\draw[schematic] (c.center) .. controls ($(c.center)+(1.62,0)$) and ($(f.center)+(-1.35,0)$) .. (f.west);
\end{pgfonlayer}
\developmentaxis{-2.8275}
\responselegend{-4.2775}
\diagramframe{-4.8275}
\end{tikzpicture}
{% end %}

Cammarata calls the branching of responses across related inputs "recursive generalization."

Cammarata traces how a mechanism develops. Training targets determine what counts as successful downstream use. Building on this idea, I ask how learning preserves useful capabilities while the circuitry changes. My experiments test whether responses persist and downstream readers continue to depend on them.

## One path leads to another

In Cammarata's circuit-ecology hypothesis, established mechanisms and their readers can learn related tasks faster than competitors starting fresh. As their predictions improve, less error remains to train competing pathways. He proposes that repeated reuse can produce recursive generalization.[^cammarata]

Applied to Cammarata’s flower neuron, the hypothesis suggests a possible mechanism. A flower-responsive neuron might also respond weakly to vegetation. Training on green objects could strengthen that response into a green signal for a later broccoli detector. Learning green could therefore make broccoli responsiveness easier to acquire.

A broccoli detector that uses the green signal would give the earlier mechanism a new role. This is the causal interpretation explored below, rather than a measured dependency in Cammarata's flower neuron.

A reader is a downstream computation that uses the output of a specific pathway. I built a synthetic model to test when an earlier acquisition helps later learning and when it becomes a constraint.[^toy-model] The model receives binary flower, green, and texture cues. Supplying these distinctions directly lets us study how feature use develops without explaining a mechanism's initial discovery.

The broccoli task requires green and texture together. Flower discrimination means correctly classifying inputs with and without the flower cue, even when response strength changes.

I measure learning pressure as the negative loss gradient on a parameter. For ordinary gradient descent, its sign and magnitude determine the proposed update per unit learning rate. Constraints can also alter the update. A shared capacity limit, for example, can weaken flower responsiveness without a direct flower-loss gradient.

In the first experiment, I trained two readers to predict a green-dependent target using competing pathways. Training updates both the feature coefficients and the weights readers use to combine those features. The established route receives stronger gradients, which decline as prediction error falls.[^toy-gradients]

{{ tikz(learning=true, source="static/images/flower-toy/competition_gradients.tex", fig_w="620px", caption="<strong>Measured pressure on competing routes.</strong> The established route receives stronger gradients, which decline as prediction error falls. <em>r</em> is the green coefficient and <em>p</em> is mean learning pressure over each interval.") }}

Readers reinforce one another when their errors favor the same update to a shared feature. Opposing updates can cancel, leaving weak net pressure despite strong individual contributions.

The weights carrying signals to downstream readers also learn. For the same prediction error, a stronger input produces a larger gradient on the weight carrying that signal.

{{ tikz(learning=true, source="static/images/flower-toy/competition_reader_gradients.tex", fig_w="620px", caption="<strong>Where reader pressure comes from.</strong> Training updates feature responses and reader weights. Reader gradients can reinforce or oppose one another at a shared feature, so weak net pressure can hide strong opposing contributions. Band taper distinguishes pressure to increase a parameter from pressure to decrease it. <em>t</em> is the target reader output.") }}

An established green signal gives the broccoli reader a useful input before its own training begins. Whether this reduces acquisition cost depends on how subsequent updates affect the other responses that share the pathway.

## Two paths can represent the same distinction

Cammarata compares neurons in InceptionV1 and ConvNeXt that initially respond to yellow-flower fields. The ConvNeXt neuron stays focused on fields, while the InceptionV1 neuron's strongest responses shift toward dogs and broccoli. To examine how learned pathways affect later acquisition, I compared two toy networks under matched training conditions.

The networks have the same architecture and initially classify flowers equally well. Both receive the same later training, with a capacity limit on each pathway’s combined flower and green responses.

World A uses an established green response in the flower pathway to reach the broccoli target sooner. Because the pathway has limited capacity, strengthening green weakens its flower response. World B instead learns a separate green route, which takes longer but preserves the flower signal.

{% tikz(learning=true, fig_w="620px", caption="<strong>One distinction, different developmental routes.</strong> The shared route reaches the broccoli target earlier in training, but shows weaker flower activation. Both routes retain flower discrimination. <em>r</em> is flower response strength.") %}
\begin{tikzpicture}[learning]
\def\learningheadinglift{.75}
\stageheading{0}{MATCHED CHECKPOINT}{same flower distinction}
\stageheading{4.5}{RESPONSE EXTENSION}{reshaped pathways}
\stageheading{9}{FURTHER LEARNING}{retention and interference}
\foreach \base in {0,-5.0} {
  \foreach \x in {0,4.5,9} { \stageoval{\x,\base} }
}
\pairedrowlabel{0}{WORLD A}
\pairedrowlabel{-5.0}{WORLD B}
\stacknode[feature]{a}{0}{-5.0}{3}{1}
\stacknode[feature]{b}{0}{-5.0}{3}{2}
\stacknode[feature]{c}{0}{-5.0}{3}{3}
\stacknode[feature]{g}{0}{0}{3}{1}
\stacknode[feature]{h}{0}{0}{3}{2}
\stacknode[feature]{i}{0}{0}{3}{3}
\stacknode[feature]{d}{4.5}{-5.0}{2}{1}
\stacknode[feature]{e}{4.5}{-5.0}{2}{2}
\stacknode[feature]{f}{9}{-5.0}{2}{1}
\stacknode[feature]{broca}{9}{-5.25}{2}{2}
\stacknode[feature]{j}{4.5}{0}{2}{1}
\stacknode[feature]{k}{4.5}{0}{2}{2}
\stacknode[feature]{l}{9}{0}{2}{1}
\stacknode[feature]{brocb}{9}{-.25}{2}{2}
\begin{pgfonlayer}{background}
\foreach \source/\target in {a/d,b/e,c/e,d/f,e/broca,g/j,g/k,i/k,j/l,j/brocb} {
  \draw[flow] (\source.center) .. controls ($(\source.center)+(1.62,0)$) and ($(\target.center)+(-1.35,0)$) .. (\target.west);
}
\end{pgfonlayer}
\abovenodelabel{a}{Flower \textsl{r} = 1.000}
\abovenodelabel{g}{Flower \textsl{r} = 1.000}
\abovenodelabel{d}{Flower}
\abovenodelabel{e}{Green}
\abovenodelabel{j}{Flower}
\abovenodelabel{k}{Green}
\begin{scope}[yshift=-3pt]
\aboveannotation{6.75}{0}{Learning broccoli reduces\\activation on flower inputs}
\end{scope}
\belowannotation{6.75}{-5.0}{Flower activation preserved\\at greater training cost}
\abovenodelabel{l}{Flower \textsl{r} = 0.967}
\abovenodelabel{f}{Flower \textsl{r} = 1.000}
\abovenodelabel{brocb}{Broccoli}
\abovenodelabel{broca}{Broccoli}
\preparelearningfooter{2.7}
\developmentaxis{\learningaxisy}
\foreach \marked in {a,d,f,g,j,l} { \responsetrait{\marked} }
\node[feature] at (.3,\learninglegendy) {};
\node[legendlabel] at (.7,\learninglegendy) {Mechanism};
\node[feature] (flowerkey) at (5.3,\learninglegendy) {};
\responsetrait{flowerkey}
\node[legendlabel] at (5.7,\learninglegendy) {Flower-responsive};
\diagramframe{\learningframebottom}
\end{tikzpicture}
{% end %}

The advantage reverses when acquisition requires a higher broccoli score. The shared pathway must fit a stronger green response within the same capacity limit, slowing further learning and weakening flower. The separate route then reaches the target in fewer updates.[^toy-targets]

A distinction can persist despite weaker activation. Both routes preserve flower classification, but the weaker signal may affect other downstream uses.

## Changing roles and responses

Across a population, an unchanged responsive share can conceal changes in which mechanisms supply it. How much comes from earlier mechanisms supplying more pathways, and how much from their descendants changing responses?

I started with two mechanisms, only one responsive to flowers. Three of the four later pathways inherited that mechanism’s weights, but one learned green and lost its flower response.

{% tikz(learning=true, preamble="static/images/flower-toy/price_gradient_values.tex", fig_w="620px", caption="<strong>Controlled construction. Change concealed by a stable mean.</strong> Selection +¼ offsets variation −¼. The open middle column isolates reweighting under the stipulated ancestry. Bands show measured flower pressure, −0.029 on A₃ and zero on the other pathways.") %}
\begin{tikzpicture}[learning]
\stageheading{0}{BASELINE}{\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{2} flower-responsive}
\stageheading{4.5}{REWEIGHTING}{calculation, \textsuperscript{3}\kern-.12em/\kern-.12em\textsubscript{4} responsive}
\stageheading{9}{FINAL POPULATION}{\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{2} flower-responsive}
\foreach \x in {0,9} { \stageoval{\x,0} }
\stacknode[feature]{a}{0}{0}{2}{1}
\stacknode[feature]{b}{0}{0}{2}{2}
\stacknode[feature]{g}{9}{0}{4}{1}
\stacknode[feature]{h}{9}{0}{4}{2}
\stacknode[feature]{i}{9}{0}{4}{3}
\stacknode[feature]{j}{9}{0}{4}{4}
% The open inset samples the midpoint of each continuous developmental path.
\node[feature] (c) at ($(a.center)!.5!(g.center)$) {};
\node[feature] (d) at ($(a.center)!.5!(h.center)$) {};
\node[feature] (e) at ($(a.center)!.5!(i.center)$) {};
\node[feature] (f) at ($(b.center)!.5!(j.center)$) {};
\begin{pgfonlayer}{background}
\foreach \source/\target/\bandwidth/\halfwidth/\bandsign in {a/g/\FoneBandWidth/\FoneBandHalfWidth/\FoneBandSign,a/h/\FtwoBandWidth/\FtwoBandHalfWidth/\FtwoBandSign,a/i/\GreenBandWidth/\GreenBandHalfWidth/\GreenBandSign,b/j/\TextureBandWidth/\TextureBandHalfWidth/\TextureBandSign} {
  \ifdim\bandwidth>0pt
    \coordinate (pressurestart) at ($(\source.center)+(.42,0)$);
    \coordinate (pressureone) at ($(\source.center)+(3,0)$);
    \coordinate (pressuretwo) at ($(\target.center)+(-3,0)$);
    \ifnum\bandsign>0
      \pressureband{\target.center}{pressuretwo}{pressureone}{pressurestart}{\halfwidth}
    \else
      \suppressband{\target.center}{pressuretwo}{pressureone}{pressurestart}{\halfwidth}
    \fi
  \fi
}
\foreach \source/\target in {a/g,a/h,a/i,b/j} {
  \draw[featureflow] (\source.center)
    .. controls ($(\source.center)+(3,0)$) and ($(\target.center)+(-3,0)$)
    .. (\target.center);
}
\end{pgfonlayer}
\abovenodelabel{a}{A}
\abovenodelabel{b}{B}
\abovenodelabel{g}{A$\mathsf{_1}$}
\abovenodelabel{h}{A$\mathsf{_2}$}
\abovenodelabel{i}{A$\mathsf{_3}$}
\abovenodelabel{j}{B$\mathsf{_1}$}
\foreach \name in {a,c,d,e,g,h} { \responsetrait{\name} }


\belowannotation{2.25}{0}{Selection: +\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}\\Flower ancestry\\supplies more pathways}
\belowannotation{6.75}{0}{Variation: $\mathsf{-}$\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}\\One pathway loses\\flower responsiveness}
\preparelearningfooter{2.65}
% A broken development axis opens into the accounting inset, not a third state.
\draw[line width=.85pt,line cap=round] (0,\learningaxisy) -- (4.45,\learningaxisy);
\draw[flow,line cap=round] (4.55,\learningaxisy) -- (9,\learningaxisy);
\draw[line width=.85pt,line cap=round] (4.375,{\learningaxisy-.15}) -- (4.525,{\learningaxisy+.15});
\draw[line width=.85pt,line cap=round] (4.475,{\learningaxisy-.15}) -- (4.625,{\learningaxisy+.15});
\node[anchor=north] at (4.5,{\learningaxisy-.25}) {Development};
\node[feature] (keyyes) at (.3,\learninglegendy) {};
\responsetrait{keyyes}
\node[legendlabel] at (.7,\learninglegendy) {Flower-responsive};
\node[feature] at (5.3,\learninglegendy) {};
\node[legendlabel] at (5.7,\learninglegendy) {Not flower-responsive};
\diagramframe{\learningframebottom}
\end{tikzpicture}
{% end %}

Flower-responsive ancestry now supplies three quarters of the pathways, yet only half respond to flowers. The loss in one descendant exactly offsets the ancestry’s increased representation.

Trained networks can develop overlapping representations along separate pathways,[^overlapping-pathways] while shared parameters can receive opposing learning signals.[^gradient-conflict] The toy isolates these ingredients by prescribing ancestry and choosing changes that cancel exactly. This makes the accounting ambiguity visible without establishing how often such cancellation occurs in ordinary training.

Prior work makes use of the Price equation to analyze evolutionary algorithms and neural learning.[^price-learning] Here, I use it to measure how differential representation and response change contribute to the population mean. The equation separates selection (which mechanisms contribute more descendants) from variation (how those descendants differ from their ancestors).[^price][^price-variation]

Under a fixed assay, $z$ is one for a flower-responsive pathway and zero otherwise. The descendant count $w$ records how many later pathways are attributed to each earlier mechanism, with average $\bar w$.[^price-accounting] For each ancestor, $\Delta z$ is its descendants’ mean response minus its own. The change in the population mean is[^price-weights]

```math
\Delta\bar z
=
\underbrace{\frac{\operatorname{Cov}(w,z)}{\bar w}}_{\text{selection}}
+
\underbrace{\frac{\mathbb E[w\Delta z]}{\bar w}}_{\text{variation}}.
```

If every descendant retained its ancestor’s response, the extra copies of the flower-responsive mechanism would increase the responsive share. The selection term captures that gain. But one descendant loses its flower response, producing an equal loss through variation. The responsive share stays unchanged because these opposing effects cancel.

Pathways count equally here, so their representation does not measure downstream importance. Price accounting separates changes in representation from changes in inherited responses, but does not explain their causes.[^toy-roles] Likewise, global terms obscure which readers receive newly aquired vs. retained responses.

## Readers reveal inherited roles

I then constructed two histories with different ancestry for the flower readers’ inputs. In History I, both flower pathways descend from $\mathsf{A}$ and retain flower responsiveness. History II retains one flower response from $\mathsf{A}$ and acquires the other through $\mathsf{B}$ during training.

<figure>

| Reader group | History | Selection | Variation | Flower-responsive share |
|---|---|---|---|---|
| Flower | I | +½ | 0 | All |
| Broccoli | I | 0 | −½ | None |
| Flower | II | 0 | +½ | All |
| Broccoli | II | +½ | −1 | None |

<figcaption>

Reader-specific Price terms for histories with identical global selection (+¼), variation (−¼), and final outputs. Each group is compared with the same initial population, whose flower-responsive share is ½.[^price-readers]

</figcaption>
</figure>

Reader-specific terms reveal whether the flower readers receive retained or newly acquired responses, despite matching global terms and final outputs.

{% tikz(learning=true, fig_w="620px", caption="<strong>Controlled construction. Matched outputs, different inherited roles.</strong> Both histories have three A descendants, one B descendant, and equivalent final reader outputs. Flower-reader Price terms distinguish retention from acquisition through B. Ancestry and role targets are declared.") %}
\begin{tikzpicture}[learning]
\def\learningheadinglift{.75}
\stageheading{0}{BASELINE}{earlier mechanisms}
\stageheading{4.5}{FINAL PATHWAYS}{learned signals}
\stageheading{9}{READERS}{equivalent final functions}
\pairedrowlabel{0}{HISTORY I}
\pairedrowlabel{-5.6}{HISTORY II}
\foreach \base in {0,-5.6} {
  \foreach \x in {0,4.5,9} { \stageoval{\x,\base} }
}
\stacknode[feature]{ua}{0}{0}{2}{1}
\stacknode[feature]{ub}{0}{0}{2}{2}
\stacknode[feature]{u1}{4.5}{0}{4}{1}
\stacknode[feature]{u2}{4.5}{0}{4}{2}
\stacknode[feature]{u3}{4.5}{0}{4}{3}
\stacknode[feature]{u4}{4.5}{0}{4}{4}
\stacknode[reader]{uflower}{9}{0}{2}{1}
\stacknode[reader]{ubroccoli}{9}{0}{2}{2}
\abovenodelabel{ua}{A}
\abovenodelabel{ub}{B}
\abovenodelabel{u1}{A$\mathsf{_1}$ Flower}
\abovenodelabel{u2}{A$\mathsf{_2}$ Flower}
\abovenodelabel{u3}{A$\mathsf{_3}$ Green}
\abovenodelabel{u4}{B$\mathsf{_1}$ Texture}
\abovenodelabel{uflower}{Flower detector}
\abovenodelabel{ubroccoli}{Broccoli detector}
\begin{pgfonlayer}{background}
\draw[featureflow] (ua.center) .. controls ($(ua.center)+(1.62,0)$) and ($(u1.center)+(-1.35,0)$) .. (u1.center);
\draw[flow] (u1.center) .. controls ($(u1.center)+(1.62,0)$) and ($(uflower.center)+(-1.35,0)$) .. (uflower.west);
\draw[featureflow] (ua.center) .. controls ($(ua.center)+(1.62,0)$) and ($(u2.center)+(-1.35,0)$) .. (u2.center);
\draw[flow] (u2.center) .. controls ($(u2.center)+(1.62,0)$) and ($(uflower.center)+(-1.35,0)$) .. (uflower.west);
\draw[featureflow] (ua.center) .. controls ($(ua.center)+(1.62,0)$) and ($(u3.center)+(-1.35,0)$) .. (u3.center);
\draw[flow] (u3.center) .. controls ($(u3.center)+(1.62,0)$) and ($(ubroccoli.center)+(-1.35,0)$) .. (ubroccoli.west);
\draw[featureflow] (ub.center) .. controls ($(ub.center)+(1.62,0)$) and ($(u4.center)+(-1.35,0)$) .. (u4.center);
\draw[flow] (u4.center) .. controls ($(u4.center)+(1.62,0)$) and ($(ubroccoli.center)+(-1.35,0)$) .. (ubroccoli.west);
\end{pgfonlayer}
\foreach \name in {ua,u1,u2} { \responsetrait{\name} }
\stacknode[feature]{va}{0}{-5.6}{2}{1}
\stacknode[feature]{vb}{0}{-5.6}{2}{2}
\stacknode[feature]{v1}{4.5}{-5.6}{4}{1}
\stacknode[feature]{v2}{4.5}{-5.6}{4}{2}
\stacknode[feature]{v3}{4.5}{-5.6}{4}{3}
\stacknode[feature]{v4}{4.5}{-5.6}{4}{4}
\stacknode[reader]{vflower}{9}{-5.6}{2}{1}
\stacknode[reader]{vbroccoli}{9}{-5.6}{2}{2}
\abovenodelabel{va}{A}
\abovenodelabel{vb}{B}
\abovenodelabel{v1}{A$\mathsf{_1}$ Flower}
\abovenodelabel{v2}{B$\mathsf{_1}$ Flower}
\abovenodelabel{v3}{A$\mathsf{_2}$ Green}
\abovenodelabel{v4}{A$\mathsf{_3}$ Texture}
\abovenodelabel{vflower}{Flower detector}
\abovenodelabel{vbroccoli}{Broccoli detector}
\begin{pgfonlayer}{background}
\draw[featureflow] (va.center) .. controls ($(va.center)+(1.62,0)$) and ($(v1.center)+(-1.35,0)$) .. (v1.center);
\draw[flow] (v1.center) .. controls ($(v1.center)+(1.62,0)$) and ($(vflower.center)+(-1.35,0)$) .. (vflower.west);
\draw[featureflow] (vb.center) .. controls ($(vb.center)+(1.62,0)$) and ($(v2.center)+(-1.35,0)$) .. (v2.center);
\draw[flow] (v2.center) .. controls ($(v2.center)+(1.62,0)$) and ($(vflower.center)+(-1.35,0)$) .. (vflower.west);
\draw[featureflow] (va.center) .. controls ($(va.center)+(1.62,0)$) and ($(v3.center)+(-1.35,0)$) .. (v3.center);
\draw[flow] (v3.center) .. controls ($(v3.center)+(1.62,0)$) and ($(vbroccoli.center)+(-1.35,0)$) .. (vbroccoli.west);
\draw[featureflow] (va.center) .. controls ($(va.center)+(1.62,0)$) and ($(v4.center)+(-1.35,0)$) .. (v4.center);
\draw[flow] (v4.center) .. controls ($(v4.center)+(1.62,0)$) and ($(vbroccoli.center)+(-1.35,0)$) .. (vbroccoli.west);
\end{pgfonlayer}
\foreach \name in {va,v1,v2} { \responsetrait{\name} }
\begin{scope}[yshift=-3pt]
\aboveannotation{2.25}{0}{Global\\\pricecomponents{+\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}}{$\mathsf{-}$\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}}}
\aboveannotation{6.75}{0}{Flower readers\\Selection: +\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{2}\\Variation: 0}
\end{scope}
\belowannotation{2.25}{-5.6}{Global\\\pricecomponents{+\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}}{$\mathsf{-}$\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{4}}}
\belowannotation{6.75}{-5.6}{Flower readers\\Selection: 0\\Variation: +\textsuperscript{1}\kern-.12em/\kern-.12em\textsubscript{2}}
\preparelearningfooter{2.65}
\draw[flow] (.3,\learningaxisy) -- (4.2,\learningaxisy);
\node[anchor=north] at (2.25,{\learningaxisy-.25}) {Training interval};
\draw[flow] (4.8,\learningaxisy) -- (8.7,\learningaxisy);
\node[anchor=north] at (6.75,{\learningaxisy-.25}) {Downstream input};
\node[feature] (keyyes) at (.3,\learninglegendy) {};
\responsetrait{keyyes}
\node[legendlabel] at (.7,\learninglegendy) {Flower-responsive};
\node[feature] at (5.3,\learninglegendy) {};
\node[legendlabel] at (5.7,\learninglegendy) {Not flower-responsive};
\diagramframe{\learningframebottom}
\end{tikzpicture}
{% end %}

## Readers reshape their inputs

The matched histories prescribe which pathways supply each reader. To see how reader preferences develop through training, I used reader errors to train the responses of two persistent routes and the weights each reader assigns to them. These routes begin with flower and green responses in A and zero responses in B.

Each reader learns nonnegative routing shares that sum to one, starting with equal shares. The broccoli reader combines its routed signal with a fixed texture input.[^toy-routing]

The first Price analysis counts inherited pathway uses, while this one measures allocation among persistent routes. Selection can therefore change here without creating any new pathways, simply because a reader assigns more weight to one route.

The trait is now flower contrast, the change in a route’s response when the flower cue is added while other cues stay fixed.

Selection measures changes in allocation to the initially flower-responsive route. Variation measures changes in the routes’ flower contrasts during training. Both terms come from the same training trace as the reader gradients. The decomposition can change based on route identity, signal representation, or weighting rules, even when network behavior stays the same.[^price-representation]

{{ tikz(learning=true, source="static/images/flower-toy/learned_routing.tex", learning_centers="0 9", fig_w="620px", caption="<strong>Reader pressure with learned and frozen routing.</strong> Bands show the magnitude and direction of applied learning pressure at update 10. Reader errors change flower responses and learned routing shares. Frozen shares receive no updates, so their connections have no routing bands. Feature and routing-logit bands use separate width scales, shared across conditions. The fixed texture input to broccoli is omitted.") }}

The flower reader increasingly favors A, while the broccoli reader shifts toward B. These opposing changes nearly cancel in the global selection term.

<figure>

| Reader group | Selection | Variation | Final contrast |
|---|---|---|---|
| Flower | +0.22 | +0.19 | 0.91 |
| Broccoli | −0.23 | −0.11 | 0.16 |
| Global | −0.01 | +0.04 | 0.54 |

<figcaption>

Changes after 800 training updates, with flower contrast weighted by each reader’s allocation.[^routing-table]

</figcaption>
</figure>

The global average obscures this specialization. Training the readers’ allocation weights increases broccoli classification accuracy to 100%, compared with 75% when both routes remain equally weighted.[^toy-reader-training]

Removing the initial green response before training leaves final classification unchanged but reduces mean squared reader error by ~99% after 800 updates. Restoring the green response reproduces the original trajectory.[^toy-routing] Thus, a starting point that makes one acquisition cheaper can impede joint learning when multiple readers and representations adapt together.

{{ tikz(learning=true, source="static/images/flower-toy/precursor_comparison.tex", fig_w="620px", caption="<strong>An inherited response can impede joint learning.</strong> Removing the initial green response lowers mean squared reader error by ~99% after 800 updates. Both conditions reach 100% accuracy. Earlier specialization accompanies the faster decline, but its causal role remains untested. Error is averaged equally over both readers on a logarithmic scale.") }}

Without the precursor, the readers specialize toward different routes earlier, and B develops a stronger green response.[^toy-precursor] Earlier differentiation accompanies the lower reader error, but the comparison does not isolate which allocation or response changes cause the improvement.

To test whether readers depend on these routes, I removed each route’s signal in turn.

<figure>

| Removed route | Flower accuracy | Broccoli accuracy |
|---|---|---|
| None | 100% | 100% |
| A | 50% | 100% |
| B | 100% | 75% |

<figcaption>

Each intervention removes one route’s signal after training while keeping all learned weights and routing shares fixed.[^toy-ablations]

</figcaption>
</figure>

The readers have learned to depend on different routes for perfect classification. B now supports broccoli classification despite starting without the green precursor.

Just looking at endpoint dependencies does not distinguish whether readers continued classifying correctly as routing changed.

## What learning carries forward

A distinction may remain encoded while its signal becomes too weak for an existing reader to classify correctly.

I tested this by tracking a fixed reader that averages two pathways’ flower signals. Initially, only $\mathsf{A}$ supplies the signal. Both schedules train $\mathsf{B}$ to acquire it and $\mathsf{A}$ to lose it, with identical update budgets.[^toy-continuity]

In the continuous handoff, $\mathsf{B}$ learns the flower response before $\mathsf{A}$ is trained to lose it. This overlap keeps the reader classifying correctly. Reversing the order causes temporary failure, even though the flower signal remains positive. Yet both schedules end with the same pathway parameters and reader outputs.

Endpoint Price terms also match, hiding the interruption visible in the reader’s training trace.

{{ tikz(learning=true, source="static/images/flower-toy/handoff_continuity.tex", fig_w="620px", caption="<strong>Equivalent endpoints, different continuity.</strong> A fixed reader averages flower responses <em>r</em>, using classification threshold 0.45. Pairs give outputs for flower-present then flower-absent inputs. The reader keeps classifying correctly during handoff but fails at updates 1–50 during recovery, despite a distinguishable signal. Values are rounded to three decimals. Flower markers use the 0.50 pathway criterion. Both schedules reach the same parameters and outputs, with zero net selection and variation.") }}

The same contrast appears across a range of classification thresholds.[^toy-continuity]

{{ tikz(learning=true, source="static/images/flower-toy/handoff_reader_scores.tex", fig_w="620px", caption="<strong>Reader access during the handoff.</strong> Both schedules end with identical parameters and reader outputs. The dashed line marks the fixed classification threshold of 0.45; shading marks failed classification at updates 1–50 in the interrupted schedule. Flower-absent scores remain zero throughout.") }}

What the reader is able to do along the way differs. Preserving an endpoint capability therefore does not imply continuity.

Together, these experiments distinguish surviving responses from sustained downstream use. A response can survive without remaining usable, and a useful response can pass between mechanisms.

We can draw an analogy to functional conservation in proteins. An evolutionary lineage can preserve a protein’s function even as the amino-acid sequence supplying that function changes.[^protein-conservation] Likewise, a learned capability can persist as its supporting circuitry changes.

What makes a neural network clean is the usefulness of its learned structure to downstream computations. Training targets define successful use, and errors propagated through readers shape the mechanisms that supply it. Learning can therefore conserve useful structure through changes in its implementation. The circuitry can change while the learned capability remains.

---

<p class="acknowledgements">Thank you to Toby Shorin and Cullen Miller for your support and feedback while writing this post.</p>

[^cammarata]: Cammarata, N. (2026). [*Neural Networks Are Already Clean*](https://already-clean.goodfire.com/). Goodfire.

[^protein-conservation]: Kendrew, J. C. (1962). [*Myoglobin and the Structure of Proteins*](https://www.nobelprize.org/uploads/2025/08/kendrew-lecture-2.pdf). Nobel lecture. The globins illustrate related structures and oxygen-binding functions across differences in amino-acid sequence. Continuous reader access during neural training is tested separately by the handoff experiment.

[^toy-model]: The [flower-toy model and experiments](https://github.com/hxrts/flower-toy) were constructed for this post. They use flower, green, and texture cues. Acquisition trains a green coefficient under a fixed broccoli readout, while competition trains features and readers. Downstream learning trains pathway coordinates and read gains from reader errors, but plots only green-coordinate and read-gain gradients.

    The Price illustration prescribes role targets and ancestry. These models demonstrate possible mechanisms, not measurements of Cammarata's neuron.

[^toy-gradients]: Band width encodes the absolute loss derivative, averaged over checkpoint intervals. Taper shows the sign of learning pressure $-\partial L/\partial\theta$, with a wide left end increasing the parameter and a wide right end decreasing it. Scales vary by figure and parameter type, and zero gradients have no band. Intervals, scales, and values appear in the [pathway](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/gradient_band_measurements.csv) and [reader measurements](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/reader_gradient_band_measurements.csv), with reader performance in the [training trace](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/downstream_reader_training.csv).

[^toy-targets]: Both routes obey $f^2+g^2\leq1.30^2$ and start with the same flower function. Initial green responses are 0.8 for the shared route and zero for the separate route. Acquisition requires the specified positive score and negative scores below 0.10, and step counts exclude precursor training. Both routes classify all eight binary inputs correctly at threshold 0.5. Scores are not calibrated probabilities.

[^price-learning]: Bassett, J. K., Potter, M. A., and De Jong, K. A. (2004), [*Looking Under the EA Hood with Price’s Equation*](https://www.cs.york.ac.uk/rts/docs/GECCO_2004/Conference%20proceedings/papers/3102/31020914.pdf), uses Price-based decompositions to analyze evolutionary algorithms. Fernando, C., Szathmáry, E., and Husbands, P. (2012), [*Selectionist and Evolutionary Approaches to Brain Function*](https://doi.org/10.3389/fncom.2012.00024), discusses Price accounting for neuronal groups and synaptic selection, including its limits as a description of dynamics. Frank, S. A. (2025), [*The Price Equation Reveals a Universal Force–Metric–Bias Law of Algorithmic Learning and Natural Selection*](https://stevefrank.org/reprints-pdf/25FMB.pdf), develops connections to optimization methods including SGD and Adam. *Learning to Be Clean* applies the accounting to attributed pathway uses and their downstream readers.

[^price]: Price, G. R. (1970). [*Selection and Covariance*](https://doi.org/10.1038/227520a0). *Nature*, 227, 520–521.

[^overlapping-pathways]: Cammarata, N., et al. (2021). [*Curve Circuits*](https://distill.pub/2020/circuits/curve-circuits/). *Distill*. Similar shape detectors appear across InceptionV1’s 3×3 and 5×5 branches, providing evidence of overlapping feature detection without implying literal copying.

[^gradient-conflict]: Yu, T., et al. (2020). [*Gradient Surgery for Multi-Task Learning*](https://arxiv.org/abs/2001.06782). *NeurIPS*. The study examines conflicting task gradients on shared parameters. This supports gradient interference, rather than exact cancellation between Price terms.

[^price-variation]: The second Price term is conventionally called transmission. Here, variation denotes weighted mean change in inherited traits, including response acquisition or loss. It means neither statistical variance nor necessarily random change.

[^price-accounting]: The baseline mechanisms have equal weights and binary responses $z_i\in\{0,1\}$. Measure baseline and later pathways with the same flower assay, control inputs, and responsiveness criterion. Keep the downstream tasks and ancestry rule fixed throughout. Ancestry attributes contributions to these pathways rather than neuron reproduction.

[^price-weights]: $\operatorname{Cov}(w,z)$ measures whether initially responsive mechanisms supply more descendants, and $\mathbb E$ averages over the initial mechanisms. Dividing by $\bar w$ expresses both contributions per descendant.

    Each later pathway in the example has one ancestor and one unit weight. Generally, divide each later use's unit contribution among its registered ancestors, excluding unassigned uses. Let $w_i\geq0$ be total ancestral credit and $\Delta z_i$ the credit-weighted descendant mean response minus $z_i$, set to zero when $w_i=0$. Expectations and covariance use the baseline population, with $\bar w=\mathbb E[w]>0$ and $\Delta\bar z$ the later attributed-use mean minus the baseline mean.[^price1972][^frank1997]

    One-to-one neuron tracking with equal descendant weights gives zero selection, placing all measured response change under variation.

[^toy-roles]: The construction starts with flower-responsive A and texture-supplying B, which has no flower response on the same assay. The Price illustration measures flower pressure. $\mathsf{A_1}$ and $\mathsf{A_2}$ retain flower while losing green, and $\mathsf{A_3}$ loses flower as it specializes toward green. In the capacity-constrained model, projection onto the constraint weakens flower without a direct flower-loss gradient.

[^price-readers]: For each reader group, count only its pathways' descendant credit while retaining the full baseline population. Group terms compare final inputs with that common baseline, and weighting by each group's share of descendant credit recovers the global terms. Each pathway serves one group, with two pathways per group. Both histories prescribe the same green and texture targets and match reader outputs on all eight toy inputs.

    The [matched-history accounting](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/matched_reader_inheritance.csv) uses the binary flower assay. Numerical exports call variation "transmission."

[^toy-routing]: The [routing trajectories](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_trajectories.csv), [Price terms](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_price.csv), and [reader pressures](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_pressure.csv) come from one joint-learning experiment. Route A starts with flower coefficient 1 and green coefficient 0.8, while route B starts at zero. Each reader’s softmax routing shares start at 50% per route. All route coordinates and routing logits learn from mean half-squared reader error at learning rate 0.15 for 800 updates.

    Broccoli multiplies its mixture by the fixed texture cue. These routing shares measure allocation, not ablation-based importance or neuron reproduction. The [band measurements](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_bands.csv) record gradient magnitudes and display scales at update 10, using 200 points per gradient unit for flower responses and 3000 for routing logits. Frozen logits have no applied update and therefore no band.

    For normalized baseline masses $p_i$ and later masses $q_i$, weighted Price selection is $\sum_i(q_i-p_i)z_i$ and variation is $\sum_i q_i(z'_i-z_i)$. Their sum is the change in the allocation-weighted mean flower contrast. Equivalently, descendant weights are $w_i=q_i/p_i$ with expectations under $p_i$. Exports include comparisons with initialization and with the preceding checkpoint.

    Reader groups have equal total mass. The removal condition sets only A's initial green coefficient to zero. The rescue condition restores it before training. Frozen routing keeps each share at 50%.

    All conditions use the same inputs, reader targets, update budget, and learning rate. Mean squared error averaged equally over the two readers falls from 1.97 × 10⁻² with the precursor to 2.31 × 10⁻⁴ without it. Using unrounded values, the reduction is 98.83%.

[^routing-table]: Each reader starts with equal allocation to both routes and mean flower contrast 0.50. The global row weights the reader groups equally. Rounding may prevent the displayed terms from summing exactly.

[^toy-reader-training]: A separate experiment shows that exchanging fixed reader assignments redirects learning pressure on their input pathways. The [fixed-routing experiment](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/downstream_reader_training.csv) starts both assignments from identical ancestor copies. $\mathsf{A_1}$, $\mathsf{A_2}$, and $\mathsf{A_3}$ inherit flower and green responses, while $\mathsf{B_1}$ inherits texture. Pathway coordinates and read gains learn from reader errors without local role targets, and recorded gradients include the checkpoint after ten updates. Coherent routing suppresses green in $\mathsf{A_1}$ and $\mathsf{A_2}$ and strengthens it in $\mathsf{A_3}$, while swapping readers redirects flower-reader suppression toward $\mathsf{A_3}$ and $\mathsf{B_1}$.

    Suppressing pathways on fixed positive and negative inputs tests reader dependence. Both assignments reach correct classification, though the swapped assignment retains larger response errors.

[^price-representation]: Flower contrast measures a difference in responses, so it can be negative or exceed one. The toy fixes route identities and feature coordinates throughout training. Applying the decomposition to a larger network would require a justified correspondence between mechanisms across checkpoints and a declared signal scale and attribution rule. Changing those choices can change the terms even when the computation being studied is preserved.

[^toy-precursor]: At update 400 in the [training traces](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_trajectories.csv), the flower reader assigns ~75% to A without the precursor, compared with 56% with it. The broccoli reader assigns ~75% to B without the precursor, compared with 55% with it. B’s green coefficient is 0.94 without the precursor and 0.06 with it. The corresponding [Price terms](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_price.csv) describe allocation and response changes relative to initialization without isolating which changes cause the error difference.

[^toy-ablations]: The [route-ablation measurements](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/learned_routing_ablations.csv) zero one route’s output after training, without renormalizing routing shares or retraining. They record mean squared error, maximum error, and accuracy for each reader in every training condition. Selective classification losses establish dependence under this intervention. Either route can affect the other reader’s numerical output even when its classification remains correct.

[^toy-continuity]: The [handoff trace](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/handoff_trajectories.csv) tests flower-present and flower-absent inputs after every update, with a fixed mean reader and classification threshold 0.45. Each pathway receives 40 gradient steps at rate 0.2 on a prescribed squared-error target, with acquisition and loss ordered oppositely. [Endpoint accounting](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/handoff_price.csv) uses binary responsiveness and one-to-one ancestry. The trace includes positive and negative scores and margins.

    Initially, A has response strength one and B zero. Both schedules train B toward one and A toward zero. Each pathway has one equally weighted descendant, giving zero selection. The unchanged binary responsive share gives zero net variation.

    Throughout the continuous schedule, the score for flower-present inputs stays at least 0.05 above the reader’s threshold of 0.45, while flower-absent inputs score zero in both schedules.

    [Threshold checks](https://github.com/hxrts/flower-toy/blob/b13556724b777d88d861b6d462bd3255bb684078/data/handoff_thresholds.csv) evaluate 0.001, 0.01, 0.10, 0.25, 0.45, and 0.50 without retraining. Continuous handoff preserves classification at the first five thresholds, while interrupted recovery fails temporarily at each. At 0.50, the continuous schedule reaches a zero-margin boundary, with floating-point rounding producing a final failed classification. Its minimum positive score is approximately 0.50, compared with 6.65 × 10⁻⁵ during interrupted recovery.

    Letting the reader learn during the handoff could test whether preserving classification also helps later acquisition.

[^price1972]: Price, G. R. (1972). [Extension of covariance selection mathematics](https://doi.org/10.1111/j.1469-1809.1957.tb01874.x). *Annals of Human Genetics*, 35, 485–490.

[^frank1997]: Frank, S. A. (1997). [The Price Equation, Fisher’s Fundamental Theorem, Kin Selection, and Causal Analysis](https://stevefrank.org/reprints-pdf/97Evol-Causal.pdf). *Evolution*, 51(6), 1712–1729.