← Writing

A Probe That Detects Can Still Fail to Silence

TL;DR. A linear probe direction is asked to do three jobs: detect the feature, steer by subtracting it, and silence by projecting it out. Only silencing is the necessity test. On four open-weights families (Llama-3-8B, Gemma-2-9B, Mistral-7B-v0.3, OLMo-2-7B) and two features (refusal and sycophancy), I find probes that detect at AUROC ≥ 0.91 and steer cleanly yet fail to silence. The failure is recipe-specific: difference-of-means (DoM) and cross-validated logistic regression (LR-CV) have disjoint failure sets, so no single fixed recipe is safe. The fix is to make the silencing test the selection step: a picker that fits both recipes and keeps whichever one actually silences. A calibration-time cosine turns out to be a cheap in-sample predictor of which recipe that will be.

The evidence is four families, two features, 35 whitening cells, a 22-cell cross-family battery, and a 5-cell held-out Qwen2.5-7B replication; whether it generalizes to novel architectures, other feature types, and non-linear monitors is the open question. Full paper here.

The test that is missing

Stand up a safety case on a linear probe and the standard checks pass: it classifies “about to refuse” versus “about to comply” at AUROC 0.94 on held-out data, and additive steering along the probe direction monotonically reduces refusal. Neither check tells you whether the direction your monitor watches is the axis the model actually uses to produce the behavior.

That property has its own test: Arditi-style projective ablation, projecting the direction out of the residual stream at every token, at the one layer where the monitor reads. If behavior drops, the direction is the causal axis at that layer. If it does not, the probe was reading a correlated proxy that you can detect with and push against, but that the model routes around whenever you are not pushing. The test certifies necessity at the deployed layer only.

My first paper measured one instance of this gap after fine-tuning. This one shows it is present before any further training, on the freshest probe fit to the current checkpoint. The gap comes from the geometry of the labeled buffer the probe is fit on, so fine-tuning is not required to produce it. It is cheap to screen for, and that is what the deployable check exploits.

Three jobs, one direction

Figure 1: a probe direction has three jobs; only silencing is the necessity test. (a) Detect: classify by sign of d̂·h. (b) Steer: subtract αd̂ at generation. (c) Silence on the causal axis: classes collapse, behavior vanishes. (d) Silence on a within-class confound: the causal axis is intact and behavior survives. Detect and steer cannot tell (c) from (d).

Detect and steer both pass on (c) the causal axis and (d) a within-class confound; only silencing separates them. A safety case built on “the probe detects and we can steer” is implicitly assuming the three collapse to one direction. Under identical-covariance Gaussian classes they do; in practice they often do not.

The recipe that silences depends on the model

I ran the three tests on two extraction recipes (DoM and LR-CV) at the per-family peak probe layer, across four families. Each family’s checkpoints run from the un-fine-tuned base model (round 0) through four rounds of benign UltraChat SFT, so the earliest cell in every family is the model before any further training. A cell is one (model, calibration checkpoint, evaluation checkpoint) triple, with both recipes fit as candidates inside it; the cross-family battery is 22 of them: 6 Llama-3 refusal, 6 Gemma-2 refusal, 4 Llama-3 sycophancy, 6 Mistral refusal. Every (model, feature, recipe) panel hits AUROC ≥ 0.91 on held-out data and produces a monotone dose-response under additive steering. The silencing verdicts diverge.

Figure 2: same silencing test, four models. Green = task-canonical (silences). Pink = anti-canonical (raises behavior). Check marks denote CIs that exclude zero in the canonical direction.

Throughout, Δ is the change in behavior rate when a direction is projected out, in percentage points. A working handle moves behavior in the direction the probe was fit to encode, and its 95% CI has to exclude zero on that side. For every refusal probe that means refusal drops, so a handle is negative; Llama-3’s sycophancy probe encodes anti-sycophancy, so its handle is positive (projecting it out raises sycophancy).

The two recipes’ failure sets are disjoint, so no fixed choice silences across the battery. Mistral is the cleanest existence proof: detection and steering stay healthy while no direction at any layer I searched silences refusal.

The mechanism: a within-class confound

The class-mean difference μ⁺ − μ⁻ is Bayes-optimal only when the two classes share a covariance. That fails when within-class variance carries a label-correlated surface feature v (formality, sentence length, anything that shifts between the feature-positive and feature-negative buffers at the probe layer). In that regime,

d̂DoM = α · (causal axis) + β · (confound)

and when β dominates, projecting plain DoM out removes v and little of the axis you meant to silence. LR-CV resists this because held-out regularization penalizes directions that generalize poorly across folds, which surface confounds tend to do.

The decomposition is testable: project out the top within-class PCs, refit DoM on the whitened activations, and three regimes should appear: recovery when β dominates, preservation when α dominates, a near-noop as β goes to zero.

Figure 3: whitening dose-response. Signed silencing Δ on refusal as a function of the number of within-class PCs projected out before refitting the class mean.

All four regimes are set by one quantity: within-class confound mass.

One cosine as a cheap screen

If the decomposition is right, one calibration-time scalar should separate the regimes: the cosine between plain DoM and DoM refit on whitened activations.

Figure 4: one cosine at calibration time tracks silencing success in-sample. Each point is one (refit, eval) cell; x-axis is cos(plain DoM, whitened DoM with top-1 PC removed); y-axis is the plain-DoM silencing effect, signed so a working handle is positive (refusal drop or sycophancy rise).

Across 35 cells spanning [0.12, 1.00] the cosine tracks silencing monotonically:

This is an in-sample correlate (Figure 4), not the instrument behind the battery counts below: one scalar that flags when plain DoM is a confound before you run any silencing experiment. Whether it holds out-of-sample well enough to gate a picker on its own is future work, and the deployable check does not depend on it.

The deployable check

The deployable protocol does not gate on the cosine. At each checkpoint, on a labeled buffer of feature-positive and feature-negative activations at the per-family probe layer:

  1. Fit both candidate recipes, plain DoM and LR-CV.
  2. Project each out at every token and measure the silencing effect with a 95% Newcombe CI on the behavior-rate change.
  3. Keep the recipe whose CI excludes zero in the task-canonical direction, breaking ties by effect size.
  4. If neither recipe’s CI excludes zero, the alarm fires: no linear direction at this layer silences this feature at this checkpoint.

This makes the silencing test itself the selection step. On the 22-cell four-family battery the picker recovers all 17 causal handles that exist and alarms on the remaining 5 Mistral cells, which have no handle under either recipe; fixed DoM alone recovers 10 of 22, fixed LR-CV alone 13 of 22. Read that 17/22 as a consistency check, not independent validation: the picker is graded on the same battery whose failures motivated it. The out-of-sample evidence is the held-out Qwen2.5-7B replication, 5 confound-light cells where both recipes silence and the picker routes all five to the larger-effect DoM without retuning. Its scope is narrow (every Qwen cell has a handle, so it only tests whether the picker over-alarms), but it is the one cross-family test the picker did not help design. The marginal cost is one extra probe fit and one extra projection per checkpoint.

What this means for probe-based monitors

Three practices fall out of it:

  1. Sweep layers per family. OLMo-2’s causal axis sits at L30 on a 32-layer model; an L12 to L18 convention would miss it. The peak detection layer is not constant across architectures.
  2. Try both recipes and keep the one that silences. Their failure sets are disjoint, so no fixed recipe is safe. The cosine between plain and whitened DoM is a cheap in-sample flag for when plain DoM is a confound, worth computing before you commit.
  3. Report a 95% CI on the silencing Δ. A CI that includes zero is an alarm to act on, and tuning until it clears defeats the purpose. Detection and steering passing does not substitute for that check; it is the whole point of the Mistral result.

Two papers, one claim

The first paper found probes failing necessity after fine-tuning; this one finds it from the start, on the freshest probe. Both failures are invisible to detection-and-steering checks, so the same rule holds: probe accuracy is not evidence of a causal handle, and deploying model-internals probes as safety interventions requires a necessity test on top of the detection score.

Limits and loose ends

The battery is refusal on four families plus sycophancy on Llama-3, 35 whitening cells in total: enough to establish the cosine signature as monotone across [0.12, 1.00], but too small to predict which regime a novel family falls into from pretraining metadata alone. Projective ablation is a necessity test with one caveat: self-repair. Later layers can partly reconstruct a feature the projection removed at the monitored layer, so a null there does not prove the feature is gone from the whole network. The operational claim is therefore about the deployed layer, where projecting the probe out does not change behavior and a monitor fails regardless of other pathways. The setup is linear throughout.

Two threads I would pull next. Name the confound: on Mistral at L14 the top within-class PC carries enough mass that plain DoM is random, and my best guess is the AdvBench-versus-Alpaca register gap (imperative and stripped-down versus conversational), but that is untested. And go beyond refusal: whether the decomposition extends to deception-detection probes, capability-elicitation probes, or SAE-feature stability is what separates a refusal-specific geometric fact from a general failure mode of linear probe-based monitors.

Paper. Code and fitted probe directions.