Fine-Tuning Silently Breaks AI Safety Monitors
TL;DR. I tracked a linear shortcut-detection probe across five rounds of rejection-sampling SFT on Qwen2.5-Coder-7B (a shortcut here is a code solution that passes the visible tests but fails hidden edge cases). Within the first SFT round the probe direction rotates 40 to 70 degrees away from where it started. Standard evaluation still looks healthy: a fresh probe refit at each checkpoint recovers AUROC at or above 0.92, and additive steering along the old direction still moves behavior. The robust evidence for that steering effect is the round-4 α-sweep, where the reduction grows monotonically from −5.6% at α=2 to −17.1% at α=10; the single-α per-round trajectory is noisier. The Arditi et al. projective-ablation test disagrees with the steering result: the frozen direction is sufficient to push behavior but no longer necessary to represent it.
I am confident the phenomenon exists and is present from the first SFT round; the adversarial version, gradient pressure against a frozen probe, is the important next experiment. Full paper here.
What a deployed monitor would show you
After fine-tuning, a safety team re-runs its checks on the probe-based monitor. Two of them look reassuring:
- Probe AUROC: 0.89, down from 0.98. Noticeable, not alarming.
- Steering test: adding −α·d̂ to the residual stream still reduces the behavior.
On that evidence the monitor ships. What the checks cannot see is that the direction being watched is no longer the axis the model uses to represent the behavior.
The same effect has surfaced outside controlled experiments. Anthropic’s Claude Mythos Preview system card flagged it qualitatively during alignment training:
“The causal effects of individual features often changed over the course of post-training, making it difficult to attribute behavioral changes merely to increases or decreases in particular feature activations.” (Mythos Preview System Card, §4.5.3.4)
Mythos reports it as an operational difficulty inside a production run. What follows is the controlled version: one probe, one trajectory, three interventions, quantified.
The one figure
Sufficiency says yes and necessity says no. Additive steering reports a large effect (−17.1% at α=10, a single n=960 measurement whose bootstrap interval [−25.8%, −7.4%] excludes zero). The Arditi ablation test, projecting d̂ out at every layer-11 token, reports essentially nothing (−0.9%, interval crosses zero).
This is the sufficiency-necessity gap: a frozen direction you can still push the model’s behavior along, even though it is no longer the axis the model uses to represent that behavior. A high-norm push along a correlated direction crosses the decision boundary without pointing at it.
Why the direction moves
The readable direction rotates within one SFT round. Across three extraction methods (LogisticRegressionCV, deterministic LR at C=1, and difference-of-means), the cosine between the base direction and every post-base direction is well below 1.0 by round 0, the first SFT round:
Meanwhile a fresh probe refit at each checkpoint recovers AUROC at or above 0.92. The feature has not decayed; if anything it has become more separable, and its coordinates have simply moved.
In 3584 dimensions, the best classifier and the actual concept direction are far from aligned. On step-0 activations, cos(LR, DoM) ≈ 0.26 with the standard scaler: far above the ~0.017 you would expect from two random directions, far below the 1.0 of identity. The LR direction carries enough norm at α=10 to cross the decision boundary by brute force even though it does not point along the axis the model uses, which is why projecting it out removes almost nothing.
A methodological aside. The initial round-4, α=5 measurement returned +0.7% (CI [−9.2, +11.8]), which almost anchored a “dose dead zone” narrative. Roughly $10 of A100 compute later, the re-measurement returned −9.8% (CI [−18.4, −0.2]). The entire framing of the paper turned on one re-run. At n=960 with single-digit effects, single-point measurements at the edge of their CI can support qualitatively different stories than the truth.
Difference-of-means recovers the causal handle, but only when refit
Running the ablation battery across all three extraction methods and both frozen (fit at base) and fresh (refit at round 4) directions:
Difference-of-means is the only method that shows necessity anywhere, but the frozen version does not clear the bar at round 4: −7.2% with a bootstrap interval of [−16.6%, +3.3%] that still crosses zero (92% of resamples come out negative). Its one frozen checkpoint that excludes zero is round 2 (Figure 4). Both LR variants stay null under ablation whether frozen or refit, including a fresh CV-LR direction that scores AUROC 0.96 held-out (median 1.0 across leave-one-problem-out folds). Detection can stay perfect after necessity is gone.
Refit at round 4, difference-of-means is the exception: the fresh DoM direction produces a −36.5% shortcut reduction under ablation, larger than any additive steering effect at any checkpoint. The feature has reorganized into a sharper, more causally concentrated subspace, and you only see it if you use difference-of-means and refit at the current checkpoint.
Difference-of-means is what Arditi et al. and RepE already default to on stability grounds, and among the methods I tested it comes closest to holding causal faithfulness under training pressure.
The gap is there from the first round
I re-ran the frozen-direction ablation at rounds 0 and 2 to locate where in training the necessity loss kicks in. It is there immediately.
Both LR directions stay null across rounds 0, 2, and 4, and DoM’s interval excludes zero only at round 2 (−10.5%); even DoM is null at round 0. The frozen CV-LR direction is magnitude-only from the first SFT round, so the necessity gap is present after even a single round of fine-tuning.
What I would change in a deployed monitor
Three things, concretely:
- Refit the probe whenever retraining occurs. Reusing a direction across retraining is the core bug. If you must freeze, difference-of-means held necessity longest in my runs, but even it was significant at only one checkpoint (round 2) and null by round 4, so freezing is a stopgap at best.
- Run the projective-ablation test alongside any steering-based validation. Additive steering alone is a sufficiency result. A monotonic dose-response is consistent with the direction being the causal axis, or with its being a correlated proxy whose large-α push happens to cross a decision boundary. Ablation is what separates the two.
- Treat AUROC as a detection check, not a causal-faithfulness check. The fresh CV-LR probe scored AUROC 0.96 held-out (median 1.0 under leave-one-problem-out) yet was causally disconnected from the feature under ablation. A separating hyperplane can exist without the model sitting on the right side of it for the right reason.
Update, July 2026: the same study under RL
I reran the whole study with GRPO in place of rejection-sampling SFT and changed nothing else: same Qwen2.5-Coder-7B, same LoRA config, same visible-test reward signal, same probe layer, and the identical probe, steering, ablation, and control battery. RL is what frontier labs actually run, so this is the version that matters for deployed monitors. Everything reproduces from committed artifacts in the repo via one comparison script.
One deviation, disclosed because the comparison rests on it. The binary pass/fail reward saturates under GRPO: the base model already passes the visible tests on essentially every sample, so within-group advantages are zero and nothing trains (reward standard deviation 0, gradient norm 0, measured directly). The fix rewards the fraction of visible tests passed, which restores within-group variance while leaving the reward-hacking structure untouched. Hidden tests stay excluded from the reward in both modes, and the fraction reward is denser than a production RLHF reward model provides, so drift here may be faster than a production run would show.
Three results:
- The rotation happens faster and lands at the same floor. Under SFT the frozen probe’s cosine to the freshly refit direction decayed gradually, 0.78 after round 0 and about 0.4 by round 2. Under GRPO it reaches 0.435 within the first 50 optimizer steps and stays near 0.44 to 0.50 for the rest of training. A split-half control puts this metric’s stability floor at roughly 0.4 (two refits with nothing but noise between them agree that much), so within 50 steps the fresh direction retains no measurable relation to the frozen one beyond refit noise.
- Detection reads healthier under RL. The frozen probe’s AUROC sits at 0.955 to 0.960 at every GRPO checkpoint. Under SFT it had at least sagged to 0.89, enough that a careful team might investigate. The monitor reports closer to full health while its geometry moved just as far.
- Refitting stops working. The one intervention that recovered a real causal handle under SFT, refitting difference-of-means on fresh data at round 4 (−15.9 points of a 44% base under ablation, a 36.5% relative reduction), recovers essentially nothing under RL: about 2.3 points of a 34% base, a 6.7% relative reduction. At matched power (base rates 30 to 36%, n=800 per cell) no frozen direction reaches significance at any checkpoint of either run.
The steering result needs the same caveat as before, and a stronger one: steering at α=5 zeroes the shortcut rate at every GRPO checkpoint, but norm-matched random directions at α=5 also zero it. At that magnitude any direction collapses the behavior, so the steering result says nothing about whether the probe direction is special.
I should disclose one measurement bug, because it almost shipped. My first necessity pass under GRPO used a harness whose problem-selection fallback picked arbitrary problems instead of shortcut-prone ones, so it measured the behavior where the behavior barely occurs (1 to 3% base rates), and the resulting “no necessity anywhere” claim was unpowered by construction. An adversarial pass over my own numbers caught it; the fix and the rerun at matched power are committed alongside the deflated originals.
Where this leaves a deployed monitor: a stack watching detection AUROC and steering response would certify this probe as fully healthy at every checkpoint of either run, and the refit-on-fresh-data fix that worked under SFT does not survive RL. If a safety case says the probe reads the feature the policy uses, RL training is where that claim needs the most scrutiny.
What’s next, and what this does not show
This is one model, one probe-derived feature (shortcut versus general code), one training recipe (iterative rejection-sampling SFT on CodeContests). Shortcut-versus-general is capability-entangled in a way that refusal or emotion directions may not be, so I would not over-generalize the specific magnitudes.
The experiment I most want to run next is the adversarial one: a training loss that directly rewards lowering the probe’s score. My prediction is that under that pressure the AUROC decay happens faster and the sufficiency-necessity dissociation appears earlier and larger. That is the version that would turn this existence proof into a quantitative monitor-robustness benchmark. Two smaller questions stay open: whether difference-of-means keeps its necessity under that same adversarial pressure, and whether there is a cheap subspace-tracking analog to full probe retraining, something that follows the feature between checkpoints instead of rediscovering it each time.