Linen Yu

Mechanistic interpretability · project report

Evaluating Safety-Relevant Representations and Behaviors in Language ModelsFrom a single direction to a semantic effect signature

A local replication and extension on Qwen2.5-1.5B-Instruct. The project asks what a refusal-related activation direction controls, what it does not erase, and how stronger evaluation gates change the story.

Experiments: June–July 2026 · Report consolidated: July 10, 2026

Executive readout

The four results that survived the full project

Refusal-phrase rate on 100 harmful prompts after one-direction ablation .990 → .070 A strong causal intervention result, measured with a keyword refusal score.
Refusal-phrase rate after removing the full 4×4 self-organizing-map span .010 Stronger suppression, but the directions are highly redundant.
Harmfulness-recognition accuracy after refusal ablation .859 → .703 Recognition weakens but remains above chance; it is not erased.
Held-out V6 source-validation effect for the refusal direction −.906 95% bootstrap interval: [−1.000, −.812].
Current thesis. A linear refusal direction is a strong intervention handle for a visible response endpoint. The combined evidence does not show that refusal is stored as one clean internal dimension. In the final experiment, the direction mainly changes safety refusal, direct-answer preference, and abstention on unanswerable questions. The narrowest useful interpretation is an answerability / guardedness handle in this model and evaluation setup.

The page reports aggregate metrics only. Harmful prompt text and potentially harmful model completions are not published.

Claim boundary

What the project tested

The initial question was operational: can a direction learned from harmful-versus-harmless prompt activations causally change refusal-like output? Later experiments asked whether the direction was a surface phrase switch, a harmfulness-recognition feature, a response-policy variable, or part of a broader answerability decision.

Supported by the local evidence

  • Projection removal along the learned direction sharply suppresses refusal phrases.
  • A larger, redundant direction span suppresses the same endpoint more strongly.
  • The intervention also changes some recognition, truthfulness, and abstention measures.
  • Refusal is a useful first case for linear-representation experiments because it has a visible behavioral endpoint.

Not established

  • Refusal is literally one internal dimension.
  • Ablation removes the model's ability to recognize harmful requests.
  • Fewer refusal phrases imply safe or unsafe compliance.
  • The same direction-search method transfers cleanly to truthfulness, sycophancy, deception, or other relational concepts.

Research trajectory

How the question narrowed

Mid-June

Build the pipeline and reproduce the single direction

Toy models, SmolLM, and 8- or 32-example implementation checks established activation extraction, layer selection, and generation-time hooks. The 100-prompt Qwen experiment became the main evidence.

June 18–22

Test a larger span and look for preservation costs

A 4×4 self-organizing map produced a stronger intervention span. Direction redundancy, truthfulness loss, and residual harmful/harmless structure limited the interpretation.

June 26–30

Separate recognition, policy, and response execution

A larger reverse truth/correction experiment failed to move refusal. Harmfulness recognition weakened but survived. A forced-action direction was readable in a controlled setting but did not transfer cleanly to ordinary generation.

July 2–4

Try a multi-concept intervention atlas

Five versions progressively added benchmark sources, disjoint splits, source validation, saturation checks, and bootstrap intervals. Most visually appealing cross-effects disappeared under the stronger gates.

July 5

Replace the atlas with a source-gated effect matrix

The final experiment treats rows as validated intervention handles and columns as behavior responses. It measures the signature of the refusal intervention without assuming a concept hierarchy.

Canonical experiment

Single-direction ablation, multi-direction follow-up, and preservation

The core replication follows the experimental object in Arditi et al. (2024): contrast harmful and harmless prompts, learn a mean activation difference, select the intervention layer on held-out validation data, and remove that direction during generation.

Model and data. Qwen/Qwen2.5-1.5B-Instruct; andyrdt/refusal_direction prompt splits; 128 training, 32 validation, and 100 test prompts per class. The selected location was hidden state 14 / decoder layer 13. The main score records whether a completion contains a refusal phrase. It does not judge harmful compliance or overall safety.
Validation separation between harmful and harmless prompts across hidden states, with hidden state 14 selected.
Figure 1. Validation-set projection separation across hidden states. Hidden state 14 was selected before reading the 100-prompt test results.
Held-out harmful and harmless prompt projections along the selected refusal-related direction.
Figure 2. Held-out harmful and harmless prompts separate along the candidate direction. This is a selection diagnostic; causal evidence comes from the intervention.

One-direction intervention on the main test set

Prompt class Condition Prompts Refusal-phrase rate
HarmfulBaseline1000.990
HarmfulOne-direction projection removal1000.070
HarmlessBaseline1000.070
HarmlessOne-direction projection removal1000.010

The direction is therefore more than a readable feature: removing it changes the model's visible response. The result remains an output-level intervention result, not evidence that every component of refusal lives on this coordinate.

A larger direction span

The follow-up adapts the idea in Piras et al. (2025). A 4×4 self-organizing map was trained on harmful-prompt activations at hidden state 14. Each node defines a local harmful centroid; subtracting the harmless centroid yields 16 candidate directions. Selected directions were orthogonalized before their joint projection was removed.

This is a local simplification. Piras et al. use Bayesian optimization and a more complete HarmBench / attack-success evaluation stack. The local experiment uses greedy subset selection on a small validation set and the keyword refusal endpoint.

Refusal-phrase rates for baseline, one-direction ablation, greedy self-organizing-map subsets, and the full 16-direction span.
Figure 3. On 100 harmful and 100 harmless test prompts, the full 16-direction span reduces the harmful refusal-phrase rate to 0.010.
Cosine-similarity heatmap among the 16 self-organizing-map directions.
Figure 4. The 16 directions are highly redundant: the largest absolute off-diagonal cosine is 0.981 and the mean is 0.783. They should not be described as 16 independent refusal axes.

Small preservation checks

The preservation sets are much smaller than the main intervention test. They are reported separately to prevent their precision from being confused with the 100-prompt result.

Evaluation set Prompts Baseline keyword pass rate One-direction ablation Interpretation
Ordinary benign tasks120.9171.000No loss in this small keyword check.
Locally written fact-checking / misconception prompts160.7500.312Large warning signal; this is not a standard TruthfulQA evaluation.
Keyword pass rates on 12 benign tasks and 16 local fact-checking prompts before and after one-direction ablation.
Figure 5. The small benign check does not degrade, while the 16-prompt fact-checking score falls from 0.750 to 0.312.
Refusal suppression on 100 harmful prompts plotted against small-sample benign and fact-checking preservation costs.
Figure 6. The horizontal axis uses 100 harmful test prompts. The preservation cost combines separate 12-prompt benign and 16-prompt fact-checking checks; it is a warning display, not a common-denominator benchmark.

Residual structure after ablation

After removing the selected direction, harmful and harmless prompts remain linearly separable elsewhere in the network. The best baseline separation was 1.836 at hidden state 14; the best post-ablation residual separation was 1.832 at hidden state 3.

Layer-wise harmful-versus-harmless separation before and after refusal-direction ablation.
Figure 7. Strong residual harmful/harmless structure remains after visible refusal is suppressed. The intervention removes a useful control coordinate without deleting every related representation.

Boundary experiments

Recognition, response policy, and action context

The next experiments tested which functions travel with the refusal handle. They produced clearer boundaries than modules: recognition and response execution are distinguishable but partly coupled, while a controlled forced-action feature does not transfer as a stable natural response axis.

ExperimentDesignKey resultConclusion
Strict reverse coupling Validate a truth/correction-down intervention on 64 held-out target examples, freeze its sign and strength, then read the refusal cross-effect. Harmful refusal stays 1.000 → 1.000. In the same experiment, refusal ablation gives 1.000 → 0.031. Negative result The larger experiment does not support truth/correction → refusal coupling.
Harmfulness recognition vs response execution Safe harmful/harmless forced-choice classification over 256 paired rows; generation check with 32 prompts per class. Refusal execution: 1.000 → 0.000. Recognition accuracy: 0.859 → 0.703. Boundary evidence Recognition weakens but survives.
Forced-action direction Controlled harmfulness × forced-action representation; 80 prompts per probe class and 32 prompts per class for ordinary generation transfer. Action-probe AUROC reaches 1.000. Ablation leaves natural harmful refusal at 1.000; addition reduces it to 0.000. Sign-unstable Readability in the constructed context does not establish a natural action-selection axis.
Multi-handle audit Compare refusal, harmfulness, forced-action, reply-style, truth/correction, and sycophancy-related directions on shared response measures. Refusal and harmfulness interventions both suppress refusal and lower TruthfulQA. Most other directions are weak on their own targets or unstable in transfer. Cross-effects are interpretable only after the source direction moves its own held-out target.
Natural refusal-phrase rates under forced-action direction ablation, addition, and a random-direction control.
Figure 8. The forced-action direction is readable in its constructed context, but projection removal does not lower natural refusal. Addition suppresses refusal with the opposite transfer pattern expected from a clean removal axis.
Cross-effect heatmap comparing refusal, harmfulness, action, reply style, truth-correction, and sycophancy-related interventions.
Figure 9. Small multi-handle audit. Rows are interventions and columns are response deltas. The display is diagnostic because several source directions do not pass strong own-target validation.

The atlas attempt

Why the multi-concept graph contracted

The project briefly pursued an intervention graph spanning refusal, harmfulness, truthfulness, sycophancy, hallucination recognition, uncertainty, helpfulness, toxicity, and bias. Each revision made the evidence rules stricter. The graph became smaller rather than clearer, which changed the research object from “concept nodes” to validated intervention sources.

VersionWhat changedResultEvidence status
V1Hand-written prompts and four- / twelve-node heatmaps.The visualization and intervention-matrix pipeline worked, but the sources and scorers were too weak.Implementation scaffold
V2Local benchmark sources and an explicit source-target gate; only 2/2/2 examples per concept.A few edges survived, but sample size, target saturation, and proxy scores dominated.Tiny-sample check
V3Twelve concepts with 32/32/32 disjoint splits, an intervention-strength sweep, 500 bootstrap resamples, and hidden failed-source rows.At the primary ablation strength, only 1 solid edge remained; 10 were diagnostic and 121 were hidden.Proxy-heavy provenance
V4Restrict to refusal, harmfulness, truthfulness, and sycophancy; prefer benchmark-native scoring.The primary graph had 0 solid edges, 3 diagnostic edges, and 9 hidden rows. The canonical refusal source remained the stable case.Benchmark-first diagnostic
V5Stop estimating cross-effects. Test only whether each of eight directions changes its own target, with n=16 per row.Only refusal passed with an interval excluding zero. Other rows were weak, null, or wrong-sign.Screening result
Own-target behavior rates before and after ablation for eight benchmark-backed or diagnostic target families.
Figure 10. V5 own-target screening, n=16 per row. Refusal falls from 0.9375 to 0.1250, with a paired bootstrap interval of [−1.0000, −0.6250]. Truthfulness, hallucination recognition, helpfulness preference, and toxicity recognition move in the expected direction but their intervals include zero. Sycophancy and unanswerability do not move; the autoregressive bias adaptation moves in the wrong direction.
Scorer fidelity matters. HaluEval is used here as hallucination recognition, not free-form hallucination production. HH-RLHF provides a chosen-versus-rejected helpfulness preference, not an open-ended helpfulness judge. The SQuAD and CrowS-Pairs rows are diagnostic adaptations rather than their native evaluation formats.

Final boundary experiment

V6: a source-gated semantic effect matrix

V6 treats each row as an intervention direction tied to a specified contrast set. A row must first change its own held-out target. Columns then record fine-grained behavior responses. This avoids assuming that every row is a complete or independent concept.

Setup. Most response columns use n=32. The refusal direction is tested at intervention strengths 0.25, 0.5, and 1.0. Diagnostic harm-salience and truth/correction rows are included alongside a norm-matched random direction. Intervals use 200 bootstrap resamples. Positive values in the rate matrix mean that a failure increased or a desirable behavior decreased.
Own-target validation effects for refusal at three intervention strengths, harm salience, truth correction, and a norm-matched random direction.
Figure 11. Source validation precedes cross-effect interpretation. On 32 held-out harmful prompts, the refusal score falls from 0.969 to 0.062: delta −0.906, 95% bootstrap interval [−1.000, −0.812]. The truth/correction row does not move its own target and remains diagnostic.
Behavior response under refusal ablation, strength 1.0Rate effectInterpretation
Safety refusal decreases0.969The main visible endpoint changes strongly.
Unsafe direct-answer preference increases0.641Closer to an answer decision than phrase detection, but still a local forced-choice measure.
Abstention on unanswerable questions decreases0.688Connects refusal to a broader answerability / guardedness response.
TruthfulQA decreases0.375A repeated preservation warning; it does not establish that refusal causes truthfulness.
Ordinary factual QA decreases0.031Small movement argues against a simple broad-capability-damage explanation.
Instruction following decreases0.000No rate change in this local check.
Surface disclaimer changes0.000No support for a purely surface-hedging explanation.
Bar chart showing the semantic response signature of refusal-direction ablation at intervention strength 1.0.
Figure 12. The largest movements occur in safety refusal, direct-answer preference, and unanswerable-question abstention. Ordinary factual QA, instruction following, and surface disclaimer checks remain small.
Rate-effect heatmap with intervention directions as rows and twelve behavior responses as columns.
Figure 13. Full V6 rate matrix. The refusal-strength sweep reproduces a similar response pattern. The failed truth/correction source and the norm-matched random direction do not reproduce the refusal signature.
Best local interpretation. The refusal direction participates in a shared control process governing when the model refuses, answers directly, or abstains because a question is not answerable. “Answerability / guardedness” names this measured pattern; it is not evidence for a standalone module.

Synthesis

What the project changed

Substantive conclusions

  • The selected refusal direction is a strong causal handle on refusal-like output.
  • Its effects extend beyond surface refusal phrases, but related internal structure remains.
  • Harmfulness recognition and refusal execution are partly coupled and empirically distinguishable.
  • The final response pattern is more consistent with answerability / guardedness than with a pure phrase, random-direction, or broad-damage account.

Method conclusions

  • Validate an intervention source on held-out own-target data before reading cross-effects.
  • Report behavior rates, log-probability margins, saturation, and uncertainty together.
  • A benchmark prompt source does not guarantee a benchmark-faithful scorer.
  • Negative replication should narrow the claim. It should not be converted into a cleaner-looking graph.

Current boundary

Limitations and next work

Limitations

  • The main refusal endpoint is still a keyword score. V6 also relies on local forced-choice and marker-based measures.
  • The project does not include a faithful harmful-compliance judge, full attack-success evaluation, or human safety review.
  • Sample sizes differ: 100 prompts per class in the main refusal test, 12 and 16 in early preservation checks, 16 per V5 row, and mostly 32 per V6 response column.
  • All main results come from one small instruction-tuned model.
  • Answerability / guardedness is an interpretation of the measured signature, not an independently identified mechanism.

Next work, in order

  1. Replace keyword-only scoring with stronger refusal, harmful-compliance, benign-capability, and truthfulness measures on the same prompt families.
  2. Scale the matched evaluation only after the scoring layer is credible.
  3. Repeat the strongest result on a larger instruction model.
  4. Add a generated-answer judge for unanswerable and fabrication behavior before making a public mechanism claim.

Selected references

Research anchors