--- name: vibe-abliteration description: "Run Heretic directional abliteration to reduce a model's refusals: set up, optimize, read the Pareto front, verify the export, then save or upload with a proper model card." category: engineering risk: critical source: self source_type: self license: MIT date_added: "2026-11-16" author: instax-dutta tags: [abliteration, heretic, model-editing, refusal-removal, optuna, model-cards] --- # Vibe Abliteration ## Overview Automate directional ablation with [Heretic](https://github.com/p-e-w/heretic), a tool that edits a model's weights to reduce refusals while measuring the cost in KL divergence from the base model. The hard part is not running the tool. It is choosing a point on the refusals-versus-drift trade-off deliberately, verifying the exported model rather than trusting the trial numbers, and documenting what was done so the result is reproducible. This skill covers setup, the parameter that actually matter, reading the Pareto front, headless flags, and verification before publishing. It edits model weights; it does not remove a licence or a terms-of-use obligation. ## When to Use - The user asks to abliterate, decensor, uncensor or remove refusals from a model. - The user wants to reproduce or improve an abliteration result and needs the refusals/KL trade-off read correctly. - The user wants to publish an abliterated model with a model card that states the parameters and the measured metrics. Before running, check whether a suitable converted model already exists; a fresh optimisation run costs GPU hours for a result someone may have published already. A directory of published Heretic conversions, including quantised formats, is at (maintained by the same author as this skill - disclose that affiliation). Its numbers are publisher-reported, not re-measured, and rows with published metrics are more trustworthy than inherited ones. ## Prerequisites A GPU with enough VRAM for the model (4-8 GB minimum for small models), Python 3.10+, and PyTorch installed for the correct CUDA version. A Hugging Face token only when the base model is gated or the result will be uploaded. Disk for checkpoints, which can be large. Installation order matters: install PyTorch first so the tool does not pull a CPU-only build, then install `heretic-llm`, which pins its own dependencies. Before spending GPU time, confirm the architecture is supported. Heretic locates layers at `model.model.language_model.layers` or `model.model.layers`, then modifies `attn.o_proj` and `mlp.down_proj`-style projections. MoE experts, hybrid linear attention and some conv blocks map onto those names. Two signatures mean "unsupported, not a bug": a missing `.layers` attribute, and `No modifiable modules found in layer`. ## How It Works ### 1. Install and authenticate Create a virtual environment, install PyTorch for the target CUDA version, then install `heretic-llm`. Log in to Hugging Face only if the model is gated or an upload is planned; keep the token in the environment, never in a config file. ### 2. Run the optimization The model id is the main argument. The levers that matter: - `--n-trials` and `--n-startup-trials`: search budget, not a quality dial on its own. - `--quantization bnb_4bit`: VRAM relief for larger models. - `--seed`: controls startup exploration and the guided search path. - `--study-checkpoint-dir`: keeps the journal so a run is not lost. Settings that are lists (scorers, modifiers, benchmarks) cannot be passed as flags. Put them in `config.toml` in the working directory. ### 3. Read the trade-off correctly Each trial applies an ablation profile and measures two objectives: refusal count on harmful prompts and KL divergence from the base model. Keep the Pareto front, not a single winner. - Attention carries most of the refusal reduction; an MLP weight of zero is normal and often optimal, not a failed trial. - A peak past the midpoint of the stack matches the published literature. - Untouched layers near the ends are the distance cutoff working as designed. - For most models the useful point sits just under 10 refusals per 100 at the lowest reachable KL. Chasing 0/100 usually costs capability. Gains flatten after roughly 50-150 trials per seed for sub-1B models. After that, a new seed or different prompt datasets beats more trials. If the entire front sits at high refusals for low KL, the refusal direction does not fit the model and more trials will not fix it. ### 4. Handle headless runs Optimization ends with interactive prompts. In a non-TTY run a missing answer raises `EOFError`; that is a missing flag, not a bug. Set the trial index as a position in the sorted Pareto front (not the printed trial label), the model action (`save` or `upload`), the save directory or repo id, and the export strategy (`merge` or `adapter`). ### 5. Verify before publishing Refusal counts are keyword-matched and the export is a different artifact from the trial. Load the saved model and check that prompts the base model refused now answer without degenerate output, that normal prompts keep their formatting and reasoning, and, when capability matters, that one benchmark holds up. KL alone is not a capability test. ### 6. Document Record the base model, tool version, trial index and parameters, KL divergence and refusal score. Save locally by default. Upload only after the user explicitly authorizes publication of that artifact to the named repository and visibility. Include the model card and publish only if the base model terms permit redistribution. ## Examples ```text User request: "Abliterate this 1.7B instruct model, keep quality high." Inputs: base model id, one 24 GB GPU, 200 trial budget, upload planned. Expected result (illustrative): Ran 200 trials at seed 42; front showed KL 0.0068-0.19 for 0-3 refusals per 100 against a 4/100 baseline. Chose the knee: KL 0.0083, 2/100 refusals, capability spot-check clean. Saved merged weights, wrote a model card with the trial parameters, and verified three previously refused prompts before upload. ``` ## Limitations - Refusal counts are keyword-matched proxies. A lower count does not prove the model is more useful, and a higher one does not prove it is safer. - KL divergence measures drift from the base model, not capability loss on a task. - Some architectures are unsupported and return a layer error rather than a workaround. - Headless runs cannot extend a finished study; plan the trial count up front or start a new seed in a new directory. - The trial-evaluation helper script lives in the upstream repository and is not bundled here. ## Safety and side effects This skill removes refusal behaviour from a model. It writes checkpoints to disk, can download tens of gigabytes, spends GPU time, and can publish a model publicly under the user's account. Upload only on explicit instruction, mark the model card accurately, and respect the base model's licence and terms. Irreversible weight edits should be done on a copy, never on an original checkpoint. Do not use the result to build something that breaks the law or a platform's terms; state that boundary once, plainly, then continue. ## Sources and related skills - Heretic: - The ablation method (Arditi et al., 2024): - Upstream skill repository: - For the training-side alternative, use `finetune-llm`; for measuring whether a model actually changed, use an evaluation skill.