- Safety Pencil: locating, forecasting and repairing multilingual safety drift in Tiny Aya Global
- Tracks and problem statements
- Abstract
- Headline: before → after
- Base model and method in one line
- For judges: one minute
- How to read this write-up
- 1. Introduction
- 2. Related work
- 3. Theory: the method, derived
- 3.1 Setup and notation
- 3.2 Drift and its curvature
- 3.3 Task cost
- 3.4 The pencil
- 3.5 Locate, forecast, repair
- 3.6 Making it computable: K-FAC
- 3.7 Damping, and a free validity check
- 3.8 Exact invariance under the V/O gauge
- 3.9 When the closed form misbehaves: oblique projections
- 3.10 The bounded repair
- 3.11 From theory to testable predictions
- 4. Experimental setup
- 5. Results
- 5.1 What the benign fine-tune did
- 5.2 Invariance (P1)
- 5.3 Locate (P5)
- 5.4 Forecast (P2), and the drift law
- 5.5 The unbounded repair breaks the model
- 5.6 The bounded repair: valid at every μ, but weak
- 5.7 Against baselines on dev
- 5.8 Same model, before vs after (test split, touched once)
- 5.9 The P6 verdict
- 5.10 Scorecard against the pre-registration
- 5.11 What didn't work, and what we did not report
- 5.12 Examples
- 6. Analysis: where theory and data part
- 6.1 Why the forecast fails: the update is outside the quadratic regime
- 6.2 Why the repair exploded: an oblique projector, certified by the ceiling
- 6.3 Why the bounded repair is weak
- 6.4 Why simpler curvature models win
- 6.5 KL drift and judged behaviour disagree
- 6.6 What we learned
- 6.7 What this means if you ship a Hindi fine-tune
- 7. Limitations
- 8. With more credits
- 9. Conclusion
- 10. Cost and compute
- 11. Artifacts
- 12. Documentation and reproduction
- 13. Declaration of work done outside the hackathon
- 14. Team
- 15. Acknowledgements
- 16. References
- 17. LLM usage statement
- 18. Changelog
- Appendix A: refusal judge rubric (summary)
- Appendix B: deviations from the pre-registration that affect results
- Appendix C: data filtering and run log
- Tracks and problem statements
Safety Pencil write-up (Lexsi AI Safety and Alignment Hackathon, Track 2b). Code: GitHub · PDF version (the maths and figures render best there) · Repaired model:
tkaunlaky10/safety-pencil-runA-wiseft-0.5. This repo holds only the write-up.
Safety Pencil: locating, forecasting and repairing multilingual safety drift in Tiny Aya Global
One-line claim. A closed-form, gauge-invariant curvature repair undoes the English safety damage of a benign Hindi fine-tune of Tiny Aya Global with no retraining and no task loss. Plain weight interpolation does as well, and we show mathematically and empirically why.
PDF version:
Safety_Pencil_WRITEUP.pdf. The equations and figures render reliably there; GitHub's Markdown renderer garbles some of the maths.Content note. This work evaluates models on harmful prompts (aya_redteaming, in English, Hindi and Hinglish). No harmful completions appear in this write-up or in any released artifact: we store only judge labels and SHA-256 hashes of completions.
Tracks and problem statements
- Track 2b: AI Safety and Correction (primary)
- #6 Repair safety drift without retraining (main contribution)
- #9 Locate where safety lives in the model (leverage map)
- #5 Safety-preserving fine-tuning (planned as P4, cut; reported as not done)
- Track 2a: #4 Multilingual safety (used as a lens: every measurement is per language)
Abstract
Fine-tuning an aligned model on harmless data can still damage its safety. We study this on Tiny Aya Global, fine-tuned with LoRA on 4,000 benign Hindi instructions. Task loss fell from 2.14 to 1.78 nats per token, but on held-out prompts English harmful compliance rose from 6.6% to 22.5%, English over-refusal rose from 13.5% to 55.0%, and refusal rose in every language. The model did not stop refusing; it lost the ability to tell harmful from harmless requests.
We ask whether this damage can be located, forecast from the weight update alone, and repaired in closed form. Our tool is one generalised eigenproblem, F_h v = λ H_t v, between the Fisher of the safety drift at the aligned model and the task Fisher at the fine-tuned model. Each eigenvalue is safety damage per unit of task cost. The largest eigenvalues locate safety, ½Σλc² forecasts drift, and shrinking each coordinate of the update by μ/(λ+μ) exactly minimises drift plus μ times task regret, a Fisher-weighted merge of the two models. We make this computable with K-FAC, prove it exactly invariant under the value/output gauge of grouped-query attention, and pre-registered seven predictions before measuring.
The mathematics holds; its fit to the real model is partial. Invariance holds to float precision (output KL ≤ 2×10⁻⁹), while Euclidean safety-subspace methods fail it (KL up to 10⁻³). Leverage concentrates in layers 32–35. The forecast fails (Spearman ρ = 0.17; 24–39× too low on the main checkpoint), and we measure where it breaks: drift along the update scales as (1−α)^0.72 [0.71, 0.74], not the (1−α)² the quadratic model assumes. The unbounded repair breaks the model, because nearly parallel eigenvectors turn it into an oblique projector with steps up to 94× the update. A per-module trust region, added after the experiment freeze and logged as a deviation, fixes this and keeps exact invariance.
The bounded repair works in absolute terms: on test it returns English harmful compliance from 22.5% to 7.4% (aligned: 6.6%) with no task loss. It is not better than simple baselines. RegMean ties it on harmful compliance; plain interpolation (WiSE-FT, α = 0.5) matches it on harmful compliance and halves its English over-refusal (15.5% vs 31.5%); on KL drift at equal task cost, WiSE-FT, RegMean and diagonal Fisher all beat it. We release the pipeline (CuratorKIT → SafeTune → AuditKIT), the repair as a SafeTune plugin, all data, adapters and results, and patches to the four Lexsi libraries.
Headline: before → after
English harmful compliance on the held-out test split (485 prompts): 22.5% after the benign fine-tune → 7.4% after the closed-form repair (aligned model: 6.6%; exact McNemar, Holm p ≈ 10⁻¹⁴). Hindi task loss stays at 1.774 nats/token (fine-tuned 1.780, aligned 2.136). Plain interpolation, WiSE-FT α = 0.5, reaches 5.4% on the same prompts.
Base model and method in one line
CohereLabs/tiny-aya-global, LoRA-fine-tuned on Hindi instructions, then repaired in closed form by shrinking the update in the eigenbasis of the pencil (F_h, H_t), with K-FAC curvature, covariant damping and a per-module trust region.
For judges: one minute
| What we did | Measured how a harmless Hindi fine-tune damages Tiny Aya Global's safety in English, Hindi and Hinglish, then built a no-retraining repair from two curvature matrices |
| What works | The repair is exactly gauge-invariant on the real model, never breaks the model once bounded, and restores English harmful compliance to the aligned level with no task loss |
| What doesn't | Second-order forecasts are 24–111× too low: real drift scales as (1−α)^0.72, not (1−α)², and on confident answers the true KL can exceed its quadratic by orders of magnitude (§6.1). And the repair is no better than plain interpolation: WiSE-FT α = 0.5 matches it on harmful compliance and halves its English over-refusal |
| Why it's useful | A pre-registered, honest negative result on curvature-based safety repair; a measured drift law and a short calculation (§6.1) that account for the failed forecasts; a multilingual drift profile (lost discrimination, not lost refusal); a confidence blind spot in second-order safety tools, with CPU pilots (§8); a tested pipeline and four library patches |
| Rigour | 7 pre-registered predictions, a dated deviation log, a test split touched once, paired exact McNemar with Holm correction, paired bootstrap for drift, 270 unit tests |
| Use it | Deploy run A's adapter interpolated at α = 0.5 (safety-pencil-runA-wiseft-0.5, §6.7); build your own with the recipe in §12.1 |
How to read this write-up
§1 states the problem and the questions. §3 is the theory: the method derived as definitions and propositions, with no data. §4 describes the experiments. §5 is the evidence: what we measured, one question at a time. §6 connects the two: where the measurements agree with the theory, where they part, and why. §6.7 says which model to deploy, and §12.1 how to build a safety-repaired fine-tune of your own. Hackathon details (kits, artifacts, reproduction, team) are in §4.7 and §10–§18.
1. Introduction
1.1 Why this problem: a real use case
Small open models like Tiny Aya are attractive for Indian-language products because teams can fine-tune them on their own data: customer support in Hindi, government-scheme Q&A, education. Most such fine-tunes use entirely benign data, yet benign fine-tuning is known to erode alignment (Qi et al., 2024). A team fine-tuning for Hindi support usually has no safety data, no red-team budget and no way to re-run alignment. It needs three things:
- to see whether its fine-tune damaged safety, and in which language;
- to predict the damage before deploying;
- to repair it cheaply while keeping what it fine-tuned for.
Tiny Aya is multilingual, so the damage need not look the same in each language. We measure English, Hindi and romanised Hinglish separately throughout.
1.2 The approach in one picture
A fine-tune moves the weights along many directions at once. Some directions carry the task, some damage safety, and some do both. Suppose we could measure, for every direction, how much it hurts safety and how much it helps the task. Then the ratio of the two tells us which parts of the update to undo: those that cost a lot of safety and buy little task performance.
Two curvature matrices provide exactly these measurements. One, F_h, measures how quickly the model's answers to harmful prompts move away from the aligned model's answers as the weights move. The other, H_t, measures how quickly the task loss changes. Their generalised eigenvectors are the directions in which both are diagonal at once, and each eigenvalue λ is the ratio "safety damage per unit of task cost" in that direction. Everything in this project is read off that one basis (Figure 1): large λ locates safety, ½Σλc² forecasts drift, and shrinking each coordinate by μ/(λ+μ) repairs the update.
Figure 1. The framework. Two curvatures, F_h at the aligned model and H_t at the fine-tuned one, define one generalised eigenproblem; locate, forecast and repair are read off its eigenvalues λ. ✓ and ✗ mark what held on the real model. Editable source: docs/figures/safety_pencil_framework.drawio.
1.3 Research questions
- Locate. Which layers and modules carry the safety behaviour that the fine-tune damaged?
- Forecast. Can the safety cost of an update be predicted from the update alone?
- Repair. Can the update be turned into a better one in closed form, removing the safety damage and keeping the task gain?
We also test a structural property: whether the repair depends only on the network's function, or also on an arbitrary choice of coordinates for its weights (§3.8). Seven predictions were pre-registered before any measurement (§3.11).
1.4 Contributions
- A multilingual drift profile of a benign fine-tune: the damage is a loss of discrimination (more refusal and more harmful compliance), and it differs by language (§5.1).
- The generalised spectrum as a leverage map for locating safety, and as the basis for a closed-form forecast and repair (§3).
- A proof and a full-scale test that the repair is exactly invariant under the GQA value/output gauge, which separates curvature merges from Euclidean "safety-subspace" methods (§3.8, §5.2).
- A measured drift law, S ∝ (1−α)^0.72, showing that a real fine-tune lies outside the second-order regime, and a short calculation (sinh(m₀)/m₀) of why Fisher forecasts run low on confident answers (§5.4, §6.1).
- A diagnosis of why the closed-form repair can explode (oblique projection, certified by an eigenvalue ceiling) and a bounded repair that provably cannot manufacture an advantage (§3.9–3.10, §5.5–5.6).
- An honest comparison against simpler merges: on this model, the most detailed curvature model is the worst merge (§5.7–5.9, §6.4).
- A confidence blind spot shared by every second-order safety tool: an aligned model's Fisher vanishes as e^(−m) on its confident refusals. CPU pilots support it: EWC fails at every λ, Fisher merging is blind, and learned module weights beat interpolation by a wide margin. These shape the plan (§8).
We do not claim the repair formula or the forecast as new; both belong to known families (§2).
2. Related work
- Benign fine-tuning erodes alignment. Qi et al. (2024) show that fine-tuning on harmless data can compromise safety. We characterise this per language on a 3B multilingual model.
- Fisher-weighted merging. Matena & Raffel (2022) merge models by Fisher-weighted averaging; RegMean (Jin et al., 2023) merges linear layers using input activation statistics; MaTS (Tam et al., 2024) generalises both. Our repair is a member of this family: a merge of the aligned and fine-tuned models weighted by two Fishers (§3.5).
- WiSE-FT (Wortsman et al., 2022) interpolates linearly between two models. We show it is the special case of our objective in which the two curvatures are proportional (§3.5).
- K-FAC (Martens & Grosse, 2015; George et al., 2018 for EK-FAC) approximates a layer's Fisher by a Kronecker product. We use it to make the generalised eigenproblem tractable.
- Safety-aware merging and projection. Safe LoRA (Hsu et al., 2024) projects updates onto a safety subspace; ReSta (Bhardwaj et al., 2024) restores safety by task arithmetic; AlignMerge (Roy et al., 2025) has an objective that overlaps ours substantially: Fisher–Rao proximity to the experts plus a penalty on motion along an alignment subspace. Safe LoRA's projection is Euclidean, which our invariance test separates from curvature geometry. Our "AlignMerge-style" control (§5.2) is our own simplification, the pencil with a Euclidean rank-r projector on the harm factors, not the authors' full method.
- Forecasting drift from curvature. Springer et al. (2026, Prop. 3.3) give the local quadratic form ½ΔᵀFΔ for alignment loss (a KL to the base policy) under small displacements; we test it on a whole fine-tune.
- Lexsi's own release.
Lexsi/Gemma-4-31B-it-Multilingual-Safety-Repairis a safety-repaired Gemma 4 31B. Its card describes a bounded, circuit-restricted weight edit evaluated in English and five Indic languages, with the full recipe in a white paper. Ours is a closed-form curvature repair on a 3B model, with a tested forecast and a per-language drift profile.
3. Theory: the method, derived
This section builds the method from first principles. It uses no measurements; §5 tests each claim.
3.1 Setup and notation
θ0 are the aligned weights, Δ is the fine-tune's update and θ1 = θ0 + Δ. D_h is a distribution of harmful prompts and D_t the task data. For a linear module with weight W of shape d_out × d_in, the update is ΔW, the repaired update ΔW′, and the repair step δ = ΔW′ − ΔW.
3.2 Drift and its curvature
Definition 1 (drift). Drift is the KL divergence from the aligned model on harmful prompts, measured on the aligned model's own answers:
Drift is zero exactly when the model answers harmful prompts as the aligned model does, needs no judge, and is smooth in the weights.
Proposition 1 (the curvature of drift is the Fisher). $\nabla S(\theta_0) = 0$ and $\nabla^2 S(\theta_0) = F_h$, where
Proof. $\nabla S = -\mathbb{E}{p_0}[\nabla\log p_\theta]$, and at θ0, $\mathbb{E}{p_0}[\nabla \log p_0] = \sum_y \nabla p_0(y) = \nabla 1 = 0$. Differentiating again and using $\nabla^2\log p = \nabla^2 p/p - \nabla\log p,\nabla\log p^\top$, the first term averages to $\nabla^2 \sum_y p_0(y) = 0$, leaving the expected outer product. ∎
It follows that, to second order, drift is a quadratic form in the update:
The labels y are sampled from the model (the "true Fisher"), not taken from a dataset. Whether a whole fine-tune is small enough for the cubic remainder to be negligible is an empirical question (§5.4).
3.3 Task cost
Definition 2 (regret). Expanding the task loss L_t around θ1, a step δ costs
H_t is the Fisher at θ1 with labels sampled from the model. For softmax cross-entropy it equals the Gauss–Newton matrix, so it is positive semidefinite.
Assumption A1. θ1 is near a task minimum, so g ≈ 0, and the extra task loss of a step is the regret $R(\delta) = \tfrac12,\delta^\top H_t,\delta$.
A1 can fail. We monitor it with the ratio $\rho_g = |g^\top\delta| / (\tfrac12\delta^\top \hat H\delta)$: above 0.1, the linear term matters and predicted regret is not trustworthy.
3.4 The pencil
Definition 3 (the pencil). The pair of quadratic forms (F_h, H_t) is a matrix pencil, with generalised eigenproblem
Proposition 2 (simultaneous diagonalisation). If H_t ≻ 0, there is a basis V with $V^\top H_t V = I$ and $V^\top F_h V = \Lambda = \mathrm{diag}(\lambda_i)$, λ_i ≥ 0.
Proof. Write $H_t = LL^\top$, eigendecompose the symmetric matrix $L^{-1}F_hL^{-\top} = Q\Lambda Q^\top$, and set $V = L^{-\top}Q$. Then $V^\top H_t V = Q^\top Q = I$ and $V^\top F_h V = \Lambda$. ∎
Each eigenvalue is a Rayleigh quotient,
and any update has coordinates c = VᵀH_tΔ, in which both costs are sums of squares: $\Delta^\top F_h\Delta = \sum_i\lambda_i c_i^2$ and $\Delta^\top H_t\Delta = \sum_i c_i^2$.
The basis is orthonormal in the task-curvature inner product, not in ordinary weight space. Two eigenvectors can point in almost the same weight direction; §3.9 shows why this matters.
3.5 Locate, forecast, repair
Locate. A module whose largest eigenvalue is huge contains a direction along which safety is very sensitive and the task barely cares. The per-module maximum λ is our leverage map.
Proposition 3 (forecast). Under Proposition 1, $S(\theta_1) \approx \tfrac12\sum_i \lambda_i c_i^2$. Each term says where the drift is.
Proposition 4 (closed-form repair). The repaired update minimising
has pencil coordinates
equivalently $\theta_{\text{rep}} = (F_h+\mu H_t)^{-1}(F_h\theta_0 + \mu H_t\theta_1)$. Every μ gives a Pareto-optimal trade-off between S and R.
Proof. Setting the gradient to zero gives $F_h\Delta' + \mu H_t(\Delta'-\Delta) = 0$; multiplying by Vᵀ decouples the coordinates: $\lambda_i z_i + \mu(z_i - c_i) = 0$. If another update had both lower S and lower R, it would have lower J. ∎
Directions with λ ≫ μ are removed and directions with λ ≪ μ are kept. The merge form shows the repair is a precision-weighted average of the aligned and fine-tuned models.
Worked example. Take F_h = diag(10, 0.1), H_t = I, Δ = (1, 1). At μ = 1 the repair keeps z = (0.091, 0.909): drift 0.083 at regret 0.417. Interpolation Δ′ = tΔ needs t = 0.354 for the same regret and leaves drift 0.633. At equal task cost, the pencil leaves 7.6× less drift. This is the advantage the method promises when its assumptions hold.
Corollary 1 (a family of merges). Other merges are this objective with a simpler curvature model:
| Method | Curvature model | Fraction of the update it keeps |
|---|---|---|
| WiSE-FT | F_h = κH_t (all λ equal) | the same μ/(κ+μ) in every direction |
| RegMean | output factors G = I (§3.6), so λ = α | μ/(α_i+μ) along each input direction |
| Diagonal Fisher | every factor replaced by its diagonal | its own fraction in [0, 1] for each weight |
| Pencil | full Kronecker on both sides | μ/(α_iγ_j+μ) along each pencil direction |
For WiSE-FT: with F_h = κH_t, stationarity gives (κ+μ)H_tΔ′ = μH_tΔ, so Δ′ = μ/(κ+μ)·Δ. So the theory predicts the pencil should beat these merges exactly when its extra detail is accurate. §5.7 tests this.
3.6 Making it computable: K-FAC
Tiny Aya has about 3.3 billion weights, so F_h cannot be stored. For one linear layer, the gradient is an outer product g aᵀ (output gradient g, input activation a). K-FAC approximates the layer's Fisher as a Kronecker product of input statistics A = E[aaᵀ] and output-gradient statistics G = E[ggᵀ]: F ≈ A ⊗ G. Quadratic forms become traces, for example $S = \tfrac12\operatorname{tr}(G_h,\Delta W,A_h,\Delta W^\top)$.
Proposition 5 (the Kronecker pencil splits). If $A_h x = \alpha A_t x$ and $G_h y = \gamma G_t y$, then x ⊗ y is a generalised eigenvector of (A_h ⊗ G_h, A_t ⊗ G_t) with eigenvalue λ = αγ.
Proof. $(A_h\otimes G_h)(x\otimes y) = (A_hx)\otimes(G_hy) = \alpha\gamma,(A_tx)\otimes(G_ty)$. ∎
So each layer needs two small eigensolves (sizes 2,048 and 11,008 on Tiny Aya) instead of one of size d_in·d_out. With $X^\top A_t X = I$ and $Y^\top G_t Y = I$, the coordinates are $C = Y^\top G_t,\Delta W,A_t X$, the shrink matrix is $M_{ji} = \mu/(\alpha_i\gamma_j + \mu)$, and the repair is applied in subtraction form,
so anything outside the solved subspaces passes through untouched. K-FAC drops cross-layer and cross-position terms. With a ~366-token chat preamble shared by every prompt, positions are strongly coupled, so the cross-position terms may be large.
3.7 Damping, and a free validity check
Task factors are only semidefinite, so we damp them covariantly, with a little of the harm factor: $A_t' = A_t + \gamma_c A_h$ and $G_t' = G_t + \gamma_c G_h$, with γ_c = 10⁻³.
Proposition 6 (the eigenvalue ceiling). Covariant damping maps each side eigenvalue α ↦ α/(1 + γ_c α) < 1/γ_c = 1000. Moreover, if G_t ⪰ 0, every solved eigenvalue
is at most 1/γ_c.
Proof. If $A_hx = \alpha A_tx$ then $(A_t + \gamma_cA_h)x = (1/\alpha + \gamma_c)A_hx$. The second claim holds because yᵀG_ty ≥ 0. ∎
A true K-FAC factor is an average of outer products, hence positive semidefinite. An eigenvalue above 1000 therefore certifies that the stored factor has gone numerically indefinite (for example through fp32 accumulation from a bf16 model). We also restrict each side to its range, project out two exactly known null vectors (LayerNorm's 1/w on the shared input factor; the all-ones vector on o_proj and down_proj outputs), keep the top 512 pairs on MLP-width sides, and retry Cholesky with increasing jitter.
3.8 Exact invariance under the V/O gauge
Grouped-query attention has a symmetry. For any invertible 128×128 matrix M per KV head, replacing W_v → M W_v and W_o → W_o M⁻¹ (on the four query heads that read that KV head) leaves the network's function exactly unchanged. A method that depends only on the function should give the same repaired function in both coordinate systems.
Proposition 7 (gauge invariance). The pencil repair commutes with the gauge: the repaired update in the new coordinates is M times the repaired update in the old ones, so the repaired function is identical.
Proof (v_proj; o_proj is analogous). Under the gauge, G → M⁻ᵀGM⁻¹ for both factors, ΔW → MΔW, and A is unchanged. The transformed output-side pencil is solved by Y′ = MY with the same eigenvalues, and the normalisation is preserved. Hence
Same coordinates, same eigenvalues, same shrink: ΔW′ maps to MΔW′. Covariant damping transforms like G_t, so it is preserved. ∎
Methods that use Euclidean projections, or that damp with νI, change under M. This gives a sharp, measurable test that separates curvature merges from safety-subspace methods (prediction P1).
3.9 When the closed form misbehaves: oblique projections
Suppose the eigenvalues split into a plateau far above μ and a bulk far below it. Then every shrink factor is close to 0 or 1, and the repair becomes a projector
Proposition 8 (size of the step).
- P is idempotent but not symmetric unless V is orthogonal: it is an oblique projector, with $\lVert P\rVert = 1/\sin\theta_{\min}$, where θ_min is the smallest angle between the removed and kept subspaces.
- In the curvature norm $\lVert U\rVert_t^2 = \operatorname{tr}(G'UA'U^\top)$ the step never exceeds the update, $\lVert\delta\rVert_t \le \lVert\Delta W\rVert_t$. In weight space it can be as large as $\sqrt{\kappa(G'),\kappa(A')}$ times the update.
Proof sketch. The coordinate map ΔW ↦ C is an isometry from the curvature norm to the Frobenius norm, and the repair multiplies coordinates by numbers in [0, 1]. Y has singular values $1/\sqrt{\mathrm{eig}(G')}$, which gives the Euclidean bound. The 1/sin θ_min norm of an oblique projector is standard. ∎
Nearly parallel removed and kept directions mean the two parts of ΔW are huge and cancel each other; deleting one leaves the other. The method's own metric cannot see this failure: its forecast keeps promising small drift while the weights move far.
3.10 The bounded repair
To rule this failure out, we bound each module's step by the fine-tune's own size on the inputs the module actually receives. (This repair was designed after seeing the failure on 12 Nov; §4.6.)
Definition 4 (trust region by scaling). For module k, let $r_k = \lVert\delta_k A_t^{1/2}\rVert / \lVert\Delta W_k A_t^{1/2}\rVert$, and replace $\delta_k \leftarrow t_k\delta_k$ with $t_k = \min(1, 1/r_k)$.
Proposition 9 (scaling cannot create an advantage). In pencil coordinates, scaling maps c ↦ (1 − t(1−M))c. Along the scaled path drift is non-increasing,
and regret scales as R(t) = t²R(1).
Proof. Direct differentiation, using 1 − M ∈ [0, 1] and t ∈ [0, 1]; R is quadratic in the step. ∎
Scaling therefore only ever shrinks steps: a real advantage survives, and an advantage that came from an oversized step disappears. In a two-parameter toy built to reproduce the failure, it removes an artefactual "win" that relied on 29× steps; in a toy with a real advantage, it changes nothing.
Two further pieces complete the bounded repair:
- Light output-side damping, G″ = G_t′ + νI with ν = 10⁻² × mean diagonal (the middle of the pre-registered grid, run once). This is exactly an added step penalty: ½tr(G″δA′δᵀ) = R(δ) + (ν/2)‖δA′^{1/2}‖², so the closed form is unchanged.
- v_proj is exempt from both, because the gauge acts on its output factor. The trust ratio for o_proj is gauge-invariant, so Proposition 7 still holds.
3.11 From theory to testable predictions
Written in PREREGISTRATION.md before the first experiment (9 Nov):
| ID | Prediction | Tests |
|---|---|---|
| P1 | The repair is invariant under the V/O gauge; Euclidean methods are not | Proposition 7 |
| P2 | The forecast ½ΔᵀF_hΔ ranks measured drift across checkpoints (Spearman ρ ≥ 0.6, CI > 0) | Propositions 1, 3 |
| P3 | Refusal and task score move with μ in the predicted direction | Proposition 4, Assumption A1 |
| P4 | Repair ≈ safety-preserving fine-tuning at μ = 1/β | Second-order equivalence (cut) |
| P5 | Leverage concentrates in late MLP blocks | §3.5 (locate) |
| P6 | The repair beats ≥ 1 weight-space baseline on worst-language refusal at equal task score | Corollary 1 |
| P7 | Per-language pencils share a subspace | (stretch) |
4. Experimental setup
4.1 Model, runs and fine-tuning
The model is CohereLabs/tiny-aya-global (Hub main at run time, revision not pinned; unquantised): 36 layers, hidden size 2,048, MLP size 11,008, 16 query and 4 KV heads, parallel attention-and-MLP block. Weights are bf16; eigen and repair maths run in float64; ΔW is computed from the LoRA adapter in fp32.
| Setting | Value |
|---|---|
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q, k, v, o, gate, up, down projections, all 36 layers |
| Learning rate, epochs | 2×10⁻⁴, 2 |
| Batch | 4 × 4 gradient accumulation (effective 16) |
| Max length | 1,024 tokens (the chat template adds a ~366-token safety preamble) |
| Runs | A: 4,000 rows, seed 0 (main run; 498 steps, 54 min on an L4). C: 1,000 rows, seed 0. D: 1,000 rows + 2% BeaverTails contamination, seed 0. Adapters saved at 25/50/75/100% of training (12 checkpoints) |
4.2 Data, splits and leak check
| Data | Source | Use | Licence |
|---|---|---|---|
| Harmful prompts, en and hi | CohereLabs/aya_redteaming |
Harm factors (F_h), dev, test | Apache-2.0 |
| Harmful prompts, Hinglish | Converted from the en and hi prompts within each split (CuratorKIT, Aya Expanse 32B) | Same | Derived, Apache-2.0 |
| Over-refusal set | XSTest safe prompts (walledai/XSTest), translated to hi and Hinglish |
Dev, test | CC-BY-4.0 |
| Task T1 | CohereLabs/aya_dataset Hindi rows, topped up from Aya Collection |
Fine-tuning, task factors (H_t), task NLL | Apache-2.0 |
| General capability | CohereLabs/Global-MMLU, en and hi, 200 questions each |
Retention check | Apache-2.0 |
| Contamination (run D only) | BeaverTails via SafeTune, 20 rows | Fine-tune variant | CC-BY-NC 4.0; not re-released |
Languages: English, Hindi (Devanagari) and Hinglish (romanised, code-mixed). The task is Hindi only.
Benchmarks. The safety benchmark is the aya_redteaming test split in all three languages (the focus task), with XSTest's safe prompts for over-refusal. Global-MMLU (en, hi) and task NLL on the held-out Hindi rows are retention checks.
Harmful prompts are split per language with a fixed seed, stratified by harm category: 40% for computing factors, 10% dev (every sweep and selection), 50% test (touched once, for final numbers).
| Split | English | Hindi | Hinglish |
|---|---|---|---|
| Factor | 388 | 362 | 510 |
| Dev | 97 | 91 | 134 |
| Test | 485 | 453 | 635 |
| XSTest dev / test | 50 / 200 | 50 / 200 | 48 / 198 |
Task T1: train_4000 (run A), train_1000 (runs C and D), heldout 300 (all human-written, for task NLL). Leak check (scripts/01_check_contamination.py): normalised exact match and word 5-gram Jaccard ≥ 0.8, within every set and between every test split and every fit split. Result: 0 exact duplicates, 0 within-set duplicates, and 45 near-duplicate pairs listed for inspection; the check passed. CuratorKIT filtering counts are in Appendix C.
4.3 Metrics and judges
| Metric | Direction | Why |
|---|---|---|
| Harmful compliance: share of harmful test prompts whose answer Llama Guard 3 marks "unsafe" | Lower is better | The behaviour that matters, and the one the fine-tune moved; primary safety metric |
| Over-refusal: share of XSTest safe prompts refused (Aya Expanse 8B rubric) | Lower is better | Catches a "repair" that just refuses everything |
| Refusal: share of harmful prompts refused | Closer to θ0 is better | The pre-registered P6 metric (see below) |
| Drift S: mean per-token KL(p_θ0 ‖ p_θ) on dev harmful prompts, teacher-forced on θ0's answers | Lower is better | Judge-free; what the repair optimises; what the forecast predicts |
| Task NLL: 300 held-out Hindi rows, nats per token | Lower is better | The task gain to keep |
| Global-MMLU accuracy, en and hi | Higher is better | General capability retention |
We pre-registered refusal expecting the fine-tune to lower it. It went up in every language, along with over-refusal (§5.1). Higher refusal therefore no longer means safer here, and we read refusal against θ0.
Judges. Llama Guard 3 8B for unsafe, whose model card lists Hindi support; Aya Expanse 8B with a pinned rubric (Appendix A; SHA-256 recorded in every result) for refusal. We did not measure agreement with human labels: the 100-labels-per-language sample was planned but not completed, so no Cohen's κ is reported (§7). Paired comparisons share a judge, so judge bias affects differences between models less than absolute rates.
Generation is greedy, 256 new tokens. The probe tier (100 dev prompts per language) is used for sweeps; the test tier (the full test split) only for final numbers.
4.4 Running the method at scale
- Factors: 256 sequences per language and per task side; batch size 1; fp32 accumulation; true Fisher with sampled labels (one backward pass of the summed sampled log-probabilities); every position, preamble included. Harm factors from θ0 on harmful prompts; task factors from θ1 on task data.
- Unbounded (covariant) repair, 12 Nov: μ ∈ {0.003, 0.01, 0.03, 0.1, 0.3, 1}.
- Bounded (scaled) repair, 13 Nov: μ placed, before any measurement, where predicted regret is {0.01, 0.02, 0.05, 0.1, 0.2, 0.4} of its maximum, giving μ ∈ {5.21, 1.56, 0.280, 0.0607, 0.0105, 0.0019}.
- Acceptance (rule 11): a repaired model is evaluated only if measured dev drift falls and every module's trust ratio is ≤ 1.
- Final μ*: fixed in advance as the smallest accepted μ whose dev task NLL keeps 90% of the task gain (NLL ≤ 1.821).
- Ablations: unbounded against bounded repair (§5.5–5.6); four curvature models in one objective (§5.7); the trust region's share of the repair (§5.6).
4.5 Baselines and matching
- WiSE-FT, θ0 + (1−α)Δ, α ∈ {0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.75}, computed natively from the fp32 adapter.
- RegMean, W1 − ΔW A_h(A_h + μA_t)⁻¹, 13 values of μ from 0.3 to 1000.
- Diagonal Fisher, from diag(G)·diag(A) of the same K-FAC factors, 11 values of μ from 10⁻³ to 3.
Each baseline is matched to the pencil's dev task NLL by regula falsi on its knob, looking only at task NLL. Because one baseline turned out to have flat task NLL near the target (§5.7), we report two comparisons: the pre-registered closest-NLL match, and a labelled post-hoc frontier match (each baseline's lowest-drift setting with task NLL no worse than the pencil's + 0.005). Neither is shown without the other.
4.6 Protocol and statistics
Proportions get Wilson 95% intervals. Paired comparisons on the same prompts use exact McNemar tests, Holm-corrected across the three languages. Drift differences use a paired bootstrap over prompts (2,000 resamples, recomputing the token-weighted mean); p = 0 is reported as p < 1/2000. The test split was touched once. Every change after pre-registration is in the dated deviation log (Appendix B). The most important: the bounded repair (§3.10) and the frontier match were designed after the experiment freeze and after seeing the unbounded repair fail.
4.7 Hackathon repos (kits) used, and how they connect
All four libraries are pinned to an upstream commit plus our patches (patches/, installed by scripts/setup_lexsi.sh): CuratorKIT cee1aed, SafeTune 1d824f0, AuditKIT 9da5838, CircuitKIT 227fafd.
- CuratorKIT builds the data:
Curator(CuratorConfig(...))pipelines with SchemaGate (token limits), exact and MinHash deduplication, TextCleaner (NFC) and deterministic ids. A customBaseGenerationTask(HinglishTask), run throughPipeline([JSONLReader, task, PromptsMetaExporter])with aProvenanceManifest, converts prompts to Hinglish and translates XSTest, using Aya Expanse 32B through the Hugging Face router. Patched: a newprompts_metaexporter that keeps id, language and source; a Hindi sentence splitter; configurable OCR language; split recorded in provenance. - SafeTune fine-tunes and hosts the repair: runs A, C and D use
lora_wrapandtokenize_datasetin our own PEFT loop (PlainSFTTrainerdiscards the adapter, which we need for ΔW).harden_contamination_pairssupplies run D's BeaverTails rows.ReStaTrainerand the Safe LoRA projection are P1's controls. The pencil ships as a SafeTune recover method,register_recover("pencil", "PencilTrainer"). Patched:SafeDeltaTrainernow passes curvature inputs (it always used H = I);RefusalDirectionTraineraccepts a config; C-ΔΘ is computed in fp32 with the correct sign; LoX handles tied tensors; ReSta checks shapes. - AuditKIT generates and judges:
AutoModel.resolvefor batched greedy generation,GuardJudge("llama_guard")for harmful compliance,LLMJudgewith Aya Expanse 8B for refusal, with one model on the GPU at a time. Patched:AUDITKIT_DISABLE_RUN_CACHE=1, because the run cache wrote every raw completion to disk with no off switch; also cache fingerprints for local checkpoints and aresolve(name=...)fix. - CircuitKIT was installed and patched (MLP node scores now take the gradient at the MLP output; the old input-side gradient agreed in sign with true patching only about half the time on a toy model), but the planned EAP-IG cross-check was not run (§7).
Settings used with each kit (everything not listed is the kit's default):
| Kit | Component | Settings |
|---|---|---|
| CuratorKIT | CuratorConfig (harmful prompts, task T1) |
min_tokens=1 (the default of 10 words drops short prompts), max_tokens=4096, dedup="minhash" (exact, then MinHash), minhash_threshold=0.85, export_formats=["prompts_meta"]; stratified 40/10/50 split by harm category done by our split_ids |
| CuratorKIT | HinglishTask (LiteLLM backend) |
Aya Expanse 32B via the HF router, temperature 0.3 (so a retry can differ), max_tokens=512; length-ratio and refusal filters |
| SafeTune | lora_wrap, tokenize_dataset |
r 16, α 32, dropout 0.05, all seven projections; max length 1,024 |
| SafeTune | harden_contamination_pairs |
20 rows (2% of run D) |
| AuditKIT | generation (AutoModel.resolve) |
greedy (temperature 0), 256 new tokens, batch 16, AUDITKIT_DISABLE_RUN_CACHE=1 |
| AuditKIT | GuardJudge("llama_guard") |
Llama Guard 3 8B, default policy; first-line verdict safe/unsafe |
| AuditKIT | LLMJudge |
Aya Expanse 8B, choices REFUSE / PARTIAL / COMPLY, pinned rubric (Appendix A) |
The pipeline. Data from CuratorKIT feeds the SafeTune fine-tunes; our K-FAC factors (F_h at θ0, H_t at θ1) feed the PencilTrainer repair; AuditKIT generates and judges every checkpoint.
4.8 Models and licences
| Role | Model | Licence |
|---|---|---|
| Model studied (θ0) | CohereLabs/tiny-aya-global |
CC-BY-NC 4.0 + Cohere Labs acceptable-use policy |
| ReSta base (P1 control) | CohereLabs/tiny-aya-base |
CC-BY-NC 4.0 |
| Harm judge | meta-llama/Llama-Guard-3-8B |
Llama 3.1 Community Licence |
| Refusal judge | CohereLabs/aya-expanse-8b |
CC-BY-NC 4.0 |
| Hinglish / translation converter | CohereLabs/aya-expanse-32b (via the HF router) |
CC-BY-NC 4.0 |
Our adapters are Tiny Aya derivatives, so they carry CC-BY-NC 4.0, and every drifted checkpoint is labelled "research artifact: unrepaired, do not deploy", as SafeTune's licence requires. Our code is MIT; the Lexsi libraries are LSAL.
5. Results
Each subsection states the question, what the theory predicts, what we measured, and the verdict.
5.1 What the benign fine-tune did
Question: how does a harmless Hindi fine-tune change safety, per language? Test split, θ0 → run A, paired by prompt:
| English (485) | Hindi (453) | Hinglish (635) | |
|---|---|---|---|
| Harmful compliance | 6.6% → 22.5% (Holm p 7×10⁻¹⁵) | 6.0% → 8.4% (n.s.) | 23.1% → 22.4% (n.s.) |
| Refusal | 59.8% → 72.8% (Holm p 4×10⁻⁹) | 51.0% → 64.7% (4×10⁻⁹) | 19.2% → 44.4% (3×10⁻²⁸) |
| Over-refusal (XSTest) | 13.5% → 55.0% | 4.0% → 9.5% | 12.1% → 30.8% |
| Global-MMLU | 58.0% → 59.0% | 50.0% → 47.5% | — |
| Task NLL (Hindi held-out) | 2.136 → 1.780 |
Finding 1: the drift is a loss of discrimination, and it differs by language. The fine-tune did its job: task loss fell by 0.36 nats per token and MMLU barely moved. But the model now refuses more and complies with harmful requests more. In English, harmful compliance tripled and over-refusal quadrupled. In Hindi and Hinglish the damage is mainly extra refusal and over-refusal. Hinglish harmful compliance was already high at θ0 (23%) and did not change. All three runs showed rising refusal at the dev probe; run A was the only one with a significant change in harmful compliance, so it became the main run (Appendix B). Measured dev drift at θ1 is S = 0.261 (en), 0.222 (hi), 0.243 (Hinglish) nats per token. The examples in §5.12 qualify the English numbers: θ1's "refusals" of safe English prompts that we inspected were translations of the prompt into Hindi, and the two judges disagree on most of the newly unsafe English answers.
5.2 Invariance (P1)
Prediction (Proposition 7): the pencil gives the same function in every V/O gauge; Euclidean methods do not. We apply a random invertible matrix per KV head to v_proj's output and undo it in o_proj's input, at condition numbers 10 and 30. We transform the update and the factors the same way, repair in both coordinate systems, and compare the two repaired models' outputs (v/o modules, fp32, μ = 0.1):
| Method | max |Δ log p| (cond 10 / 30) | Output KL (cond 10 / 30) | Invariant? |
|---|---|---|---|
| ReSta (linear; positive control) | 1.6e-4 / 1.4e-4 | 2.0e-11 / 2.2e-11 | yes |
| RegMean | 1.5e-4 / 1.5e-4 | 1.9e-11 / 2.2e-11 | yes |
| Pencil, covariant | 2.0e-3 / 4.0e-3 | 8.2e-10 / 1.7e-9 | yes |
| Pencil, bounded (final method) | 2.0e-3 / 4.0e-3 | 8.2e-10 / 1.75e-9 | yes |
| Diagonal Fisher | 6.0e-2 / 1.5e-1 | 6.0e-7 / 2.1e-6 | no |
| Pencil, Euclidean damping | 7.7e-2 / 1.7e-1 | 1.2e-6 / 8.0e-6 | no |
| AlignMerge-style, our simplification (Euclidean, rank 64) | 0.48 / 0.50 | 8.5e-5 / 1.0e-4 | no |
| Safe LoRA (negative control) | 1.1 / 5.0 | 4.7e-5 / 1.1e-3 | no |
The invariant methods sit 2–4 orders of magnitude below the rest and stay at float precision as conditioning worsens (ReSta and RegMean flat; the pencil's KL roughly doubles from 8.2e-10 to 1.75e-9). Every Euclidean method's error grows. The analytic factor transform matches factors re-estimated on the reparameterised model to a median 2.2%. Verdict: P1 passes, at full scale.
5.3 Locate (P5)
Prediction: leverage concentrates in late MLP blocks. Pooled over languages, the ten highest-leverage modules are all in layers 32–35: up_proj and gate_proj in each, plus layer 35's down_proj and o_proj. The highest is layer 35 up_proj (λ = 8.2×10⁵). English shares 9 of its own top 10 with the pooled list and Hinglish 8, but Hindi shares only 2: Hindi's safety leverage sits elsewhere.
Caveat from Proposition 6: layer 35's gate and up output factors break the 1/γ_c ceiling (eigenvalues 1.4×10³ and 2.2×10³; leverage_pooled_runA.json), so the very top of the map is partly numerical. The refusal-direction cosine and the CircuitKIT comparison were not run. Verdict: P5 partial.
5.4 Forecast (P2), and the drift law
Prediction (Propositions 1 and 3): ½ΔᵀF_hΔ ranks measured drift. Over 36 checkpoint–language pairs (runs A, C, D × 4 checkpoints × 3 languages):
| Forecast against | Spearman ρ [95% CI] | Meets ρ ≥ 0.6, CI > 0? |
|---|---|---|
| Refusal drop (pre-registered) | 0.17 [−0.17, 0.49] | no |
| Measured KL drift S | 0.50 [0.22, 0.70] | no |
| Change in harmful compliance | 0.14 [−0.18, 0.43] | no |
The forecast is also far too small: 24–39× too low on run A's final checkpoint (0.0068 / 0.0075 / 0.0100 predicted against 0.262 / 0.218 / 0.238 measured in the drift job) and 24–111× across all 36 pairs. It also orders the languages wrongly.
To see why, we measured drift along the straight line from θ1 back to θ0 (WiSE-FT, θ0 + (1−α)Δ) and fitted S(α)/S(θ1) = (1−α)^p:
| 1 − α | 0.95 | 0.9 | 0.8 | 0.7 | 0.6 | 0.5 | 0.25 |
|---|---|---|---|---|---|---|---|
| Measured S/S(θ1) | 0.962 | 0.924 | 0.85 | 0.778 | 0.708 | 0.636 | 0.354 |
| Quadratic model (1−α)² | 0.902 | 0.81 | 0.64 | 0.49 | 0.36 | 0.25 | 0.062 |
Finding 2: real drift is far from quadratic. p = 0.72 [0.71, 0.74] (en 0.71, hi 0.70, Hinglish 0.77; bootstrap over prompts), against the p = 2 that Proposition 1 implies near θ0. Halving the update removes only 36% of the drift, not 75%. Verdict: P2 fails. §6.1 explains the connection.
5.5 The unbounded repair breaks the model
Prediction (Proposition 4): the repair lowers drift at a predictable task cost. Unbounded covariant repair, 12 Nov:
| μ | Measured S (θ1: 0.24) | Task NLL (θ1: 1.79) | Predicted S | Max step / ΔW |
|---|---|---|---|---|
| 0.003 | 31.4 | 31.0 | 0.003 | 94 |
| 0.1 | 31.3 | 31.0 | 0.003 | 94 |
| 1 | 31.3 | 31.2 | 0.003 | 94 |
Every μ destroyed the model and was rejected by rule 11. At μ = 1, 53 of 252 modules (17 at μ = 0.003) took steps more than twice the fine-tune's size; the worst were layer 35's up, gate and down projections (94, 62 and 63×). The model predicted drift ≈ 0.003 throughout, exactly the blind spot of Proposition 8. The two modules whose factors broke the eigenvalue ceiling are among the three with the largest steps; L35 down_proj also stepped 63×, but its step on task inputs was tiny. Verdict: P3 fails for the unbounded repair.
5.6 The bounded repair: valid at every μ, but weak
The bounded repair (§3.10), run on 13 Nov after the freeze:
| μ | Predicted regret fraction | Dev S (en / hi / Hinglish) | Task NLL | Modules scaled | ρ_g | Rule 11 |
|---|---|---|---|---|---|---|
| θ1 | — | 0.242 (0.261 / 0.222 / 0.243) | 1.786 | — | — | — |
| 5.21 | 0.01 | 0.208 (0.223 / 0.196 / 0.203) | 1.784 | 10 | 14.4 | pass |
| 1.56 | 0.02 | 0.202 (0.218 / 0.192 / 0.197) | 1.784 | 22 | 0.72 | pass |
| 0.280 | 0.05 | 0.195 (0.212 / 0.187 / 0.188) | 1.782 | 38 | 7.0 | pass |
| 0.0607 | 0.1 | 0.191 (0.208 / 0.183 / 0.183) | 1.782 | 48 | 6.7 | pass |
| 0.0105 | 0.2 | 0.188 (0.203 / 0.180 / 0.179) | 1.781 | 59 | 5.9 | pass |
| 0.0019 = μ* | 0.4 | 0.185 (0.200 / 0.178 / 0.176) | 1.780 | 54 | 4.7 | pass |
- Every μ is a valid repair: drift falls in every language, task loss does not rise, and invariance is unchanged (KL 8.2×10⁻¹⁰ / 1.75×10⁻⁹).
- At μ* the repair removes 24% of the measured drift. Its own exact prediction was 64% (0.0081 → 0.0029).
- The trust region is not what limits it. At μ*, the 54 scaled modules (median t = 0.82) carry only 6% of the predicted drift, and scaling holds back about 0.6% of it. The ten modules carrying the most predicted drift are all unscaled.
- ρ_g is 0.7–14 at every μ, far above 0.1: Assumption A1 fails. Moving from θ1 toward θ0 even improves dev task loss (WiSE-FT reaches 1.775 at α = 0.3).
Verdict: P3 partial. Drift falls with μ as predicted; task loss does not follow predicted regret.
5.7 Against baselines on dev
Prediction (Corollary 1): the pencil, the most detailed curvature model, beats the simpler merges. Post-hoc frontier match: each baseline's lowest-drift dev setting with task NLL no worse than the pencil μ*'s + 0.005. Differences S(pencil) − S(baseline) per language, paired bootstrap over prompts:
| Baseline (setting, dev S) | Task NLL vs pencil | English | Hindi | Hinglish |
|---|---|---|---|---|
| WiSE-FT α = 0.5 (0.154) | 0.004 better | +0.032 [+0.027, +0.039] | +0.034 [+0.031, +0.037] | +0.027 [+0.024, +0.031] |
| RegMean μ = 0.3 (0.148) | 0.003 worse | +0.034 [+0.029, +0.040] | +0.036 [+0.034, +0.039] | +0.039 [+0.035, +0.043] |
| Diagonal Fisher μ = 1.35 (0.158) | 0.004 better | +0.027 [+0.022, +0.033] | +0.029 [+0.027, +0.032] | +0.023 [+0.020, +0.027] |
All nine differences are positive (Holm p < 1/2000). Finding 3: on dev KL drift at equal task cost, the pencil is dominated by all three simpler merges, in every language.
Under the pre-registered closest-NLL rule, the pencil beats only WiSE-FT at α = 0.1, an almost arbitrary pick: WiSE-FT's dev NLL is flat (1.775–1.783 for α 0.05–0.5), and its own α = 0.5 beats the pencil. The pencil loses to RegMean and diagonal Fisher under either rule.
Further along the task range, diagonal Fisher reaches S = 0.060 at task NLL 1.803 (75% of the drift removed, inside the 90%-task-gain limit), and S = 0.0016 at 1.868 just outside it (99% removed while keeping 77% of the task gain). Drift and task gain are separable weight by weight.
5.8 Same model, before vs after (test split, touched once)
| Metric (test) | θ0 aligned | θ1 fine-tuned | Pencil μ* | RegMean (matched) | WiSE-FT α = 0.5 |
|---|---|---|---|---|---|
| Harmful compliance, English ↓ | 6.6% | 22.5% | 7.4% | 5.8% | 5.4% |
| Harmful compliance, Hindi ↓ | 6.0% | 8.4% | 5.5% | 4.9% | 4.9% |
| Harmful compliance, Hinglish ↓ | 23.1% | 22.4% | 23.0% | 23.8% | 21.4% |
| Over-refusal, English ↓ | 13.5% | 55.0% | 31.5% | 23.5% | 15.5% |
| Over-refusal, Hindi ↓ | 4.0% | 9.5% | 7.5% | 7.5% | 6.5% |
| Over-refusal, Hinglish ↓ | 12.1% | 30.8% | 34.8% | 28.3% | 29.3% |
| Refusal, English (vs θ0) | 59.8% | 72.8% | 62.5% | 63.9% | 67.0% |
| Refusal, Hindi (vs θ0) | 51.0% | 64.7% | 58.3% | 61.1% | 62.0% |
| Refusal, Hinglish (vs θ0) | 19.2% | 44.4% | 39.2% | 34.3% | 38.4% |
| Global-MMLU, en / hi ↑ | 58.0 / 50.0% | 59.0 / 47.5% | 61.0 / 46.0% | 58.5 / 47.5% | 58.5 / 47.5% |
| Task NLL, Hindi held-out ↓ | 2.136 | 1.780 | 1.774 | 1.775 | 1.771 |
Paired exact McNemar tests, Holm-corrected across the three languages:
- Fine-tuned → pencil: English harmful compliance 22.5% → 7.4% (Holm p ≈ 10⁻¹⁴); English over-refusal 55.0% → 31.5% (Holm p 2×10⁻¹⁰); Hindi harmful compliance 8.4% → 5.5% (p 0.04).
- Pencil vs RegMean: harmful compliance not different in any language; English over-refusal lower for RegMean (23.5% vs 31.5%, Holm p 0.011).
- Pencil vs WiSE-FT α = 0.5 (post-hoc frontier match): harmful compliance not different in any language; English over-refusal half for WiSE-FT (15.5% vs 31.5%, Holm p 1×10⁻⁸). WiSE-FT's English refusal is further from θ0 (67.0% vs 62.5%; θ0 59.8%; Holm p 0.004), and its English MMLU is lower (58.5% vs 61.0%).
Finding 4: the repair works in absolute terms but is not better than simple baselines. With no retraining, it returns English (and Hindi) harmful compliance to the aligned model's level, keeps the whole task gain and MMLU, and removes about half the added English over-refusal. RegMean ties it on harmful compliance, and plain interpolation matches it on harmful compliance with half its English over-refusal.
5.9 The P6 verdict
P6's pre-registered metric is worst-language refusal at equal task score. The worst language is Hinglish, and there the pencil's 39.2% beats RegMean's 34.3% (exact McNemar p 0.011, Holm 0.034). Read literally, P6 is met.
We do not claim it. The fine-tune raised refusal in every language, so on this metric the unrepaired fine-tuned model (44.4%) would beat the aligned model (19.2%): the metric rewards the damage. The pencil's extra Hinglish refusal comes with more Hinglish over-refusal (34.8% vs RegMean's 28.3%, raw p 0.04), Hinglish harmful compliance does not differ, and WiSE-FT α = 0.5 ties the pencil on this metric (38.4% vs 39.2%, p 0.72). Verdict: P6 not met in substance. We report the literal result and our verdict side by side.
5.10 Scorecard against the pre-registration
| ID | Outcome | Verdict |
|---|---|---|
| P1 | Pencil, RegMean, ReSta invariant (KL ≤ 2×10⁻⁹); Euclidean methods not (KL 6×10⁻⁷ to 1×10⁻³) | Pass |
| P2 | ρ = 0.17 [−0.17, 0.49] vs refusal drop; 0.50 [0.22, 0.70] vs KL drift; 0.14 vs harmful compliance | Fail |
| P3 | Unbounded: every μ rejected. Bounded: drift falls with μ as predicted; task loss does not follow predicted regret | Fail (unbounded); partial (bounded) |
| P4 | Cut on 9 Nov | Not run |
| P5 | Top 10 modules all in layers 32–35, 9 of them MLP; alignment with the refusal direction not measured | Partial |
| P6 | Literally met vs RegMean on Hinglish refusal, but the metric rewards the damage; ties on harmful compliance; no better than WiSE-FT | Not met in substance |
| P7 | Stretch goal | Not run |
5.11 What didn't work, and what we did not report
- The quadratic forecast (P2): wrong in magnitude (24–111×) and in ranking.
- The unbounded closed-form repair: oblique steps up to 94×, destroyed models.
- Curvature-selective repair against interpolation: no safety advantage over WiSE-FT α = 0.5.
- Pre-registered refusal as a safety metric for this fine-tune (§5.9).
- Predicted regret: the first-order term dominates (ρ_g 0.7–14).
Not reported: results on a quantised model (§7). Harmful-prompt outputs are never shown, because by design only judge labels and SHA-256 hashes are stored for them; §5.12 shows benign examples in full and harmful ones as labels.
5.12 Examples
Figure 3 shows what the models do. Panel A compares all four models on harmful prompts; panel B follows benign prompts through the aligned model, the fine-tuned model and the WiSE-FT baseline.
- Harmful prompts appear only as counts. By design we store judge labels and a SHA-256 hash per answer, never the text.
- Benign prompts: θ0, θ1 and WiSE-FT were regenerated as text with the evaluation settings (greedy, 256 tokens, bf16). The pencil's merged checkpoint was not saved, so it has no text to show and appears only in panel A.
- Hash check: θ1's and WiSE-FT's answers to the chess, Gandalf and poverty prompts are byte-identical to the answers the judges scored (11 of 21 regenerated answers match overall). The task row has no stored answer, because the task is scored by NLL.
Figure 3. Examples. A: share of the 485 harmful English test prompts judged unsafe by Llama Guard 3, for all four models. B: benign prompts as answered by θ0, θ1 and WiSE-FT, with the refusal judge's label. Editable source: docs/figures/safety_pencil_examples.drawio.
Of the 90 prompts the fine-tune broke, the pencil fixes 81 and WiSE-FT 84. But for 84 of those 90, the refusal judge said REFUSE, so the two judges disagree on most of the harmful-compliance increase. On the benign prompts, the pencil's stored labels say it answers the chess and Gandalf prompts, is PARTIAL on the Hinglish one, and still refuses "How do I snipe the best Black Friday deals?", one case behind its higher English over-refusal than WiSE-FT's.
6. Analysis: where theory and data part
Section 3's propositions are exact statements about quadratic models. Section 5 shows which of them describe the real model. This section connects the two.
6.1 Why the forecast fails: the update is outside the quadratic regime
Proposition 1 makes drift quadratic near θ0, so moving a fraction t = 1−α of the way along Δ should give S ∝ t². The measured exponent is 0.72 over 25–95% of the update (§5.4), so the second-order regime, if it holds anywhere, ends before a quarter of the update. That alone does not fix the direction of the error: a curve that starts quadratic and then bends down to t^0.72 would make the forecast too high. The forecast is 24–39× too low, so before t = 0.25 drift must grow faster than quadratically, or K-FAC underestimates the curvature, or both.
The first is expected on an aligned model's confident answers. If logits move linearly along Δ, the curvature of a token's KL is the variance of its logit change under the current distribution: tiny at θ0, where the model is confident, and largest when its answer flips. For a token with margin m₀ that the update reverses, the exact KL is sinh(m₀)/m₀ times its second-order forecast: 11× at p₀ = 0.99, 72× at p₀ = 0.999, which matches the size of the measured gap. The second explanation, K-FAC dropping cross-position terms (§3.6) that the shared 366-token preamble makes large, is not ruled out. The plan in §8 (experiment E1) separates the two with exact Jacobian–vector products.
6.2 Why the repair exploded: an oblique projector, certified by the ceiling
Proposition 8 predicts that when shrink factors are near 0 or 1 and the eigenvectors are nearly parallel, the repair takes huge, self-cancelling steps while its own metric reports nothing unusual. That is exactly what §5.5 shows: steps up to 94× (so θ_min ≤ 0.61° in layer 35 up_proj), and predicted drift 0.003 against measured 31. Proposition 6 gives an independent, data-free warning: the only two factors that broke the 1/γ_c ceiling, and so were provably indefinite, are among the three modules that exploded.
6.3 Why the bounded repair is weak
Proposition 9 guarantees that the trust region cannot create an advantage, and §5.6 shows it is not what limits the repair: scaling holds back about 0.6% of the predicted removal. The gap between predicted removal (64%) and measured removal (24%) has three sources:
- The quadratic model misattributes drift (§6.1): directions it ranks as harmless carry real drift.
- Top-512 truncation and the output-side damping lower λ on some directions.
- Assumption A1 fails (ρ_g 0.7–14). Regret is charged for moving toward θ0, a move that is actually free: WiSE-FT improves task loss on the way back.
6.4 Why simpler curvature models win
Corollary 1 places all four merges in one objective, differing only in curvature detail. The theory says more detail should help if it is accurate. On this model the ranking is reversed: the full two-sided Kronecker model is dominated on drift by input-only (RegMean), diagonal and isotropic (WiSE-FT) models (§5.7). Diagonal Fisher uses the same K-FAC factors, and it separates drift from task gain weight by weight (99% of drift removed while keeping 77% of the task gain). So the information is in the factors, and the full two-sided eigensolve is what loses it. Its output-side eigenbasis is where the indefinite factors and the exploding steps were found (§5.5).
6.5 KL drift and judged behaviour disagree
The repair optimises KL drift. Removing only a quarter of it fully restored English harmful compliance (§5.8), and methods that differ clearly on KL (§5.7) tie on judged harmful compliance. KL drift on the aligned model's answers is a blunt proxy for the behaviour that matters: much of it may reflect changes in wording or confidence rather than in whether the model complies.
6.6 What we learned
- Benign fine-tuning damaged discrimination, not refusal. The model refused more and complied more, and how this showed up depended on the language.
- Exact invariance is a cheap, decisive test. It separates curvature merges from Euclidean safety-subspace methods, at full scale.
- Second-order models are the wrong scale for a whole fine-tune. Drift scales as (1−α)^0.72 over most of the update, and on confident answers the true KL exceeds its quadratic by sinh(m₀)/m₀.
- More curvature detail was worse, not better, and the output-side eigenbasis is where the numerical trouble lived.
- KL drift and judged behaviour disagree, so a safety repair should be judged on behaviour.
- Pre-registration earned its keep. It stopped us from claiming P6 on a metric that, for this fine-tune, rewards the damage.
6.7 What this means if you ship a Hindi fine-tune
For a team that fine-tunes Tiny Aya on its own benign data, the measurements come down to four practical points.
1. Audit every language, in both directions. Our fine-tune kept its task gain and its MMLU, yet English harmful compliance tripled (6.6% → 22.5%) and English over-refusal quadrupled (13.5% → 55.0%). One number captures both: the refusal Youden index, J = P(refuse | harmful) − P(refuse | safe). Computed post hoc from §5.8, in points:
| J (points) | θ0 aligned | θ1 fine-tuned | Pencil μ* | RegMean | WiSE-FT α = 0.5 |
|---|---|---|---|---|---|
| English | 46.3 | 17.8 | 31.0 | 40.4 | 51.5 |
| Hindi | 47.0 | 55.2 | 50.8 | 53.6 | 55.5 |
| Hinglish | 7.1 | 13.6 | 4.4 | 6.0 | 9.1 |
2. Interpolate before trying anything else. Halving the update (WiSE-FT α = 0.5) needs no data and no GPU beyond evaluation. It:
- keeps the whole task gain (Hindi NLL 1.771, against 1.780 for the fine-tuned model);
- returns English harmful compliance to the aligned level (5.4%; aligned 6.6%);
- cuts English over-refusal from 55.0% to 15.5%;
- brings English J back to the aligned model's level (51.5 vs 46.3 points; the difference is not tested).
It ties or beats every curvature repair we ran (§5.7–5.8). Corollary 1 says why: it is the same objective with the simplest curvature model, and the simplest model is also the least noisy (§6.4). The pencil's remaining advantages are a higher English MMLU (61.0% vs 58.5%) and English refusal closer to θ0, at twice the English over-refusal.
3. What no repair here fixes: Hinglish. Hinglish harmful compliance stays at 21–24% for every model, the aligned one included, and the aligned model barely discriminates there (J = 7.1). Any repair that pulls toward θ0 inherits that, so Hinglish needs its own safety data or a target other than θ0. The judges have not been checked against human labels (§7).
4. The checkpoint to deploy from this work is run A's adapter interpolated at α = 0.5. It is published as the LoRA adapter tkaunlaky10/safety-pencil-runA-wiseft-0.5 on CohereLabs/tiny-aya-global. It is an evaluated research artifact, not certified for deployment: red-team it in your own languages first. §12.1 shows how to build the same for your own fine-tune.
7. Limitations
- No human validation of the judges. No Cohen's κ; absolute rates depend on Llama Guard 3 and Aya Expanse 8B, especially in Hinglish.
- One seed, one main run. Run B (seed 1) was cut, so seed variance is unknown; runs C and D differ from A in data mix as well as size.
- The final method was designed after seeing data. The bounded repair and the frontier match came after the experiment freeze; both are in the deviation log, and the pre-registered results are reported beside them.
- K-FAC approximations. Cross-position and cross-layer terms are dropped; fp32 factors from a bf16 model go indefinite in places (two ceiling breaks, two jitter retries).
- The pencil was never measured at the 90%-task-gain limit, because its predicted-regret grid stopped at task NLL 1.780. Its drift moved only from 0.208 to 0.185 across three decades of μ, so it is unlikely to reach diagonal Fisher's 0.060 at 1.803, but we did not test it.
- Not done: P4 (preserve during fine-tuning), P7, the refusal-direction and CircuitKIT checks for P5, further baselines on dev (Safe Delta, LoX, SafeMERGE, task arithmetic), a quantised model, a public demo Space.
- Hinglish is synthetic: converted by Aya Expanse 32B, with ~31% of conversions rejected.
- The judges measure refusal and harm, not answer quality. The refusal rubric scores a Hindi translation of an English prompt as REFUSE, so part of θ1's English over-refusal is the model translating instead of answering (§5.12). The refusal judge also labels 84 of the 90 newly unsafe English answers as REFUSE, so the harm and refusal judges disagree on most of the harmful-compliance increase. We have not measured either effect across the full test split.
8. With more credits
In one paragraph. Our failures share one cause. Every second-order safety tool relies on the curvature (Fisher information), and that curvature is almost blind to the refusals an aligned model is most sure about. Our forecast, our repair and EWC, the standard penalty for fine-tuning safely, all inherit this blind spot. With more credits we would replace the curvature approximation with the exact quantity it approximates, which costs only extra forward passes. Small CPU pilots already support each step. We commit to running and reporting every experiment, whatever the outcome. Derivations are in docs/WITH_MORE_CREDITS.md, pilot code in pilots/.
8.1 The idea
Take a refusal the model gives with 99.9% confidence. Its curvature weight is p(1 − p) ≈ 0.001, almost zero. So a curvature-based tool sees almost no cost in moving it, until the fine-tune flips it and the real damage turns out far larger than predicted.
In symbols, let K be the exact drift (the KL divergence from the aligned model on harmful prompts) and Q = ½δᵀF_hδ its quadratic approximation. Q weights each answer token by about e^(−m), where m is the model's confidence margin. For a refusal that the update flips,
which is the size of our 24–111× forecast gap (§6.1).
The direction: keep K, drop Q. Wherever round one used Q, we use K, or the refusal margin it protects, instead (Figure 2). Unlike Q, the curvature of K is measured at the current model, so its pull grows as a refusal weakens rather than staying frozen at the aligned model's tiny value.
Figure 2. The research direction. Each use of the quadratic approximation Q in round one (left) is replaced by the exact drift K or the refusal margin (right), with the pilot result for each. Editable source: docs/figures/safety_pencil_direction.drawio.
8.2 Evidence from CPU pilots
The pilot model is a 2-layer transformer trained on a synthetic language in which prompts with a trigger token must be refused. A benign LoRA fine-tune on another task destroys the refusal (harmful compliance 0% → 97–100%), as our Hindi fine-tune did. At this size everything can be computed exactly. The pilots use 3 seeds, except the fine-tuning pilot (1 seed), and contain no natural-language or harmful text.
| Question | Pilot answer |
|---|---|
| Is the quadratic forecast too low on confident answers? | Yes: 1,200–12,000× too low, as the blind spot predicts |
| Does EWC protect refusals during fine-tuning? | No: harmful compliance stays at 78–100% at every strength. An exact-KL or refusal-margin penalty gives 0% with the task kept |
| Do learned merge weights beat plain interpolation (WiSE-FT)? | Yes: 0.2–8.6% harmful compliance against 25–99%, at the same task gain |
| Does cross-fitting fix noisy curvature? | Yes: it loses 4–12% of the possible improvement instead of 18–50%, and never overshoots |
8.3 What we would run
- E1. Diagnose first. On Tiny Aya, split the forecast gap exactly into its three causes (the curvature approximation, the confidence blind spot and non-linearity), using Jacobian–vector products. The result decides what runs next.
- E2. Learned safety merge. Learn one weight per module (252) on the exact objective, starting from the best interpolation point. Each weight can only shrink the update, so the repair cannot blow up as round one's did.
- E9. Fine-tune safely. Train the Hindi fine-tune with an exact-KL or refusal-margin penalty instead of EWC, and compare all three on the same data.
- E3. Better curvature, where still used. Cross-fitted eigenvalues, fp64 factors and more data.
- Also: a discrimination metric (the refusal Youden index, E4), predicting the best interpolation point in advance (E5), quantised models (E6), cross-lingual transfer (E7), and human labels with more seeds (E8).
8.4 What we promise, and what we expect
Promises, which hold whatever the results:
- Pre-register every prediction and its failure condition, and report every outcome, including refutations.
- E1 always produces the full three-way split of the gap, because the split is an exact identity.
- E2 and E3 never take steps larger than the original update allows: E2 by construction, E3 by a proven bound. E2 is also never worse than interpolation on its fitting objective.
- Run E9, E6 and E8 as described, and release everything new under the same rule as round one: labels and hashes only for harmful prompts.
Forecasts, which the experiments may refute:
| Quantity | Now | Expected |
|---|---|---|
| Share of the forecast gap due to the blind spot (E1) | unmeasured | a factor ≥ 10× |
| English harmful compliance with the learned merge (E2) | pencil 7.4% | ≤ 5% |
| Harmful compliance after the Hindi fine-tune with a KL or margin penalty (E9) | 22.5% | ≤ 8%, with EWC worse |
| English over-refusal (E4) | pencil 31.5% | ≤ 20% |
A forecast that fails is a result, not a broken promise.
9. Conclusion
A single generalised eigenproblem gives a clean mathematical account of how to locate, forecast and repair the safety damage of a benign fine-tune, and its exact gauge invariance holds on the real 3B model. The quantitative parts do not survive contact with a full fine-tune: drift is far from quadratic, and the most detailed curvature model is the worst merge, beaten by RegMean, diagonal Fisher and plain interpolation. The bounded repair still restores English harmful compliance to the aligned level with no retraining and no task loss, but so does halving the update. For practitioners, simple interpolation is the strong baseline to beat. For method designers, the two measurements to take before trusting a curvature-based safety repair are the drift exponent along the update and the eigenvalue ceiling.
10. Cost and compute
| Where | Hugging Face Jobs, paid from hackathon and personal HF credits |
| GPUs | NVIDIA L4 (24 GB) and L40S (48 GB) |
| Hours | About 32 GPU-hours: roughly 19 on L4 and 13 on L40S, including runs that ran out of memory |
| Cost | Roughly USD 38 at list prices (USD 0.80/h L4, USD 1.80/h L40S), estimated from job runtimes; plus a few cents of Inference Providers calls for the Hinglish conversion |
| Biggest jobs | Bounded repair + baselines + test tier: 7.7 h on an L40S, ≈ USD 14. Unbounded sweep + invariance: 3.5 h, ≈ USD 6.3 |
| The repair itself | One basis solve (about an hour on an L40S, |
We did not run an ROI comparison against a closed model.
11. Artifacts
- HF datasets collection: Safety Pencil: data. Repos:
tkaunlaky10/safety-pencil-data(processed splits, leak-check report) andtkaunlaky10/safety-pencil-results(every result JSON, run ledger and masked logs, figures, summary). - HF models collection: Safety Pencil: models. Repos:
safety-pencil-lora-runA,safety-pencil-lora-runC,safety-pencil-lora-runD(LoRA adapters at 25/50/75/100% of training, labelled "research artifact: unrepaired, do not deploy"). - Repaired model to use:
tkaunlaky10/safety-pencil-runA-wiseft-0.5, the run A adapter interpolated at α = 0.5 (§6.7), with the test numbers on its card. - GitHub repo: https://github.com/tkaunlaky10/safety-pencil
- Explainer:
docs/MATH_STEP_BY_STEP.md, the method from first principles with worked examples. - Demo / Space: none.
12. Documentation and reproduction
Environment. Python 3.12; torch with a CUDA build matching the driver; transformers 5.21.1; the four Lexsi libraries at the pinned commits plus our patches. Everything is pinned in requirements-main.lock.txt. Hardware: one L4 for fine-tuning and evaluation; one L40S (48 GB GPU, 62 GB RAM) for the repair sweep. Factor files take about 52 GB per side.
| Path | What it is |
|---|---|
PREREGISTRATION.md |
Predictions P1–P7, pass criteria, dated deviation log |
EXPERIMENTS.md |
Every GPU job: code version, command, outcome |
configs/experiment.yaml |
Models, data, runs, LoRA, factors, pencil settings |
src/pencil/ |
The library: factors, generalised eigensolver, repair, invariance, baselines, judges, statistics |
scripts/00–12_*.py |
One script per stage, each with --dry-run |
jobs/ |
Hugging Face Jobs launcher, stage scripts, sync, report, publishing |
tests/ |
270 tests (CPU, under a minute and a half) |
patches/ |
Our fixes to the four Lexsi libraries |
verification/ |
Toy-transformer checks of the maths |
pilots/ |
CPU pilots behind §8: forecast decomposition, learned merge weights, EWC vs KL vs margin penalties, cross-fitted curvature, quantisation (JAX; pilots/README.md) |
git clone https://github.com/tkaunlaky10/safety-pencil && cd safety-pencil
python3.12 -m venv .venv && . .venv/bin/activate && pip install -U pip uv
uv pip install -r requirements-main.lock.txt
bash scripts/setup_lexsi.sh main # patched SafeTune, AuditKIT, CuratorKIT
uv pip install -e . --no-deps
python -m pytest # 270 passed, 8 skipped
bash scripts/dry_run_all.sh # whole pipeline on a tiny offline model
python jobs/launch.py calibrate # access checks, data, fine-tunes A, C, D
python jobs/launch.py evaluate theta0 # and evaluate A: probe and test tiers
python jobs/launch.py invariance_forecast A # factors, P1, P2
python jobs/launch.py repair_scaled A --gpu l40sx1 # sweep, baselines, P6, test tier (~8 h)
python jobs/launch.py wiseft_test A_0p5 --gpu l4x1 # WiSE-FT α = 0.5 on test
python jobs/report.py # outputs/summary.md from the result JSONs
12.1 Recipe: build a safety-repaired fine-tune of your own
This is the pipeline as a practitioner would run it on their own benign data. Times are from our runs on Tiny Aya Global.
Step 1. Configure. In configs/experiment.yaml, set the aligned model, your task data, the languages to audit and the LoRA settings (ours: r 16, α 32, all seven projections, 2 epochs, lr 2×10⁻⁴).
Step 2. Build and check the data (CuratorKIT). You need:
- the task rows;
- harmful prompts for the audit;
- benign-but-risky prompts for over-refusal;
- conversions into every language you will serve.
The scripts are 01_build_task_t1.py, 01_build_harmful.py, 01_build_xstest.py and 01_make_hinglish.py; then run the leak check, 01_check_contamination.py. The calibrate job runs them, and starts the hinglish stage for 01_make_hinglish.py automatically. Split the audit prompts into dev and test, and touch test once.
Step 3. Fine-tune (SafeTune). python scripts/03_finetune.py --run A takes 54 min on an L4 for 4,000 rows. Keep the fp32 LoRA adapter: every repair below starts from it.
Step 4. Audit before and after (AuditKIT). Run python scripts/02_evaluate.py --checkpoint CohereLabs/tiny-aya-global --tier probe --out-name probe_theta0, then the same with --checkpoint <your adapter folder>. Per language, compare harmful compliance, over-refusal, J and task NLL. If any language lost J or gained harmful compliance, repair.
Step 5. Repair by interpolation first. WiSE-FT keeps a fraction 1 − α of the update. Scale every lora_B in the adapter by 1 − α to get the result as an adapter, with no GPU. Or save a merged checkpoint with python scripts/11_wiseft_checkpoint.py --run A --alpha 0.5. Choose α on dev: take the smallest α that brings harmful compliance and J back to the aligned model's level in every language, as long as the task loss stays where you need it. On our run that was α = 0.5.
Step 6. Optional: the curvature repair. The factor jobs are scripts/04_factors.py --side harm --lang <lang> (at θ0) and --side task --run A (at θ1); jobs/stages/lib.sh has the exact calls. Then run the bounded repair with python scripts/06_repair.py sweep --run A --method scaled --mu <μ> --save <μ>. Use the bounded (scaled) method: the unbounded one breaks the model (§5.5). On our model it was no better than step 5 (§5.7), so its main value is diagnostic: the leverage map shows where safety moved, and the invariance check shows whether a method depends on arbitrary coordinates.
Step 7. Verify once on test and publish. Run python scripts/02_evaluate.py --checkpoint <chosen checkpoint> --tier full --out-name test_<name> once. Publish the adapter with a model card that gives the per-language before/after numbers and says the model is not certified for deployment.
Preventing the drift during fine-tuning is the other route, for example by mixing in safety data or with SafeTune's train-time harden methods. We did not test it (P4 was cut).
13. Declaration of work done outside the hackathon
None. All code, data processing, experiments and writing were done during the hackathon (1–15 November 2026). The techniques we build on (K-FAC, Fisher merging, RegMean, WiSE-FT) are published prior work, cited in §2.
14. Team
Team of three:
| Name | Contact |
|---|---|
| Tanay Kulkarni | tanaykulkarni34@gmail.com · HF / GitHub @tkaunlaky10 |
| Mukund Garg | gargmd19@gmail.com |
| Abdul Jawad Khan | khanabduljawad@gmail.com · HF @obluda420 |
15. Acknowledgements
Lexsi Labs for the hackathon, the four libraries and Hugging Face credits; Cohere Labs for Tiny Aya, Aya Expanse, aya_redteaming, aya_dataset and Global-MMLU; Meta for Llama Guard 3; the XSTest and BeaverTails authors.
16. References
- Qi, Zeng, Xie, Chen, Jia, Mittal & Henderson (2024). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024. arXiv:2310.03693.
- Matena & Raffel (2022). Merging Models with Fisher-Weighted Averaging. NeurIPS 2022. arXiv:2111.09832.
- Jin, Ren, Preotiuc-Pietro & Cheng (2023). Dataless Knowledge Fusion by Merging Weights of Language Models (RegMean). ICLR 2023. arXiv:2212.09849.
- Tam, Bansal & Raffel (2024). Merging by Matching Models in Task Parameter Subspaces (MaTS). TMLR. arXiv:2312.04339.
- Wortsman et al. (2022). Robust fine-tuning of zero-shot models (WiSE-FT). CVPR 2022. arXiv:2109.01903.
- Martens & Grosse (2015). Optimizing Neural Networks with Kronecker-factored Approximate Curvature. ICML 2015. arXiv:1503.05671.
- George et al. (2018). Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis (EK-FAC). NeurIPS 2018. arXiv:1806.03884.
- Hsu et al. (2024). Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models. NeurIPS 2024. arXiv:2405.16833.
- Bhardwaj, Do & Poria (2024). Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic (ReSta). ACL 2024. arXiv:2402.11746.
- Röttger et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024. arXiv:2308.01263.
- Roy, Patel, Chadha, Jain & Das (2025). AlignMerge: Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints. arXiv:2512.16245.
- Springer, Lee, Metevier, Castleman, Turbal, Jung, Shen & Korolova (2026). The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety. arXiv:2602.15799.
- Lexsi Labs. Gemma-4-31B-it-Multilingual-Safety-Repair. Hugging Face model card,
Lexsi/Gemma-4-31B-it-Multilingual-Safety-Repair. - Lexsi Labs: SafeTune, AuditKIT, CuratorKIT, CircuitKIT. https://github.com/Lexsi-Labs
17. LLM usage statement
All of the work in this project was AI-assisted. The code, the experiment and job scripts, the analysis and this write-up were drafted with Claude (including Claude Code), under the authors' direction, and reviewed by the authors. Every number comes from a logged run and a result file in the public results repo, and the examples in §5.12 come from a saved generation file with hash checks.
18. Changelog
- v1 (15 Nov 2026): first submission.
Appendix A: refusal judge rubric (summary)
The Aya Expanse 8B judge labels only the response's behaviour, not whether the request is harmful, in English, Hindi or Hinglish. REFUSE: does not provide what was asked, explicitly or implicitly (declining, lecturing, redirecting to a professional, or answering a different safe question). PARTIAL: provides some of what was asked, with heavy hedging or omissions. COMPLY: provides what was asked. The full rubric is configs/refusal_rubric.txt; its SHA-256 is recorded in every result.
Appendix B: deviations from the pre-registration that affect results
| Date | Change | Why |
|---|---|---|
| 9 Nov | Run B, P4 and the T2 task cut | Five-day plan |
| 12 Nov | Run A made the main run against the letter of Gate 1; harmful compliance made the primary safety metric | Refusal rose instead of falling |
| 12 Nov | Rule 11 judged by the input-weighted step ratio (chosen after seeing data) | The Euclidean ratio flags harmless steps along directions the task never visits |
| 13 Nov | Post-freeze: bounded repair (trust region + output damping), μ grid from predicted regret, backup μ* rule, native WiSE-FT, harmful-input ratio logged | The unbounded repair broke the model |
| 13 Nov | P6 reported two ways, pre-registered closest-NLL and post-hoc frontier, never the latter alone | WiSE-FT's dev NLL is flat near the target |
| 13 Nov | WiSE-FT α = 0.5 added to the test tier (post-hoc) | The frontier match |
Appendix C: data filtering and run log
CuratorKIT filtering.
| Step | Before | After | What was removed and why |
|---|---|---|---|
| aya_dataset Hindi rows | 1,153 raw | 889 usable | Filtered by the build's checks. Too few for 4,000 training rows, so the rest came from Aya Collection: train_4000 is ~15% human-written, train_1000 ~59% |
| Hinglish harmful, factor / dev / test | 750 / 188 / 938 | 510 / 134 / 635 | ~16% failed the length-ratio check; ~15% were refusals by the converter model |
| XSTest, Hindi / Hinglish | 250 / 250 | 250 / 246 | Failed conversions |
| Run A training rows | 4,000 | 3,984 | 16 over the 1,024-token limit |
What ran, in order (EXPERIMENTS.md has every job):
| When (IST) | Job | Outcome |
|---|---|---|
| 11 Nov | Data build, leak check, Hinglish conversion | Done; leak check passed |
| 11 Nov | Fine-tunes A, C, D (L4) | Done; each probe-evaluated at 4 checkpoints |
| 12 Nov | Test evaluation of θ0 and run A | Done (§5.1) |
| 12 Nov | Factors, leverage map, forecast | Done (§5.3, §5.4) |
| 12 Nov | Several jobs out of GPU or CPU memory | Fixed: batch-1 probe; bf16 base model; one basis set per job; moved to an L40S |
| 12–13 Nov | Invariance + unbounded repair sweep (L40S, 3.5 h) | P1 pass; every μ rejected (§5.5) |
| 13 Nov | Post-freeze bounded repair: factors, sweep, invariance recheck, baselines, matching, P6, test tier (L40S, 7.7 h) | Every μ valid; P6 dev fails; test tier done |
| 13 Nov | WiSE-FT α = 0.5 on test (L4, 66 min) | Done |



