Improve flow of AFFECTOR purpose and research context
Browse files- docs/AFFECTOR.pdf +2 -2
- paper/AFFECTOR-source.zip +2 -2
- paper/AFFECTOR.pdf +2 -2
- paper/AFFECTOR.tex +13 -22
- paper/PAPER-SHA256SUMS.txt +3 -3
docs/AFFECTOR.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3125eeed5e35b5e01506ebe58adfd229b434521f8c1d02b62eee45f35c2a6e65
|
| 3 |
+
size 1948813
|
paper/AFFECTOR-source.zip
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c984b4107783190bc7d334589c8d2554b2c499ded337d837db51bd9c7aba5377
|
| 3 |
+
size 4207585
|
paper/AFFECTOR.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3125eeed5e35b5e01506ebe58adfd229b434521f8c1d02b62eee45f35c2a6e65
|
| 3 |
+
size 1948813
|
paper/AFFECTOR.tex
CHANGED
|
@@ -23,8 +23,19 @@
|
|
| 23 |
\setcounter{topnumber}{3}
|
| 24 |
\setcounter{bottomnumber}{3}
|
| 25 |
\setcounter{totalnumber}{5}
|
| 26 |
-
\setlength{\textfloatsep}{12pt
|
| 27 |
-
\setlength{\intextsep}{12pt
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
\raggedbottom
|
| 29 |
|
| 30 |
\geometry{top=0.75in,bottom=0.65in,includeheadfoot}
|
|
@@ -98,9 +109,7 @@ can be inward and sadness for another person outward. Intensity
|
|
| 98 |
describes strength of affect, not pleasantness.
|
| 99 |
|
| 100 |
For intervention, intensity is centered:
|
| 101 |
-
|
| 102 |
\[c(t)=(v(t),d(t),2i(t)-1).\]
|
| 103 |
-
|
| 104 |
Coordinates are linearly interpolated between points, with endpoint
|
| 105 |
clamping outside the stored range. Geometry imports that lack a
|
| 106 |
supported direction omit that point; interpolating across the resulting
|
|
@@ -162,14 +171,10 @@ The capture site is the final chat-formatted prompt position immediately
|
|
| 162 |
before the post-attention normalization that feeds the feed-forward
|
| 163 |
sublayer. It is not a mean over all content tokens. For layer \nolinkurl{l}
|
| 164 |
and axis \nolinkurl{j}, the mean contrast is:
|
| 165 |
-
|
| 166 |
\[u_{\ell,j}=\frac14\sum_k(h^+_{\ell,j,k}-h^-_{\ell,j,k}).\]
|
| 167 |
-
|
| 168 |
With \texttt{RMS(x)\ =\ sqrt(mean\_r(x{[}r{]}\^{}2))}, normalization
|
| 169 |
gives:
|
| 170 |
-
|
| 171 |
\[B_{\ell,j}=\frac{u_{\ell,j}}{\max(\operatorname{RMS}(u_{\ell,j}),10^{-8})}.\]
|
| 172 |
-
|
| 173 |
The resulting \texttt{B{[}l{]}} is a 3 by 1,024 matrix. These are the
|
| 174 |
editable steering-module weights. Defaults select layers 8, 16, and 23,
|
| 175 |
with weights 1, 1, and 0.5. Geometry files retain the calibration
|
|
@@ -197,9 +202,7 @@ computation}\label{coordinate-to-activation-computation}
|
|
| 197 |
\end{figure}
|
| 198 |
Axis temperature \nolinkurl{T} transforms each coordinate magnitude while
|
| 199 |
retaining its sign:
|
| 200 |
-
|
| 201 |
\[q_j=\operatorname{sign}(c_j)|c_j|^{1/T}.\]
|
| 202 |
-
|
| 203 |
The implementation ranks axes by magnitude, retains at most axis top-k,
|
| 204 |
and stops adding axes once cumulative magnitude reaches the configured
|
| 205 |
mass fraction. Surviving coefficients are not renormalized. These
|
|
@@ -208,19 +211,13 @@ vocabulary distribution.
|
|
| 208 |
|
| 209 |
For current final-token hidden state \nolinkurl{h}, strength \nolinkurl{s},
|
| 210 |
and layer weight \texttt{w{[}l{]}}, the proposed perturbation is:
|
| 211 |
-
|
| 212 |
\[\delta_{\mathrm{raw}}=(q B_\ell)\operatorname{RMS}(h)s w_\ell.\]
|
| 213 |
-
|
| 214 |
Inertia \nolinkurl{rho} combines this proposal with the previous applied
|
| 215 |
perturbation within the same chunk:
|
| 216 |
-
|
| 217 |
\[\delta_{\mathrm{mix}}=\rho\delta_{\mathrm{previous}}+(1-\rho)\delta_{\mathrm{raw}}.\]
|
| 218 |
-
|
| 219 |
The first step uses the raw proposal directly. A relative RMS budget
|
| 220 |
\nolinkurl{beta} then bounds the perturbation:
|
| 221 |
-
|
| 222 |
\[\delta=\delta_{\mathrm{mix}}\min\left(1,\frac{\beta\operatorname{RMS}(h)}{\max(\operatorname{RMS}(\delta_{\mathrm{mix}}),10^{-12})}\right).\]
|
| 223 |
-
|
| 224 |
Zero strength, budget, or layer weight gives an exact zero perturbation.
|
| 225 |
Each chunk resets inertia history. The settings are applied as
|
| 226 |
requested; the release does not automatically lower them after poor
|
|
@@ -239,13 +236,9 @@ transformer}\label{intervention-within-the-transformer}
|
|
| 239 |
Let \nolinkurl{h} denote a block's residual state after its attention or
|
| 240 |
sequence-mixing stage. A conventional feed-forward residual update has
|
| 241 |
the form:
|
| 242 |
-
|
| 243 |
\[y=h+\operatorname{FFN}(\operatorname{Norm}(h)).\]
|
| 244 |
-
|
| 245 |
AFFECTOR implements:
|
| 246 |
-
|
| 247 |
\[y=(h+\delta)+\operatorname{FFN}(\operatorname{Norm}(h+\delta)).\]
|
| 248 |
-
|
| 249 |
A pre-hook on the feed-forward normalization adds \nolinkurl{delta} to its
|
| 250 |
input, and a post-hook on the block output adds the same \nolinkurl{delta}
|
| 251 |
to the residual branch. This is a forward-pass intervention, not a
|
|
@@ -326,9 +319,7 @@ estimates, not psychometric measurements.
|
|
| 326 |
|
| 327 |
For assessed coordinate \nolinkurl{p=(v,d,2i-1)} and target \nolinkurl{c}, the
|
| 328 |
reported similarity is:
|
| 329 |
-
|
| 330 |
\[\operatorname{similarity}=1-\frac{\lVert p-c\rVert_2}{2\sqrt3}.\]
|
| 331 |
-
|
| 332 |
The denominator is the maximum distance across the three-dimensional
|
| 333 |
\texttt{{[}-1,1{]}} cube. Similarity is not a probability. Axis errors
|
| 334 |
are also reported. Unknown direction makes the three-axis distance
|
|
|
|
| 23 |
\setcounter{topnumber}{3}
|
| 24 |
\setcounter{bottomnumber}{3}
|
| 25 |
\setcounter{totalnumber}{5}
|
| 26 |
+
\setlength{\textfloatsep}{12pt}
|
| 27 |
+
\setlength{\intextsep}{12pt}
|
| 28 |
+
\setlength{\floatsep}{12pt}
|
| 29 |
+
\captionsetup[figure]{skip=6pt}
|
| 30 |
+
% Keep display spacing equal, including after short introductory lines.
|
| 31 |
+
\makeatletter
|
| 32 |
+
\g@addto@macro\normalsize{%
|
| 33 |
+
\setlength{\abovedisplayskip}{7pt}%
|
| 34 |
+
\setlength{\belowdisplayskip}{7pt}%
|
| 35 |
+
\setlength{\abovedisplayshortskip}{7pt}%
|
| 36 |
+
\setlength{\belowdisplayshortskip}{7pt}%
|
| 37 |
+
}
|
| 38 |
+
\makeatother
|
| 39 |
\raggedbottom
|
| 40 |
|
| 41 |
\geometry{top=0.75in,bottom=0.65in,includeheadfoot}
|
|
|
|
| 109 |
describes strength of affect, not pleasantness.
|
| 110 |
|
| 111 |
For intervention, intensity is centered:
|
|
|
|
| 112 |
\[c(t)=(v(t),d(t),2i(t)-1).\]
|
|
|
|
| 113 |
Coordinates are linearly interpolated between points, with endpoint
|
| 114 |
clamping outside the stored range. Geometry imports that lack a
|
| 115 |
supported direction omit that point; interpolating across the resulting
|
|
|
|
| 171 |
before the post-attention normalization that feeds the feed-forward
|
| 172 |
sublayer. It is not a mean over all content tokens. For layer \nolinkurl{l}
|
| 173 |
and axis \nolinkurl{j}, the mean contrast is:
|
|
|
|
| 174 |
\[u_{\ell,j}=\frac14\sum_k(h^+_{\ell,j,k}-h^-_{\ell,j,k}).\]
|
|
|
|
| 175 |
With \texttt{RMS(x)\ =\ sqrt(mean\_r(x{[}r{]}\^{}2))}, normalization
|
| 176 |
gives:
|
|
|
|
| 177 |
\[B_{\ell,j}=\frac{u_{\ell,j}}{\max(\operatorname{RMS}(u_{\ell,j}),10^{-8})}.\]
|
|
|
|
| 178 |
The resulting \texttt{B{[}l{]}} is a 3 by 1,024 matrix. These are the
|
| 179 |
editable steering-module weights. Defaults select layers 8, 16, and 23,
|
| 180 |
with weights 1, 1, and 0.5. Geometry files retain the calibration
|
|
|
|
| 202 |
\end{figure}
|
| 203 |
Axis temperature \nolinkurl{T} transforms each coordinate magnitude while
|
| 204 |
retaining its sign:
|
|
|
|
| 205 |
\[q_j=\operatorname{sign}(c_j)|c_j|^{1/T}.\]
|
|
|
|
| 206 |
The implementation ranks axes by magnitude, retains at most axis top-k,
|
| 207 |
and stops adding axes once cumulative magnitude reaches the configured
|
| 208 |
mass fraction. Surviving coefficients are not renormalized. These
|
|
|
|
| 211 |
|
| 212 |
For current final-token hidden state \nolinkurl{h}, strength \nolinkurl{s},
|
| 213 |
and layer weight \texttt{w{[}l{]}}, the proposed perturbation is:
|
|
|
|
| 214 |
\[\delta_{\mathrm{raw}}=(q B_\ell)\operatorname{RMS}(h)s w_\ell.\]
|
|
|
|
| 215 |
Inertia \nolinkurl{rho} combines this proposal with the previous applied
|
| 216 |
perturbation within the same chunk:
|
|
|
|
| 217 |
\[\delta_{\mathrm{mix}}=\rho\delta_{\mathrm{previous}}+(1-\rho)\delta_{\mathrm{raw}}.\]
|
|
|
|
| 218 |
The first step uses the raw proposal directly. A relative RMS budget
|
| 219 |
\nolinkurl{beta} then bounds the perturbation:
|
|
|
|
| 220 |
\[\delta=\delta_{\mathrm{mix}}\min\left(1,\frac{\beta\operatorname{RMS}(h)}{\max(\operatorname{RMS}(\delta_{\mathrm{mix}}),10^{-12})}\right).\]
|
|
|
|
| 221 |
Zero strength, budget, or layer weight gives an exact zero perturbation.
|
| 222 |
Each chunk resets inertia history. The settings are applied as
|
| 223 |
requested; the release does not automatically lower them after poor
|
|
|
|
| 236 |
Let \nolinkurl{h} denote a block's residual state after its attention or
|
| 237 |
sequence-mixing stage. A conventional feed-forward residual update has
|
| 238 |
the form:
|
|
|
|
| 239 |
\[y=h+\operatorname{FFN}(\operatorname{Norm}(h)).\]
|
|
|
|
| 240 |
AFFECTOR implements:
|
|
|
|
| 241 |
\[y=(h+\delta)+\operatorname{FFN}(\operatorname{Norm}(h+\delta)).\]
|
|
|
|
| 242 |
A pre-hook on the feed-forward normalization adds \nolinkurl{delta} to its
|
| 243 |
input, and a post-hook on the block output adds the same \nolinkurl{delta}
|
| 244 |
to the residual branch. This is a forward-pass intervention, not a
|
|
|
|
| 319 |
|
| 320 |
For assessed coordinate \nolinkurl{p=(v,d,2i-1)} and target \nolinkurl{c}, the
|
| 321 |
reported similarity is:
|
|
|
|
| 322 |
\[\operatorname{similarity}=1-\frac{\lVert p-c\rVert_2}{2\sqrt3}.\]
|
|
|
|
| 323 |
The denominator is the maximum distance across the three-dimensional
|
| 324 |
\texttt{{[}-1,1{]}} cube. Similarity is not a probability. Axis errors
|
| 325 |
are also reported. Unknown direction makes the three-axis distance
|
paper/PAPER-SHA256SUMS.txt
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
322c798131b9aadcf7ada359c42f32c22adc44ab0dbd7e066c73cbe77ecfdf55 SCREENSHOTS.json
|
| 2 |
d14edda139a36c0bac8b56d02e5cda63ab7558c6db1424f1087189a9429bfaf0 LITERATURE_SOURCES.md
|
| 3 |
-
|
| 4 |
9257f38423bd7e496f1a0916c0fbaefbfb32ab544a684ca9a411508201f0535e references.bib
|
| 5 |
97a2a8a88f99be148a9b8a5774cd276cc38317712132ba024bd0b73b88a05760 source.md
|
| 6 |
97a2a8a88f99be148a9b8a5774cd276cc38317712132ba024bd0b73b88a05760 TECHNICAL_PAPER.md
|
| 7 |
-
|
| 8 |
d78577482f74cba4163db18d88af82786c923d72cd09607bde62cb747c645c5d AFFECTOR.bbl
|
| 9 |
eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
|
| 10 |
7f1c5ae6a0a8d61880debe26cc0b0a91ac74dd51841495920ae0cdd7ed23f83f README.md
|
|
@@ -24,4 +24,4 @@ a6a497a38f4ba9de8088f410535bee13dd862d1a97f4175d205df768e1d2c1c9 figures/illust
|
|
| 24 |
8d267fd6d50e2146279f1f33e07f88a3cbf76d023dd9ee36b29e390ac0473cd1 figures/screenshot-02.png
|
| 25 |
759dabd81a2ce67f01a3ec96a01b014468402baee5774792fccd5b7fe53af80a figures/screenshot-03.png
|
| 26 |
ff1ace53310f4f8ea57dedb7a202edd7404886091ed34b1bcc70cbb6c93b2779 figures/screenshot-01.png
|
| 27 |
-
|
|
|
|
| 1 |
322c798131b9aadcf7ada359c42f32c22adc44ab0dbd7e066c73cbe77ecfdf55 SCREENSHOTS.json
|
| 2 |
d14edda139a36c0bac8b56d02e5cda63ab7558c6db1424f1087189a9429bfaf0 LITERATURE_SOURCES.md
|
| 3 |
+
633baa667a0dc91028ef23e8483a1d57a6a7c32c89ffc3515441246cbad357b9 AFFECTOR.tex
|
| 4 |
9257f38423bd7e496f1a0916c0fbaefbfb32ab544a684ca9a411508201f0535e references.bib
|
| 5 |
97a2a8a88f99be148a9b8a5774cd276cc38317712132ba024bd0b73b88a05760 source.md
|
| 6 |
97a2a8a88f99be148a9b8a5774cd276cc38317712132ba024bd0b73b88a05760 TECHNICAL_PAPER.md
|
| 7 |
+
3125eeed5e35b5e01506ebe58adfd229b434521f8c1d02b62eee45f35c2a6e65 AFFECTOR.pdf
|
| 8 |
d78577482f74cba4163db18d88af82786c923d72cd09607bde62cb747c645c5d AFFECTOR.bbl
|
| 9 |
eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
|
| 10 |
7f1c5ae6a0a8d61880debe26cc0b0a91ac74dd51841495920ae0cdd7ed23f83f README.md
|
|
|
|
| 24 |
8d267fd6d50e2146279f1f33e07f88a3cbf76d023dd9ee36b29e390ac0473cd1 figures/screenshot-02.png
|
| 25 |
759dabd81a2ce67f01a3ec96a01b014468402baee5774792fccd5b7fe53af80a figures/screenshot-03.png
|
| 26 |
ff1ace53310f4f8ea57dedb7a202edd7404886091ed34b1bcc70cbb6c93b2779 figures/screenshot-01.png
|
| 27 |
+
c984b4107783190bc7d334589c8d2554b2c499ded337d837db51bd9c7aba5377 AFFECTOR-source.zip
|