ALSC: Asymmetric Log-Sigmoid Cap (Novel)
Mathematical Construction
ALSC(x; cap, alpha, beta) = cap * [sigma(alpha(x/cap + beta)) - sigma(alpha*beta)]
/ [sigma(alpha*beta) * (1 - sigma(alpha*beta))]
where sigma(z) = 1/(1+e^{-z})
Comparison Table
| Property |
tanh |
soft_sign |
ALSC (novel) |
| Bound |
[-cap, cap] |
(-cap, cap) |
[0, cap] (asymmetric) |
| Gradient at 0 |
1 |
1 |
alpha/4 (tunable) |
| Gradient tail |
exp(-2 |
x |
/cap) |
| Asymmetry |
No |
No |
Yes (alpha, beta control) |
| Invertible |
Yes |
Yes |
Yes (analytic inverse) |
| Zero-centered |
Yes |
Yes |
No (intentional bias) |
Why Novel
Asymmetric capping matches attention's natural asymmetry (queries attend to keys,
not symmetric). The beta shift creates a "dead zone" near zero for sparsity;
alpha controls gradient sharpness independently of bound.
Gradient Properties
d/dx ALSC(x) = (alpha/cap) * sigma'(alpha(x/cap + beta))
/ [sigma(alpha*beta) * (1 - sigma(alpha*beta))]
- At x=0: gradient = alpha/4 (tunable via alpha)
- Tail: exponential decay exp(-alpha|x|/cap) -- tunable rate
- No vanishing gradient problem at moderate inputs (unlike tanh)
Analytic Inverse
ALSC^{-1}(y) = cap * (logit(y/cap * denom + sigma(alpha*beta)) / alpha - beta)
where denom = sigma(alpha*beta) * (1 - sigma(alpha*beta))
Useful for: quantization-aware training, debugging, invertible normalizing flows.
Correctness Conditions (Proof Obligations)
| Obligation |
Statement |
| Cap Bound |
forall x, method, params: |
| Cap Monotonicity |
forall x1 < x2: cap_forward(x1) <= cap_forward(x2) |
| Gradient Consistency |
cap_grad(x) = d/dx cap_forward(x) (verified by autodiff) |
| Softmax Invariant |
l_i = sum_j exp(qk_j - m_i) maintained per block |
| Mask Correctness |
ranker_mask implements: history<->all, candidate<->self-only |
| Backward Match |
jax.grad(unified_attention) approx _unified_bwd (numerical) |
Complexity Analysis
| Metric |
Triton Kernel |
Mosaic Kernel |
| Time |
O(QKVD / (block_q*block_kv)) |
Same with 2x compute overlap |
| Shared Mem |
O(block_qD + block_kvD) |
O(2block_qD + max_concurrentblock_kvD) |
| Registers |
~120 |
~232 (compute WG) / ~40 (memory WG) |
| Occupancy |
Limited by shared mem |
3 WGs/SM, pipeline hides latency |
Novelty Status: POSSIBLY_NOVEL
- No collision found in: "bounded activation functions", "asymmetric tanh alternatives",
"log-sigmoid capping"
- Closest prior art: soft-sign (symmetric), tanh (symmetric), hard-tanh (non-smooth)
- ALSC is the first asymmetric, smooth, invertible cap with independent gradient/shape control
