File size: 2,863 Bytes
677e207 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | # ALSC: Asymmetric Log-Sigmoid Cap (Novel)
## Mathematical Construction
```
ALSC(x; cap, alpha, beta) = cap * [sigma(alpha(x/cap + beta)) - sigma(alpha*beta)]
/ [sigma(alpha*beta) * (1 - sigma(alpha*beta))]
where sigma(z) = 1/(1+e^{-z})
```
## Comparison Table
| Property | tanh | soft_sign | ALSC (novel) |
|----------|------|-----------|--------------|
| Bound | [-cap, cap] | (-cap, cap) | [0, cap] (asymmetric) |
| Gradient at 0 | 1 | 1 | alpha/4 (tunable) |
| Gradient tail | exp(-2|x|/cap) | 1/x^2 | exp(-alpha|x|/cap) (tunable) |
| Asymmetry | No | No | Yes (alpha, beta control) |
| Invertible | Yes | Yes | Yes (analytic inverse) |
| Zero-centered | Yes | Yes | No (intentional bias) |
## Why Novel
Asymmetric capping matches attention's natural asymmetry (queries attend to keys,
not symmetric). The `beta` shift creates a "dead zone" near zero for sparsity;
`alpha` controls gradient sharpness independently of bound.
## Gradient Properties
```
d/dx ALSC(x) = (alpha/cap) * sigma'(alpha(x/cap + beta))
/ [sigma(alpha*beta) * (1 - sigma(alpha*beta))]
```
- At x=0: gradient = alpha/4 (tunable via alpha)
- Tail: exponential decay exp(-alpha|x|/cap) -- tunable rate
- No vanishing gradient problem at moderate inputs (unlike tanh)
## Analytic Inverse
```
ALSC^{-1}(y) = cap * (logit(y/cap * denom + sigma(alpha*beta)) / alpha - beta)
where denom = sigma(alpha*beta) * (1 - sigma(alpha*beta))
```
Useful for: quantization-aware training, debugging, invertible normalizing flows.
## Correctness Conditions (Proof Obligations)
| Obligation | Statement |
|------------|-----------|
| Cap Bound | forall x, method, params: |cap_forward(x, method, params)| <= params.cap |
| Cap Monotonicity | forall x1 < x2: cap_forward(x1) <= cap_forward(x2) |
| Gradient Consistency | cap_grad(x) = d/dx cap_forward(x) (verified by autodiff) |
| Softmax Invariant | l_i = sum_j exp(qk_j - m_i) maintained per block |
| Mask Correctness | ranker_mask implements: history<->all, candidate<->self-only |
| Backward Match | jax.grad(unified_attention) approx _unified_bwd (numerical) |
## Complexity Analysis
| Metric | Triton Kernel | Mosaic Kernel |
|--------|---------------|---------------|
| Time | O(Q*KV*D / (block_q*block_kv)) | Same with 2x compute overlap |
| Shared Mem | O(block_q*D + block_kv*D) | O(2*block_q*D + max_concurrent*block_kv*D) |
| Registers | ~120 | ~232 (compute WG) / ~40 (memory WG) |
| Occupancy | Limited by shared mem | 3 WGs/SM, pipeline hides latency |
## Novelty Status: POSSIBLY_NOVEL
- No collision found in: "bounded activation functions", "asymmetric tanh alternatives",
"log-sigmoid capping"
- Closest prior art: soft-sign (symmetric), tanh (symmetric), hard-tanh (non-smooth)
- ALSC is the first asymmetric, smooth, invertible cap with independent gradient/shape control
|