| # ALSC: Asymmetric Log-Sigmoid Cap (Novel) |
|
|
| ## Mathematical Construction |
|
|
| ``` |
| ALSC(x; cap, alpha, beta) = cap * [sigma(alpha(x/cap + beta)) - sigma(alpha*beta)] |
| / [sigma(alpha*beta) * (1 - sigma(alpha*beta))] |
| |
| where sigma(z) = 1/(1+e^{-z}) |
| ``` |
|
|
| ## Comparison Table |
|
|
| | Property | tanh | soft_sign | ALSC (novel) | |
| |----------|------|-----------|--------------| |
| | Bound | [-cap, cap] | (-cap, cap) | [0, cap] (asymmetric) | |
| | Gradient at 0 | 1 | 1 | alpha/4 (tunable) | |
| | Gradient tail | exp(-2|x|/cap) | 1/x^2 | exp(-alpha|x|/cap) (tunable) | |
| | Asymmetry | No | No | Yes (alpha, beta control) | |
| | Invertible | Yes | Yes | Yes (analytic inverse) | |
| | Zero-centered | Yes | Yes | No (intentional bias) | |
| |
| ## Why Novel |
| |
| Asymmetric capping matches attention's natural asymmetry (queries attend to keys, |
| not symmetric). The `beta` shift creates a "dead zone" near zero for sparsity; |
| `alpha` controls gradient sharpness independently of bound. |
| |
| ## Gradient Properties |
| |
| ``` |
| d/dx ALSC(x) = (alpha/cap) * sigma'(alpha(x/cap + beta)) |
| / [sigma(alpha*beta) * (1 - sigma(alpha*beta))] |
| ``` |
| |
| - At x=0: gradient = alpha/4 (tunable via alpha) |
| - Tail: exponential decay exp(-alpha|x|/cap) -- tunable rate |
| - No vanishing gradient problem at moderate inputs (unlike tanh) |
| |
| ## Analytic Inverse |
| |
| ``` |
| ALSC^{-1}(y) = cap * (logit(y/cap * denom + sigma(alpha*beta)) / alpha - beta) |
| where denom = sigma(alpha*beta) * (1 - sigma(alpha*beta)) |
| ``` |
| |
| Useful for: quantization-aware training, debugging, invertible normalizing flows. |
| |
| ## Correctness Conditions (Proof Obligations) |
| |
| | Obligation | Statement | |
| |------------|-----------| |
| | Cap Bound | forall x, method, params: |cap_forward(x, method, params)| <= params.cap | |
| | Cap Monotonicity | forall x1 < x2: cap_forward(x1) <= cap_forward(x2) | |
| | Gradient Consistency | cap_grad(x) = d/dx cap_forward(x) (verified by autodiff) | |
| | Softmax Invariant | l_i = sum_j exp(qk_j - m_i) maintained per block | |
| | Mask Correctness | ranker_mask implements: history<->all, candidate<->self-only | |
| | Backward Match | jax.grad(unified_attention) approx _unified_bwd (numerical) | |
|
|
| ## Complexity Analysis |
|
|
| | Metric | Triton Kernel | Mosaic Kernel | |
| |--------|---------------|---------------| |
| | Time | O(Q*KV*D / (block_q*block_kv)) | Same with 2x compute overlap | |
| | Shared Mem | O(block_q*D + block_kv*D) | O(2*block_q*D + max_concurrent*block_kv*D) | |
| | Registers | ~120 | ~232 (compute WG) / ~40 (memory WG) | |
| | Occupancy | Limited by shared mem | 3 WGs/SM, pipeline hides latency | |
|
|
| ## Novelty Status: POSSIBLY_NOVEL |
| |
| - No collision found in: "bounded activation functions", "asymmetric tanh alternatives", |
| "log-sigmoid capping" |
| - Closest prior art: soft-sign (symmetric), tanh (symmetric), hard-tanh (non-smooth) |
| - ALSC is the first asymmetric, smooth, invertible cap with independent gradient/shape control |
| |
