XHToken

/

Spark-X2.5-4B

Like 1.44k

Follow

SparkLLM 1.33k

Architectural Proposal: KV-BSS (Key-Value Binding Softmax Sharpening)

Hi Spark team and community,

We have been conducting extensive latent manifold and attention audits on Spark-X2.5-4B across its 36 layers (particularly studying sliding-window vs. full-attention bridge layers 3, 7, 11, 15, 19, 23, 27, 31, 35).

During long-context code synthesis and structured associative mapping (e.g. ["key"] => "value", AST binding, function signature-to-call matching), we observed a phenomenon where diffuse low-magnitude attention logits create an "attention haze" that degrades the hallucination threshold.

To address this, we developed and empirically validated KV-BSS (Key-Value Binding Softmax Sharpening).

1. Mathematical Formulation

In standard multi-head / grouped-query attention:
$$A_{\text{logits}}=\frac{Q{K}^T}{\sqrt{d_k}}A\_\{\backslash text\{logits\}\}\; =\; \backslash frac\{Q\; K^T\}\{\backslash sqrt\{d\_k\}\}$$

KV-BSS applies two targeted regularizations:

1. Focus Factor Scaling ((\tau_{\text{focus}} = 1.10)):
   $$A_{\text{logits}}=\frac{Q{K}^T}{\sqrt{d_k}}\times {\tau }_{\text{focus}}A\_\{\backslash text\{logits\}\}\; =\; \backslash frac\{Q\; K^T\}\{\backslash sqrt\{d\_k\}\}\; \backslash times\; \backslash tau\_\{\backslash text\{focus\}\}$$
   A modest 10% steepening of the logit geometry sharpens peak Softmax probability around exact antecedent bindings without causing numerical instability.

2. Attention Haze Truncation:
   
   Any attention logit falling more than 12.0 nats below the active sequence maximum is strictly clipped to $-\infty$. Because $\exp(-12) \approx 6.14 \times 10^{-6}$, these logits carry zero semantic signal but collectively dilute probability mass over large context windows.

2. Implementation in modeling_spark.py

The modification requires only a few lines in Spark2_5Attention:

# In Spark2_5Attention.__init__:
kv_focus = getattr(config, "kv_focus_factor", 1.10) if getattr(config, "key_value_binding_sharpening", False) else 1.0
self.scaling = float(kv_focus) / math.sqrt(self.head_dim)

# In eager_attention_forward (before Softmax):
if getattr(self.config, "attention_sink_suppression", True):
    max_logits = torch.amax(attn_weights, dim=-1, keepdim=True)
    haze_mask = attn_weights < (max_logits - 12.0)
    attn_weights = attn_weights.masked_fill(haze_mask, float("-inf"))

3. Empirical Results & Live Proof-of-Concept

We validated KV-BSS in combination with our DV-SSQ (Dense-Vectorized Subspace Salience Quantization) pipeline on Spark-X2.5-4B over a 119-token recursive algorithmic execution sequence.

A fully working public model checkpoint incorporating this architecture, complete with safetensors weights and benchmark suite, is available here:
🔗 F-Labs/Spark-X2.5-4B-Hadamard-GSQ

We would love to hear the authors' thoughts on adopting or standardizing KV-BSS for future Spark-X releases!