Custom KDA adapter: decay bound, L2 epsilon, and beta dtype

#2
by Omar161000 - opened

The custom adapter at revision cae62774492b589eea4aab2d6d36fb167a67d90c differs from the reference KDA calculations:

  • The recurrent update omits the configured kda_lower_bound.
  • The L2 normalization epsilon differs from the reference.
  • The beta logits reach sigmoid in the input dtype instead of float32.

The attached five-edit patch addresses these calculations. Stock mlx-lm already contains them. The package selects its custom adapter through model_file.

The synthetic float32 CPU comparison uses 32 heads, 128 dimensions, nine tokens, and two input scales. Output differences are zero. Cache and split-state comparisons pass their tolerances. The environment uses MLX 0.32.2 and mlx-lm 0.32.1.dev4+g53b9af378.

Patch, portable comparison script, and results.

Full-model, BF16, PyTorch, and Metal equivalence remain unproved. The separate output-gate cast also needs a BF16 comparison. TensorFold Zig has no Ling backend at 88c424e; the native branch cannot validate this adapter.

Sign up or log in to comment