Grawnch's picture
Publish tencent/Hy4-flash pure-BF16 reference logits (2K + 8K BF16 safetensors + exact FP32 factorized + OPT 64x2048)
cfd1f91 verified
|
Raw History Blame Contribute Delete
3.74 kB
---
license: other
license_name: tencent-hy
license_link: LICENSE
base_model: tencent/Hy4-flash
tags:
- tencent
- hy4-flash
- reference-logits
- quantization
- evaluation
- kld
---
# `tencent/Hy4-flash` Pure-BF16 Reference Logits & Teacher Hidden States
Exact pure-`BF16` teacher reference logits and post-`RMSNorm` hidden states from **`tencent/Hy4-flash`** (`220.25B` total / `155.5B` backbone, `fp8_exact` Engram, `120,832` vocab) for evaluating quantized checkpoints (`KLD`, `Top-1`, `Top-5`, `Delta PPL`) without needing `440+ GB` of VRAM/disk to run the full BF16 teacher.
## Included Files
| File | Size | Contents |
| :--- | :---: | :--- |
| `blind_val_2k_logits_bf16.safetensors` | `3.96 GB` | Direct pre-materialized `logits_bf16 [8, 2048, 120832]` (`bfloat16`) + `input_ids [8, 2048]` (`int64`). Rows `0:2` = WikiText-2 `test` prose, `2:4` = Held-Out Code, `4:8` = On-Policy Math/Code CoT. |
| `blind_val_8k_logits_bf16.safetensors` | `3.96 GB` | Direct pre-materialized `logits_bf16 [2, 8192, 120832]` (`bfloat16`) + `input_ids [2, 8192]` (`int64`, WikiText-2 `test` 8K long-context). |
| `factorized_fp32_exact.safetensors` | `2.72 GB` | **Zero-loss `FP32` teacher factorization** (`logits_f32 = hidden_f32 @ lm_head_weight_bf16.float().T`, `0.0` bit error vs full BF16 teacher): `blind_val_2k_{ids,hidden_f32}`, `blind_val_8k_{ids,hidden_f32}`, `opt_64x2048_{ids,hidden_f32}` (`131,072` disjoint calibration tokens: `25%` C4 + `25%` WT2 + `25%` Code + `25%` CoT), and `lm_head_weight_bf16 [120832, 3072]`. |
| `bf16_reference_metadata.json` | `< 4 KB` | Verified `BF16` PPL baselines, slice boundaries, and tensor shapes. |
| `tokenizer.json` / `tokenizer_config.json` / `chat_template.jinja` | `9.4 MB` | Official `tencent/Hy4-flash` tokenizer files for convenience. |
## Verified Pure-BF16 Teacher Baselines
- **Blind-Val `2K` (`8 x 2,048`, `16,376` next-token targets):** `BF16 PPL = 13.2992` (`WT2 Prose = 29.2078`, `Code = 42.3502`, `Self-Gen Math/Code = 5.1036`)
- **Blind-Val `8K` (`2 x 8,192`, `16,382` next-token targets):** `BF16 PPL = 34.8908` (`Early 0..512 = 11.9276`, `Mid 4096..4608 = 52.4458`, `Late 7679..8191 = 33.1118`)
## Quickstart (`safetensors` + PyTorch)
### Option A: Load Direct `[B, S, 120832]` `BF16` Logits
```python
import torch
import torch.nn.functional as F
from safetensors.torch import load_file
d2k = load_file("blind_val_2k_logits_bf16.safetensors")
input_ids = d2k["input_ids"] # [8, 2048] int64
tea_logits = d2k["logits_bf16"] # [8, 2048, 120832] bfloat16
```
### Option B: Exact `FP32` Logits (`0.0` Bit Loss, Includes `64 x 2,048` Calibration Set)
```python
import math, torch
import torch.nn.functional as F
from safetensors.torch import load_file
fac = load_file("factorized_fp32_exact.safetensors")
W_lm = fac["lm_head_weight_bf16"].cuda().float() # [120832, 3072]
hid = fac["blind_val_2k_hidden_f32"] # [8, 2048, 3072] float32
ids = fac["blind_val_2k_ids"] # [8, 2048] int64
# Streamed exact FP32 KLD(P_bf16 || P_quant) & Top-1 agreement against your student logits `stu_logits [8, 2048, 120832]`
kld_sum, top1_hits, n_tok = 0.0, 0, 0
for b in range(hid.shape[0]):
for s0 in range(0, hid.shape[1] - 1, 256):
s1 = min(hid.shape[1] - 1, s0 + 256)
t_lg = hid[b, s0:s1].cuda() @ W_lm.T
s_lg = stu_logits[b, s0:s1].cuda().float()
lpt = F.log_softmax(t_lg, dim=-1)
lps = F.log_softmax(s_lg, dim=-1)
kld_sum += float((lpt.exp() * (lpt - lps)).sum().item())
top1_hits += int((t_lg.argmax(-1) == s_lg.argmax(-1)).sum().item())
n_tok += (s1 - s0)
print(f"Blind-Val 2K KLD = {kld_sum / n_tok:.5f} | Top-1 = {100.0 * top1_hits / n_tok:.2f}%")
```