|
Download README.md from Grawnch/Hy4-Flash-BF16-Reference-Logits: direct link, hf CLI and curl.
- Browser
- Download file 3.74 kB
-
https://huggingface.co/Grawnch/Hy4-Flash-BF16-Reference-Logits/resolve/main/README.md
- Command line
-
hf download hf://Grawnch/Hy4-Flash-BF16-Reference-Logits/README.md
-
curl -L -o README.md https://huggingface.co/Grawnch/Hy4-Flash-BF16-Reference-Logits/resolve/main/README.md
3.74 kB
metadata
license: other
license_name: tencent-hy
license_link: LICENSE
base_model: tencent/Hy4-flash
tags:
- tencent
- hy4-flash
- reference-logits
- quantization
- evaluation
- kld
tencent/Hy4-flash Pure-BF16 Reference Logits & Teacher Hidden States
Exact pure-BF16 teacher reference logits and post-RMSNorm hidden states from tencent/Hy4-flash (220.25B total / 155.5B backbone, fp8_exact Engram, 120,832 vocab) for evaluating quantized checkpoints (KLD, Top-1, Top-5, Delta PPL) without needing 440+ GB of VRAM/disk to run the full BF16 teacher.
Included Files
| File | Size | Contents |
|---|---|---|
blind_val_2k_logits_bf16.safetensors |
3.96 GB |
Direct pre-materialized logits_bf16 [8, 2048, 120832] (bfloat16) + input_ids [8, 2048] (int64). Rows 0:2 = WikiText-2 test prose, 2:4 = Held-Out Code, 4:8 = On-Policy Math/Code CoT. |
blind_val_8k_logits_bf16.safetensors |
3.96 GB |
Direct pre-materialized logits_bf16 [2, 8192, 120832] (bfloat16) + input_ids [2, 8192] (int64, WikiText-2 test 8K long-context). |
factorized_fp32_exact.safetensors |
2.72 GB |
Zero-loss FP32 teacher factorization (logits_f32 = hidden_f32 @ lm_head_weight_bf16.float().T, 0.0 bit error vs full BF16 teacher): blind_val_2k_{ids,hidden_f32}, blind_val_8k_{ids,hidden_f32}, opt_64x2048_{ids,hidden_f32} (131,072 disjoint calibration tokens: 25% C4 + 25% WT2 + 25% Code + 25% CoT), and lm_head_weight_bf16 [120832, 3072]. |
bf16_reference_metadata.json |
< 4 KB |
Verified BF16 PPL baselines, slice boundaries, and tensor shapes. |
tokenizer.json / tokenizer_config.json / chat_template.jinja |
9.4 MB |
Official tencent/Hy4-flash tokenizer files for convenience. |
Verified Pure-BF16 Teacher Baselines
- Blind-Val
2K(8 x 2,048,16,376next-token targets):BF16 PPL = 13.2992(WT2 Prose = 29.2078,Code = 42.3502,Self-Gen Math/Code = 5.1036) - Blind-Val
8K(2 x 8,192,16,382next-token targets):BF16 PPL = 34.8908(Early 0..512 = 11.9276,Mid 4096..4608 = 52.4458,Late 7679..8191 = 33.1118)
Quickstart (safetensors + PyTorch)
Option A: Load Direct [B, S, 120832] BF16 Logits
import torch
import torch.nn.functional as F
from safetensors.torch import load_file
d2k = load_file("blind_val_2k_logits_bf16.safetensors")
input_ids = d2k["input_ids"] # [8, 2048] int64
tea_logits = d2k["logits_bf16"] # [8, 2048, 120832] bfloat16
Option B: Exact FP32 Logits (0.0 Bit Loss, Includes 64 x 2,048 Calibration Set)
import math, torch
import torch.nn.functional as F
from safetensors.torch import load_file
fac = load_file("factorized_fp32_exact.safetensors")
W_lm = fac["lm_head_weight_bf16"].cuda().float() # [120832, 3072]
hid = fac["blind_val_2k_hidden_f32"] # [8, 2048, 3072] float32
ids = fac["blind_val_2k_ids"] # [8, 2048] int64
# Streamed exact FP32 KLD(P_bf16 || P_quant) & Top-1 agreement against your student logits `stu_logits [8, 2048, 120832]`
kld_sum, top1_hits, n_tok = 0.0, 0, 0
for b in range(hid.shape[0]):
for s0 in range(0, hid.shape[1] - 1, 256):
s1 = min(hid.shape[1] - 1, s0 + 256)
t_lg = hid[b, s0:s1].cuda() @ W_lm.T
s_lg = stu_logits[b, s0:s1].cuda().float()
lpt = F.log_softmax(t_lg, dim=-1)
lps = F.log_softmax(s_lg, dim=-1)
kld_sum += float((lpt.exp() * (lpt - lps)).sum().item())
top1_hits += int((t_lg.argmax(-1) == s_lg.argmax(-1)).sum().item())
n_tok += (s1 - s0)
print(f"Blind-Val 2K KLD = {kld_sum / n_tok:.5f} | Top-1 = {100.0 * top1_hits / n_tok:.2f}%")