Hugging Face's logo

tiny-random
/
flint

Safetensors
sglang
flint
Model card Files
xet
 Community
yujiepan commited on
Commit
e130b21
·
verified ·
1 Parent(s): 70bf47a

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +7 -3
  2. config.json +3 -55
  3. model.safetensors +2 -2
README.md CHANGED
@@ -13,8 +13,8 @@ Structural comparison with `RecursiveSI/flint`:
13
  | Loading implementation | source SGLang plugin | same source plugin wheel |
14
  | Stored precision | BF16 | BF16 |
15
  | Vocabulary / embedding and LM-head ties | 150,144 / untied | same |
16
- | Layers / dense prefix | 49 / layers 0-1 | same |
17
- | Attention cadence / RoPE-free layers | 49-layer source schedule | same |
18
  | Hidden size / Q and KV heads / head dimension | 2,048 / 32 and 4 / 128 | 32 / 1 and 1 / 32 |
19
  | Routed experts / active experts | 256 / 6 | 64 / 6 |
20
  | MoE / shared-expert widths | 512 / 1,024 | 32 / 64 |
@@ -27,7 +27,7 @@ validates TP=1 only, so this model carries no TP=4 claim.
27
 
28
  | File path | Size |
29
  |------|------|
30
- | model.safetensors | 40.1MB |
31
 
32
 
33
  ### Example usage:
@@ -71,10 +71,14 @@ raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=s
71
  # The fixed 150,144-row untied embedding and LM head make hidden_size=32 the
72
  # smallest practical BF16 geometry; one 32-wide attention head retains the
73
  # required LLM head-dimension floor.
 
74
  raw.update(
75
  hidden_size=32, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
76
  intermediate_size=128, moe_intermediate_size=32, shared_expert_intermediate_size=64,
77
  num_experts=64, num_experts_per_tok=6, max_position_embeddings=4096,
 
 
 
78
  )
79
  (folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8")
80
  generator = torch.Generator(device="cpu").manual_seed(42)
 
13
  | Loading implementation | source SGLang plugin | same source plugin wheel |
14
  | Stored precision | BF16 | BF16 |
15
  | Vocabulary / embedding and LM-head ties | 150,144 / untied | same |
16
+ | Layers / dense prefix | 49 / layers 0-1 | 8 / layers 0-1 |
17
+ | Attention cadence / RoPE-free layers | 49-layer source schedule | first 8 source layers |
18
  | Hidden size / Q and KV heads / head dimension | 2,048 / 32 and 4 / 128 | 32 / 1 and 1 / 32 |
19
  | Routed experts / active experts | 256 / 6 | 64 / 6 |
20
  | MoE / shared-expert widths | 512 / 1,024 | 32 / 64 |
 
27
 
28
  | File path | Size |
29
  |------|------|
30
+ | model.safetensors | 21.9MB |
31
 
32
 
33
  ### Example usage:
 
71
  # The fixed 150,144-row untied embedding and LM head make hidden_size=32 the
72
  # smallest practical BF16 geometry; one 32-wide attention head retains the
73
  # required LLM head-dimension floor.
74
+ layers = 8
75
  raw.update(
76
  hidden_size=32, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
77
  intermediate_size=128, moe_intermediate_size=32, shared_expert_intermediate_size=64,
78
  num_experts=64, num_experts_per_tok=6, max_position_embeddings=4096,
79
+ num_hidden_layers=layers,
80
+ layer_types=raw["layer_types"][:layers],
81
+ no_rope_layer_indices=[index for index in raw["no_rope_layer_indices"] if index < layers],
82
  )
83
  (folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8")
84
  generator = torch.Generator(device="cpu").manual_seed(42)
config.json CHANGED
@@ -26,48 +26,7 @@
26
  "sliding_attention",
27
  "full_attention",
28
  "sliding_attention",
29
- "sliding_attention",
30
- "sliding_attention",
31
- "full_attention",
32
- "sliding_attention",
33
- "sliding_attention",
34
- "sliding_attention",
35
- "full_attention",
36
- "sliding_attention",
37
- "sliding_attention",
38
- "sliding_attention",
39
- "full_attention",
40
- "sliding_attention",
41
- "sliding_attention",
42
- "sliding_attention",
43
- "full_attention",
44
- "sliding_attention",
45
- "sliding_attention",
46
- "sliding_attention",
47
- "full_attention",
48
- "sliding_attention",
49
- "sliding_attention",
50
- "sliding_attention",
51
- "full_attention",
52
- "sliding_attention",
53
- "sliding_attention",
54
- "sliding_attention",
55
- "full_attention",
56
- "sliding_attention",
57
- "sliding_attention",
58
- "sliding_attention",
59
- "full_attention",
60
- "sliding_attention",
61
- "sliding_attention",
62
- "sliding_attention",
63
- "full_attention",
64
- "sliding_attention",
65
- "sliding_attention",
66
- "sliding_attention",
67
- "full_attention",
68
- "sliding_attention",
69
- "sliding_attention",
70
- "full_attention"
71
  ],
72
  "max_position_embeddings": 4096,
73
  "mlp_only_layers": [
@@ -79,23 +38,12 @@
79
  "num_attention_heads": 1,
80
  "num_experts": 64,
81
  "num_experts_per_tok": 6,
82
- "num_hidden_layers": 49,
83
  "num_key_value_heads": 1,
84
  "no_rope_layer_indices": [
85
  0,
86
  1,
87
- 5,
88
- 9,
89
- 13,
90
- 17,
91
- 21,
92
- 25,
93
- 29,
94
- 33,
95
- 37,
96
- 41,
97
- 45,
98
- 48
99
  ],
100
  "output_router_logits": false,
101
  "pad_token_id": 150001,
 
26
  "sliding_attention",
27
  "full_attention",
28
  "sliding_attention",
29
+ "sliding_attention"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  ],
31
  "max_position_embeddings": 4096,
32
  "mlp_only_layers": [
 
38
  "num_attention_heads": 1,
39
  "num_experts": 64,
40
  "num_experts_per_tok": 6,
41
+ "num_hidden_layers": 8,
42
  "num_key_value_heads": 1,
43
  "no_rope_layer_indices": [
44
  0,
45
  1,
46
+ 5
 
 
 
 
 
 
 
 
 
 
 
47
  ],
48
  "output_router_logits": false,
49
  "pad_token_id": 150001,
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5ef11a79c57fef7fd870084e5af2fd6f69d0a142d1bfda9f7aeafd205744bbd6
3
- size 40059474
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e38ba949d64adbabd3772099563273468f7f0a428a8274b9ae15b0f18ec03faa
3
+ size 21938168