| --- |
| library_name: sglang |
| base_model: |
| - RecursiveSI/flint |
| --- |
| |
| This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint). |
|
|
| Structural comparison with `RecursiveSI/flint`: |
|
|
| | Feature | Source | Tiny | |
| |---|---:|---:| |
| | Loading implementation | source SGLang plugin | same source plugin wheel | |
| | Stored precision | BF16 | BF16 | |
| | Vocabulary / embedding and LM-head ties | 150,144 / untied | same | |
| | Layers / dense prefix | 49 / layers 0-1 | 8 / layers 0-1 | |
| | Attention cadence / RoPE-free layers | 49-layer source schedule | first 8 source layers | |
| | Hidden size / Q and KV heads / head dimension | 2,048 / 32 and 4 / 128 | 32 / 1 and 1 / 32 | |
| | Routed experts / active experts | 256 / 6 | 64 / 6 | |
| | MoE / shared-expert widths | 512 / 1,024 | 32 / 64 | |
| | Sigmoid router, top-k renormalization, learned sinks, QK norms | present | same checkpoint schema | |
|
|
| Direct checkpoint generation is required because Transformers does not implement |
| Flint. Tensor names and shapes come from the pinned source safetensors headers |
| and the source-provided `flint_sglang` loader wheel. The source plugin explicitly |
| validates TP=1 only, so this model carries no TP=4 claim. |
|
|
| | File path | Size | |
| |------|------| |
| | model.safetensors | 21.9MB | |
|
|
|
|
| ### Example usage: |
|
|
| - SGLang (unverified) |
|
|
| - The source Flint plugin validates one GPU per replica only; it explicitly marks tensor/expert/pipeline parallelism as unvalidated. The tiny checkpoint therefore does not claim TP=4 support. |
|
|
| - Install the source-provided plugin wheel before this command. Its learned attention sinks, sigmoid-router top-k, shared expert, tool grammar and reasoning parser are not implemented by stock SGLang. |
|
|
| ```bash |
| pip install ./sglang/flint_sglang-0.1.4-py3-none-any.whl |
| SGLANG_EXTERNAL_MODEL_PACKAGE=flint_sglang FLASHINFER_USE_CUDA_NORM=1 \ |
| python -m sglang.launch_server \ |
| --model-path tiny-random/flint --served-model-name flint --tp-size 1 \ |
| --attention-backend fa3 --moe-runner-backend triton --enable-fp32-lm-head \ |
| --tool-call-parser flint --reasoning-parser flint --grammar-backend flint |
| ``` |
|
|
| ### Codes to create this repo: |
|
|
| <details><summary>Click to expand</summary> |
|
|
| ```python |
| import json |
| from pathlib import Path |
| from shutil import copyfile |
| |
| import torch |
| from huggingface_hub import hf_hub_download |
| from safetensors.torch import save_file |
| |
| source_model_id = "RecursiveSI/flint" |
| save_folder = "/tmp/tiny-random/flint" |
| source_revision = "2a247be0c0aec895ccb0ac7ca92229ec6df37288" |
| folder = Path(save_folder) |
| folder.mkdir(parents=True, exist_ok=True) |
| raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text()) |
| # Transformers has no Flint model class. The source SGLang plugin is the |
| # authoritative executable schema, so create its public checkpoint tensors directly. |
| # The fixed 150,144-row untied embedding and LM head make hidden_size=32 the |
| # smallest practical BF16 geometry; one 32-wide attention head retains the |
| # required LLM head-dimension floor. |
| layers = 8 |
| raw.update( |
| hidden_size=32, head_dim=32, num_attention_heads=1, num_key_value_heads=1, |
| intermediate_size=128, moe_intermediate_size=32, shared_expert_intermediate_size=64, |
| num_experts=64, num_experts_per_tok=6, max_position_embeddings=4096, |
| num_hidden_layers=layers, |
| layer_types=raw["layer_types"][:layers], |
| no_rope_layer_indices=[index for index in raw["no_rope_layer_indices"] if index < layers], |
| ) |
| (folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8") |
| generator = torch.Generator(device="cpu").manual_seed(42) |
| |
| def random_tensor(*shape: int) -> torch.Tensor: |
| return torch.randn(shape, generator=generator, dtype=torch.bfloat16).mul_(0.02) |
| |
| hidden = raw["hidden_size"] |
| head_dim = raw["head_dim"] |
| experts = raw["num_experts"] |
| state: dict[str, torch.Tensor] = { |
| "model.embed_tokens.weight": random_tensor(raw["vocab_size"], hidden), |
| "lm_head.weight": random_tensor(raw["vocab_size"], hidden), |
| "model.norm.weight": torch.ones(hidden, dtype=torch.bfloat16), |
| } |
| dense_layers = set(raw["mlp_only_layers"]) |
| for layer in range(raw["num_hidden_layers"]): |
| prefix = f"model.layers.{layer}" |
| state[f"{prefix}.input_layernorm.weight"] = torch.ones(hidden, dtype=torch.bfloat16) |
| state[f"{prefix}.post_attention_layernorm.weight"] = torch.ones(hidden, dtype=torch.bfloat16) |
| state[f"{prefix}.self_attn.q_proj.weight"] = random_tensor(hidden, hidden) |
| state[f"{prefix}.self_attn.k_proj.weight"] = random_tensor(head_dim, hidden) |
| state[f"{prefix}.self_attn.v_proj.weight"] = random_tensor(head_dim, hidden) |
| state[f"{prefix}.self_attn.o_proj.weight"] = random_tensor(hidden, hidden) |
| state[f"{prefix}.self_attn.q_norm.weight"] = torch.ones(head_dim, dtype=torch.bfloat16) |
| state[f"{prefix}.self_attn.k_norm.weight"] = torch.ones(head_dim, dtype=torch.bfloat16) |
| state[f"{prefix}.self_attn.softmax_offset"] = random_tensor(raw["num_attention_heads"]) |
| if layer in dense_layers: |
| for projection in ("gate_proj", "up_proj"): |
| state[f"{prefix}.mlp.{projection}.weight"] = random_tensor(raw["intermediate_size"], hidden) |
| state[f"{prefix}.mlp.down_proj.weight"] = random_tensor(hidden, raw["intermediate_size"]) |
| continue |
| state[f"{prefix}.mlp.gate.weight"] = random_tensor(experts, hidden) |
| for projection in ("gate_proj", "up_proj"): |
| state[f"{prefix}.mlp.shared_expert.{projection}.weight"] = random_tensor( |
| raw["shared_expert_intermediate_size"], hidden |
| ) |
| state[f"{prefix}.mlp.shared_expert.down_proj.weight"] = random_tensor( |
| hidden, raw["shared_expert_intermediate_size"] |
| ) |
| for expert in range(experts): |
| expert_prefix = f"{prefix}.mlp.experts.{expert}" |
| for projection in ("gate_proj", "up_proj"): |
| state[f"{expert_prefix}.{projection}.weight"] = random_tensor(raw["moe_intermediate_size"], hidden) |
| state[f"{expert_prefix}.down_proj.weight"] = random_tensor(hidden, raw["moe_intermediate_size"]) |
| save_file(state, folder / "model.safetensors", metadata={"format": "pt"}) |
| for filename in ( |
| "LICENSE", "chat_template.jinja", "generation_config.json", "tokenizer.json", "tokenizer_config.json", |
| "sglang/flint_sglang-0.1.4-py3-none-any.whl", "sglang/serve_h200.sh", "sglang/README.md", |
| ): |
| destination = folder / filename |
| destination.parent.mkdir(parents=True, exist_ok=True) |
| copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), destination) |
| ``` |
|
|
| </details> |
|
|
| ### Test environment: |
|
|
| - huggingface_hub: 1.27.0 |
| - safetensors: 0.8.0 |
| - tokenizers: 0.23.0rc0 |
| - torch: 2.13.0+cu126 |
| - transformers: 5.15.0.dev0 |
| |
| ### Change log: |
| |
| - 2026-10-23: Initial version. |
| |
| This repo is generated by GPT-6.1-Sol |
