Upload folder using huggingface_hub

LICENSE.custom.md

@@ -0,0 +1,149 @@
+# BaiHu-V1-Flash Custom License
+
+**Version 1.0 | Effective date: 2026-09-29**
+
+Copyright (c) 2026 NovaAI6868. All rights reserved.
+
+This License governs the **BaiHu-V1-Flash** model (the "Model"), including its weights,
+configuration, inference code and accompanying documentation. **By downloading, copying,
+installing, invoking or otherwise using the Model, you confirm that you have read,
+understood and agree to be bound by all terms of this License.** If you do not agree,
+stop using the Model immediately and delete all copies.
+
+---
+
+## 1. Definitions
+
+- **"Personal Use"** means use by a natural person for their own learning, research,
+  teaching, experimentation, hobby projects or other non-commercial purposes, where such
+  use does **not** directly or indirectly generate commercial revenue and does **not**
+  provide paid services to third parties based on the Model.
+- **"Commercial Use"** means any use for profit, any use in the business operations of a
+  for-profit entity, or any use that generates commercial revenue. This includes, without
+  limitation: internal production systems of a company, paid APIs or SaaS offered to third
+  parties, integration into paid products, advertising or commercial analytics,
+  client-facing delivery work, and any paid service built on the Model.
+- **"You"** means the natural person or legal entity exercising rights under this License.
+- **"Derivative Model"** means any model obtained by fine-tuning, continued training,
+  distillation, quantization, pruning, merging or otherwise modifying the Model.
+
+## 2. Free Grant: Personal Use
+
+Subject to this License, **Personal Use is free of charge**, with no fee and no prior
+application required. You may:
+
+1. download, install, run and copy the Model;
+2. modify the Model and create or train Derivative Models;
+3. use the Model freely in personal, non-commercial projects, including publicly
+   releasing your research results and demos.
+
+## 3. Paid Grant: Commercial Use
+
+**Any Commercial Use requires prior written commercial authorization.** Commercial Use
+without such authorization is not permitted.
+
+Commercial licenses are negotiated based on scale of use, deployment model and term, and
+may include custom terms. To obtain one, contact:
+
+> **Business contact: novaweb6868@outlook.com**
+>
+> Please briefly state: your entity name, intended use case, expected volume or deployment
+> scale, and whether redistribution rights are required.
+
+Until you and the copyright holder reach a written agreement on commercial licensing,
+this License grants you **no** right of Commercial Use.
+
+## 4. General Restrictions (applies to both Personal and Commercial Use)
+
+Whether your use is Personal Use or commercially licensed, you must not:
+
+1. remove, obscure or alter copyright notices, license identifiers or attribution in the Model;
+2. use the Model or any Derivative Model for any activity that violates applicable laws
+   or regulations;
+3. use the Model to generate or distribute malicious code, fraudulent content, or content
+   that infringes the lawful rights of others;
+4. sublicense the Model or any Derivative Model to third parties under terms that conflict
+   with this License (except where redistribution rights are expressly granted in a
+   commercial license);
+5. claim original authorship of the Model, or make any promise or warranty on behalf of
+   the copyright holder.
+
+## 5. Licensing of Derivative Models
+
+Derivative Models you create **remain subject to this License**; your modifications do not
+remove them from its scope. Sections 2, 3 and 4 therefore apply equally to Derivative
+Models: Personal Use is free, Commercial Use requires a paid license. When distributing a
+Derivative Model you must include the full text of this License and prominently state that
+it is built upon BaiHu-V1-Flash.
+
+## 6. Upstream License and Third-Party Components
+
+The Model is an architectural retrofit and continued-pretraining product of
+`Qwen/Qwen3-0.6B-Base` (Apache License 2.0). This License **only** governs the rights the
+copyright holder holds in the portions newly added to the Model, and **does not alter**
+the original license terms of upstream components. All use of the Model must also comply
+with the licenses applicable to those upstream components. In the event of a conflict,
+the upstream license prevails with respect to the upstream components.
+
+## 7. What Counts as Commercial Use
+
+The following **are** Commercial Use and require authorization (this list is not
+exhaustive):
+
+- use in the business of a company, studio, sole proprietorship or other for-profit
+  entity, even if no fee is charged directly;
+- delivering projects or deliverables that incorporate the Model to clients;
+- use to generate content, products or services sold externally;
+- offering a free service built on the Model while monetizing through advertising,
+  traffic diversion, data monetization or similar.
+
+The following are **generally not** Commercial Use (final determination rests with the
+copyright holder):
+
+- personal study, coursework, or academic research that is publicly published;
+- use by non-profit organizations for non-commercial purposes.
+
+## 8. Disclaimer of Warranty
+
+**The Model is provided "AS IS", without warranty of any kind**, express or implied,
+including but not limited to the warranties of merchantability, fitness for a particular
+purpose, accuracy and non-infringement. You bear all risks and consequences arising from
+your use of the Model. In no event shall the copyright holder be liable for any direct,
+indirect, incidental, special or consequential damages arising from the use of, or
+inability to use, the Model. The Model may produce inaccurate, biased or otherwise
+inappropriate output; you are responsible for evaluating it and for the consequences.
+
+## 9. Termination
+
+If you breach any term of this License, the rights granted to you under it **terminate
+automatically**, without notice. Upon termination you must immediately stop using the
+Model and delete all copies and Derivative Models. Sections 4, 5, 8 and 10 survive
+termination.
+
+## 10. Governing Law and Dispute Resolution
+
+The formation, validity, interpretation and dispute resolution of this License are
+governed by the laws of the copyright holder's jurisdiction. The parties shall first seek
+an amicable resolution of any dispute arising from this License.
+
+## 11. Changes to This License
+
+The copyright holder reserves the right to revise this License from time to time. A
+revised License takes effect upon publication; your continued use of the Model after such
+publication constitutes acceptance of the revised terms.
+
+---
+
+## Quick Summary (for convenience only; the full text above controls)
+
+| Use case | Fee | Notes |
+|---|---|---|
+| Personal study / research / experimentation / hobby | **Free** | No application required |
+| Academic research with public publication | **Free** | Please cite the source |
+| Non-profit, non-commercial use by non-profits | **Free** | No application required |
+| Internal use by a company (even without direct fees) | **License required** | Contact novaweb6868@outlook.com |
+| Paid API / SaaS / product integration | **License required** | Contact novaweb6868@outlook.com |
+| Client-facing deliverables | **License required** | Contact novaweb6868@outlook.com |
+| Distributing fine-tuned / quantized versions | Same license | Free for personal, paid for commercial |
+
+**Commercial licensing contact: novaweb6868@outlook.com**

README.md

@@ -0,0 +1,237 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - zh
+  - en
+license: other
+license_name: baihu-custom-license
+license_link: https://huggingface.co/NovaAI6868/BaiHu-V1-Flash/blob/main/LICENSE.custom.md
+base_model: Qwen/Qwen3-0.6B-Base
+tags:
+  - sparse-attention
+  - subq
+  - ssa
+  - long-context
+  - commercial-license-required
+  - text-generation
+---
+
+# BaiHu-V1-Flash
+
+**BaiHu-V1-Flash** is a retrofit of the dense-attention model `Qwen/Qwen3-0.6B-Base` into an
+**SSA (Sparse-attention + SubQ)** architecture, obtained by continued pretraining.
+
+- Base model: `Qwen/Qwen3-0.6B-Base` (28 layers / 16 Q heads / 8 KV heads / head_dim 128 / 32K context / tied embeddings)
+- Parameters: 598.8M
+- Training data: mixed Chinese + English (Fineweb-Edu-Chinese-V2.1 + fineweb-edu, 50/50)
+- License: **free for personal use; a paid license is required for commercial use** (see "License" below)
+
+---
+
+## 1. Architecture: SSA (three paths, each with its own softmax, then summed)
+
+Every layer keeps the base model's MLP / RMSNorm weights and replaces full attention with an
+SSA layer built from three parallel paths:
+
+| Path | Role | Complexity |
+|---|---|---|
+| `shared` | every query sees all **completed** blocks through one compressed vector per block | `O(T·T/B)` |
+| `local` | dense causal attention over the most recent window | `O(T·w)` |
+| `sparse` (**SubQ**) | only 4 of 16 query heads produce block scores, shared across the whole head group; real attention is computed only for the selected top-k blocks | `O(T·k·B)` |
+
+### Hyperparameters
+
+| Parameter | Value | Meaning |
+|---|---|---|
+| `ssa_block_size` | 64 | block size B |
+| `ssa_top_k` | 8 | number of blocks selected by the sparse path |
+| `ssa_local_blocks` | 2 | local window = 3 × 64 = 192 tokens |
+| `ssa_num_subq_heads` | 4 | SubQ heads, r = 16 / 4 = 4 |
+| `ssa_router_dim` | 128 | router subspace dimension |
+| `ssa_compress_dim` | 128 | block compression dimension |
+
+Only **2.75M** parameters are new (≈0.46% of the model); all other weights are inherited
+from the base model.
+
+---
+
+## 2. Training
+
+| Item | Setting |
+|---|---|
+| Starting point | a conversion checkpoint that is **bit-exact** with the base model (`max\|Δlogit\| = 0.000e+00`) |
+| Tokens seen | 5.0M (≈0.25 epoch of the corpus) |
+| Sequence length | 512 (must be a multiple of `ssa_block_size = 64`) |
+| Effective batch | 8192 tokens (batch 2 × grad_accum 8) |
+| Precision | fp32 |
+| Optimizer | SGD with momentum 0.9 |
+| Learning rate | 5e-4, 50-step warmup, cosine decay to 10% |
+| Hardware | single NVIDIA GTX TITAN X (Maxwell, sm_52, 12.9 GB) |
+| Throughput | ≈274 tokens/s |
+
+### Critical issues found and fixed during this retrofit
+
+Several defects silently break training and are worth documenting:
+
+1. **Both new branches had identically zero gradients (blocking).** To make the converted
+   model bit-exact with the base model, `compress_out` and `router_out` were initialized to
+   exactly zero, and the branches were skipped entirely by a gate. The branch output was
+   therefore always zero, so the back-propagated gradient was also always zero: all 2.75M
+   SSA parameters **stayed frozen for the entire run** and the sparse attention was dead
+   code. The fix is a small non-zero initialization.
+2. **Routing was non-differentiable.** `top-k` produces hard indices, and indexing is not
+   differentiable. If the routing scores are used only to decide *which* blocks to read and
+   never enter the softmax, the gradients of `router_q` / `router_k` are **exactly zero** —
+   the router can never learn to route. The fix is to feed the selected blocks' scores,
+   squashed through `tanh` and gently scaled, into the attention logits as an additive bias.
+3. **The shared summary was a sum, not a mean.** Its magnitude grew linearly with the
+   prefix, and because that branch is injected at full weight (a single-element softmax has
+   probability exactly 1), it swamped the residual stream: hidden states grew from 0.2 to
+   about 7 in layer 0 and to about 1900 by layer 27, and validation loss went 3.56 → 11.38.
+   The fix is to divide by the token count.
+4. **The shared branch needs an explicit gate.** With a single-element softmax the
+   probability is always 1, so the initialization scale of `compress_out` cannot control the
+   injection strength at all (measured: scales from 1e-4 to 0.03 all left the loss at
+   exactly 7.3526). A learnable scalar gate, initialized to a small positive value, lets the
+   optimizer decide how far to open it.
+
+---
+
+## 3. Evaluation
+
+### 3.1 Language modeling perplexity (validation set, identical windows)
+
+| Sequence length | Qwen3-0.6B-Base | BaiHu-V1-Flash |
+|---|---|---|
+| 512 | 3.6726 / ppl 39.354 | 4.4108 / ppl 82.335 |
+| 1024 | 3.4691 / ppl 32.109 | 4.2575 / ppl 70.631 |
+| 2048 | 3.0801 / ppl 21.761 | 3.9213 / ppl 50.467 |
+
+### 3.2 Standard benchmarks (lm-evaluation-harness)
+
+| Task | Qwen3-0.6B-Base | BaiHu-V1-Flash | Delta |
+|---|---|---|---|
+| arc_easy | 0.5550 | 0.6250 | +0.0700 |
+| hellaswag | 0.5350 | 0.5200 | -0.0150 |
+| piqa | 0.7050 | 0.6950 | -0.0100 |
+| winogrande | 0.6300 | 0.6300 | +0.0000 |
+
+### 3.3 Inference compute and resource usage
+
+| Metric | Qwen3-0.6B-Base | BaiHu-V1-Flash |
+|---|---|---|
+| Parameters (M) | 596.0500 | 598.8000 |
+| Prefill peak memory (GB) | 9.6200 | 8.0210 |
+| Generation peak memory (GB) | 9.9120 | 9.0050 |
+| Prefill latency (s) | 0.5030 | 1.4310 |
+| TTFT (ms) | 502.7 | 1431.2 |
+| TPOT (ms) | 36.6 | 80.5 |
+| Decode throughput (tok/s) | 27.3300 | 12.4200 |
+| Attention FLOPs/token (GFLOPs) | 0.1176 | 0.1057 |
+| Attention key accesses vs full attention | 1.0000 | 0.8993 |
+| GPU utilization mean/max (%) | 93.9 | 43.3 |
+| Power mean/max (W) | 179.1 | 124.3 |
+
+Positive findings: peak inference memory is lower (generation 9.005 vs 9.912 GB, −9.2%), and the
+model draws less power because it is not compute-bound.
+
+Negative findings, stated plainly:
+
+- **Decode is 2.2× slower** (12.42 vs 27.33 tok/s) and **prefill is 2.8× slower**
+  (TTFT 1431 vs 503 ms), despite the sparse path reading fewer keys. The current
+  implementation loops over query blocks in Python and issues many small kernels, so
+  launch overhead dominates the FLOPs saved. **The sparse attention does not yet pay off
+  on this hardware.**
+- **Attention key accesses are still 89.9% of full attention** at this sequence length.
+  The reason is structural: the local window already covers 3 blocks (192 tokens) and the
+  sparse path reads up to `top_k + 1 = 9` blocks from a grid that only has 16 blocks at
+  1024 tokens, so the selected set is almost the whole grid. Sparsity only becomes a real
+  saving once the sequence is long relative to `top_k × block_size` (i.e. well beyond
+  10k tokens).
+- **Language modeling perplexity is clearly worse than the base model** at every length
+  tested (ppl 82.3 vs 39.4 at 512; 50.5 vs 21.8 at 2048). This is the honest cost of
+  shrinking the dense local window from the full prefix to 192 tokens while the new
+  long-range branches are still very weakly trained.
+- **Standard benchmarks are roughly neutral but not better**: arc_easy improves
+  (+0.070 acc_norm), winogrande is unchanged, while hellaswag (−0.015) and piqa (−0.010)
+  regress slightly.
+
+### 3.4 Sparsity
+
+Two measurements are reported because they answer different questions and are **not**
+interchangeable:
+
+| Scenario | Average blocks selected per query | Key access ratio vs full attention |
+|---|---|---|
+| Chunked forward over a 512-token validation window | 0.38 | 0.0938 |
+| Prefill of 1024 tokens (steady state) | up to `top_k` | 0.8993 |
+
+The first number averages over *all* query blocks including the early ones, which have no
+completed blocks available to select and therefore read nothing through the sparse path.
+The second is the steady-state ratio for later queries, and it is the one that matters for
+efficiency — see the note in section 3.3: at these sequence lengths the sparse path is not
+yet saving meaningful work.
+
+---
+
+## 4. Known Limitations
+
+1. **Trained for very little.** Only 5.0M tokens (≈0.25 epoch). The new branches have not
+   converged; ppl is well above the base model and decode is slower (see 3.3). Continuing
+   to 200M tokens or more is required before the SSA layers can genuinely take over
+   long-range modeling.
+2. **The shared branch does not appear to help and was actively suppressed by the
+   optimizer.** Its learnable gate *decreased* over training (0.0100 → 0.0129 at step 100 →
+   0.0122 at step 610) instead of growing, meaning the optimizer found the single
+   prefix-mean summary not worth injecting. This is the single most important thing to
+   change next: replace it with **one compressed vector per block** (same `O(T·T/B)` cost,
+   far more information retained).
+3. **Sparse attention is not yet a net win on this hardware.** It reads fewer keys but runs
+   2.2× slower because the implementation loops over query blocks in Python and issues many
+   small kernels. It needs kernel-level batching (or a fused implementation) before the
+   sparsity can translate into speed.
+4. **Sparsity only pays off at long sequences.** With `top_k=8` and `block_size=64`, the
+   sparse path can read up to 9 blocks = 576 tokens; at 1024 tokens the grid only has 16
+   blocks, so the selected set covers most of the context and the local window already
+   covers the rest. Real savings require sequences well beyond 10k tokens.
+5. **Routing quality is not fully validated.** The distribution of selected top-k blocks
+   should be checked for degeneration (e.g. always selecting the same blocks). The
+   non-differentiable-routing bug that would have made this *impossible* to learn has been
+   fixed (see section 2), but the learned policy has not been analyzed in detail.
+6. **Block size B = 64 was not ablated.** Limited by memory and the Windows WDDM watchdog on
+   this machine; B ∈ {32, 64, 128} should be swept on a larger GPU.
+7. **Document-boundary packing.** The current packing strategy places multiple documents in
+   one sequence (separated by EOS).
+
+---
+
+## 5. License
+
+**Free for personal use; a paid license is required for commercial use.**
+
+- ✅ Personal study, research, teaching, hobby projects: **free**, no application required
+- ✅ Academic research with public publication: **free** (please cite the source)
+- 💰 Internal company/studio use, paid API/SaaS, product integration, client deliverables:
+  **commercial license required**
+
+**Commercial licensing contact: novaweb6868@outlook.com**
+
+Full terms: [LICENSE.custom.md](./LICENSE.custom.md).
+
+This model is an architectural retrofit of `Qwen/Qwen3-0.6B-Base` (Apache License 2.0).
+This license governs only the newly added portions and does not alter the upstream
+component's original license.
+
+---
+
+## 6. Citation
+
+```bibtex
+@misc{baihu-v1-flash,
+  title  = {BaiHu-V1-Flash: An SSA (Sparse-attention + SubQ) Retrofit of Qwen3-0.6B-Base},
+  author = {NovaAI6868},
+  year   = {2026},
+  url    = {https://huggingface.co/NovaAI6868/BaiHu-V1-Flash}
+}
+```

config.json

@@ -0,0 +1,79 @@
+{
+  "architectures": [
+    "BaiHuSSAForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "baihu_ssa",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "ssa_block_size": 64,
+  "ssa_component_init": 0.01,
+  "ssa_component_seed": 1234,
+  "ssa_compress_dim": 128,
+  "ssa_donor_model": "Qwen3-0.6B-Base",
+  "ssa_donor_revision": null,
+  "ssa_force_full_window": false,
+  "ssa_local_blocks": 2,
+  "ssa_num_subq_heads": 4,
+  "ssa_router_bias_scale": 0.1,
+  "ssa_router_dim": 128,
+  "ssa_router_init_scale": 0.01,
+  "ssa_shared_init_scale": 0.01,
+  "ssa_shared_kv": true,
+  "ssa_top_k": 8,
+  "ssa_use_router_key": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": false,
+  "eos_token_id": 151643,
+  "max_new_tokens": 2048,
+  "transformers_version": "4.37.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40f5a9f413e8ce888f0e44e8bf3c18acd9d711cf2d0c76482096d43ec473c4c6
+size 2395267832

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set content = message.content %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in message.content %}\n                {%- set content = message.content.split('</think>')[-1].lstrip('\\n') %}\n                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}