Add model card

.gitattributes

@@ -34,3 +34,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+media/SWE-rebench-V2-pipeline.png filter=lfs diff=lfs merge=lfs -text
+media/glm47flash_throughput.png filter=lfs diff=lfs merge=lfs -text
+media/learned_patterns.png filter=lfs diff=lfs merge=lfs -text
+media/overlay_sandbox.png filter=lfs diff=lfs merge=lfs -text
+media/reasoning-tool-calls-logprobs.png filter=lfs diff=lfs merge=lfs -text
+media/score_vs_cost.png filter=lfs diff=lfs merge=lfs -text
+media/trace_overview.png filter=lfs diff=lfs merge=lfs -text
+media/traffic_recorder.png filter=lfs diff=lfs merge=lfs -text
+media/train_loss_decomposition.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,288 @@
+---
+license: mit
+base_model: zai-org/GLM-4.7-Flash
+datasets:
+- whitecircle/swe-rebench-v2-glm-5.1-pi-agent-successful-traces
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# Introducing ⚡ GLM-4.7-Flash-Coder ⚡
+
+We fine-tuned GLM-4.7-Flash to push its agentic coding capabilities further: from 33.15% to 41.65% on SWE-rebench-V2 (+26% relative improvement) at 14% lower inference cost.
+
+This 30B Mixture-of-Experts model beats competitors 3.5-4× its size — Qwen3.5-122B-A10B and Devstral-2 — and surpasses Ling-2.6-Flash by a narrow margin.
+
+![Score vs cost-per-task scatter plot on SWE-rebench-V2, 500 Python tasks: GLM-4.7-Flash-Coder at 0.417 sits 26% above GLM-4.7-Flash at 0.332, at a lower cost per task](media/score_vs_cost.png)
+
+## More reasoning, fewer turns
+
+Although GLM-4.7-Flash-Coder does 2.4× more reasoning than the base model, it completes tasks faster: 45% fewer turns, which translates to 14% lower inference cost.
+
+![Box plots for base vs SFT model: median reasoning tokens per response 63 vs 150, assistant turns per task 82.5 vs 44, trace token length 52k vs 39k, average cost per task $0.0443 vs $0.0382](media/trace_overview.png)
+
+## The key to turn efficiency is concurrent tool calls
+
+![Frequency of responses with 2+ read, bash, and edit tool calls: the SFT model issues simultaneous tool calls 6–90× more often than the base model](media/simultaneous_tool_calls.png)
+
+## SWE-rebench-V2
+
+We trained and evaluated models on [nebius/SWE-rebench-V2](https://huggingface.co/datasets/nebius/SWE-rebench-V2) — a dataset of 32k agentic coding tasks. It consists of real issues from public repositories, closely representing real-world SWE challenges.
+
+![SWE-rebench-V2 pipeline: a repository and an issue enter an agentic loop in a sandbox, the generated patch is checked by the test-suite verifier, and the reward is 1 if all tests pass](media/SWE-rebench-V2-pipeline.png)
+
+## Teacher — GLM-5.1
+
+The training data comes from GLM-5.1 acting as a teacher. We ran it on the train split with 4 rollouts per task and kept only trajectories that passed the tests — 7,777 successful multi-turn traces after rejection sampling.
+
+We’re releasing the [training dataset](https://huggingface.co/datasets/whitecircle/swe-rebench-v2-glm-5.1-pi-agent-successful-traces) alongside the [train/test splits](https://huggingface.co/datasets/whitecircle/swe-rebench-v2-clean-python-tasks).
+
+All models run inside PI — a lightweight agent harness with four tools: `bash`, `read`, `edit`, `write`.
+
+## Training
+
+We trained Coder with Supervised Fine-Tuning (SFT) using [Halo](https://github.com/whitecircle/halo), our post-training framework we’re open-sourcing today 🎉
+
+- Parallelism mode: DP=16 (multi-node training)
+- Context length: 75k
+- Packing: Enabled
+- Global batch: 16
+- Learning rate scheduling: Cosine, with linear warm-up
+- Training time: 3 hours on 16 B300 GPUs
+
+<details open>
+<summary><h3>Halo 😇 is driven entirely by a single YAML config</h3></summary>
+
+```yaml
+model_name_or_path: zai-org/GLM-4.7-Flash
+trust_remote_code: true
+attn_implementation: flash_attention_4_hybrid
+
+dataset:
+- whitecircle/swe-rebench-v2-glm-5.1-pi-agent-successful-traces
+test_size: 0.01
+
+conversation_field: messages
+tools_field: tools
+interleaved_thinking: true
+
+max_length: 75000
+packing: true
+
+learning_rate: 1e-5
+lr_scheduler_type: cosine
+warmup_steps: 25
+per_device_train_batch_size: 1
+per_device_eval_batch_size: 1
+
+moe_balancing: none
+liger_kernel_config:
+  fused_linear_cross_entropy: true
+
+assistant_message_template: "<|assistant|>"
+
+output_dir: /mnt/research/GLM-4.7-Flash-Coder
+save_steps: 50
+save_only_model: true
+
+eval_strategy: steps
+eval_steps: 50
+logging_steps: 1
+logging_first_step: true
+log_decoded_samples: true
+
+report_to: wandb
+project_name: agentic-sft
+run_name: glm-4.7-flash-coder
+
+enable_efficiency_metrics: true
+```
+
+</details>
+
+## Emergent skills
+
+The model picked up several of the teacher’s skills and patterns during fine-tuning:
+
+- A 10× higher rate of self-corrections in reasoning,
+- Self-directed questions (“But wait, could this ever be True?”),
+- Advanced tool calling patterns, e.g. setting timeout for the `bash` tool.
+
+![Bar charts of self-corrections, question marks in reasoning, git stash usage, and timeouts passed to bash across base, SFT, and teacher models](media/learned_patterns.png)
+
+Perhaps the most surprising transfer was the `git stash` usage:
+
+> *Hmm, actually these look like pre-existing test differences unrelated to my fix. Let me revert my fix and run the test again.*
+>
+> ```bash
+> git stash && python -m pytest tests/test_markdown.py::test_markdown_table -xvs 2>&1 | tail -20
+> ```
+
+## Loss decomposition into Reasoning & Tool Calls
+
+We only train on assistant responses. Of 177 million tokens in the dataset, only 46 million carry loss — the remaining 131 million (system prompts and tool responses) are masked.
+
+Assistant messages are split into two parts — reasoning and tool calls. Naturally, tool call tokens have much lower entropy. These parts are very different in nature:
+
+![A training trace rendered with per-token probabilities: reasoning tokens in green, tool call tokens in orange, lighter shades meaning higher probability](media/reasoning-tool-calls-logprobs.png)
+
+Although tool call tokens make up 43.3% of input tokens, they only contribute to 16.1% of overall loss. To track convergence separately, we decompose loss into two components — negative log-likelihood on reasoning and tool calling tokens, respectively.
+
+![Train and eval loss curves decomposed into reasoning and tool-call components with epoch boundaries; every eval loss reaches its global minimum at step 300](media/train_loss_decomposition.png)
+
+Training loss charts show 10-20% epoch cliffs, indicating benign memorization of repeated data.
+
+Although it’s possible to set different learning rates for two components, we observed that on our data, eval losses reach their global minima simultaneously — at the end of epoch 2.
+
+## Test scores
+
+Below is the comparison of GLM-4.7-Flash, GLM-4.7-Flash-Coder, and its teacher GLM-5.1 on test set of SWE-rebench-V2:
+
+| metric | GLM-4.7-Flash | GLM-4.7-Flash-Coder | Δ | Teacher GLM-5.1 |
+| --- | --- | --- | --- | --- |
+| pass@1 | 0.3315 | 0.4165 | +25.6% | 0.5650 |
+| pass@2 | 0.4087 | 0.5023 | +22.9% | 0.6063 |
+| pass@4 | 0.4620 | 0.5720 | +23.8% | 0.6300 |
+
+## How we enabled FA4
+
+FlashAttention-4 is a fast attention kernel — but it’s incompatible with GLM-4.7-Flash’s architecture:
+
+> *FlashAttention-4 produces NaN gradients for this model’s head_dim-256 partial-rotary attention; falling back to `attn_implementation='sdpa'`.*
+
+We found that the incompatibility only appears in the backward pass — the forward computation is correct. So we designed a hybrid scheme: FA4 handles the forward pass, while FA2 takes over for the backward. The hybrid approach gave a 1.36× increase in attention throughput.
+
+*We’ll have a separate post on FA4-fwd-FA2-bwd in [our research blog](https://whitecircle.com/#research); stay tuned! 💚*
+
+## Kernels
+
+GLM-4.7-Flash is a Mixture-of-Experts model. Tokens are distributed unevenly across experts — that means MLP blocks receive tensors of varying width.
+
+The Grouped-GEMM kernel batches matrix multiplications of varying shapes into a single kernel launch. Enabling it gave a 1.6× speedup on the FFN step.
+
+Additionally, with Liger Fused Linear Cross-Entropy, we were able to compute loss without full logit matrix materialization in VRAM, preventing OOM on long sequences.
+
+Together, these improvements yield up to a 1.63× training speedup over TRL, raising throughput to 11.1k tokens/s on a single B300 GPU on context length 4k:
+
+![GLM-4.7-Flash SFT throughput on a single B300 across context lengths from 4k to 128k: Halo FA4-hybrid leads Halo FA2 and TRL at every length; TRL OOMs at 128k](media/glm47flash_throughput.png)
+
+## Agentic infra challenge
+
+We faced a real challenge with agentic evaluation infrastructure.
+
+At a scale of thousands of concurrent agents, Docker introduces high startup overhead and heavy disk pressure, and quickly saturates the local network — resulting in a high rate of environment startup errors.
+
+Kubernetes struggled at high concurrency on a single host as well: it couldn’t set up and tear down pods fast enough, triggering timeouts. The problem was worst on powerful hosts: they could support enormous concurrency, but pod scheduling couldn’t keep up, so we never reached full CPU or RAM utilization.
+
+## Overlay sandboxes
+
+Our solution is the Overlay Sandbox: instead of a container image, the agent runs on the bare host, in an isolated filesystem and process group.
+
+It has three layers: a base filesystem with the OS and repository, a hidden layer with the verifier suite, and an upper layer holding all edits and pip-installed packages.
+
+Because overlay mounts are lazy (they don’t even index the filesystem!) and only track modified files, sandbox startup is essentially free.
+
+![Overlay sandbox diagram: a read-only base filesystem, hidden verifier files not visible to the agent, and a writable lazy overlay mount holding edits and new files](media/overlay_sandbox.png)
+
+Since SWE-rebench-V2 is built from public repositories, agents must run without network access — otherwise, a model can simply fetch a real fix from GitHub. Before restricting network, we caught Fable 5 and Sonnet 5 doing exactly this — retrieving the upstream patches directly on 20-40% of tasks.
+
+## Traffic recorder
+
+Our sandbox also includes a traffic recorder — a tiny proxy that forwards requests to the provider (OpenRouter / vLLM / SGLang) and records every response. It’s a harness-agnostic way to assemble full trajectories in OpenAI format and collect performance metrics.
+
+![Traffic recorder diagram: a proxy between the agent and the provider records raw requests and assembles them into an OpenAI-format trace](media/traffic_recorder.png)
+
+## Evaluation
+
+Every agent runs with `max_turns = 200`, medium reasoning effort, and default sampling parameters.
+
+Inference costs are calculated from OpenRouter pricing tables. We report the theoretical price under the assumption of perfect caching — actual OpenRouter costs were higher due to inconsistent routing. For locally served GLM-4.7-Flash-Coder, we applied the per-request pricing of GLM-4.7-Flash via OpenRouter’s DeepInfra provider.
+
+<details>
+<summary>OpenRouter providers per model</summary>
+<br>
+
+| model | provider | input $/Mtok | output $/Mtok | cache_read $/Mtok | cache_write $/Mtok |
+| --- | --- | --- | --- | --- | --- |
+| Fable-5 | google-vertex/global | 10 | 50 | 1 | 12.5 |
+| Opus-4.8 | anthropic | 5 | 25 | 0.5 | 6.25 |
+| GLM-5.1 | wafer/fp4 | 1 | 3.2 | 0.1 |  |
+| MiniMax-M3 | minimax/fp8 | 0.3 | 1.2 | 0.06 |  |
+| DeepSeek-V4-Pro | deepinfra/fp4 | 1.3 | 2.6 | 0.1 |  |
+| GPT-5.5 | azure | 5 | 30 | 0.5 |  |
+| Kimi-K2.6 | siliconflow/fp8 | 0.77 | 3.4 | 0.14 |  |
+| Haiku-4.5 | anthropic | 1 | 5 | 0.1 | 1.25 |
+| Qwen3.5-397B-A17B | deepinfra/fp8 | 0.45 | 3 | 0.22 |  |
+| GPT-5.4-Mini | azure | 0.75 | 4.5 | 0.075 |  |
+| Gemini-3.1-Pro | google-ai-studio | 2 | 12 | 0.2 | 0.375 |
+| GLM-4.7-Flash-Coder | deepinfra/bf16 | 0.06 | 0.4 | 0.01 |  |
+| Ling-2.6-Flash | novita | 0.1 | 0.3 | 0.02 |  |
+| Devstral-2 | mistral | 0.4 | 2 | 0.04 |  |
+| Qwen3.5-122B-A10B | alibaba | 0.4 | 3.2 |  |  |
+| GLM-4.7-Flash | deepinfra/bf16 | 0.06 | 0.4 | 0.01 |  |
+| Gemini-3.5-Flash | google-vertex/global | 1.5 | 9 | 0.15 | 0.08333 |
+| Qwen3-32B | deepinfra/fp8 | 0.08 | 0.28 |  |  |
+| Qwen3.6-35B-A3B | parasail/fp8 | 0.15 | 1 | 0.05 |  |
+| Gemini-2.5-Flash | google-vertex/global | 0.3 | 2.5 | 0.03 | 0.08333 |
+| Nemotron-3-Super | digitalocean | 0.21 | 0.455 | 0.06 |  |
+| Solar-Pro-3 | upstage | 0.15 | 0.6 | 0.015 |  |
+
+</details>
+
+<details>
+<summary>Prompt template</summary>
+<br>
+
+```markdown
+You are solving a real GitHub issue in the `{repo}` repository. The repo is already cloned and set up in the current working directory.
+
+## Workflow
+
+1. **Understand the issue.** Read the problem statement carefully. Identify the expected vs actual behavior.
+2. **Explore the codebase.** Find the relevant files using grep, find, or by reading the project structure. Read the code that needs to change.
+3. **Reproduce the bug** if possible — run the relevant test file or write a small script to confirm the issue exists.
+4. **Implement a minimal fix.** Change only what is necessary. Do not refactor unrelated code, do not add features beyond what the issue asks for.
+5. **Verify your fix.** Run the project's existing test suite to make sure you haven't introduced regressions. Look at the project's test configuration (Makefile, tox.ini, pyproject.toml, CI config) to find the right test command.
+6. **Mark the task complete** only after you've verified existing tests pass with your changes.
+
+## Issue
+
+{problem_statement}
+
+## Expected Interfaces
+
+The following interfaces are expected by the test suite. Pay close attention to function names, parameter signatures, and return types — the tests verify these exactly.
+
+{interface}
+```
+
+</details>
+
+<details>
+<summary>Inference with SGLang 0.5.14, CUDA 13</summary>
+<br>
+
+```bash
+python3 -m sglang_router.launch_server \
+  --model-path whitecircle/GLM-4.7-Flash-Coder \
+  --trust-remote-code \
+  --host 0.0.0.0 --port 30000 \
+  --context-length 202752 \
+  --prefill-attention-backend trtllm_mla \
+  --decode-attention-backend triton \
+  --speculative-draft-attention-backend triton \
+  --page-size 64 \
+  --tool-call-parser glm47 --reasoning-parser glm45 \
+  --router-policy cache_aware \
+  --tp-size 1 --dp-size 8
+```
+
+</details>
+
+## Team
+
+- Project lead: Oleg Smirnov
+- Agentic sandboxes: Konstantin Korolev, Sergei Bratchnikov
+- Contributors: Mohamed Mekkouri, Dmitrii Kharlapenko, Daniil Oblakov
+
+A [White Circle](https://whitecircle.com) project