Blame for README.md

---
model_name: PhAI-IDE-9B
base_model: Qwen/Qwen3.5-9B
base_model_relation: finetune
datasets:
- AItonomy/ScienceIDE
library_name: transformers
pipeline_tag: image-text-to-text
license: apache-2.0
tags:
- phai-ide
- science
- code
- tool-use
- sft
- lora
- safetensors
---

# PhAI-IDE

**PhAI-IDE** is a family of models for scientific coding and interaction with tools, available in **4B, 9B, and 72B** sizes. Each model is supervised fine-tuned with [ms-swift](https://github.com/modelscope/ms-swift) and released as full BF16 weights with the final LoRA adapter merged, together with its configuration and tokenizer.

The **training dataset is Codex trajectories**, sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE).

## Models

| Model | Base model | BF16 weights | License |
| --- | --- | ---: | --- |
| [PhAI-IDE-4B](https://huggingface.co/AItonomy/PhAI-IDE-4B) | [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | 9.08 GB | Apache-2.0 |
| [PhAI-IDE-9B](https://huggingface.co/AItonomy/PhAI-IDE-9B) | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | 18.82 GB | Apache-2.0 |
| [PhAI-IDE-72B](https://huggingface.co/AItonomy/PhAI-IDE-72B) | [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) | 145.41 GB | [Qwen](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/495f39366efef23836d0cfae4fbe635880d2be31/LICENSE) |

Weight sizes are approximate; inference also requires memory for runtime allocations and the KV cache.

## ScienceAccelBench performance

**PhAI-IDE-9B** improves the task-pooled pass rate from **17.09% to 20.89% (+3.80 percentage points)** across all **27 ScienceAccelBench environments**, using **158 valid paired tasks**.

Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Each row compares the initial model with its corresponding fine-tuned model on identical tasks. Pass rates are percentages; gains are percentage points.

| Model | Environment | Tasks | Initial model | PhAI-IDE | Gain (pp) |
| --- | --- | ---: | ---: | ---: | ---: |
| 4B | PLUTO-Particles-Dust | 3 | 0.00 | **33.33** | **+33.33** |
| 9B | LAPS | 16 | 31.25 | **50.00** | **+18.75** |
| 9B | MITgcm-biogeo | 8 | 0.00 | **12.50** | **+12.50** |
| 9B | PLUTO-RMHD | 7 | 0.00 | **28.57** | **+28.57** |

## Comparison with published models

Scores (%), grouped by benchmark and model size. Each reference entry gives its published score and the **PhAI-IDE score difference in percentage points**. Reference models are approximately the same size: 3–4B, 7–9B, and 67–72B, respectively.

| PhAI-IDE | Benchmark | Score | Reference models: score (difference) |
| --- | --- | ---: | --- |
| 4B | BBH multistep-arithmetic-two | **97.60** | [Llama-3.2-3B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.21B): 53.2 (**+44.40**); [Phi-3.5-mini-8k-instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.82B): 95.6 (**+2.00**) |
| 9B | BBH word-sorting | **60.40** | [Llama-3.1-8B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (8.03B): 51.2 (**+9.20**); [Qwen2.5-7B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (7.62B): 15.6 (**+44.80**) |
| 9B | MATH-500 | **92.20** | [InternLM3-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 83 (**+9.20**); [Qwen2.5-7B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (7B): 72.4 (**+19.80**); [Llama-3.1-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 48.4 (**+43.80**) |
| 72B | AQuA-RAT | **77.56** | [Llama-2-70B-Chat](https://openreview.net/pdf?id=FvfhHucpLd) (70B): 31.32 (**+46.24**) |
| 72B | ARC-Easy | **84.64** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 76.5 (**+8.14**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 81.6 (**+3.04**) |
| 72B | ARC-Challenge | **64.42** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 59.5 (**+4.92**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 64.1 (**+0.32**) |

Reference scores come from the linked publications, model cards, and independent evaluation reports; evaluation settings and sample counts vary by source. Differences describe reported scores across evaluations, rather than matched-protocol head-to-head gains. BBH entries refer to the named tasks.

## Quick start

Use Transformers 5.16.1, PyTorch and Accelerate. Set `model_id` to any model in the table above; the example selects the matching model class.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForImageTextToText

model_id = "AItonomy/PhAI-IDE-4B"
loader = AutoModelForCausalLM if model_id.endswith("72B") else AutoModelForImageTextToText
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = loader.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain how to verify a numerical simulation."}],
    add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Training procedure

ScienceIDE demonstrations were collected with **GPT-5.6-sol** and filtered using a **numerical-equivalence verifier**. They capture code inspection, tool use, and responses to execution feedback.

All three models use **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges each final checkpoint's adapter into its base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The trajectories retain the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.

| Shared setting | Value |
| --- | --- |
| Training dataset | Codex trajectories |
| Training examples / tasks | 4,567 segments / 564 tasks |
| Validation examples / tasks | 544 segments / 81 tasks |
| Train/validation task overlap | 0 |
| Training epochs | 3 |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| Released weights | LoRA merged into BF16 Safetensors |

Long trajectories are organized into segments. Source partition assignments are preserved, with no task identifiers shared between training and validation.

For **4B and 9B**, fine-tuning uses text trajectories with the vision tower and aligner frozen. Their additional settings are:

| Setting | 4B and 9B |
| --- | --- |
| Maximum training sequence length | 36,864 tokens |
| Learning rate / schedule | 2e-5 / cosine |
| Warmup ratio | 0.03 |
| Batch size per device / accumulation | 1 / 16 |
| LoRA targets | Linear modules in the unfrozen language model |
| Optimizer steps | 216 |
| Packing | Disabled |

### Framework versions

The release was validated with the following environment.

| Component | Version |
| --- | --- |
| Python | 3.11 |
| ms-swift | 4.5.3 |
| Transformers | 5.16.1 |
| PyTorch | 2.6.0+cu124 |
| PEFT | 0.20.0 |
| Datasets | 4.8.4 |
| Tokenizers | 0.23.2 |
| Accelerate | 1.14.0 |

## Citations

The training framework is [ms-swift](https://github.com/modelscope/ms-swift), and adapter training uses [LoRA](https://arxiv.org/abs/2106.09685).

```bibtex
@misc{zhao2024swift,
  title = {SWIFT: A Scalable lightWeight Infrastructure for Fine-Tuning},
  author = {Yuze Zhao and others},
  year = {2024},
  eprint = {2408.05517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05517}
}

@misc{hu2021lora,
  title = {LoRA: Low-Rank Adaptation of Large Language Models},
  author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
  year = {2021},
  eprint = {2106.09685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.09685}
}
```