Instructions to use PTTREP/appworld-native-standard-qwen3-4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use PTTREP/appworld-native-standard-qwen3-4b with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
AppWorld native ReAct agent: Qwen3-4B LoRA trained with the original (standard) API docs
15 LoRA adapters on Qwen/Qwen3-4B: 5 scenario-level cross-validation folds ร 3 training seeds. Each adapter was trained on the teacher episodes of the tasks outside its fold and evaluated on the tasks inside it. No adapter was trained on all 147 tasks.
These are the "standard docs" control of a study on how the format of tool documentation (NL prose / JSON / Python
stubs) affects what an agent learns. Code and design: branch cross-rep-agents-native of the
GitHub repo zhenwu0831/PTTREP (private; docs/AGENT_HANDOFF.md).
Setup
- Agent protocol: AppWorld's own ReAct code agent: a Python REPL, API discovery via
apis.api_docs.*, 50 steps, AppWorld's default prompt (2 demo API docs included). - Training data: gpt-5.4 (medium reasoning) episodes on AppWorld train+dev (147 tasks, 49 scenarios). The teacher read the untouched AppWorld docs; 141 successful episodes passed replay checks. Each fold trains on about 113 episodes and validates on about 28.
- Training: loss on the agent's actions only (observations and prompt are context); LoRA r=16, alpha=32, dropout
0.05 on all attention and MLP projections; 3 epochs, lr 1e-4 cosine, effective batch 1, max length 40,960 tokens,
bf16, thinking disabled (
enable_thinking: false). Each folder holds the epoch-3 (final) adapter. - Folds:
folds.jsonlists the held-out task ids of every fold.
Layout
fold{k}/seed{s}/ adapter_model.safetensors, adapter_config.json, tokenizer files,
train_config.json, train_manifest.json (data hashes, losses), length_filter.json
folds.json held-out task ids per fold
Results
Task goal completion (TGC, %) on each adapter's held-out tasks, one sample per task at temperature 0.7. Columns are the doc format shown at test time: standard (AppWorld's own docs), or the same content rendered as NL, STRUCT (JSON) or CODE (Python stubs).
| fold | seed | held-out tasks | standard | NL | STRUCT | CODE |
|---|---|---|---|---|---|---|
| 0 | 0 | 30 | 16.7 | 20.0 | 13.3 | 26.7 |
| 0 | 1 | 30 | 16.7 | 13.3 | 20.0 | 20.0 |
| 0 | 2 | 30 | 20.0 | 16.7 | 20.0 | 20.0 |
| 1 | 0 | 30 | 26.7 | 30.0 | 30.0 | 33.3 |
| 1 | 1 | 30 | 40.0 | 36.7 | 23.3 | 43.3 |
| 1 | 2 | 30 | 36.7 | 50.0 | 36.7 | 33.3 |
| 2 | 0 | 30 | 6.7 | 20.0 | 13.3 | 13.3 |
| 2 | 1 | 30 | 6.7 | 16.7 | 20.0 | 26.7 |
| 2 | 2 | 30 | 3.3 | 6.7 | 3.3 | 3.3 |
| 3 | 0 | 30 | 23.3 | 40.0 | 30.0 | 40.0 |
| 3 | 1 | 30 | 33.3 | 23.3 | 40.0 | 33.3 |
| 3 | 2 | 30 | 36.7 | 26.7 | 26.7 | 40.0 |
| 4 | 0 | 27 | 14.8 | 37.0 | 29.6 | 29.6 |
| 4 | 1 | 27 | 18.5 | 25.9 | 18.5 | 22.2 |
| 4 | 2 | 27 | 33.3 | 18.5 | 22.2 | 22.2 |
Pooled over folds and seeds: 22.2 / 25.4 / 23.1 / 27.2. For reference, the untrained Qwen3-4B scores about 3%, and the same recipe trained on the NL / STRUCT / CODE renderings scores about 11% on those formats.
Run-to-run variance is large: per (fold, seed) scores range from 3% to 50%, and folds differ in difficulty. Compare adapters only within a fold.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, sub = "PTTREP/appworld-native-standard-qwen3-4b", "fold0/seed0"
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, repo, subfolder=sub)
With vLLM: download a folder and serve it with --enable-lora --max-lora-rank 16 --lora-modules ckpt=<local folder>.
The adapters expect AppWorld's ReAct prompt and observations formatted as in training (Output:\n```\n...\n```` user turns); see cross_rep/native/agent.pyandscripts/appworld_native_eval.py` in the code branch.
- Downloads last month
- -