AppWorld native ReAct agent: Qwen3-4B LoRA trained with the original (standard) API docs

15 LoRA adapters on Qwen/Qwen3-4B: 5 scenario-level cross-validation folds ร— 3 training seeds. Each adapter was trained on the teacher episodes of the tasks outside its fold and evaluated on the tasks inside it. No adapter was trained on all 147 tasks.

These are the "standard docs" control of a study on how the format of tool documentation (NL prose / JSON / Python stubs) affects what an agent learns. Code and design: branch cross-rep-agents-native of the GitHub repo zhenwu0831/PTTREP (private; docs/AGENT_HANDOFF.md).

Setup

  • Agent protocol: AppWorld's own ReAct code agent: a Python REPL, API discovery via apis.api_docs.*, 50 steps, AppWorld's default prompt (2 demo API docs included).
  • Training data: gpt-5.4 (medium reasoning) episodes on AppWorld train+dev (147 tasks, 49 scenarios). The teacher read the untouched AppWorld docs; 141 successful episodes passed replay checks. Each fold trains on about 113 episodes and validates on about 28.
  • Training: loss on the agent's actions only (observations and prompt are context); LoRA r=16, alpha=32, dropout 0.05 on all attention and MLP projections; 3 epochs, lr 1e-4 cosine, effective batch 1, max length 40,960 tokens, bf16, thinking disabled (enable_thinking: false). Each folder holds the epoch-3 (final) adapter.
  • Folds: folds.json lists the held-out task ids of every fold.

Layout

fold{k}/seed{s}/   adapter_model.safetensors, adapter_config.json, tokenizer files,
                   train_config.json, train_manifest.json (data hashes, losses), length_filter.json
folds.json         held-out task ids per fold

Results

Task goal completion (TGC, %) on each adapter's held-out tasks, one sample per task at temperature 0.7. Columns are the doc format shown at test time: standard (AppWorld's own docs), or the same content rendered as NL, STRUCT (JSON) or CODE (Python stubs).

fold seed held-out tasks standard NL STRUCT CODE
0 0 30 16.7 20.0 13.3 26.7
0 1 30 16.7 13.3 20.0 20.0
0 2 30 20.0 16.7 20.0 20.0
1 0 30 26.7 30.0 30.0 33.3
1 1 30 40.0 36.7 23.3 43.3
1 2 30 36.7 50.0 36.7 33.3
2 0 30 6.7 20.0 13.3 13.3
2 1 30 6.7 16.7 20.0 26.7
2 2 30 3.3 6.7 3.3 3.3
3 0 30 23.3 40.0 30.0 40.0
3 1 30 33.3 23.3 40.0 33.3
3 2 30 36.7 26.7 26.7 40.0
4 0 27 14.8 37.0 29.6 29.6
4 1 27 18.5 25.9 18.5 22.2
4 2 27 33.3 18.5 22.2 22.2

Pooled over folds and seeds: 22.2 / 25.4 / 23.1 / 27.2. For reference, the untrained Qwen3-4B scores about 3%, and the same recipe trained on the NL / STRUCT / CODE renderings scores about 11% on those formats.

Run-to-run variance is large: per (fold, seed) scores range from 3% to 50%, and folds differ in difficulty. Compare adapters only within a fold.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, sub = "PTTREP/appworld-native-standard-qwen3-4b", "fold0/seed0"
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, repo, subfolder=sub)

With vLLM: download a folder and serve it with --enable-lora --max-lora-rank 16 --lora-modules ckpt=<local folder>. The adapters expect AppWorld's ReAct prompt and observations formatted as in training (Output:\n```\n...\n```` user turns); see cross_rep/native/agent.pyandscripts/appworld_native_eval.py` in the code branch.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for PTTREP/appworld-native-standard-qwen3-4b

Finetuned
Qwen/Qwen3-4B
Adapter
(1,267)
this model