astraforge-8b-TCR β€” v3.0.0

A LoRA (r=16, alpha=32) on unsloth/Meta-Llama-3.1-8B-Instruct that emits tool calls in the ATEM XML surface (<atem:function_calls> / <atem:invoke name=...> / <atem:parameter name=...>), including multi-call and parallel turns, and answers without calling when no tool fits.

This release replaces v2.0.0 (astraforge-v4/checkpoint-2079) with the all-domain distillation arm astraforge-distill4/checkpoint-54.

Evaluation β€” head-to-head, same splits, same serving session

Both adapters were served in one vLLM session on the same base and scored on the identical BFCL-v4-derived splits (n=840), so the comparison is apples to apples. Full JSON for both runs is in benchmarks/.

split n v3.0.0 (this) v2.0.0 (ck2079)
bfcl_simple 100 0.940 0.860
bfcl_multiple 100 0.900 0.850
bfcl_parallel 100 0.850 0.750
bfcl_parallel_multiple 100 0.850 0.730
bfcl_live_simple 100 0.690 0.660
bfcl_live_multiple 100 0.750 0.600
bfcl_live_parallel 16 0.688 0.312
bfcl_live_parallel_multiple 24 0.625 0.458
bfcl_irrelevance 100 0.930 0.010
bfcl_live_irrelevance 100 0.790 0.050
multi_domain 100 0.730 0.740
unseen_family 100 1.000 1.000
bfcl_all 840 0.829 [0.802, 0.853] 0.556 [0.522, 0.589]

The decisive change is the irrelevance heads. v2.0.0 made a tool call on essentially every row where the correct behaviour is to call nothing (bfcl_irrelevance 0.010, bfcl_live_irrelevance 0.050); v3.0.0 abstains on 0.930 / 0.790. The two 95% CIs on bfcl_all do not overlap.

Where it still stands. Against the DeepSeek-V4.1-Flash teacher on the same splits β€” 0.857 [0.832, 0.879] β€” v3.0.0 is 0.028 below and the CIs overlap, so this is not a claim of parity.

Training

  • Distillation from a local DeepSeek-V4.1-Flash teacher: the teacher is given the same system prompt and offered catalogue and produces up to 3 tool-calling rounds, each fed the real recorded tool result, then a written answer.
  • Base unsloth/Meta-Llama-3.1-8B-Instruct; LoRA r=16, alpha=32 on all seven projection modules; LR 5e-6.
  • Corpus: corpus_distill_astra_frozen1 (419 rows) β€” this line is corpus-bound, not step-bound; checkpoint 25/50/54 sit within noise of one another.

Use

Load the base with the ATEM chat template (train/templates/llama31_atem.jinja) and this adapter on top. The adapter config's base_model_name_or_path is unsloth/Meta-Llama-3.1-8B-Instruct.

Provenance

Adapter and benchmark JSONs are byte-identical to the local artifacts; the safetensors LFS sha256 was verified against the local file after upload.

Downloads last month
27
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for 79Labs/astraforge-8b-TCR