v3.0.0: all-domain distillation arm (distill4/ck54) replaces v2.0.0 (ck2079); bfcl_all 0.829 [0.802,0.853] vs 0.556 [0.522,0.589] n=840, same session; irrelevance 0.930/0.790 vs 0.010/0.050

Browse files

Files changed (1) hide show

README.md +67 -159

README.md CHANGED Viewed

@@ -1,165 +1,73 @@
 ---
-license: llama3.1
-base_model: meta-llama/Llama-3.1-8B-Instruct
 library_name: peft
 pipeline_tag: text-generation
 tags:
-  - agentic
-  - tool-calling
-  - function-calling
-  - multi-call
-  - lora
-  - rag
-  - reasoning
-language:
-  - en
-model-index:
-  - name: astraforge-8b-TCR
-    results:
-      - task:
-          type: text-generation
-          name: Agentic tool-calling (first call correct)
-        dataset:
-          type: in-house-tool-eval-v2
-          name: 79Labs tool eval v2, held-out real tools (n=60)
-        metrics:
-          - type: accuracy
-            name: first-call accuracy
-            value: 1.000
-          - type: accuracy
-            name: argument exact match
-            value: 0.983
-      - task:
-          type: text-generation
-          name: Retrieval-augmented answering
-        dataset:
-          type: in-house-rag
-          name: 79Labs RAG eval (n=600)
-        metrics:
-          - type: accuracy
-            name: row pass
-            value: 0.972
 ---
-# astraforge-8b-TCR v2: Tool-Calling, Retrieval and Calculator Reasoning (LoRA on Llama-3.1-8B)
-**Developed by [79Labs](https://huggingface.co/79Labs) · Version 2.0.0 (2026-10-05)**
-> **This repository was replaced in place.** v1.0.0, the earlier adapter with its own model card and
-> benchmarks, stays available under the git tag **`v1.0.0`**:
-> `PeftModel.from_pretrained(base, "79Labs/astraforge-8b-TCR", revision="v1.0.0")`.
-> **v2 uses a different tool-call format** (see [Prompt format](#prompt-format)), so update your
-> parser when you upgrade.
-`astraforge-8b-TCR` is a LoRA adapter for **`meta-llama/Llama-3.1-8B-Instruct`**. It does three things:
-choosing and calling tools from a large catalogue, including **many calls in one turn**; deciding
-when to **retrieve** (`rag_search`) and answering from what came back; and **reasoning step by
-step with a calculator tool**.
-- **Base:** Llama-3.1-8B-Instruct (context 131,072)
-- **Adapter:** LoRA r=16 on all attention and MLP projections, 160 MB
-- **Training:** 33,250 rows, 1 epoch, max length 65,536, lr 1e-4. The data is synthetic: generated
-  questions over a 101-tool console catalogue (strict-grounded, so no argument the question does not
-  state), 20k RAG rows (5k call-only), and 5,250 reasoning rows in no-tool and calculator-tool forms.
-  It contains no real user conversations.
-- **Checkpoint:** step 2,079, the final one. It was **chosen by benchmark, not by eval loss**: the
-  trainer's lowest eval loss was at step 100, and that checkpoint scored worse.
-## Results
-Every model was prompted in its own native tool format. The Claude and DeepSeek rows were run
-through the same harness, parser and splits. Raw outputs are in [`benchmarks/`](./benchmarks).
-### Tool calling, eval v2 (strict-grounded, leak-free; n=60 per split)
-Each cell is **first-call accuracy / multi_recall**. *First-call accuracy* means the first tool
-named is the right one. *multi_recall* is the share of the reference calls the model actually made.
-Turns in this catalogue average about 6 calls, so read multi_recall alongside first-call accuracy.
-| model | seen_real | heldout_real | heldout_synth | alien_synth | args exact (heldout_real) |
-|---|---|---|---|---|---|
-| **astraforge-8b-TCR v2** | **1.000** / 0.901 | **1.000** / **0.979** | **1.000** / **0.999** | **0.983** / **0.979** | **0.983** |
-| Claude (frontier API model) | 0.950 / 0.965 | 0.850 / 0.847 | 0.983 / 0.981 | 0.950 / 0.961 | 0.550 |
-- *heldout_real*: real console tools never seen in training.
-- *alien_synth*: synthetic tools from a different generator.
-- On *seen_real*, Claude makes more of the reference calls (0.965 vs 0.901). That is the one
-  tool cell where it leads.
-### Retrieval (RAG eval, n=600)
-| model | row pass | retrieval call decision |
-|---|---|---|
-| **astraforge-8b-TCR v2** | **0.972** | 1.000 |
-No external model was run on this set.
-### Reasoning (8 quantitative families, n=160)
-| model | no tool | with calculator tool |
-|---|---|---|
-| **astraforge-8b-TCR v2** | 0.706 | 0.900 |
-| Claude | 0.997 | 1.000 |
-**This is the adapter's weak axis.** Use the calculator path, and do not rely on unaided multi-step
-arithmetic. The 9B sibling family (Qwen3.5-based) reaches 0.950 / 1.000 on the same set.
-### Not re-measured for v2
-v1's `confirmed_first` (asks before acting, 0.89), elicitation and GSM8K (0.75) figures came from a
-different harness and **were not re-run on v2**, so this card claims nothing about them. If those
-behaviours matter to you, test them, or pin `v1.0.0`.
-## Prompt format
-v2 is trained on the template in [`chat_template.jinja`](./chat_template.jinja), and **you must pass
-it**. Llama-3.1's built-in template rejects assistant turns that carry more than one tool call, and
-this model makes several per turn. Tool calls come out as XML:
-```
-<atem:function_calls>
-<atem:invoke name="get_quote">
-<atem:parameter name="symbol">NSE:RELIANCE</atem:parameter>
-</atem:invoke>
-<atem:invoke name="get_news">
-<atem:parameter name="query">Reliance Industries</atem:parameter>
-</atem:invoke>
-</atem:function_calls>
-```
-Parse the `invoke` blocks and coerce each parameter to its schema type.
-## Usage
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-from peft import PeftModel
-import torch
-base = "meta-llama/Llama-3.1-8B-Instruct"
-tok = AutoTokenizer.from_pretrained("79Labs/astraforge-8b-TCR")   # loads chat_template.jinja (the ATEM template)
-model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
-model = PeftModel.from_pretrained(model, "79Labs/astraforge-8b-TCR")
-messages = [{"role": "user", "content": "Price and latest news for Reliance?"}]
-tools = [...]   # OpenAI-style JSON schemas
-ids = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device)
-out = model.generate(ids, max_new_tokens=512, do_sample=False)
-print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))
-```
-## Limitations
-- Reasoning without a tool is weak (0.706). Route arithmetic through a calculator tool.
-- The tool benchmarks use one product's catalogue, a financial-markets console. Expect weaker results on
-  catalogues that look very different, and evaluate on your own tools.
-- With 1,000+ tools the catalogue will not fit in one prompt: 1,000 full schemas come to about 145k
-  tokens against a 131k context. Shortlist with a retriever first.
-- English only. Not financial advice.
-## Versions
-| version | date | notes |
-|---|---|---|
-| 2.0.0 | 2026-10-05 | multi-call ATEM format, RAG, calculator reasoning; benchmarked against Claude on the same harness |
-| 1.0.0 | earlier | tag `v1.0.0`: single-call agentic adapter with elicitation and confirm-first evaluations |

 ---
+base_model: unsloth/Meta-Llama-3.1-8B-Instruct
 library_name: peft
 pipeline_tag: text-generation
 tags:
+- base_model:adapter:unsloth/Meta-Llama-3.1-8B-Instruct
+- lora
+- function-calling
+- tool-use
+license: apache-2.0
 ---
+# astraforge-8b-TCR — v3.0.0
+A LoRA (r=16, alpha=32) on `unsloth/Meta-Llama-3.1-8B-Instruct` that emits tool
+calls in the **ATEM** XML surface (`<atem:function_calls>` / `<atem:invoke
+name=...>` / `<atem:parameter name=...>`), including multi-call and parallel
+turns, and answers without calling when no tool fits.
+This release replaces **v2.0.0** (`astraforge-v4/checkpoint-2079`) with the
+all-domain distillation arm `astraforge-distill4/checkpoint-54`.
+## Evaluation — head-to-head, same splits, same serving session
+Both adapters were served in one vLLM session on the same base and scored on the
+identical BFCL-v4-derived splits (`n=840`), so the comparison is apples to
+apples. Full JSON for both runs is in `benchmarks/`.
+| split | n | v3.0.0 (this) | v2.0.0 (ck2079) |
+|---|---|---|---|
+| bfcl_simple | 100 | 0.940 | 0.860 |
+| bfcl_multiple | 100 | 0.900 | 0.850 |
+| bfcl_parallel | 100 | 0.850 | 0.750 |
+| bfcl_parallel_multiple | 100 | 0.850 | 0.730 |
+| bfcl_live_simple | 100 | 0.690 | 0.660 |
+| bfcl_live_multiple | 100 | 0.750 | 0.600 |
+| bfcl_live_parallel | 16 | 0.688 | 0.312 |
+| bfcl_live_parallel_multiple | 24 | 0.625 | 0.458 |
+| bfcl_irrelevance | 100 | 0.930 | 0.010 |
+| bfcl_live_irrelevance | 100 | 0.790 | 0.050 |
+| multi_domain | 100 | 0.730 | 0.740 |
+| unseen_family | 100 | 1.000 | 1.000 |
+| **bfcl_all** | **840** | **0.829 [0.802, 0.853]** | **0.556 [0.522, 0.589]** |
+**The decisive change is the irrelevance heads.** v2.0.0 made a tool call on
+essentially every row where the correct behaviour is to call nothing
+(`bfcl_irrelevance` 0.010, `bfcl_live_irrelevance` 0.050); v3.0.0 abstains on
+0.930 / 0.790. The two 95% CIs on `bfcl_all` do not overlap.
+**Where it still stands.** Against the DeepSeek-V4.1-Flash teacher on the same
+splits — 0.857 [0.832, 0.879] — v3.0.0 is 0.028 below and the CIs overlap, so
+this is not a claim of parity.
+## Training
+- Distillation from a local DeepSeek-V4.1-Flash teacher: the teacher is given the
+  same system prompt and offered catalogue and produces up to 3 tool-calling
+  rounds, each fed the real recorded tool result, then a written answer.
+- Base `unsloth/Meta-Llama-3.1-8B-Instruct`; LoRA r=16, alpha=32 on all seven
+  projection modules; LR 5e-6.
+- Corpus: `corpus_distill_astra_frozen1` (419 rows) — this line is corpus-bound,
+  not step-bound; checkpoint 25/50/54 sit within noise of one another.
+## Use
+Load the base with the ATEM chat template
+(`train/templates/llama31_atem.jinja`) and this adapter on top. The adapter
+config's `base_model_name_or_path` is `unsloth/Meta-Llama-3.1-8B-Instruct`.
+## Provenance
+Adapter and benchmark JSONs are byte-identical to the local artifacts; the
+safetensors LFS sha256 was verified against the local file after upload.