README.md

---
language:
  - hi
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - indic
  - small-language-model
  - qwen3
  - tool-calling
  - function-calling
  - banking
---

# NextToken-model-2 — SAM (Small Action Model)

A 272.7M-parameter Hindi-English model fine-tuned into a **small action
model** for banking/NBFC customer service: given a user utterance and a
system prompt listing the available tools, it decides whether one applies
and extracts the arguments.

This checkpoint (`sep19_round1`, end of epoch 1) replaces `sep18_round2`.
**Updated in place** — this repository tracks the current best checkpoint;
check back for updates.

## What changed vs. `sep18_round2`

Same 5-tool + no-tool scope, but a substantially larger and more
categorized training set: **15,894 examples** (14,196 train / 1,698 val),
organized into 8 explicit buckets — 5 tools, `negative` (with 4
sub-categories: chitchat, definitional, info, statement), `no_tool`, and a
**`general_capability` bucket** (non-banking text: continuation,
summarization, lists, creative writing) specifically to keep the model
useful outside the banking domain and reduce over-firing a tool on
unrelated prompts.

**Early-stopped again, same lesson as before**: trained to epoch 1 (this
checkpoint), then epoch 2 showed the same overfitting signature already
seen in this project — val loss rose (1.5426 → 1.6514) while train loss
kept falling — so training was stopped before epoch 3. The published
weights are the end-of-epoch-1 checkpoint, not a later one.

## Verified accuracy — held-out 80-example eval, by category

This is the training team's own held-out evaluation (`eval_epoch1_results.json`,
never trained on), broken down here rather than reported as one aggregate
number:

| Bucket | n | JSON valid | Action correct | Tool correct | Args exact |
|---|---|---|---|---|---|
| get_loan_balance | 8 | 100% | 100% | 100% | 100% |
| get_next_due_date | 8 | 100% | 100% | 100% | 100% |
| get_payment_status | 8 | 100% | 100% | 75% | 75% |
| get_recent_transactions | 8 | 100% | 100% | 100% | 100% |
| get_transaction_details | 8 | 100% | 100% | 100% | 100% |
| general_capability | 8 | 100% | 100% | — | — |
| negative_definitional | 8 | 100% | 100% | — | — |
| negative_info | 8 | 100% | 100% | — | — |
| **negative_chitchat** | 8 | 100% | **62.5%** | — | — |
| **negative_statement** | 8 | 100% | **0%** | — | — |
| **Overall** | **80** | **100%** | **86.2%** | **95.0%** (of 40 tool-target rows) | **95.0%** |

**`negative_statement` is a real, systematic failure, not noise**: all 8
examples in this bucket are past-tense/statement framings (e.g. "just
checking in about my home loan"), and the model fired the identical
`get_loan_balance`/`get_payment_status`-style tool call on every single
one instead of giving a natural-language acknowledgment — the same
"inconsistent negative-suppression for past-tense/third-person phrasing"
limitation flagged on the previous checkpoint's card, now precisely
quantified rather than described qualitatively.

Independently re-verified here (not just taking the training team's
numbers on faith): re-ran two real training examples (one `get_loan_balance`
positive, one `negative_info`) through the converted HF checkpoint —
both reproduced their exact target string.

## Architecture

Same base architecture as `somasekhar-dev/NextToken-model-1`:

| | |
|---|---|
| Parameters | 272.7M |
| Attention | GQA — 16 query heads, 4 KV heads |
| Normalization | RMSNorm + per-head QK-norm |
| FFN | SwiGLU |
| Context length | 2048 |
| Tokenizer | [`sarvamai/sarvam-1`](https://huggingface.co/sarvamai/sarvam-1), vocab 68,096 |

## Usage

Tokenize each message segment separately and concatenate — do not
re-tokenize a merged prompt string (a real train/inference tokenization
mismatch was found and fixed earlier in this project's history):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("somasekhar-dev/NextToken-model-2")
tokenizer = AutoTokenizer.from_pretrained("somasekhar-dev/NextToken-model-2")

system_prompt = (
    "You are a banking assistant for an Indian NBFC. Today's date is 2026-09-19 (Saturday).\n"
    "Given the user's message, decide whether one of the tools below applies. If it does, "
    "call it: reply with a short natural-language line if appropriate, followed by a JSON "
    "object {\"tool\": \"<name>\", \"arguments\": {...}} on its own line. Only use arguments "
    "that are stated in the conversation or can be computed from today's date -- never invent "
    "a value. If no tool applies, reply in natural language only, or with {\"tool\": null} if "
    "a bare JSON response is expected.\n\n"
    "Available tools:\n"
    '{"name": "get_loan_balance", "description": "Outstanding principal on a loan", '
    '"parameters": {"type": "object", "properties": {"loan_type": {"type": "string", '
    '"enum": ["home", "car", "personal", "gold", "business"]}}, "required": []}}'
)
user_utterance = "मेरे लोन का कितना बकाया है, कृपया मुझे बताएं।"

segments = [f"System: {system_prompt}\n", f"User: {user_utterance}\n", "Assistant: "]
ids = []
for seg in segments:
    ids.extend(tokenizer.encode(seg, add_special_tokens=False))
ids_t = torch.tensor([ids])

out = model.generate(ids_t, max_new_tokens=60, do_sample=False,
                      eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0][ids_t.shape[1]:], skip_special_tokens=True))
# -> {"tool": "get_loan_balance", "arguments": {}}
```

## Training

Full fine-tune (not LoRA), 15,894 examples (`kashyap/sep19/raw/*.jsonl`),
8 buckets (5 tools, negative x4 sub-types, no_tool, general_capability),
val loss 4.8757 -> 1.5426 by end of epoch 1. Stopped before epoch 3 on
detecting overfitting (see above).

## Status and limitations

- **`negative_statement` framing is unreliable — 0% on the held-out eval.**
  Do not rely on this checkpoint to correctly ignore past-tense/statement
  mentions of a loan or account; it is likely to fire a tool call anyway.
- **`negative_chitchat` is inconsistent (62.5%)** — casual conversational
  openers sometimes still trigger a tool call.
- Tool scope is deliberately narrow: 5 tools + no-tool. The full 41-tool
  schema (`model2_tools.json`) does not fit this model's 2048-token
  context at all (3,359 tokens alone) — this is an architectural
  constraint, not just a training choice.
- Free-text argument values and multi-turn slot-filling are not covered
  by this round's eval — treat as unverified until tested.
- **Any state-changing action should be confirmed before executing**,
  not trusted blindly from a single model output.
- Conversion to this `Qwen3ForCausalLM` checkpoint was verified
  numerically (100% argmax agreement vs. the original training-format
  checkpoint on random inputs, a full save/reload round-trip, and two
  real training examples reproduced exactly).