Card: training recipe and considerations

README.md

@@ -238,12 +238,97 @@ GPU. Generating the rationale adds roughly 20–25 tokens.
 - As with any language model, decisions can be wrong; keep a fallback for high-stakes actions (payments, account
   changes, cancellations).
 
-## Training data
-
-Fine-tuned on a multilingual mix of the public datasets listed in this card's metadata (intent routing, typed
-decisions, NLI, NER, slot and function-argument extraction, explanation data), each used under its own licence, plus
-proprietary conversational routing data from Ringg AI, which is not released. Evaluation rows above are disjoint from
-the training rows. Belebele and the Kev suites were never used in training.
+## Training
+
+### Data
+
+A multitask mix in one shared format (system prompt per task, JSON input, JSON answer):
+
+| task family | public sources |
+|---|---|
+| Intent routing | MASSIVE (multilingual), Banking77, CLINC-OOS, Bitext customer support, Hindi prompt routing, Hinglish-TOP |
+| Typed decisions (choice / yes-no-unknown / score) | Open-Jev, jev-bench, jev-distill, tasksource-jev, typed-decisions-synth |
+| NLI and yes/no, English + 10 Indic languages | IndicXNLI, BoolQ-Indic, BoolQ, MultiNLI |
+| Explanations | e-SNLI, ECQA |
+| Entity extraction | Naamapadam, HiNER, MultiCoNER v2 |
+| Slots, function selection and arguments | SGD, MASSIVE-Agents, BFCL-Hi, ToolACE, Hermes JSON mode, xLAM irrelevance, X-RiSAWOZ |
+| Extractive QA | IndicQA |
+
+On top of these, Ringg's own conversational routing data (not released) teaches the voice-agent setting: transcribed
+multilingual calls, multi-step flows, and when to stay versus move. Its rationales are short English sentences.
+
+About half of the public rows are in Indian languages or code-mixed text. Telugu, Kannada and Gujarati are
+oversampled because they are underrepresented in the sources. Every row passed automatic format checks (the gold id
+is among the options, ids are unique, JSON is valid), and a sample of every source was reviewed for label quality.
+Sources whose labels did not hold up in review were left out.
+
+### Recipe
+
+- **Base:** `google/gemma-4-E2B-it`, text only. Parameter-efficient fine-tuning (LoRA on the language-model attention
+  and MLP projections), then merged into the base weights. The vision and audio towers are untouched.
+- **Objective:** next-token loss on the answer only. The option id is weighted up, so the decision is learned first;
+  rationale tokens are weighted down.
+- **Order invariance:** options are reshuffled every epoch, and ids are sometimes replaced by opaque codes, so the
+  model reads the descriptions instead of trusting positions or id names. Some examples appear twice in an epoch with
+  different option orders and the same answer.
+- **Robustness:** a "none of these" option is added to some public choice tasks, and fields to extract are reshuffled.
+- **Length:** prompts up to about 2.5k tokens. Long conversations keep their latest turns.
+- **Selection:** the checkpoint was chosen on routing accuracy for held-out agents (conversation flows never seen in
+  training), not on the public benchmarks above. Training stopped once that score stopped improving.
+- **Precision:** bf16 throughout, on a single GPU.
+
+Evaluation rows in this card are disjoint from training rows. Belebele and the Kev suites were never used in training.
+
+## Considerations
+
+### Designing options
+
+- **Write options as conditions.** "User reports a failed or pending payment" works; "intent = payments" does not.
+  The model never sees your agent's system prompt, so any rule it must follow has to be in the option text.
+- **Put the call stage in the text.** For example: "user has no further questions after being helped" or "user agreed
+  to talk to a live specialist". Wrap-up and hand-off moves are the ones the model misses when the option is vague.
+- **Include an explicit stay option**, and a "none of these" option for open-ended menus.
+- **Use unique, short ids.** Up to 24 options per decision were seen in training.
+- **Feed data the model cannot hear as text** in the state or option (e.g. "account status: not registered"). Do not
+  expect the model to infer hidden facts.
+
+### Deployment
+
+- **Precision:** use bfloat16. float16 corrupts Gemma-4 outputs.
+- **Attention kernels:** Gemma-4's 512-wide attention heads need a GPU with enough shared memory. Ada/Hopper (L4,
+  L40S, H100) work with vLLM's Triton backend; T4 does not (use transformers in bf16 there, which is slow).
+- **Multimodal memory:** with vLLM, pass `limit_mm_per_prompt={"image": 0, "video": 0, "audio": 0}` to skip
+  reserving memory for inputs you don't send.
+- **Context:** the model was trained on the most recent turns of a conversation (about five lines). Longer histories
+  add latency more than accuracy.
+- **Constrained decoding:** to guarantee a valid id, constrain the branch to your option ids (e.g. vLLM structured
+  outputs with a `choice` list), or map the decoded text back to the nearest id.
+- **Calibration:** the first-token log-probabilities of the option ids give a usable ranking. Calibrate a stay/move
+  threshold on your own traffic before you act on low-margin decisions.
+
+### Safety and responsible use
+
+- **Prompt injection:** the model is trained to treat the `state` as data, not instructions, but it is not a
+  security boundary. Keep permission checks and irreversible actions (payments, cancellations, account changes)
+  behind your own logic and confirmations.
+- **Imperfect output:** routing and extraction mistakes happen, and more often on noisy transcripts, rare languages
+  and long option lists. Log the decisions and keep a human or LLM fallback for ambiguous cases.
+- **Uneven language coverage:** English, Hindi and Hinglish are strongest. Some Indic languages have less training
+  data. Evaluate on your own languages and domains before production.
+- **Personal data:** extraction pulls personal details (names, numbers, addresses) out of conversations when asked.
+  Handle outputs under your data-protection obligations. Do not use the model to profile people or to make decisions
+  with legal or similarly significant effects without human review.
+- **Rationales** are short explanations of the chosen option, useful for logs and debugging. They are not guaranteed
+  to reflect the model's internal reasoning.
+
+### Evaluation caveats
+
+- **Small samples:** each source contributes up to 150 held-out rows (60 for validation), so single-dataset numbers
+  carry a few points of noise. Task-family averages are steadier.
+- **Extraction scoring** uses exact match after normalisation, so a correct value in different wording counts as
+  wrong. This penalises all three models equally.
+- **Same prompts for all three models:** the base models are scored zero-shot on this card's prompt format, and a
+  different prompt could raise their scores.
 
 ## License