Add TwIL-LM2 (Granite-3.3-2B): model card, licences, config and tokenizer

LICENSE.md

@@ -0,0 +1,36 @@
+webAI Non-Commercial License ver. 1.0
+
+
+ 1. Definitions
+
+ “Licensor” means any person or entity that distributes its Work. “Work” means (a) the original work of authorship made available under this license, which may include software, documentation, or other files, and (b) any additions to or derivative works thereof that are made available under this license. The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this license, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work. Works are “made available” under this license by including in or with the Work either (a) a copyright notice referencing the applicability of this license to the Work, or (b) a copy of this license.
+
+
+ 2. License Grant
+
+ 2.1 Copyright Grant. Subject to the terms and conditions of this license, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to use, reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form.
+
+
+ 3. Limitations
+
+ 3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this license, (b) you include a complete copy of this license with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work.
+
+ 3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this license (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself.
+
+ 3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. As used herein, “non-commercially” means for non-commercial research and educational purposes only.
+
+ 3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this license from such Licensor (including the grant in Section 2.1) will terminate immediately.
+
+ 3.5 Trademarks. This license does not grant any rights to use any Licensor's or its affiliates' names, logos, or trademarks, except as necessary to reproduce the notices described in this license. Nothing in this license shall be construed as permission to use the trade names, trademarks, service marks, or product names of webAI, Inc. or its affiliates to endorse, promote, or imply association with any derivative work, product, service, or entity, without prior written consent from webAI, Inc. Use of the Work does not imply endorsement by webAI, Inc. of any derivative work, product, service, or entity.
+
+ 3.6 Termination. If you violate any term of this license, then your rights under this license (including the grant in Section 2.1) will terminate immediately.
+
+
+ 4. Disclaimer of Warranty.
+
+ THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE.
+
+
+ 5. Limitation of Liability.
+
+ EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

README.md

@@ -0,0 +1,372 @@
+---
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: ibm-granite/granite-3.3-2b-instruct
+license: other
+license_name: webai-non-commercial-license-ver.-1.0
+license_link: https://huggingface.co/webAI-Official/TwIL-LM2/blob/main/LICENSE.md
+tags:
+  - granite
+  - formal-logic
+  - reasoning
+  - lora
+  - model-merging
+  - wise-ft
+  - reinforcement-learning
+  - grpo
+  - twil-lm
+  - gguf
+---
+
+# TwIL-LM2
+
+A 2.5B reasoning model for **formal logic** tasks, built from
+[`ibm-granite/granite-3.3-2b-instruct`](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct)
+through LoRA supervised fine-tuning, WiSE-FT weight interpolation (λ = 0.25) and entropy-weighted
+GRPO reinforcement learning (MGPO, step 1400).
+
+It is the **balanced** point of its training arm. On the in-domain macro gate it scores **0.4178**,
+within 0.004 of TwIL-LM3 (0.4218) at about 18% fewer parameters, and it keeps a 10-dataset
+held-out macro of **0.6759** — second only to TwIL-LM3 (0.7339) among the models compared below.
+It also decodes **1.57x faster than TwIL-LM3** under identical forced work.
+
+The price of that balance is structured-output strictness. Its strict-7 score (0.1214) and strict
+MCQ accuracy (0.0000) are well below the λ = 0.5 sibling from the same arm (0.2457 and 0.2850),
+which gives back roughly 0.044 of held-out macro to get them. This release is the held-out-friendly
+end of that trade; the other end was not published. See [Limitations](#limitations-and-caveats).
+
+## Highlights
+
+* **Gate parity with TwIL-LM3 at a smaller size.** Macro gate 0.4178 against 0.4218, at 2.53B
+  against 3.08B parameters. Read the parity as approximate: TwIL-LM3's figure is understated by
+  truncation (4.4% of its Track A rows hit the token cap), so its true gate is somewhat higher.
+* **Held-out capability is largely retained.** 10-dataset Track B macro 0.6759, against 0.6317 for
+  the λ = 0.5 sibling and 0.4333 for the SmolLM2-1.7B-based TwIL-LM2. The gap to the λ = 0.5 arm is
+  widest on the math sets: GSM8K 0.7867 against 0.6767 and GSM-Symbolic 0.7233 against 0.5033.
+* **A cleaner measurement.** Only 0.9% of Track A generations hit the 2048-token cap, under the 2%
+  threshold our protocol requires to mark a comparison `rankable`. TwIL-LM3 (4.4%) and the
+  SmolLM2-1.7B-based model (6.9%) are both above it.
+* **Fast.** 21,369 decode tokens/s on one H100 in a controlled bench — 1.57x TwIL-LM3 and 1.22x the
+  SmolLM2-1.7B-based model — because Granite's architecture decodes quickly, not because it answers
+  short.
+* **Lowest `lm_corpus` perplexity in the comparison** (1.9808), and 3.3073 on `math_corpus`. Read
+  these with the tokenizer caveat under [Limitations](#limitations-and-caveats).
+* **Runs anywhere.** 2.53B parameters in bf16 (4.72 GiB), with a Q4\_K\_M GGUF at 1.44 GiB for CPU.
+
+It is not a general assistant, and it is not the strongest structured-output model here:
+`lean_critic` accuracy is 0.3100 against 0.6600 for TwIL-LM3.
+
+## Model Details
+
+| Property                  | Value                                                                                           |
+| ------------------------- | ----------------------------------------------------------------------------------------------- |
+| Model ID                  | `webAI-Official/TwIL-LM2`                                                                       |
+| Base model                | [`ibm-granite/granite-3.3-2b-instruct`](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct) |
+| Total parameters          | 2.53B (2,533,539,840; tied input/output embeddings)                                             |
+| Architecture              | Granite decoder-only transformer; 40 layers, hidden size 2048, 32 attention heads, 8 KV heads   |
+| Input / output            | Text / text                                                                                     |
+| Language                  | English                                                                                         |
+| Vocabulary size           | 49,159 embedding rows                                                                           |
+| Context window            | 131,072 tokens (inherited from the base; see note below)                                        |
+| Checkpoint precision      | bfloat16 (4.72 GiB), plus Q4\_K\_M / Q5\_K\_M / Q6\_K / Q8\_0 / F16 GGUF builds                  |
+| Post-training             | LoRA SFT → WiSE-FT (λ = 0.25) → MGPO reinforcement learning (step 1400)                         |
+| Chat template             | Granite chat template (`<\|start_of_role\|>…<\|end_of_role\|>`), EOS `<\|end_of_text\|>`         |
+| Reasoning format          | Answers directly under the default chat template; no `<think>` block was observed               |
+| Evaluated decoding        | Greedy; 2048 new tokens (Track A), 4096 new tokens (Track B)                                    |
+| Specialisation            | Formal logic: FOL translation, entailment, semantic parsing, Lean formalisation and critique    |
+| License                   | webAI Non-Commercial License ver. 1.0                                                           |
+
+The base model's 131,072-token context is carried through unchanged, but every score on this card
+was measured with generation budgets of 2048 (Track A) or 4096 (Track B) tokens. Longer contexts are
+inherited rather than validated here. Granite's optional `thinking=True` chat-template mode is also
+inherited from the base and was not evaluated for this card.
+
+## Results
+
+Every column below is a checkpoint we trained or released, scored through the same harness and
+decoding settings described under [Evaluation protocol](#evaluation-protocol):
+
+* **this model** — the λ = 0.25, MGPO step-1400 checkpoint in this repository;
+* **λ = 0.25, step 300** — the next-best gate in the same arm;
+* **λ = 0.5, step 800** — the arm's highest Track A gate, with weaker held-out results (not released);
+* **TwIL-LM3** — [`webAI-Official/TwIL-LM3`](https://huggingface.co/webAI-Official/TwIL-LM3),
+  SmolLM3-3B, WiSE-FT λ = 0.25, MGPO step 2071;
+* **TwIL-LM2 (SmolLM2-1.7B)** — the earlier, SmolLM2-1.7B-Instruct-based TwIL-LM2 (WiSE-FT
+  λ = 0.75, MGPO step 1680), using its published figures.
+
+### Track A — in-domain formal logic
+
+| lane / metric              | **This model** | λ=0.25 s300 | λ=0.5 s800 | TwIL-LM3   | TwIL-LM2 (SmolLM2-1.7B) |
+| -------------------------- | -------------- | ----------- | ---------- | ---------- | ----------------------- |
+| parameters                 | 2.53B          | 2.53B       | 2.53B      | 3.08B      | 1.7B                    |
+| `lean_formalize` token-F1  | 0.5159         | 0.5455      | **0.7175** | 0.5869     | 0.6199                  |
+| `rule_induction` derivation | 0.3292        | 0.3471      | **0.5516** | 0.3192     | 0.5136                  |
+| `entailment_label` accuracy | 0.5300        | 0.5000      | **0.6250** | 0.5750     | 0.5850                  |
+| `mcq_answer` accuracy      | 0.0000         | 0.0000      | **0.2850** | 0.1100     | 0.1600                  |
+| `semantic_parse` token-F1  | 0.4013         | 0.4171      | 0.6171     | 0.4416     | **0.8428**              |
+| `lean_critic` accuracy     | 0.3100         | 0.3400      | 0.5350     | **0.6600** | 0.5250                  |
+| `procedural` accuracy      | 0.0100         | 0.0150      | **0.0750** | 0.0300     | 0.0100                  |
+| `fol_translation` primary  | 0.0000         | 0.0000      | 0.1000     | 0.0000     | **0.1150**              |
+| `lm_corpus` perplexity ↓   | **1.9808**     | 1.9948      | 1.9982     | 2.8972     | 2.2981                  |
+| `math_corpus` perplexity ↓ | 3.3073         | 3.3731      | 3.2403     | 3.8229     | **3.0390**              |
+| average, 6 lanes           | 0.3477         | 0.3583      | **0.5552** | 0.4488     | 0.5410                  |
+| **macro gate**             | 0.4178         | 0.4114      | **0.4393** | 0.4218     | 0.3927                  |
+| **strict-7**               | 0.1214         | 0.1221      | **0.2457** | 0.1971     | 0.2386                  |
+| macro\_primary             | 0.4400         | 0.4275      | 0.4113     | **0.4475** | 0.3625                  |
+| mean generation length ↓ (tokens) | 517     | 505         | **213**    | 443        | 460                     |
+| truncation at 2048 tokens  | 0.9%           | 0.6%        | 0.5%       | 4.4%       | 6.9%                    |
+| `rankable` (< 2% truncation) | yes          | yes         | yes        | no         | no                      |
+
+`average, 6 lanes`, `macro gate`, `macro_primary` and `strict-7` are the harness aggregates defined
+on the [TwIL-LM3 model card](https://huggingface.co/webAI-Official/TwIL-LM3); this card does not
+redefine them. They are not interchangeable, and the ordering changes between them — this model
+leads neither `macro_primary` nor `strict-7`.
+
+**Reading it.** This is a mid-table formal-logic model whose main asset is held-out retention.
+Against TwIL-LM3 it wins on `rule_induction` (0.3292 against 0.3192) and on both perplexities, and
+loses on the rest, heavily on `lean_critic` (0.3100 against 0.6600). Its strict-7 (0.1214) is well
+below TwIL-LM3's (0.1971), so the near-parity is on the gate, which credits loose matches on some
+lanes, and not on strict scoring.
+
+The λ = 0.5 sibling is the stronger *formal-logic* model — it leads nine of the fifteen scored rows
+above — and the SmolLM2-1.7B-based model is the semantic-parsing specialist at 0.8428. Neither
+keeps held-out capability the way this checkpoint does; see Track B.
+
+Two things are worth stating plainly. Strict MCQ accuracy is **0.0000** for both λ = 0.25
+checkpoints: they do not emit the exactly-requested answer form on that lane. And harness
+answers-per-second (12.07 for this model, 17.57 for λ = 0.5) reflects how long each model chooses
+to write, not how fast it decodes; use the controlled bench below for speed.
+
+### Track B — held-out benchmarks
+
+Nothing in this suite was trained on. Ten chain-of-thought datasets, 300 randomly sampled examples
+each, identical rows for every column.
+
+| dataset                     | **This model** | λ=0.25 s300 | λ=0.5 s800 | TwIL-LM3   | TwIL-LM2 (SmolLM2-1.7B) |
+| --------------------------- | -------------- | ----------- | ---------- | ---------- | ----------------------- |
+| `gsm8k`                     | 0.7867         | 0.8000      | 0.6767     | **0.8733** | 0.4633                  |
+| `svamp`                     | 0.8200         | 0.8400      | 0.7333     | **0.8500** | 0.3833                  |
+| `gsm_symbolic`              | 0.7233         | 0.7400      | 0.5033     | **0.7567** | 0.2600                  |
+| `arc_cot`                   | 0.7367         | 0.7300      | 0.7067     | **0.8467** | 0.5200                  |
+| `logicbench`                | 0.6900         | 0.6933      | 0.6833     | **0.7167** | 0.5400                  |
+| `strategyqa`                | 0.6933         | 0.6733      | **0.7233** | 0.6500     | 0.5900                  |
+| `drop`                      | 0.5833         | 0.6000      | 0.5700     | **0.7467** | 0.4367                  |
+| `csqa`                      | 0.6900         | 0.6767      | 0.7233     | **0.7367** | 0.4333                  |
+| `musr` (mean of 3)          | 0.4957         | 0.4877      | 0.4904     | **0.4958** | 0.3131                  |
+| `mmlu_redux`                | 0.5400         | 0.5167      | 0.5067     | **0.6667** | 0.3933                  |
+| **macro (10 CoT datasets)** | 0.6759         | 0.6758      | 0.6317     | **0.7339** | 0.4333                  |
+
+**TwIL-LM3 leads this table**, on nine of ten datasets and on the macro. This model is second on
+the macro (and ties with the s300 checkpoint), ahead of the λ = 0.5 arm by 0.044 and of the
+SmolLM2-1.7B-based model by 0.243. It is close to TwIL-LM3 on `musr` (0.4957 against 0.4958) and
+`logicbench` (0.6900 against 0.7167), and furthest behind on `drop` (−0.163) and `mmlu_redux`
+(−0.127).
+
+The 14-dataset macro (adding `ifeval`, `rudas_ood`, `bbh_logic` and `math500`) was not run for the
+Granite checkpoints in a directly comparable tree, so it is not reported here, and **no
+instruction-following result is claimed**. It is also not known from these runs how this model
+compares with its own base, `granite-3.3-2b-instruct`: no paired base-versus-tuned evaluation is
+included in this card.
+
+### Speed
+
+Harness tokens-per-second and answers-per-second are confounded by how much each model writes. For
+an actual speed comparison, each model ran alone on one idle H100 (vLLM 0.19.1, torch 2.10.0+cu128,
+transformers 5.15.0) over the same 128 prompts with `ignore_eos` and a hard 512-token cap, so every
+model emitted exactly 65,536 output tokens.
+
+| controlled decode bench         | **This model** | TwIL-LM3 | TwIL-LM2 (SmolLM2-1.7B) |
+| ------------------------------- | -------------- | -------- | ----------------------- |
+| decode tokens/s                 | **21,369**     | 13,623   | 17,542                  |
+| 512-token completions/s         | **41.7**       | 26.6     | 34.2                    |
+| decode wall seconds (65,536 tok) | **3.07**      | 4.81     | 3.74                    |
+| relative to this model          | 1.00x          | 0.64x    | 0.82x                   |
+
+The three Granite checkpoints decode within 1.1% of one another (21,236 to 21,467 tokens/s), so
+the speed advantage comes from the architecture, not from a particular checkpoint. The shared
+prompt file is Granite-templated, so prefill differs slightly by tokenizer (30.7K tokens for
+Granite and SmolLM3, 38.2K for SmolLM2); it is a small share of the forced output and, if anything,
+slightly handicaps the SmolLM2-based model.
+
+### Choosing a checkpoint within the arm
+
+All four evaluated λ = 0.25 checkpoints sit within 0.0006 of one another on Track B, so held-out
+performance cannot separate them; the ordering comes from Track A and the probe.
+
+| MGPO step | Track B R10 | Track A macro gate |
+| --------- | ----------- | ------------------ |
+| 300       | 0.6758      | 0.4114             |
+| 1100      | 0.6764      | 0.4019             |
+| **1400**  | 0.6759      | **0.4178**         |
+| 1700      | 0.6761      | 0.4059             |
+
+Step 1400 is the checkpoint chosen by probe Pass@1 and has the highest gate of the four.
+
+### Where it sits in the wider Granite family
+
+For context, an internal, **unreleased** Granite 4.1 3B MGPO checkpoint (fusion + WiSE-FT λ = 0.5,
+step 600), scored on the same Track A and Track B protocol, reaches a macro gate of 0.5105,
+strict-7 of 0.2886 and Track B R10 of 0.7761 — 9.3, 16.7 and 10.0 percentage points above this
+model respectively. This model is the smaller, faster option; it is not the strongest Granite
+checkpoint we have trained.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "webAI-Official/TwIL-LM2"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto"
+)
+
+messages = [{"role": "user", "content":
+             "Does 'All dogs are mammals. Rex is a dog.' entail 'Rex is a mammal'? "
+             "Answer entailment, contradiction, or neutral."}]
+inputs = tok.apply_chat_template(
+    messages, add_generation_prompt=True,
+    return_tensors="pt", return_dict=True,
+).to(model.device)
+
+out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
+
+`return_dict=True` matters on transformers 5.x, where `apply_chat_template` returns a
+`BatchEncoding` rather than a bare tensor; the above works on both 4.x and 5.x.
+
+For the example above, greedy decoding with the bf16 weights (transformers 5.14.1, CPU) produced,
+in 57 tokens and ending on EOS:
+
+> Entailment. The statement "All dogs are mammals" implies that any individual dog, such as Rex,
+> must also be a mammal. Therefore, the conclusion "Rex is a mammal" is entailed by the premises.
+
+The reported numbers use **greedy decoding** (`do_sample=False`) and a **2048-token** generation
+budget for Track A. The shipped `generation_config.json` carries no sampling defaults, so greedy is
+what you get unless you ask for otherwise. Track A generations average about 517 tokens and 0.9%
+reach the 2048-token cap, so keep the budget at 2048 or more for formal-logic prompts.
+
+### GGUF / llama.cpp
+
+Quantized GGUF builds ship in this repository alongside the safetensors weights. The `granite`
+architecture is supported by llama.cpp, and the chat template is embedded in the GGUF metadata, so
+chat mode needs no extra flags.
+
+| file                   | quant    | size     | bits/weight | notes                                             |
+| ---------------------- | -------- | -------- | ----------- | ------------------------------------------------- |
+| TwIL-LM2-Q4\_K\_M.gguf | Q4\_K\_M | 1.44 GiB | 4.88        | recommended default; runs on CPU                  |
+| TwIL-LM2-Q5\_K\_M.gguf | Q5\_K\_M | 1.68 GiB | 5.70        | a little more headroom than Q4\_K\_M              |
+| TwIL-LM2-Q6\_K.gguf    | Q6\_K    | 1.94 GiB | 6.57        | close to Q8\_0 quality at about three quarters of the size |
+| TwIL-LM2-Q8\_0.gguf    | Q8\_0    | 2.51 GiB | 8.51        | near-lossless, for quality-sensitive use          |
+| TwIL-LM2-F16.gguf      | F16      | 4.72 GiB | 16.01       | unquantized, for requantization or reference runs |
+
+```bash
+llama-cli -m TwIL-LM2-Q4_K_M.gguf -cnv --temp 0 -n 2048
+```
+
+Pass `--temp 0`, because the evaluation is greedy, and leave the generation budget at 2048 tokens
+or more.
+
+The published Track A and Track B numbers were measured on the **bf16** weights through vLLM, not
+on any of these GGUF builds, so expect small deviations — most likely at Q4\_K\_M — that have not
+been quantified here. Note also that F16 is not bit-identical to the bf16 weights: the two formats
+carry the same 16 bits but trade exponent range against mantissa precision.
+
+## How it was built
+
+Three stages on top of the base model:
+
+1. **LoRA supervised fine-tuning** on a synthetic formal-logic corpus covering the Track A
+   objectives (first-order-logic translation, entailment labelling, semantic parsing, Lean
+   formalisation and critique, procedural reasoning, rule induction), using the project's v5
+   SFT recipe.
+2. **WiSE-FT interpolation** toward the pretrained base,
+   `W = (1 − λ)·W_base + λ·W_finetuned` with **λ = 0.25** — only a quarter of the fine-tuned delta
+   is retained. This conservative λ is the main reason held-out capability survives; the λ = 0.5
+   sibling retains twice the delta and shows it in the strict formal-logic lanes and in the loss of
+   held-out math.
+3. **MGPO** — entropy-weighted GRPO reinforcement learning against a programmatic verifier, with
+   partial credit for loose matches and token-F1 so that all-fail prompt groups still produce
+   gradient. Published checkpoint is **step 1400**.
+
+Unlike TwIL-LM3, there is no checkpoint-fusion stage between SFT and WiSE-FT in this arm.
+
+## Limitations and caveats
+
+**Strict output form.** Strict MCQ accuracy is 0.0000 and `procedural` accuracy 0.0100;
+`fol_translation` primary score is 0.0000; strict-7 is 0.1214. The model reasons near the required
+form without reliably emitting it. If you need exactly-formatted formal objects, the λ = 0.5
+checkpoint's profile (strict-7 0.2457, MCQ 0.2850) is better, at the cost of held-out math.
+
+**Lean critique.** `lean_critic` accuracy is 0.3100, well below TwIL-LM3 (0.6600) and the
+SmolLM2-1.7B-based model (0.5250). This is the weakest of the Track A lanes relative to its peers.
+
+**No base comparison.** This card does not include a paired evaluation of
+`granite-3.3-2b-instruct` on the same harness, so it makes no claim about how much the tuning
+improved or regressed the base model on either track.
+
+**Perplexity across tokenizers.** `lm_corpus` and `math_corpus` perplexity is a per-token quantity,
+and the columns use different tokenizers (Granite and SmolLM2 have about 49K entries each but
+distinct vocabularies; SmolLM3 has 128K). The perplexity rows are informative within a family and
+only indicative across families.
+
+**Result trees.** For Track B, TwIL-LM3 and the SmolLM2-1.7B-based model are scored from the
+results tree that matches their published cards (rope-fixed), and the Granite checkpoints from the
+default tree, with vLLM 0.19.1; the two TwIL macros reproduce their published values exactly
+(0.7339 and 0.4333). Track A figures for all five columns come from GATE 2 reports at n = 200 per
+lane and a 2048-token cap.
+
+**Truncation of the comparison arms.** TwIL-LM3 and the SmolLM2-1.7B-based model are formally
+`rankable: false` at 4.4% and 6.9% truncation. A truncated response scores zero regardless of
+reasoning quality, so their Track A figures are understated: wherever this model is ahead of them
+the true margin is smaller, and wherever it is behind, the true deficit is larger.
+
+**Scope.** Tuned for formal logic. The Track B suite reported here does not cover code generation
+or tool use, and no claim is made about either. Granite's base tool-calling and document-grounded
+chat-template features are inherited but were not evaluated.
+
+**Not a chat model.** It was optimised against automatic verifiers on logic tasks. It has had no
+safety tuning beyond whatever the base model carries, and no instruction-following alignment work.
+
+**GGUF builds.** Scores were measured on the bf16 weights only; the quantized builds have not been
+evaluated.
+
+## Evaluation protocol
+
+* Track A: `n = 200` per objective, greedy (`temperature = 0`), `max_new_tokens = 2048`, one retry
+  at 4096 for truncated rows.
+* Track B: 300 examples per task, greedy, 4096 generation tokens, chat template applied, vLLM 0.19.1
+  backend. `musr` is the mean of the murder, object and team splits.
+* Controlled decode bench: one idle H100 per model, 128 shared prompts, `ignore_eos`, hard 512-token
+  cap, engine initialisation excluded from the rate.
+* All columns are scored on the same sampled rows within each track.
+
+Track B is sampled at 300 examples per dataset for compute reasons. Absolute scores can shift on
+the full sets, but the comparative ordering across models is expected to be stable.
+
+## Relationship to TwIL-LM
+
+**TwIL-LM3** ([`webAI-Official/TwIL-LM3`](https://huggingface.co/webAI-Official/TwIL-LM3)) is the
+SmolLM3-3B-based member of the family. It is stronger on held-out benchmarks and on `lean_critic`;
+this model matches its in-domain gate at a smaller size and decodes 1.57x faster.
+
+The name **TwIL-LM2** was previously used for a SmolLM2-1.7B-Instruct-based model. It is the
+semantic-parsing specialist in the tables above, and the weakest of these models on held-out
+benchmarks. This repository is a different model — Granite-based, 2.5B — that carries the TwIL-LM2
+name; the SmolLM2-1.7B-based figures above are that model's published numbers, included as a
+reference point and not as this model's results.
+
+## License and attribution
+
+Released under the **webAI Non-Commercial License ver. 1.0** — see `LICENSE.md` in this repository.
+
+The base model,
+[`ibm-granite/granite-3.3-2b-instruct`](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct),
+is Apache 2.0; its licence text is retained as `apache-2.0-LICENSE.txt` and all credit for the base
+model goes to the IBM Granite team. Apache 2.0 permits distributing derivative works under
+different terms provided attribution is preserved, which is what the pair of licence files in this
+repository does.

apache-2.0-LICENSE.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

chat_template.jinja

@@ -0,0 +1,62 @@
+{# Alias tools -> available_tools #}
+{%- if tools and not available_tools -%}
+    {%- set available_tools = tools -%}
+{%- endif -%}
+{%- if messages[0]['role'] == 'system' %}
+     {%- set system_message = messages[0]['content'] %}
+     {%- set loop_messages = messages[1:] %}
+ {%- else %}
+     {%- set system_message = "Knowledge Cutoff Date: April 2024.
+Today's Date: " + strftime_now('%B %d, %Y') + ".
+You are Granite, developed by IBM." %}
+     {%- if available_tools and documents %}
+         {%- set system_message = system_message + " You are a helpful assistant with access to the following tools. When a tool is required to answer the user's query, respond only with <|tool_call|> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.
+Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data." %}
+     {%- elif available_tools %}
+         {%- set system_message = system_message + " You are a helpful assistant with access to the following tools. When a tool is required to answer the user's query, respond only with <|tool_call|> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request." %}
+     {%- elif documents %}
+         {%- set system_message = system_message + " Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data." %}
+    {%- elif thinking %}
+    {%- set system_message = system_message + " You are a helpful AI assistant.
+Respond to every user query in a comprehensive and detailed way. You can write down your thoughts and reasoning process before responding. In the thought process, engage in a comprehensive cycle of analysis, summarization, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. In the response section, based on various attempts, explorations, and reflections from the thoughts section, systematically present the final solution that you deem correct. The response should summarize the thought process. Write your thoughts between <think></think> and write your response between <response></response> for each user query." %}
+     {%- else %}
+         {%- set system_message = system_message + " You are a helpful AI assistant." %}
+     {%- endif %}
+     {%- if 'citations' in controls and documents %}
+         {%- set system_message = system_message + '
+Use the symbols <|start_of_cite|> and <|end_of_cite|> to indicate when a fact comes from a document in the search result, e.g <|start_of_cite|> {document_id: 1}my fact <|end_of_cite|> for a fact from document 1. Afterwards, list all the citations with their corresponding documents in an ordered list.' %}
+     {%- endif %}
+     {%- if 'hallucinations' in controls and documents %}
+         {%- set system_message = system_message + '
+Finally, after the response is written, include a numbered list of sentences from the response with a corresponding risk value that are hallucinated and not based in the documents.' %}
+     {%- endif %}
+     {%- set loop_messages = messages %}
+ {%- endif %}
+ {{- '<|start_of_role|>system<|end_of_role|>' + system_message + '<|end_of_text|>
+' }}
+ {%- if available_tools %}
+     {{- '<|start_of_role|>available_tools<|end_of_role|>' }}
+     {{- available_tools | tojson(indent=4) }}
+     {{- '<|end_of_text|>
+' }}
+ {%- endif %}
+ {%- if documents %}
+     {%- for document in documents %}
+         {{- '<|start_of_role|>document {"document_id": "' + document['doc_id'] | string + '"}<|end_of_role|>
+' }}
+         {{- document['text'] }}
+         {{- '<|end_of_text|>
+' }}
+              {%- endfor %}
+ {%- endif %}
+ {%- for message in loop_messages %}
+     {{- '<|start_of_role|>' + message['role'] + '<|end_of_role|>' + message['content'] + '<|end_of_text|>
+' }}
+     {%- if loop.last and add_generation_prompt %}
+         {{- '<|start_of_role|>assistant' }}
+             {%- if controls %}
+                 {{- ' ' + controls | tojson()}}
+             {%- endif %}
+         {{- '<|end_of_role|>' }}
+     {%- endif %}
+ {%- endfor %}

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "GraniteForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attention_multiplier": 0.015625,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "embedding_multiplier": 12.0,
+  "eos_token_id": 0,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "logits_scaling": 8.0,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "granite",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "residual_multiplier": 0.22,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": true,
+  "vocab_size": 49159
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0"
+}

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|end_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|end_of_text|>",
+  "errors": "replace",
+  "is_local": true,
+  "model_max_length": 9223372036854775807,
+  "pad_token": "<|end_of_text|>",
+  "padding_side": "left",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|end_of_text|>",
+  "vocab_size": 49152
+}