| --- |
| language: |
| - en |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-30B-A3B-Instruct-2507 |
| base_model_relation: finetune |
| pipeline_tag: text-classification |
| tags: |
| - banyan |
| - semantic-verification |
| - policy-verification |
| - mixture-of-experts |
| - research-preview |
| - custom-code |
| --- |
| |
| # Banyan Sapling |
|
|
| **Semantic verification for policy-driven AI systems.** |
|
|
| Banyan Sapling is a research preview from **MubitLabs** and the first model in |
| the **Banyan** family. It evaluates whether a claim is supported by a supplied |
| policy and evidence, with separate outputs for policy conflict, missing |
| information, and inconsistent context. |
|
|
| Sapling is designed as a verification component for developers building policy |
| checks, evidence review, and claim-validation workflows. Its structured output |
| makes both the preferred classification and calibrated uncertainty available to |
| the surrounding application. |
|
|
| ## Model overview |
|
|
| | Property | Specification | |
| |---|---| |
| | Model | `MubitLabs/banyan-sapling` | |
| | Release | Research preview | |
| | Task | Policy, evidence, and claim classification | |
| | Base model | Qwen3-30B-A3B-Instruct-2507 | |
| | Base revision | `0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe` | |
| | Stored parameters | 23,042,164,740 | |
| | Architecture | Pruned MoE backbone; 96 retained experts per layer; TriVector interaction head | |
| | Adaptation | Attention LoRA and verification head; routers frozen during the final run | |
| | Precision | BF16 base; FP32 adapters, routers, and head; adapters remain unmerged | |
| | Input limit | 16,384 encoded tokens across all fields and formatting | |
| | Language evaluated | English | |
|
|
| Policy, evidence, and claim spans are pooled separately. The interaction head |
| compares their representations to produce four class logits. Native positional |
| encoding is preserved. The evaluated inference path processes one example at a |
| time. |
|
|
| ## Inputs and outputs |
|
|
| Provide three nonempty strings: `policy`, `evidence`, and `claim`. |
|
|
| | Label | Interpretation | |
| |---|---| |
| | `supported` | The supplied evidence supports the claim under the policy. | |
| | `violation` | The claim conflicts with the policy and supplied evidence. | |
| | `insufficient_information` | The supplied material lacks information needed to establish the claim. | |
| | `inconsistent_context` | Relevant facts within the supplied context conflict. | |
|
|
| `raw_label` is the highest-scoring class. Calibrated probabilities and the |
| prediction set describe the model's uncertainty. `status` is the calibrated |
| result and may be `unsure` when no unique decision is available. An unresolved |
| result is distinct from the substantive `insufficient_information` class. |
| A narrow rounding-related empty-set fallback preserves the argmax result; |
| it does not resolve a prediction set containing multiple labels. |
|
|
| Sapling classifies the supplied text. Retrieval, source authentication, generated |
| explanations, and final business decisions belong to the surrounding workflow. |
|
|
| ## Run the model |
|
|
| The weights package includes the checkpoint, tokenizer, calibration, bundled |
| inference code, pinned requirements, and file checksums. Follow the |
| [weights guide](docs/weights.md) to download and run an example. Use the bundled |
| loader: the verification head and factorized adapters require reconstruction |
| beyond loading the base Qwen architecture. |
|
|
| The reference environment uses Python 3.12, CUDA 13.0, and an NVIDIA H200. |
| The BF16 checkpoint occupies approximately 46 GB on disk, with additional |
| memory required for inference. Hardware and input length affect memory use. |
| The reference runtime is recorded in `evidence/training_runtime.json`; the |
| weights package includes the complete pinned dependency list. |
|
|
| ## Training and calibration |
|
|
| The final run completed one epoch over **50,000 examples**: 42,809 controlled |
| synthetic cases and 7,191 ContractNLI training claims, across 21,333 optimizer |
| updates. Training adapted the attention projections and interaction head while |
| keeping routers frozen. |
|
|
| Temperature fitting and conformal calibration use separate partitions of 1,360 |
| examples each, balanced between synthetic and contract cases. Temperature is |
| bounded below by 0.5; the fitted value is 5.171578. The shared nonconformity |
| threshold is 0.995478, taking the more conservative domain threshold. |
|
|
| ## Evaluation |
|
|
| | Partition | Cases | Raw accuracy | Calibrated answers | Answer coverage | |
| |---|---:|---:|---:|---:| |
| | Combined | 1,731 | 94.22% | 779 | 45.00% | |
| | Controlled synthetic | 1,000 | 100.00% | 773 | 77.30% | |
| | ContractNLI | 731 | 86.32% | 6 | 0.82% | |
|
|
| All 779 answered cases matched their reference labels in this evaluation; 952 |
| cases were unresolved. There were zero false `supported` decisions in the |
| calibrated output and 14 in raw predictions. These figures describe this |
| particular evaluation and should be read together with answer coverage. |
|
|
| The domain gap is material: the current shared calibration leaves most contract |
| cases unresolved. This preview is intended for integration experiments and |
| domain evaluation, with controlled policy scenarios providing the stronger |
| initial fit. Broader contract use requires additional data and calibration work. |
|
|
| The evaluation includes prior benchmark exposure and correlated document |
| claims. It is a regression and development characterization rather than an |
| independent estimate of performance on unseen organizations or document |
| populations. Real-world accuracy and nominal conformal coverage are not |
| established by these results. |
|
|
| Additional diagnostics include five insurance-claim variants that all return |
| `unsure`; the eligible case's raw label is `insufficient_information` rather than |
| `supported`. This identifies a target for improving evidence interpretation, |
| in addition to calibration. Details and machine-readable summaries are in |
| [the evaluation report](docs/results.md) and `evidence/`. |
|
|
| ## Runtime validation |
|
|
| Independent reload reproduced logits exactly on 53 reference cases, including |
| reversed and mixed request orders. API parity checks also passed. On those |
| references, sequential pod-local HTTP latency was 97.08 ms median and 111.33 ms |
| p95 on the reference H200 runtime. These observations describe that request set |
| and runtime; sustained serving capacity was not measured. |
|
|
| The public package loader was additionally checked on eight short reference |
| cases: logits matched exactly, including reversed request order. Its example |
| execution and scoped validation are recorded in |
| [evidence/package_validation.json](evidence/package_validation.json). |
|
|
| ## Feedback and development |
|
|
| We invite developers and researchers to evaluate Sapling on representative |
| workflows and share results through the model's Hugging Face Community tab. |
| Please include a redacted or synthetic reproduction, expected label, observed |
| output, domain, and runtime version. Cases where `unsure` prevents a useful |
| answer are particularly valuable. |
|
|
| The next development priorities are domain diversity, harder partial-evidence |
| examples, document-level evaluation splits, and improved answer coverage while |
| retaining explicit uncertainty. |
|
|
| ## License and provenance |
|
|
| Model weights are distributed under Apache 2.0, retaining the Qwen base model's |
| attribution. Bundled project code retains its MIT license. Dataset text is not |
| included in the weights package. See [third-party notices](THIRD_PARTY_NOTICES.md). |
|
|
| The checkpoint is the completed v7 model, named Banyan Sapling for this release. |
| The original release identity is |
| `245970b0e5b0b22d64c1434ee21dd221b1f211f06bda5465f8fc108f70cd5c3f`. |
| The public distribution has its own file manifest, with an unchanged weights |
| checksum. See [provenance](docs/provenance.md). |
|
|
