README.md

---
language:
- en
- fr
- es
library_name: transformers
license: other
license_name: lfm1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/9e6c6ccf47cd318696e137d381a7ded8fe4df09f/LICENSE
tags:
- classification
- inference-only
- structured-generation
- constrained-decoding
- lfm2
- apple-silicon
- pytorch
---

# LFM2.5-350M-RLCD

Parallel structured inference with **unchanged [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) weights**. Prefill the context once, reuse attention and convolution state across candidate branches, score allowed values in a batch, and assemble JSON in Python.

**Inference only: no training or fine-tuning, and no reproduction of TypeSafe.ai's proprietary Jev training method.** This repository includes code, benchmark results, and a byte-for-byte copy of the original model weights and tokenizer. `Engine` below continues to load the pinned original model for reproducibility.

## Performance: 28 fields

One synthetic input with 28 boolean fields, two warmups per method, and three measured repetitions. Both methods use FP16 and the same prompt. Times are mean end-to-end request latency; AR means autoregressive generation.

| Hardware | Constrained mean | Autoregressive mean | Speedup | JSON valid (constrained / AR) | Schema compliant (constrained / AR) | Field accuracy (constrained / AR) |
|---|---:|---:|---:|---:|---:|---:|
| M2 Max | 393.20 ms | 3326.95 ms | **8.46×** | 100% / 100% | 100% / 100% | 60.7% / 53.6% |
| L40S | 54.12 ms | 3404.48 ms | **62.91×** | 100% / 100% | 100% / 100% | 64.3% / 53.6% |
| H100 | 43.76 ms | 2586.97 ms | **59.12×** | 100% / 100% | 100% / 100% | 60.7% / 53.6% |

**Valid JSON does not mean correct answers:** neither method produced a fully correct 28-field object. The constrained method guarantees structure through programmatic assembly, not model accuracy.

Results depend on the workload. On the 12-case diagnostic suite, speedups were 6.25×–9.68×, but field accuracy fell from 80.6% to 77.8%. With 255 candidates on the Mac, constrained inference was 3.23× slower. See [all results and raw measurements](results/REPORT.md).

**Both methods use PyTorch eager attention and the reference PyTorch convolution implementation. Optimized causal-conv1d, torch.compile, FlashAttention and MLX are not enabled. This isolates an inference-method comparison; it does not establish the fastest attainable backend on any hardware.**

## Setup

Python 3.11; pinned dependencies include PyTorch 2.14.0 and Transformers 5.17.0. Run from the repository root:

```bash
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.txt
source .venv/bin/activate
```

## Use

This example reads a customer support message and selects three attributes: its topic, whether immediate action is needed, and whether the customer requests a refund. Define the allowed values in a JSON Schema, then pass that schema and the message to the engine.

```python
import json
from rlcd.engine import Engine

schema = {
    "type": "object",
    "properties": {
        "topic": {
            "type": "string",
            "description": "The main issue in the customer's message",
            "enum": ["billing", "technical", "shipping"],
        },
        "urgent": {
            "type": "boolean",
            "description": "True only if the customer explicitly needs immediate action",
        },
        "refund_requested": {
            "type": "boolean",
            "description": "Whether the customer asks for a refund",
        },
    },
    "required": ["topic", "urgent", "refund_requested"],
    "additionalProperties": False,
}

# Load the original base model once and reuse the engine for multiple requests.
engine = Engine(device="mps", dtype="float16")  # Apple Silicon; use "cuda" on NVIDIA

message = "I was charged twice. Please refund the duplicate charge. No hurry."
result = engine.constrained(message, schema)

# A JSON object containing only the three declared fields and allowed value types.
print(result["text"])

# Parse it to use the model's decisions in your application.
attributes = json.loads(result["text"])
print("Selected topic:", attributes["topic"])
print("Refund requested:", attributes["refund_requested"])

# Optional: inspect raw candidate log-likelihoods (not calibrated confidence).
print(result["scores"])
```

Change the field descriptions and enum values to define your own finite-choice extraction task. The engine selects the values; the program assembles the JSON. A schema-valid result can still contain an incorrect decision.

Supported schemas are flat objects, with all properties required and `additionalProperties: false`. Each field must be a boolean or a nonempty string enum. Unsupported constraints are rejected. Candidate strings may share token prefixes; full-sequence scoring distinguishes them. Enum cardinality is not capped in code, but large sets can exhaust memory. Scores depend on candidate wording, tokenization, length, and a newline terminator. They are not calibrated confidence. No calibration evaluation was performed and no normalized probability API is exposed.

For successful calls, `json.dumps` plus selection from typed, allowed values guarantees JSON syntax and compliance with this restricted schema subset. Those guarantees come from the program, **not from learned decision quality**. Fields are decided independently; output truth and cross-field consistency are not guaranteed.

## Bundled base weights

The bundled files are unchanged from `LiquidAI/LFM2.5-350M` revision `9e6c6ccf47cd318696e137d381a7ded8fe4df09f`. Their verified checksums are in [BASE_MODEL_MANIFEST.json](BASE_MODEL_MANIFEST.json). No fine-tuning or quantization was performed.

You can load the bundled base model directly:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "notnotsamuel/LFM2.5-350M-RLCD"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto").eval()
```

This loads the original language model; parallel constrained inference still uses the `Engine` example above. To recreate the bundle locally, run `python -m scripts.bundle_base` followed by `python -m scripts.prepare_release`.

## Reproduce and inspect

[Benchmark commands and methodology](docs/BENCHMARKING.md) cover M2 Max, L40S and H100, including warmups, precision, hardware versions and validation. [Implementation notes](docs/IMPLEMENTATION_REVIEW.md) explain hybrid-cache handling and candidate scoring. [Execution notes](docs/EXECUTION_NOTES.md) record numerical differences and run limitations.

## License

Inference code: [MIT](LICENSE-CODE). Bundled LiquidAI model files: [LFM Open License v1.0](LICENSE). The original model files retain their upstream license; no adapters or trained modifications are included.