Instructions to use notnotsamuel/LFM2.5-350M-RLCD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use notnotsamuel/LFM2.5-350M-RLCD with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="notnotsamuel/LFM2.5-350M-RLCD")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("notnotsamuel/LFM2.5-350M-RLCD")
model = AutoModelForCausalLM.from_pretrained("notnotsamuel/LFM2.5-350M-RLCD", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use notnotsamuel/LFM2.5-350M-RLCD with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "notnotsamuel/LFM2.5-350M-RLCD"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "notnotsamuel/LFM2.5-350M-RLCD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/notnotsamuel/LFM2.5-350M-RLCD

SGLang

How to use notnotsamuel/LFM2.5-350M-RLCD with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "notnotsamuel/LFM2.5-350M-RLCD" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "notnotsamuel/LFM2.5-350M-RLCD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "notnotsamuel/LFM2.5-350M-RLCD" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "notnotsamuel/LFM2.5-350M-RLCD",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use notnotsamuel/LFM2.5-350M-RLCD with Docker Model Runner:
```
docker model run hf.co/notnotsamuel/LFM2.5-350M-RLCD
```

LFM2.5-350M-RLCD / README.md

notnotsamuel

Bundle verified unchanged LiquidAI base weights and tokenizer

deb589d verified about 1 month ago

preview code

Raw History Blame Contribute Delete

6.87 kB

	---
	language:
	- en
	- fr
	- es
	library_name: transformers
	license: other
	license_name: lfm1.0
	license_link: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/9e6c6ccf47cd318696e137d381a7ded8fe4df09f/LICENSE
	tags:
	- classification
	- inference-only
	- structured-generation
	- constrained-decoding
	- lfm2
	- apple-silicon
	- pytorch
	---

	# LFM2.5-350M-RLCD

	Parallel structured inference with unchanged [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) weights. Prefill the context once, reuse attention and convolution state across candidate branches, score allowed values in a batch, and assemble JSON in Python.

	Inference only: no training or fine-tuning, and no reproduction of TypeSafe.ai's proprietary Jev training method. This repository includes code, benchmark results, and a byte-for-byte copy of the original model weights and tokenizer. `Engine` below continues to load the pinned original model for reproducibility.

	## Performance: 28 fields

	One synthetic input with 28 boolean fields, two warmups per method, and three measured repetitions. Both methods use FP16 and the same prompt. Times are mean end-to-end request latency; AR means autoregressive generation.

	\| Hardware \| Constrained mean \| Autoregressive mean \| Speedup \| JSON valid (constrained / AR) \| Schema compliant (constrained / AR) \| Field accuracy (constrained / AR) \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| M2 Max \| 393.20 ms \| 3326.95 ms \| 8.46× \| 100% / 100% \| 100% / 100% \| 60.7% / 53.6% \|
	\| L40S \| 54.12 ms \| 3404.48 ms \| 62.91× \| 100% / 100% \| 100% / 100% \| 64.3% / 53.6% \|
	\| H100 \| 43.76 ms \| 2586.97 ms \| 59.12× \| 100% / 100% \| 100% / 100% \| 60.7% / 53.6% \|

	Valid JSON does not mean correct answers: neither method produced a fully correct 28-field object. The constrained method guarantees structure through programmatic assembly, not model accuracy.

	Results depend on the workload. On the 12-case diagnostic suite, speedups were 6.25×–9.68×, but field accuracy fell from 80.6% to 77.8%. With 255 candidates on the Mac, constrained inference was 3.23× slower. See [all results and raw measurements](results/REPORT.md).

	Both methods use PyTorch eager attention and the reference PyTorch convolution implementation. Optimized causal-conv1d, torch.compile, FlashAttention and MLX are not enabled. This isolates an inference-method comparison; it does not establish the fastest attainable backend on any hardware.

	## Setup

	Python 3.11; pinned dependencies include PyTorch 2.14.0 and Transformers 5.17.0. Run from the repository root:

	```bash
	uv venv --python 3.11 .venv
	uv pip install --python .venv/bin/python -r requirements.txt
	source .venv/bin/activate
	```

	## Use

	This example reads a customer support message and selects three attributes: its topic, whether immediate action is needed, and whether the customer requests a refund. Define the allowed values in a JSON Schema, then pass that schema and the message to the engine.

	```python
	import json
	from rlcd.engine import Engine

	schema = {
	"type": "object",
	"properties": {
	"topic": {
	"type": "string",
	"description": "The main issue in the customer's message",
	"enum": ["billing", "technical", "shipping"],
	},
	"urgent": {
	"type": "boolean",
	"description": "True only if the customer explicitly needs immediate action",
	},
	"refund_requested": {
	"type": "boolean",
	"description": "Whether the customer asks for a refund",
	},
	},
	"required": ["topic", "urgent", "refund_requested"],
	"additionalProperties": False,
	}

	# Load the original base model once and reuse the engine for multiple requests.
	engine = Engine(device="mps", dtype="float16") # Apple Silicon; use "cuda" on NVIDIA

	message = "I was charged twice. Please refund the duplicate charge. No hurry."
	result = engine.constrained(message, schema)

	# A JSON object containing only the three declared fields and allowed value types.
	print(result["text"])

	# Parse it to use the model's decisions in your application.
	attributes = json.loads(result["text"])
	print("Selected topic:", attributes["topic"])
	print("Refund requested:", attributes["refund_requested"])

	# Optional: inspect raw candidate log-likelihoods (not calibrated confidence).
	print(result["scores"])
	```

	Change the field descriptions and enum values to define your own finite-choice extraction task. The engine selects the values; the program assembles the JSON. A schema-valid result can still contain an incorrect decision.

	Supported schemas are flat objects, with all properties required and `additionalProperties: false`. Each field must be a boolean or a nonempty string enum. Unsupported constraints are rejected. Candidate strings may share token prefixes; full-sequence scoring distinguishes them. Enum cardinality is not capped in code, but large sets can exhaust memory. Scores depend on candidate wording, tokenization, length, and a newline terminator. They are not calibrated confidence. No calibration evaluation was performed and no normalized probability API is exposed.

	For successful calls, `json.dumps` plus selection from typed, allowed values guarantees JSON syntax and compliance with this restricted schema subset. Those guarantees come from the program, not from learned decision quality. Fields are decided independently; output truth and cross-field consistency are not guaranteed.

	## Bundled base weights

	The bundled files are unchanged from `LiquidAI/LFM2.5-350M` revision `9e6c6ccf47cd318696e137d381a7ded8fe4df09f`. Their verified checksums are in [BASE_MODEL_MANIFEST.json](BASE_MODEL_MANIFEST.json). No fine-tuning or quantization was performed.

	You can load the bundled base model directly:

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	repo = "notnotsamuel/LFM2.5-350M-RLCD"
	tokenizer = AutoTokenizer.from_pretrained(repo)
	model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto").eval()
	```

	This loads the original language model; parallel constrained inference still uses the `Engine` example above. To recreate the bundle locally, run `python -m scripts.bundle_base` followed by `python -m scripts.prepare_release`.

	## Reproduce and inspect

	[Benchmark commands and methodology](docs/BENCHMARKING.md) cover M2 Max, L40S and H100, including warmups, precision, hardware versions and validation. [Implementation notes](docs/IMPLEMENTATION_REVIEW.md) explain hybrid-cache handling and candidate scoring. [Execution notes](docs/EXECUTION_NOTES.md) record numerical differences and run limitations.

	## License

	Inference code: [MIT](LICENSE-CODE). Bundled LiquidAI model files: [LFM Open License v1.0](LICENSE). The original model files retain their upstream license; no adapters or trained modifications are included.