ukkathva/telecom-oss-8b-vllm-fp8

telecom-oss-8b vLLM FP8 Dynamic

This model is a vLLM-oriented FP8 Dynamic quantized version of:

https://huggingface.co/Tapask/telecom-oss-8b-merged

The purpose of this checkpoint is to make the telecom OSS/BSS model easier to serve with vLLM, especially where reducing model weight memory is useful.

Courtesy and attribution

Full credit for the original merged telecom model goes to:

• Tapask/telecom-oss-8b-merged
• Original base/model lineage referenced by the upstream model card
• The authors and maintainers of the underlying Llama 3 ecosystem

This repository does not introduce new training data, fine-tuning, domain adaptation, or model behavior changes.

It is only a quantized derivative intended for serving and deployment experiments.

Please refer to the upstream model card for original model details, intended use, and licensing context.

What changed

The original FP16 checkpoint was converted using LLM Compressor into a vLLM-compatible FP8 Dynamic checkpoint.

Conversion recipe:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)

Quantization details

• Quantization method: FP8_DYNAMIC
• Format: compressed-tensors
• Target modules: Linear
• lm_head: ignored / kept unquantized
• Weights: FP8
• Activations: dynamic FP8 at inference where supported by vLLM
• No additional fine-tuning
• No dataset changes
• No model architecture changes

A100 deployment note

This checkpoint is intended to be loadable with vLLM on A100-class systems.

A100 is Ampere, so do not expect the same native FP8 W8A8 execution path as H100.

The main expected benefit on A100 is reduced model weight memory, which can help with model packing, concurrency, and serving footprint.

Example vLLM serving command

vllm serve ukkathva/telecom-oss-8b-vllm-fp8 \
  --dtype auto \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Example API test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ukkathva/telecom-oss-8b-vllm-fp8","messages":[{"role":"user","content":"Explain alarm correlation in telecom OSS."}],"max_tokens":128,"temperature":0.1}'

Recommended validation

After downloading or serving the model, verify that config.json contains a quantization_config section and that vLLM loads the checkpoint without falling back to a normal FP16 load.

Conversion timestamp

Generated on: 2026-05-03T01:49:08.832791+00:00

Disclaimer

This is an unofficial quantized derivative of the upstream model.

Use it only after validating quality, latency, memory usage, and correctness in your own telecom OSS/BSS workload.