Instructions to use Code4me2/elephant-1-NVFP4-MLP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Code4me2/elephant-1-NVFP4-MLP with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Code4me2/elephant-1-NVFP4-MLP")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Code4me2/elephant-1-NVFP4-MLP") model = AutoModelForMultimodalLM.from_pretrained("Code4me2/elephant-1-NVFP4-MLP", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download README.md from Code4me2/elephant-1-NVFP4-MLP: direct link, hf CLI and curl.
- Browser
- Download file 11.6 kB
-
https://huggingface.co/Code4me2/elephant-1-NVFP4-MLP/resolve/main/README.md
- Command line
-
hf download hf://Code4me2/elephant-1-NVFP4-MLP/README.md
-
curl -L -o README.md https://huggingface.co/Code4me2/elephant-1-NVFP4-MLP/resolve/main/README.md
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: baselabs/elephant-1
base_model_relation: quantized
language:
- en
tags:
- nvfp4
- compressed-tensors
- mixed-precision
- decision-model
- experimental
Elephant-1 NVFP4 — 96-MLP experimental checkpoint
Experimental, not production-qualified. Released publicly for testing and feedback. A second, independent evaluation and native-FP4 execution results (both added 2026-11) are reported below; neither establishes production qualification.
This is a mixed-precision NVFP4 W4A4 derivative of Base Labs / Baseten's Elephant-1, made from upstream revision b3f8a93462fb1051a9de031e94a7ff380e254072. The compressed checkpoint was successfully exported and reloaded. Its limited public benchmark did not establish the predeclared accuracy non-inferiority criterion; a second independent evaluation narrowly misses it on the same path and passes on two others (see below). Native-FP4 speed and memory have been measured in one independent engine on one GPU only; no claim is made for other runtimes or hardware.
What is quantized?
- Exactly 96 text MLP projections: gate/up/down projections across 32 layers.
- Stock llmcompressor basic
oneshotNVFP4: RTN weights, groups of 16, FP8 block scales, calibrated static-minmax activation global scales. - Attention/recurrent projections, embeddings, norms, and LM head remain unquantized. All 664 non-quantized tensors were compared with the source and verified bit-for-bit unchanged (331 text and 333 unused vision tensors).
lm_headremains BF16 on disk and during calibration; numerical scoring used an FP32 head.- Tensor-file storage: 18,819,722,392 → 11,873,990,288 bytes, about 36.9% smaller. This is disk storage, not measured native GPU-memory savings.
Execution and evaluation were text-only. Stock saving retains the original composite configuration and unused BF16 vision tensors. The shard named model_mtp.safetensors contains retained vision tensors; its name does not establish MTP functionality. Multimodal behavior has not been validated.
Evaluation status
Public held-out benchmark — one original paired evaluation
278 labeled tasks: 128 BANKING77, 128 LEDGAR, and 22 eligible LongBench-v2 questions. There were also 556 unlabeled support questions, excluded from accuracy and confident labeled-task gating.
| Metric | BF16 source | NVFP4 |
|---|---|---|
| Correct | 221/278 | 221/278 |
| Error rate | 20.50% | 20.50% |
| BANKING77 correct | 118/128 | 118/128 |
| LEDGAR correct | 93/128 | 95/128 |
| LongBench-v2 correct | 10/22 | 8/22 |
- Accuracy difference: 0 percentage points, paired record-bootstrap 95% interval [−2.1583, +1.7986] pp.
- The predeclared lower-bound requirement was ≥−2.0 pp. It was narrowly missed: insufficient evidence for that non-inferiority tolerance, not proof of a substantial average accuracy decline.
- Four correct answers became incorrect, four incorrect became correct, and one wrong answer changed to a different wrong answer.
- Confident labeled agreement: 229/230, using BF16 probability margin ≥0.5; this component passed its 98% criterion.
- Maximum probability drift: 41.19 pp on one labeled question. This is not an overall error rate.
- The 22 long-document labels provide limited evidence; this is not validation for any specific legal, litigation, or other production workflow.
Development repeatability — three fresh-process runs per model
128 records / 384 questions, with 72 explicit labels. This is a different set from the held-out benchmark.
| Model | Errors in runs 1 / 2 / 3 | Error rate, every run |
|---|---|---|
| BF16 | 14 / 14 / 14 | 19.44% |
| NVFP4 | 15 / 15 / 15 | 20.83% |
- Correct/incorrect status was unchanged on every labeled question across same-model repeats. One quantized answer varied between different wrong options.
- Across new processes, maximum probability drift reached 0.8141 pp for BF16 and 5.8135 pp for NVFP4.
- Against historical development scores, NVFP4 drift reached 13.3819 pp, without changing labeled correctness; startup/execution history differed.
- Short/long sentinels had exactly matching logits within each process. Only those two records were tested repeatedly within a process.
- The runtime cause is unresolved. Repeated cases do not count as additional independent labels, and three runs do not establish a universal variability bound.
Second independent evaluation (added 2026-11)
A second labeled set was built the same way from the same public sources: 278 labeled tasks (128 BANKING77, 128 LEDGAR, 22 eligible LongBench-v2) plus 556 unlabeled support questions. It is equivalent in composition to the set above, not the same records. Records, prompts and labels are not uploaded.
Quantization alone. Both arms ran in the same author-patched qev reference. The NVFP4 arm used the simulated-QDQ loading described under "Loading and use limitations". Each kernel-shape path was paired with the matching BF16 path:
| Reference path | BF16 correct | NVFP4 correct | Accuracy difference, 95% CI | Brier difference, 95% CI |
|---|---|---|---|---|
| Per question, batch one (the validated path above) | 227/278 | 225/278 | −0.72 pp [−2.52, +1.08] | +0.0175 [+0.0030, +0.0329] |
| Batched | 227/278 | 227/278 | 0.00 pp [−1.44, +1.44] | +0.0098 [−0.0032, +0.0237] |
| Sequential | 227/278 | 227/278 | 0.00 pp [−1.44, +1.44] | +0.0111 [−0.0041, +0.0264] |
- On the per-question path the lower bound again misses the predeclared ≥−2.0 pp requirement. On the batched and sequential paths it meets it.
- Every interval includes zero. This set resolves gross regressions, not differences of one or two points.
- Calibration: NVFP4 has a higher Brier score than BF16 on every path. On the per-question path the interval excludes zero.
- The answer with the highest probability differs from BF16 on 1.7–2.9% of all 834 questions. The largest single-option probability change is 0.37–0.49.
- Changing only the NVFP4 kernel shape (four QDQ paths) moves top-1 answers about as much: on 1.6–2.8% of questions. BF16 paths never disagree with one another on a top-1 answer.
Native FP4 execution in an independent engine. The same set was run through a separate, not-yet-public Rust inference engine that executes this checkpoint's MLP projections with native block-scaled FP4 GEMMs. It quantizes activations in the same way as the compressed-tensors QDQ reference; the activation quantizer was validated bit-exact against it. Other projections run in BF16. Measured on one NVIDIA GB10 (sm_121):
| BF16 reference | NVFP4, native FP4 engine | |
|---|---|---|
| Correct | 227/278 | 227/278 |
| BANKING77 / LEDGAR / LongBench-v2 | 116 / 102 / 9 | 116 / 103 / 8 |
| Brier / ECE | 0.2475–0.2481 / 0.062 | 0.2495 / 0.064 |
- The engine's outputs stayed within the statistical tolerance derived beforehand from the four QDQ reference paths. That tolerance was committed before the engine was evaluated.
- Its results are bit-identical whether a request runs alone or packed with others.
- Speed, same engine and GPU, 16,384-token packed steps: about 8,900 tokens/s, against 4,900 for BF16 (about 1.8×).
- Weight memory, text weights only: 8.3 GiB, against 14.8 GiB for BF16.
- These are single-device measurements of one engine, not a general speed claim.
Not yet tested: vLLM or any other serving runtime; any other GPU, including other Blackwell parts (sm_100, sm_120) and non-Blackwell GPUs, where FP4 kernels may be unavailable or fall back to other paths; the unlabeled score/noul question types for accuracy (they were compared for agreement only); requests longer than 16,384 tokens.
An activation-off W4A16 diagnostic was attempted previously but stopped on a W4A4 replay mismatch before disabling activation quantization. This repository contains W4A4, not W4A16.
Loading and use limitations
Elephant is a decision model, not an interchangeable causal text-generation checkpoint. Preserve the author's rendering, option ordering, slot readout, block-bidirectional full attention and causal recurrent layers.
- Documented trained/request limit: 16,384 total rendered tokens, including questions/options/instructions. The inherited positional configuration does not establish trained 128K/256K support.
- Validated numerical path: author-patched OneJev
qev, independent full forward per question, batch one,use_cache=False, temperature 1, TF32 matmul disabled. - Quantized loading used stock
CompressedTensorsConfig(run_compressed=False)to materialize quantized weights into BF16 and retain activation QDQ. This simulates W4A4 numerics, not native FP4 kernels. - Quantization metadata is retained in both the root composite config and extracted text config for the author text loader.
- The scoring integration materializes the protected LM head out of compressed-tensors offload dispatch before the author's FP32 upcast. A plain default loader is not a demonstrated substitute.
- The included
engine.pyis the unchanged upstream Decision Index adapter, retained for provenance. Its default shared-prefix/batched serving path was not the quantized evaluation path and is not certified for this artifact. Do not treat it as a ready-to-use quantized serving integration. - Upstream
setup.shtargets a different runtime. It is not included as an installation prescription for this quantized checkpoint. A portable runtime/serving package is not supplied here.
The isolated numerical evaluation used Python 3.12.3, Torch 2.13.0+cu132, Transformers 5.19.0, llmcompressor 0.15.0, compressed-tensors 0.20.1, flash-linear-attention/fla-core 0.5.2, causal-conv1d 1.7.0, accelerate 1.15.0, and author-patched OneJev commit 81ce62f1597c91e46767d4d02d6ac2e18534fe94. These are tested-environment identifiers, not a guarantee that arbitrary installations or native serving runtimes will work.
Provenance and contents
Calibration used 256 disjoint public-source records / 768 rendered questions / 2,477,543 forwarded tokens, with no truncation or gold-guided scale selection. Public sources included LexGLUE, BANKING77, arXiv summarization, PG19, CNN/DailyMail, Hermes function-calling and US tax-law QA. LongBench-v2 was evaluation-only. No evaluation/calibration corpus records, rendered prompts, gold-label files, score vectors, logs, credentials, or local operational receipts are uploaded. Public upstream tokenizer/template files and the upstream model card's illustrative example are retained.
- Original serialized checkpoint/config/tokenizer/readout/recipe bytes are retained.
provenance.jsonrecords source revision, method, runtime, and aggregate results without local operational paths.SHA256SUMSrecords file byte identities. Checksums do not establish authenticity or portability.UPSTREAM_MODEL_CARD.mdpreserves the source model card and attribution. Its original performance and usage claims concern the upstream BF16 model, not a new validation of this derivative.
Attribution and licensing
The upstream Elephant-1 model card declares MIT licensing. This derivative retains that metadata and credits Base Labs at Baseten. Elephant-1 was post-trained from Qwen/Qwen3.5-9B; retain and comply with applicable upstream notices and terms. See the pinned upstream model card for its attribution and citation. This is an independent quantized derivative, not an official Baseten release.