m3kro/openjev-NVFP4-W4A4

Quantization

This repository contains an unofficial NVIDIA NVFP4 W4A4 quantization of openjev/openjev.

The checkpoint was produced with LLM Compressor using NVFP4 weight and activation quantization on eligible Linear layers. Components excluded from the FP4 recipe, including the visual encoder, lm_head, and Qwen3.5 linear-attention / Gated DeltaNet projections, remain at higher precision.

The uploaded weights were calibrated using 1,024 text-only examples selected to resemble the decision workloads exposed by OpenJev's helper API.

Calibration datasets

The PTQ calibration set contained:

Dataset / source	Examples	Purpose
MASSIVE	320	Intent classification and routing-style decisions
MAUD	288	Legal / merger-agreement classification and binary decisions
XNLI	160	Natural-language inference, including three-way and binary entailment-style decisions
QuALITY	96	Long-context reading comprehension and multiple-choice reasoning
Synthetic browser / DOM cases	80	Generated structured DOM-like states with action/button choices
Synthetic operations cases	80	Generated invoices, policies, binary decisions, routing tasks, and ordinal scoring cases
Total	1,024

The synthetic browser examples are not MiniWoB trajectories and do not represent real browser execution. They are text-shaped structured examples intended to exercise OpenJev-style state, instruction, and candidate formatting.

The synthetic operations examples similarly exercise structured decision formats rather than representing a specific external benchmark.

PTQ calibration methodology

The 1,024 calibration examples are used to estimate activation quantization statistics for NVFP4 W4A4.

They are not used as supervised training examples, and the model is not fine-tuned on this data.

During calibration, the OpenJev helper request is captured before inference and converted into the exact prompt and token representation consumed by the model. LLM Compressor receives these model inputs for activation calibration; labels and synthetic assistant responses are not used to train the model.

This is distinct from optional probability/readout fitting, which uses a separate labelled fitting split after quantization.

Dataset separation

Calibration, probability-fitting, and evaluation examples are kept separate.

Grouping is performed at a source-appropriate level to reduce leakage between splits:

• MASSIVE examples are grouped by translation ID.
• XNLI examples are grouped by parallel-example ID.
• MAUD examples are grouped by contract.
• QuALITY examples are grouped by article.
• Identical shared states detected across groups are excluded from later partitions.

The evaluation set was not included in the PTQ calibration corpus.

Public training examples provide calibration/fitting data where applicable, while evaluation uses held-out partitions selected by the pipeline. QuALITY uses its labelled development split for evaluation.

These precautions reduce split leakage, but they cannot establish whether any public examples were present in the original OpenJev or underlying model training data.

Evaluation

The uploaded checkpoint was evaluated on a separate 2,048-example text/DOM regression suite against the BF16 openjev/openjev reference.

Metric	BF16	NVFP4 W4A4
Accuracy	90.2344%	89.9902%
Accuracy delta	—	-0.244 pp
ECE, 15 bins	0.02644	0.02569
Brier score	0.14283	0.14227
Median latency	325.24 ms	197.73 ms

Additional results:

• Answer flip rate: 2.34%
• Mean per-example probability MAE: 0.01596
• 95th percentile probability MAE: 0.0814
• Group-bootstrap 95% interval for accuracy delta: [-0.848, +0.341] percentage points

The measured median latency decreased from 325.24 ms to 197.73 ms, approximately a 39% reduction in this sequential evaluation setup.

The latency numbers are sequential warm-service measurements that include client/helper overhead and may benefit from prefix caching. They should not be interpreted as maximum batched serving throughput.

The bootstrap confidence interval means that this evaluation is not a statistical proof of ≤0.5 percentage-point noninferiority.

Evaluation scope

This evaluation is not the official OpenJev benchmark.

In particular:

• It does not reproduce OpenJev's published held-out 10,000-question benchmark.
• MAUD is used as a representative legal dataset; this does not imply that MAUD is part of OpenJev's official benchmark.
• The browser examples are generated textual DOM-like fixtures, not real browser-agent trajectories.
• No screenshot or vision evaluation was performed.
• Retaining the visual encoder at higher precision does not establish vision equivalence.

The reported numbers should therefore be interpreted as results on the 2,048-example text/DOM regression suite described above.

Tested serving environment

The uploaded checkpoint has been tested with:

• Hardware: NVIDIA DGX Spark
• GPU: NVIDIA GB10 / SM121
• vLLM: 0.29.0
• CUDA runtime: 13
• NVFP4 linear backend: flashinfer_b12x

Example serving configuration:

vllm serve /path/to/openjev-NVFP4-W4A4 \
  --served-model-name openjev \
  --dtype bfloat16 \
  --max-model-len 16384 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.75 \
  --enable-prefix-caching \
  --max-logprobs 64 \
  --gdn-prefill-backend triton \
  --linear-backend flashinfer_b12x \
  --trust-remote-code \
  --enforce-eager

On NVIDIA GB10 / SM121, the tested serving path uses a CUDA 13 environment with flashinfer_b12x.

Reproducibility and provenance

This repository includes metadata produced during quantization and verification:

• recipe.yaml — quantization recipe
• ARTIFACT.json — checkpoint artifact metadata
• OPENJEV_NVFP4_RECEIPT.json — quantization receipt and provenance
• VERIFIED.json — verification record for the uploaded checkpoint
• verification/ — retained verification artifacts
• SOURCE.json — source-model information
• UPSTREAM_MODEL_CARD.md — retained upstream model-card information

These files are included to make the quantization configuration, source model, and verification process easier to inspect.

License and attribution

This is a derivative quantization of openjev/openjev.

Please review the upstream model's license, attribution requirements, and usage restrictions before using or redistributing this checkpoint. Dataset-specific licenses and terms also apply to their respective source datasets.