Inkling 2 Small β EXL3 2.59375-bit experts
A calibrated EXL3 export for text inference across two 128 GB NVIDIA DGX Sparks. The original indexed artifact set is 208,361,252,076 bytes (194.052 GiB). The bitrate applies to expert weights; dense tensors retain their published precision. This is a custom Inkling adapter checkpoint, not a drop-in model for stock Transformers, vLLM, or ExLlamaV3's standard server.
Use the standalone two-Spark recipe at kortexa-ai/inkling2-exl3-spark. The recipe uses public runtime sources and plain shell launchers.
Quantization
Source: thinkingmachines/Inkling-2-Small,
revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b.
All 48 layers and 384 experts per layer are included. Layer 0 and layers 40β47
use K3. In layers 1β39, the 192 experts with the most calibration routes use
K3 and the rest K2. Mean expert bitrate: 2.59375.
Native NVFP4/MXFP8 activation quantization is retained. Unembedding produces
FP32 outputs. Vision/draft auxiliary tensors are preserved, but this runtime
currently supports text inference only.
Measured validation
A disjoint held-out panel used 8,192 real-text tokens from 17 conversations, with 1,088 fixed comparison positions. Source FP4 accumulation was emulated; these numbers do not establish parity with a full upstream inference engine.
| Metric | Value |
|---|---|
| Native-reference NLL | 1.341903329 |
| Converted NLL | 1.348285437 |
| NLL increase | 0.006382108 |
| Mean KL divergence | 0.022117095 |
| Top-1 agreement | 95.956% |
An actual two-Spark trial passed with 8,128 prompt tokens, up to 64 new
tokens admitted within 8,192 total. It recovered early, middle, and late
needles and answered the arithmetic question: orchid, 47, nebula, 42.
Prefill took 324.924 seconds; total generation took 334.030 seconds for 13
output tokens. Peak allocated GPU memory was 92.228 / 98.198 GiB; peak reserved
memory was 92.307 / 98.283 GiB. Available host memory at completion was
23.260 / 16.373 GiB. The memory guard stayed clear. See
validation/spark-8k.json.
8K is the validated full-model context. 512K is the next runtime target and has not yet passed a full-model trial. The source metadata advertises a 1,048,576-token maximum. Larger limits require attention workspace changes and separate validation; changing a command-line limit alone is insufficient. No throughput or long-context quality claim is made beyond these measurements.
Standalone two-Spark validation
The public-source recipe at commit 80b4bd64c2cb9888f5d2beb3a990b667d13132b2 passed its own clean 8K
trial: 8,128 prompt tokens, 11 generated tokens, and the final answer
orchid, 47, nebula, 42. Prefill took 409.514 seconds;
total generation including prefill took 418.313 seconds
(loading and setup excluded). Peak GPU allocations were
91.090 / 97.754 GiB.
The host-memory guard stayed clear.
The native K2/K3 codec checks passed 20 projection/row cases per host, with
maximum relative L2 error 1.75e-06 against independent
fixtures and exact NVFP4/MXFP8 activation bytes. The Gloo residual round trip
was exact. Each GB10 also held all four global 512K FP8 caches and passed
single-token and 128-row attention checks, with a 4.187 GiB
peak allocation. These isolated capacity checks do not qualify full-model
512K quality or throughput. The CLI explicitly opens the assistant text block
for thinking effort zero. See validation/portable-spark-8k.json for the receipt and validation/portable-spark-8k-prompt.txt for the exact synthetic
prompt. The downloader pins the verified immutable weight revision 05c6cd0dff9df5bf20b10419c663a1e2adc7eb64.
Integrity and layout
checkpoint.json indexes all 2,363 weight/tokenizer artifacts and their
SHA-256 hashes. ranks/rank-0.json and ranks/rank-1.json select the two
24-layer stages. The original conversion manifest is retained, compressed,
in provenance/conversion-manifest.json.gz; its uncompressed SHA-256 is
f92ee205c5154d1c255ea2316cb3b0e33c179b523cda21caacba83d7d772a239.
The inference manifest omits conversion-only per-expert statistics without
changing any weight bytes. Download only the rank inventory assigned to each
Spark. Keep at least 125 GiB of free disk per host for its model shard, in
addition to Docker/build space.
Original weights are Apache-2.0; see LICENSE and NOTICE. EXL3's codec is MIT.
Native TensorFold and optional DSpark
The experimental native two-Spark recipe
uses whole-expert EP2, attention TP2 and exact BF16 KV. It serves one greedy
text request. Its optional five-layer BF16 DSpark adapter uses the preserved
MTP file and an additional 103,326,656-byte folded head outside the original
checkpoint manifest. The downloader verifies its SHA256 and pins revision
f82bc6f03ec8c61a1d3afd155c820b71bf05bf7c; target/MTP files remain pinned to
05c6cd0dff9df5bf20b10419c663a1e2adc7eb64.
Matched A-B-B-A on two GB10s, with 256-token outputs and a 512K allocation:
| Request | Plain tok/s | DSpark tok/s |
|---|---|---|
| Short prose | 16.59 | 18.33 |
| Code | 16.59 | 19.76 |
| 8K | 11.77 | 12.37 |
| 32K | 7.47 | 6.74 |
All 4096 output token IDs match. The byte audit and cancellation/reset canary also pass. Short prefill is about 5.1% slower; long prefill stays within 1%. Drafting remains opt-in because the 32K regression fails the promotion bar. The selected plain profile is preserved. Minimum observed free host memory is 6.17 GiB; allocating 512K does not qualify full 512K prefill or retrieval quality. See the draft recipe and full receipt.
Model tree for kortexa-ai/Inkling-2-Small-EXL3-2.59bpw
Base model
thinkingmachines/Inkling-2-Small