Spaces:

PYTHAI
/

bankml

Running

bankml / docs /research.md

bankML: the whole source (github.com/cryptoAGI/bankml @ 12ae409) and its page, with the bankML persona; the live engine (Dockerfile, hf/start.sh) ready for Docker hardware

28c70af verified 26 days ago

preview code

Raw History Blame Contribute Delete

16.8 kB

# Research: where bankml stands

*A survey of Rust-native LLM inference, 1-bit and ternary CPU inference, and verifiable inference, made on
2026-09-28/29 to answer one question: is bankml at the cutting edge, and where is it not? Every link below was
checked to resolve on those dates unless marked. bankml's own numbers are its own measurements
([TECHNICAL.md](TECHNICAL.md), [PERFORMANCE.md](PERFORMANCE.md), [`testing/results/`](../testing/results/)); no third
party has benchmarked bankml.*

*Updated 2026-10-06 for bankml's own status only: v0.3.6 is the latest release, and 0.3.7–0.3.9 are unreleased
([CHANGELOG.md](../CHANGELOG.md)). The survey of other projects is as found on 2026-09-28/29 and has not been redone;
their state may have moved since. Where bankml's status changed, the original statement is kept and the change is
marked.*

## The answer in brief

**Where bankml is at or near the edge**

1. **Bit-exact reproduction of the compiled llama.cpp library, tested by an oracle, from Rust.** bankml's `Q1_0` and
   `Q2_0_g64` kernels are compared against the shipped ggml shared objects of llama.cpp b11192. The check covers every
   weight, every `q8_0` activation row and every dot product, and matches the compiler's fused multiply-add rounding.
   No other project was found that does this against the *binary*. The nearest are:
   - [OxiLLaMa](https://github.com/cool-japan/oxillama), which claims logit parity within a tolerance;
   - [Frink](https://github.com/antonellof/frink), whose quantizer output is byte-identical but whose matmul is not
     claimed to be;
   - [bitnet-rs](https://github.com/lilyco-42/bitnet-rs), which has byte-level unit tests against its C++ baseline.
   llama.cpp's own [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348) shows why the FMA detail matters:
   its kernel matched bit for bit in isolation, but perplexity still shifted once the compiler contracted it
   differently.
2. **A plain-AVX2 kernel for `Q2_0_g64`, a gap upstream has not closed.** At b11192, llama.cpp has only scalar code
   for `Q2_0` on x86:
   - [PR #24448](https://github.com/ggml-org/llama.cpp/pull/24448) (merged 2026-07-07) is "CPU only, ARM NEON +
     generic scalar fallback";
   - the x86 kernel in [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348) needs AVX-VNNI and is still
     open.
   bankml's AVX2 kernel is 9.5–9.8× the reference per projection, bit-exact. **Caveat:** that speedup is measured
   against a scalar baseline. PrismML's own fork has AVX2 kernels for its *other* ternary format (`PQ2_0`, [PR
   #206](https://github.com/PrismML-Eng/llama.cpp/pull/206), about 3× over its own baseline), which is a different
   format and not directly comparable.
3. **A pinned-model gateway with a receipt on every answer, in one zero-dependency static binary.** Model hashes
   and answer hashes are common inside TEE and attestation stacks, but rare as a simple local gate. **Caveat:**
   bankml's receipts are not signed or hardware-attested. They prove what this gateway served, not where or by whom
   (§3 below).

**Where others are clearly ahead**

- **A full forward pass in Rust.** [candle](https://github.com/huggingface/candle),
  [mistral.rs](https://github.com/EricLBuehler/mistral.rs), [Frink](https://github.com/antonellof/frink),
  [OxiLLaMa](https://github.com/cool-japan/oxillama) and [bitnet-rs](https://github.com/lilyco-42/bitnet-rs) run
  whole models natively. bankml answers through llama-server; its own Qwen3 forward pass is phase P3.
  *Since then:* bankml's own forward pass generates llama.cpp's tokens since 0.2.7 (1-bit) and 0.2.8 (ternary), and
  `bankml serve --native` answers from it since 0.3.0; the Llama graph followed in 0.3.4
  ([TECHNICAL.md](TECHNICAL.md) §IV.4). Unlike the engines above, it is token-identical to llama-server b11192 in the
  release gate ([oracles.md](oracles.md)).
- **GPUs.** mistral.rs, Frink and PrismML's fork run on CUDA and/or Metal; bankml is CPU-only by design.
  *Since then:* a Vulkan worker with bankml's own SPIR-V takes a share of each 1-bit product, used only after it is
  bit-exact on the card (0.2.12–0.2.14). On the one integrated card measured (a Radeon Vega 3) decode did not change
  within noise ([modules/gpu.md](modules/gpu.md)). The engines above remain far ahead on GPUs.
- **Breadth of formats.** OxiLLaMa lists K, IQ, TQ1_0, TQ2_0 and Q1_0_G128 (all "Alpha"); bankml has its own kernels
  for two formats and serves the rest through the reference. *Since then:* a third, F16 (0.3.4); every other format
  is still served only through the reference.
- **ARM.** Upstream `Q1_0`/`Q2_0` have NEON; bankml's kernels are x86 AVX2 only.
- **End-to-end speed leadership.** It is not shown: bankml's gains are per kernel, and at the whole-token level
  the decode-budget ratio against the reference ranges from 0.87× to 1.62× across thread counts in the 0.1.5 gate
  record, measured on a laptop that was also serving a model, so it is noisy and not a claim of end-to-end leadership.
  *Since then:* with its own forward pass bankml decodes the ternary model at 2.32–2.41 tokens/s against
  llama-server's 0.30 on the same laptop (CHANGELOG 0.2.8). On the 1-bit model llama-server was still faster at 0.3.0
  (2.8 against 1.9–2.0 tokens/s), and parity there is not yet claimed ([PERFORMANCE.md](PERFORMANCE.md)).
- **Attestation.** [EigenAI](https://arxiv.org/abs/2602.00182) (bit-exact GPU inference on a modified llama.cpp,
  re-executed in a TEE) and TEE serving stacks give receipts a third party can trust. bankml's receipts do not.

## 1. Rust-native inference engines

| project | what it runs on CPU | own kernels? | vs llama.cpp on CPU | exactness testing |
|---|---|---|---|---|
| [candle](https://github.com/huggingface/candle) (Hugging Face) | GGUF K-quants ([k_quants.rs](https://github.com/huggingface/candle/blob/main/candle-core/src/quantized/k_quants.rs)); x86 repack in progress, aarch64 in [PR #3697](https://github.com/huggingface/candle/pull/3697) | yes (AVX2/NEON) | no published parity numbers | none found |
| [mistral.rs](https://github.com/EricLBuehler/mistral.rs) | GGUF Q/K, ISQ, GPTQ, AWQ, HQQ, FP8 | yes (on candle) | published wins are CUDA; no CPU comparison | none found |
| [burn](https://github.com/tracel-ai/burn) / [models](https://github.com/tracel-ai/models) | a general deep-learning framework | yes | not a GGUF-quant CPU engine | — |
| [rustformers/llm](https://github.com/rustformers/llm) | archived, "Unmaintained" (last push 2024-06) | called ggml | — | — |
| [lm.rs](https://github.com/samuel-vitorino/lm.rs) | a minimal engine (last push 2024-10) | yes | — | — |
| [ratchet](https://github.com/huggingface/ratchet) | browser / WebGPU | — | — | — |
| [Crane](https://github.com/lucasjinreal/Crane), [kalosm](https://github.com/floneum/kalosm) | built on candle | — | — | — |
| [OxiLLaMa](https://github.com/cool-japan/oxillama) (2026) | K, IQ, TQ1_0/TQ2_0, Q1_0_G128, all "Alpha" | yes (AVX2/AVX-512/NEON) | targets ≥ 80 % of llama.cpp | top-1 logit parity 32/32, within tolerance, not bit-exact |
| [Frink](https://github.com/antonellof/frink) (2026; [blog](https://www.fratepietro.com/2026/frink-rust-gguf-inference-engine/)) | K, IQ, MXFP4, MoE; CPU, Metal, CUDA | yes | 1.41–5.06× **slower** than llama.cpp on CPU, by its author | quantizer bytes identical on Q8_0, IQ4_NL |
| [llama-gguf](https://github.com/Lexmata/llama-gguf) | GGUF | yes | 0.3 tok/s on Mistral-7B; "correctness over speed" | — |

## 2. 1-bit and ternary inference

**Upstream llama.cpp** ([releases](https://github.com/ggml-org/llama.cpp/releases); latest b11243):
- `Q1_0` arrived in [PR #21273](https://github.com/ggml-org/llama.cpp/pull/21273) (merged 2026-04-06, NEON and scalar
  only), and x86 SSSE3→AVX2+FMA in [PR #21636](https://github.com/ggml-org/llama.cpp/pull/21636) (merged 2026-04-20).
- `Q2_0` is in [PR #24448](https://github.com/ggml-org/llama.cpp/pull/24448) (merged 2026-07-07; x86 "later").
- The only x86 `Q2_0` kernel is VNNI-only, in [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348)
  (open; 2.1–3.6×), which does not help a plain-AVX2 CPU such as bankml's test Ryzen 3200U.
- The older ternary types `TQ1_0`/`TQ2_0` come from [PR #10010](https://github.com/ggml-org/llama.cpp/pull/10010),
  and the formats are discussed in [#22019](https://github.com/ggml-org/llama.cpp/discussions/22019).
- [Issue #29351](https://github.com/ggml-org/llama.cpp/issues/29351) (2026-09-24) found an x86 `Q8_0` sign bug by
  checking against the generic C kernel, which is the method bankml's oracle uses.

**PrismML** (the Bonsai models): [Bonsai-8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), [announcement](https://prismml.com/news/bonsai-8b),
[docs](https://docs.prismml.com/run/llamacpp), and the [fork](https://github.com/PrismML-Eng/llama.cpp), which has
x86 kernels for PrismML's newer formats:
- `PQ2_0` AVX2/AVX-VNNI: [PR #206](https://github.com/PrismML-Eng/llama.cpp/pull/206), merged 2026-09-21, about 3×
  decode on Ternary-Bonsai-2-27B;
- `Q1_0` 4×8 repack: [PR #233](https://github.com/PrismML-Eng/llama.cpp/pull/233);
- `PQ1_0`: [PR #255](https://github.com/PrismML-Eng/llama.cpp/pull/255);
- `PTQ1_0`: [PR #250](https://github.com/PrismML-Eng/llama.cpp/pull/250).

**Microsoft:**
- [bitnet.cpp](https://github.com/microsoft/BitNet) has kernels I2_S, TL1 (2 weights → 4-bit index) and TL2
  (3 weights → 5-bit). It reports 2.37–6.17× on x86 and 72–82 % less energy.
- [T-MAC](https://github.com/microsoft/T-MAC) does lookup-table mixed-precision GEMM, up to 6.6× per kernel and 2.8×
  end to end over llama.cpp.
- Both work on BitNet-style ternary weights, a different format from ggml's `Q2_0`.

**In Rust:**
- [bitnet-rs](https://github.com/lilyco-42/bitnet-rs) (NEON);
- [alice-aegis](https://github.com/Aefinity-AI/alice-aegis) (2026), a `no_std` UEFI BitNet engine with frozen
  integer semantics, bit-identical digests across ISAs and **SHA-256 receipts chaining the logits**. It is the
  closest peer to bankml's receipts, but it checks against itself, not against llama.cpp;
- [bitnet-toy](https://github.com/tidynest/bitnet-toy);
- [bitnet-llm](https://crates.io/crates/bitnet-llm) (FFI bindings).

No Rust engine other than bankml was found with `Q2_0_g64`; OxiLLaMa lists `Q1_0_G128`.

## 3. Verifiable and attested inference

| approach | projects | what it proves | bankml's relation |
|---|---|---|---|
| zero-knowledge proofs | [EZKL](https://github.com/zkonduit/ezkl) (Rust, small models); [zkLLM](https://arxiv.org/abs/2404.16109); [DeepProve](https://eprint.iacr.org/2026/1112); [OpenLLM](https://eprint.iacr.org/2026/1578) | that a computation was done, with no trust in the prover; [Hollow-LLM](https://arxiv.org/abs/2607.28884) shows ZK correctness alone does not prove a real large model ran | none; far heavier |
| optimistic re-execution | [opML](https://github.com/ora-io/opml) ([paper](https://arxiv.org/abs/2401.17555)); [Optimistic TEE-Rollups](https://arxiv.org/abs/2512.20176) | fraud is challengeable by re-running | bankml's bit-exactness is what re-execution needs, but it has no dispute protocol |
| TEEs | [EigenAI](https://arxiv.org/abs/2602.00182); [OpenPCC](https://arxiv.org/abs/2606.11145); [Phala private AI inference](https://phala.com/solutions/private-ai-inference) | signed receipts from attested hardware | bankml's receipts are unsigned, with no hardware root |
| model pin + answer hash | bankml | which verified file served, and that the text and request are unchanged | honest scope: integrity between a client and its own gateway |

## 4. What this means for bankml's roadmap

- **P3 (its own forward pass) is where the field already is**, so it is not optional if bankml is to stand beside
  candle and OxiLLaMa. Its distinct contribution would be a forward pass *bit-exact against the compiled reference*,
  which nobody else claims. *Done since:* 0.2.1–0.3.0 ([BUILD_HISTORY.md](BUILD_HISTORY.md)).
- **A plain-AVX2 `Q2_0` kernel is worth upstreaming** (or offering to PrismML): the gap is real, current and
  documented upstream. *Prepared since:* `upstream/q2_0_avx2.c` (0.2.0), bit-exact against the shipped library on
  200,000 of 200,000 random cases; submitting it is the authors' decision, and nothing has been submitted.
- **Receipts:** signing them (an operator key) and binding them to a re-executable bit-exact run would move them
  from integrity toward verifiability, the direction EigenAI shows.
- **NEON** would bring the kernels to ARM, where upstream `Q1_0`/`Q2_0` already are.
## Papers
**Binary and ternary networks**
1. Courbariaux, Bengio, David. [*BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations*](https://arxiv.org/abs/1511.00363). NeurIPS 2015.
2. Rastegari, Ordonez, Redmon, Farhadi. [*XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks*](https://arxiv.org/abs/1603.05279). ECCV 2016.
3. Li, Zhang, Liu. [*Ternary Weight Networks*](https://arxiv.org/abs/1605.04711). 2016.
4. Zhu, Han, Mao, Dally. [*Trained Ternary Quantization*](https://arxiv.org/abs/1612.01064). ICLR 2017.
**Quantized and 1-bit language models**
5. Dettmers, Lewis, Belkada, Zettlemoyer. [*LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale*](https://arxiv.org/abs/2208.07339). NeurIPS 2022.
6. Frantar, Ashkboos, Hoefler, Alistarh. [*GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers*](https://arxiv.org/abs/2210.17323). ICLR 2023.
7. Wang et al. [*BitNet: Scaling 1-bit Transformers for Large Language Models*](https://arxiv.org/abs/2310.11453). 2023.
8. Ma et al. [*The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits*](https://arxiv.org/abs/2402.17764). 2024.
9. Ma et al. [*BitNet b1.58 2B4T Technical Report*](https://arxiv.org/abs/2504.12285). 2025.
**CPU kernels for 1-bit and ternary inference**
10. Wang, Zhou, Song et al. [*1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs*](https://arxiv.org/abs/2410.16144). 2024.
11. Wang, Zhou, Song et al. [*Bitnet.cpp: Efficient Edge Inference for Ternary LLMs*](https://arxiv.org/abs/2502.11880). 2025.
12. Wei et al. [*T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge*](https://arxiv.org/abs/2407.00088). EuroSys 2025.
13. Li, Yin, Wang et al. [*Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference*](https://arxiv.org/abs/2512.06443). 2025.
**Verifiable and attested inference**
14. Sun, Li, Zhang. [*zkLLM: Zero Knowledge Proofs for Large Language Models*](https://arxiv.org/abs/2404.16109). CCS 2024.
15. Conway, So, Yu et al. [*opML: Optimistic Machine Learning on Blockchain*](https://arxiv.org/abs/2401.17555). 2024.
16. Chan, Ding, Chen et al. [*Optimistic TEE-Rollups*](https://arxiv.org/abs/2512.20176). 2025.
17. Alves, Patankar, Pereira et al. [*EigenAI: Deterministic Inference, Verifiable Results*](https://arxiv.org/abs/2602.00182). 2026.
18. Cankaya. [*Bit-Exact AI Inference Verification Without Performance Tradeoffs*](https://arxiv.org/abs/2606.00279). 2026 (vLLM and Hugging Face on GPU, not llama.cpp).
19. Gailly et al. [*DeepProve: Verifiable End-to-End LLM Inference*](https://eprint.iacr.org/2026/1112). IACR ePrint 2026/1112.
20. Yang, Ren, Xu, Zhang et al. [*OpenLLM: Modular and Scalable zkSNARKs for Verifiable LLM Inference*](https://eprint.iacr.org/2026/1578). IACR ePrint 2026/1578.
21. Gong, Liu, Li. [*Hollow-LLM Attack*](https://arxiv.org/abs/2607.28884). 2026.
22. Zhou, Zhao, Wang et al. [*OpenPCC: Open and Confidential LLM Serving on Commodity TEEs*](https://arxiv.org/abs/2606.11145). 2026.
**Retrieval, embedding and commitments used by bankml**
23. Cormack, Clarke, Büttcher. [*Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods*](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf). SIGIR 2009. [doi:10.1145/1571941.1572114](https://doi.org/10.1145/1571941.1572114) (the DOI refuses scripts; the author's PDF resolves).
24. Chen, Xiao, Zhang et al. [*M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation*](https://arxiv.org/abs/2402.03216) (BGE-M3). 2024.
25. Laurie, Langley, Kasper. [*RFC 6962: Certificate Transparency*](https://www.rfc-editor.org/rfc/rfc6962). 2013.
26. Laurie, Messeri, Stradling. [*RFC 9162: Certificate Transparency Version 2.0*](https://www.rfc-editor.org/rfc/rfc9162). 2021.
*Unverified details, stated as found:*
- The author lists of papers 3, 4, 12, 25 and 26 are as commonly cited, not re-fetched.
- The venues of papers 12 and 14 come from secondary pages.
- Every project's own speed claims are the project's, not reproduced here.

	# Research: where bankml stands

	*A survey of Rust-native LLM inference, 1-bit and ternary CPU inference, and verifiable inference, made on
	2026-09-28/29 to answer one question: is bankml at the cutting edge, and where is it not? Every link below was
	checked to resolve on those dates unless marked. bankml's own numbers are its own measurements
	([TECHNICAL.md](TECHNICAL.md), [PERFORMANCE.md](PERFORMANCE.md), [`testing/results/`](../testing/results/)); no third
	party has benchmarked bankml.*

	*Updated 2026-10-06 for bankml's own status only: v0.3.6 is the latest release, and 0.3.7–0.3.9 are unreleased
	([CHANGELOG.md](../CHANGELOG.md)). The survey of other projects is as found on 2026-09-28/29 and has not been redone;
	their state may have moved since. Where bankml's status changed, the original statement is kept and the change is
	marked.*

	## The answer in brief

	Where bankml is at or near the edge

	1. Bit-exact reproduction of the compiled llama.cpp library, tested by an oracle, from Rust. bankml's `Q1_0` and
	`Q2_0_g64` kernels are compared against the shipped ggml shared objects of llama.cpp b11192. The check covers every
	weight, every `q8_0` activation row and every dot product, and matches the compiler's fused multiply-add rounding.
	No other project was found that does this against the binary. The nearest are:
	- [OxiLLaMa](https://github.com/cool-japan/oxillama), which claims logit parity within a tolerance;
	- [Frink](https://github.com/antonellof/frink), whose quantizer output is byte-identical but whose matmul is not
	claimed to be;
	- [bitnet-rs](https://github.com/lilyco-42/bitnet-rs), which has byte-level unit tests against its C++ baseline.

	llama.cpp's own [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348) shows why the FMA detail matters:
	its kernel matched bit for bit in isolation, but perplexity still shifted once the compiler contracted it
	differently.
	2. A plain-AVX2 kernel for `Q2_0_g64`, a gap upstream has not closed. At b11192, llama.cpp has only scalar code
	for `Q2_0` on x86:
	- [PR #24448](https://github.com/ggml-org/llama.cpp/pull/24448) (merged 2026-07-07) is "CPU only, ARM NEON +
	generic scalar fallback";
	- the x86 kernel in [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348) needs AVX-VNNI and is still
	open.

	bankml's AVX2 kernel is 9.5–9.8× the reference per projection, bit-exact. Caveat: that speedup is measured
	against a scalar baseline. PrismML's own fork has AVX2 kernels for its other ternary format (`PQ2_0`, [PR
	#206](https://github.com/PrismML-Eng/llama.cpp/pull/206), about 3× over its own baseline), which is a different
	format and not directly comparable.
	3. A pinned-model gateway with a receipt on every answer, in one zero-dependency static binary. Model hashes
	and answer hashes are common inside TEE and attestation stacks, but rare as a simple local gate. Caveat:
	bankml's receipts are not signed or hardware-attested. They prove what this gateway served, not where or by whom
	(§3 below).

	Where others are clearly ahead

	- A full forward pass in Rust. [candle](https://github.com/huggingface/candle),
	[mistral.rs](https://github.com/EricLBuehler/mistral.rs), [Frink](https://github.com/antonellof/frink),
	[OxiLLaMa](https://github.com/cool-japan/oxillama) and [bitnet-rs](https://github.com/lilyco-42/bitnet-rs) run
	whole models natively. bankml answers through llama-server; its own Qwen3 forward pass is phase P3.
	Since then: bankml's own forward pass generates llama.cpp's tokens since 0.2.7 (1-bit) and 0.2.8 (ternary), and
	`bankml serve --native` answers from it since 0.3.0; the Llama graph followed in 0.3.4
	([TECHNICAL.md](TECHNICAL.md) §IV.4). Unlike the engines above, it is token-identical to llama-server b11192 in the
	release gate ([oracles.md](oracles.md)).
	- GPUs. mistral.rs, Frink and PrismML's fork run on CUDA and/or Metal; bankml is CPU-only by design.
	Since then: a Vulkan worker with bankml's own SPIR-V takes a share of each 1-bit product, used only after it is
	bit-exact on the card (0.2.12–0.2.14). On the one integrated card measured (a Radeon Vega 3) decode did not change
	within noise ([modules/gpu.md](modules/gpu.md)). The engines above remain far ahead on GPUs.
	- Breadth of formats. OxiLLaMa lists K, IQ, TQ1_0, TQ2_0 and Q1_0_G128 (all "Alpha"); bankml has its own kernels
	for two formats and serves the rest through the reference. Since then: a third, F16 (0.3.4); every other format
	is still served only through the reference.
	- ARM. Upstream `Q1_0`/`Q2_0` have NEON; bankml's kernels are x86 AVX2 only.
	- End-to-end speed leadership. It is not shown: bankml's gains are per kernel, and at the whole-token level
	the decode-budget ratio against the reference ranges from 0.87× to 1.62× across thread counts in the 0.1.5 gate
	record, measured on a laptop that was also serving a model, so it is noisy and not a claim of end-to-end leadership.
	Since then: with its own forward pass bankml decodes the ternary model at 2.32–2.41 tokens/s against
	llama-server's 0.30 on the same laptop (CHANGELOG 0.2.8). On the 1-bit model llama-server was still faster at 0.3.0
	(2.8 against 1.9–2.0 tokens/s), and parity there is not yet claimed ([PERFORMANCE.md](PERFORMANCE.md)).
	- Attestation. [EigenAI](https://arxiv.org/abs/2602.00182) (bit-exact GPU inference on a modified llama.cpp,
	re-executed in a TEE) and TEE serving stacks give receipts a third party can trust. bankml's receipts do not.

	## 1. Rust-native inference engines

	\| project \| what it runs on CPU \| own kernels? \| vs llama.cpp on CPU \| exactness testing \|
	\|---\|---\|---\|---\|---\|
	\| [candle](https://github.com/huggingface/candle) (Hugging Face) \| GGUF K-quants ([k_quants.rs](https://github.com/huggingface/candle/blob/main/candle-core/src/quantized/k_quants.rs)); x86 repack in progress, aarch64 in [PR #3697](https://github.com/huggingface/candle/pull/3697) \| yes (AVX2/NEON) \| no published parity numbers \| none found \|
	\| [mistral.rs](https://github.com/EricLBuehler/mistral.rs) \| GGUF Q/K, ISQ, GPTQ, AWQ, HQQ, FP8 \| yes (on candle) \| published wins are CUDA; no CPU comparison \| none found \|
	\| [burn](https://github.com/tracel-ai/burn) / [models](https://github.com/tracel-ai/models) \| a general deep-learning framework \| yes \| not a GGUF-quant CPU engine \| — \|
	\| [rustformers/llm](https://github.com/rustformers/llm) \| archived, "Unmaintained" (last push 2024-06) \| called ggml \| — \| — \|
	\| [lm.rs](https://github.com/samuel-vitorino/lm.rs) \| a minimal engine (last push 2024-10) \| yes \| — \| — \|
	\| [ratchet](https://github.com/huggingface/ratchet) \| browser / WebGPU \| — \| — \| — \|
	\| [Crane](https://github.com/lucasjinreal/Crane), [kalosm](https://github.com/floneum/kalosm) \| built on candle \| — \| — \| — \|
	\| [OxiLLaMa](https://github.com/cool-japan/oxillama) (2026) \| K, IQ, TQ1_0/TQ2_0, Q1_0_G128, all "Alpha" \| yes (AVX2/AVX-512/NEON) \| targets ≥ 80 % of llama.cpp \| top-1 logit parity 32/32, within tolerance, not bit-exact \|
	\| [Frink](https://github.com/antonellof/frink) (2026; [blog](https://www.fratepietro.com/2026/frink-rust-gguf-inference-engine/)) \| K, IQ, MXFP4, MoE; CPU, Metal, CUDA \| yes \| 1.41–5.06× slower than llama.cpp on CPU, by its author \| quantizer bytes identical on Q8_0, IQ4_NL \|
	\| [llama-gguf](https://github.com/Lexmata/llama-gguf) \| GGUF \| yes \| 0.3 tok/s on Mistral-7B; "correctness over speed" \| — \|

	## 2. 1-bit and ternary inference

	Upstream llama.cpp ([releases](https://github.com/ggml-org/llama.cpp/releases); latest b11243):
	- `Q1_0` arrived in [PR #21273](https://github.com/ggml-org/llama.cpp/pull/21273) (merged 2026-04-06, NEON and scalar
	only), and x86 SSSE3→AVX2+FMA in [PR #21636](https://github.com/ggml-org/llama.cpp/pull/21636) (merged 2026-04-20).
	- `Q2_0` is in [PR #24448](https://github.com/ggml-org/llama.cpp/pull/24448) (merged 2026-07-07; x86 "later").
	- The only x86 `Q2_0` kernel is VNNI-only, in [PR #26348](https://github.com/ggml-org/llama.cpp/pull/26348)
	(open; 2.1–3.6×), which does not help a plain-AVX2 CPU such as bankml's test Ryzen 3200U.
	- The older ternary types `TQ1_0`/`TQ2_0` come from [PR #10010](https://github.com/ggml-org/llama.cpp/pull/10010),
	and the formats are discussed in [#22019](https://github.com/ggml-org/llama.cpp/discussions/22019).
	- [Issue #29351](https://github.com/ggml-org/llama.cpp/issues/29351) (2026-09-24) found an x86 `Q8_0` sign bug by
	checking against the generic C kernel, which is the method bankml's oracle uses.

	PrismML (the Bonsai models): [Bonsai-8B](https://huggingface.co/prism-ml/Bonsai-8B-gguf), [announcement](https://prismml.com/news/bonsai-8b),
	[docs](https://docs.prismml.com/run/llamacpp), and the [fork](https://github.com/PrismML-Eng/llama.cpp), which has
	x86 kernels for PrismML's newer formats:
	- `PQ2_0` AVX2/AVX-VNNI: [PR #206](https://github.com/PrismML-Eng/llama.cpp/pull/206), merged 2026-09-21, about 3×
	decode on Ternary-Bonsai-2-27B;
	- `Q1_0` 4×8 repack: [PR #233](https://github.com/PrismML-Eng/llama.cpp/pull/233);
	- `PQ1_0`: [PR #255](https://github.com/PrismML-Eng/llama.cpp/pull/255);
	- `PTQ1_0`: [PR #250](https://github.com/PrismML-Eng/llama.cpp/pull/250).

	Microsoft:
	- [bitnet.cpp](https://github.com/microsoft/BitNet) has kernels I2_S, TL1 (2 weights → 4-bit index) and TL2
	(3 weights → 5-bit). It reports 2.37–6.17× on x86 and 72–82 % less energy.
	- [T-MAC](https://github.com/microsoft/T-MAC) does lookup-table mixed-precision GEMM, up to 6.6× per kernel and 2.8×
	end to end over llama.cpp.
	- Both work on BitNet-style ternary weights, a different format from ggml's `Q2_0`.

	In Rust:
	- [bitnet-rs](https://github.com/lilyco-42/bitnet-rs) (NEON);
	- [alice-aegis](https://github.com/Aefinity-AI/alice-aegis) (2026), a `no_std` UEFI BitNet engine with frozen
	integer semantics, bit-identical digests across ISAs and SHA-256 receipts chaining the logits. It is the
	closest peer to bankml's receipts, but it checks against itself, not against llama.cpp;
	- [bitnet-toy](https://github.com/tidynest/bitnet-toy);
	- [bitnet-llm](https://crates.io/crates/bitnet-llm) (FFI bindings).

	No Rust engine other than bankml was found with `Q2_0_g64`; OxiLLaMa lists `Q1_0_G128`.

	## 3. Verifiable and attested inference

	\| approach \| projects \| what it proves \| bankml's relation \|
	\|---\|---\|---\|---\|
	\| zero-knowledge proofs \| [EZKL](https://github.com/zkonduit/ezkl) (Rust, small models); [zkLLM](https://arxiv.org/abs/2404.16109); [DeepProve](https://eprint.iacr.org/2026/1112); [OpenLLM](https://eprint.iacr.org/2026/1578) \| that a computation was done, with no trust in the prover; [Hollow-LLM](https://arxiv.org/abs/2607.28884) shows ZK correctness alone does not prove a real large model ran \| none; far heavier \|
	\| optimistic re-execution \| [opML](https://github.com/ora-io/opml) ([paper](https://arxiv.org/abs/2401.17555)); [Optimistic TEE-Rollups](https://arxiv.org/abs/2512.20176) \| fraud is challengeable by re-running \| bankml's bit-exactness is what re-execution needs, but it has no dispute protocol \|
	\| TEEs \| [EigenAI](https://arxiv.org/abs/2602.00182); [OpenPCC](https://arxiv.org/abs/2606.11145); [Phala private AI inference](https://phala.com/solutions/private-ai-inference) \| signed receipts from attested hardware \| bankml's receipts are unsigned, with no hardware root \|
	\| model pin + answer hash \| bankml \| which verified file served, and that the text and request are unchanged \| honest scope: integrity between a client and its own gateway \|

	## 4. What this means for bankml's roadmap

	- P3 (its own forward pass) is where the field already is, so it is not optional if bankml is to stand beside
	candle and OxiLLaMa. Its distinct contribution would be a forward pass bit-exact against the compiled reference,
	which nobody else claims. Done since: 0.2.1–0.3.0 ([BUILD_HISTORY.md](BUILD_HISTORY.md)).
	- A plain-AVX2 `Q2_0` kernel is worth upstreaming (or offering to PrismML): the gap is real, current and
	documented upstream. Prepared since: `upstream/q2_0_avx2.c` (0.2.0), bit-exact against the shipped library on
	200,000 of 200,000 random cases; submitting it is the authors' decision, and nothing has been submitted.
	- Receipts: signing them (an operator key) and binding them to a re-executable bit-exact run would move them
	from integrity toward verifiability, the direction EigenAI shows.
	- NEON would bring the kernels to ARM, where upstream `Q1_0`/`Q2_0` already are.

	## Papers

	Binary and ternary networks
	1. Courbariaux, Bengio, David. [BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations](https://arxiv.org/abs/1511.00363). NeurIPS 2015.
	2. Rastegari, Ordonez, Redmon, Farhadi. [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://arxiv.org/abs/1603.05279). ECCV 2016.
	3. Li, Zhang, Liu. [Ternary Weight Networks](https://arxiv.org/abs/1605.04711). 2016.
	4. Zhu, Han, Mao, Dally. [Trained Ternary Quantization](https://arxiv.org/abs/1612.01064). ICLR 2017.

	Quantized and 1-bit language models
	5. Dettmers, Lewis, Belkada, Zettlemoyer. [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339). NeurIPS 2022.
	6. Frantar, Ashkboos, Hoefler, Alistarh. [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323). ICLR 2023.
	7. Wang et al. [BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453). 2023.
	8. Ma et al. [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764). 2024.
	9. Ma et al. [BitNet b1.58 2B4T Technical Report](https://arxiv.org/abs/2504.12285). 2025.

	CPU kernels for 1-bit and ternary inference
	10. Wang, Zhou, Song et al. [1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144). 2024.
	11. Wang, Zhou, Song et al. [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880). 2025.
	12. Wei et al. [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://arxiv.org/abs/2407.00088). EuroSys 2025.
	13. Li, Yin, Wang et al. [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference](https://arxiv.org/abs/2512.06443). 2025.

	Verifiable and attested inference
	14. Sun, Li, Zhang. [zkLLM: Zero Knowledge Proofs for Large Language Models](https://arxiv.org/abs/2404.16109). CCS 2024.
	15. Conway, So, Yu et al. [opML: Optimistic Machine Learning on Blockchain](https://arxiv.org/abs/2401.17555). 2024.
	16. Chan, Ding, Chen et al. [Optimistic TEE-Rollups](https://arxiv.org/abs/2512.20176). 2025.
	17. Alves, Patankar, Pereira et al. [EigenAI: Deterministic Inference, Verifiable Results](https://arxiv.org/abs/2602.00182). 2026.
	18. Cankaya. [Bit-Exact AI Inference Verification Without Performance Tradeoffs](https://arxiv.org/abs/2606.00279). 2026 (vLLM and Hugging Face on GPU, not llama.cpp).
	19. Gailly et al. [DeepProve: Verifiable End-to-End LLM Inference](https://eprint.iacr.org/2026/1112). IACR ePrint 2026/1112.
	20. Yang, Ren, Xu, Zhang et al. [OpenLLM: Modular and Scalable zkSNARKs for Verifiable LLM Inference](https://eprint.iacr.org/2026/1578). IACR ePrint 2026/1578.
	21. Gong, Liu, Li. [Hollow-LLM Attack](https://arxiv.org/abs/2607.28884). 2026.
	22. Zhou, Zhao, Wang et al. [OpenPCC: Open and Confidential LLM Serving on Commodity TEEs](https://arxiv.org/abs/2606.11145). 2026.

	Retrieval, embedding and commitments used by bankml
	23. Cormack, Clarke, Büttcher. [Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf). SIGIR 2009. [doi:10.1145/1571941.1572114](https://doi.org/10.1145/1571941.1572114) (the DOI refuses scripts; the author's PDF resolves).
	24. Chen, Xiao, Zhang et al. [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://arxiv.org/abs/2402.03216) (BGE-M3). 2024.
	25. Laurie, Langley, Kasper. [RFC 6962: Certificate Transparency](https://www.rfc-editor.org/rfc/rfc6962). 2013.
	26. Laurie, Messeri, Stradling. [RFC 9162: Certificate Transparency Version 2.0](https://www.rfc-editor.org/rfc/rfc9162). 2021.

	Unverified details, stated as found:
	- The author lists of papers 3, 4, 12, 25 and 26 are as commonly cited, not re-fetched.
	- The venues of papers 12 and 14 come from secondary pages.
	- Every project's own speed claims are the project's, not reproduced here.