yannelli/code-review-decisions-GGUF

Code Review Decisions GGUF

GGUF builds of two decision models that review JavaScript and TypeScript code, for llama.cpp and Ollama. Give a model a snippet and the task the snippet implements, and it answers 13 typed questions with probabilities: which of 10 code review flaws the code has, whether it has any flaw, which flaw matters most, and how mergeable it is. The models score answer options directly and generate no text.

Both models are fine-tuned on JS/TS Code Review Decisions. Test-split results of the source models (PyTorch, 15,600 decisions):

Source model	Params	Macro F1 @ 0.5	Accuracy	ECE	Builds here
d1-3B Code Review Decisions	3.1B	0.917	0.940	0.017	BF16, Q8_0, Q4_K_M
d1-omni-600M Code Review Decisions	587M	0.771	0.882	0.028	F16, Q8_0

The source model cards hold the full results, training details and limitations.

Files

File	Size	Runtime
d1-3b-code-review-decisions-Q8_0.gguf	2.9 GB	llama.cpp llama-server, /v1/systemone
d1-3b-code-review-decisions-Q4_K_M.gguf	1.7 GB	llama.cpp llama-server, /v1/systemone
d1-3b-code-review-decisions-BF16.gguf	5.4 GB	llama.cpp llama-server, /v1/systemone
d1-omni-600m-code-review-decisions-Q8_0.gguf	0.4 GB	llama.cpp llama-server, /v1/systemone
d1-omni-600m-code-review-decisions-F16.gguf	0.8 GB	llama.cpp llama-server, /v1/systemone
ollama/d1-3b-code-review-decisions-Q4_K_M.gguf + ollama/Modelfile	1.7 GB	Ollama, with ollama/ollama_d1.py
ollama/d1-3b-code-review-decisions-Q8_0.gguf + ollama/Modelfile.Q8_0	2.9 GB	Ollama, with ollama/ollama_d1.py

Q8_0 and BF16 score the same as the PyTorch model on the full test split. Q4_K_M loses 0.004 macro F1 and changes 0.8% of top answers (see Benchmarks), so use it only when memory is tight. d1-omni is an encoder with a decision head, so it runs only in llama.cpp.

Questions

The models answer the 13 questions of the dataset's decisions config, stored in the dataset's manifest.json: one yes/no question (noul) for each of the 10 flaws, has_flaw, primary_flaw (a choice over none and the 10 flaws), and quality (a score from 0 "Should not merge." to 4 "Exemplary. Merge as is."). Each question's criteria carry the flaw definition the labels follow. The models accept other questions in the same schema, but they were trained and evaluated only on these 13.

llama.cpp

llama-server serves both models natively through its TypeSafe-compatible /v1/systemone endpoint. It reads the snippet once and answers all 13 questions from that prefix.

hf download yannelli/code-review-decisions-GGUF d1-3b-code-review-decisions-Q8_0.gguf --local-dir .
llama-server -m d1-3b-code-review-decisions-Q8_0.gguf -c 8192
# d1-omni reads each question in one batch, so the batch must cover the longest prompt:
# llama-server -m d1-omni-600m-code-review-decisions-Q8_0.gguf -c 16384 -b 4096 -ub 4096

import json
import urllib.request

from huggingface_hub import hf_hub_download

# The 13 questions the models were trained on, from the dataset's manifest.json
manifest = hf_hub_download("yannelli/js-ts-code-review-decisions", "manifest.json", repo_type="dataset")
questions = json.load(open(manifest))["questions"]

state = {
    "task": "Define payroll bank account fields and decode a JSON account record, "
            "accepting only a nine-digit routing number and a four-digit account suffix.",
    "language": "TypeScript",
    "code": """import { z } from 'zod';

const bankAccountSchema = z.object({
    routingNumber: z.string().regex(/^\\d{9}$/),
    accountSuffix: z.string().regex(/^\\d{4}$/)
});
export type PayrollBankAccount = z.infer<typeof bankAccountSchema>;
export function decodeBankAccount(serialized: string): PayrollBankAccount {
    const account_payload = JSON.parse(serialized);
    return account_payload;
}""",
}

body = json.dumps({"state": state, "questions": questions}).encode()
request = urllib.request.Request("http://127.0.0.1:8080/v1/systemone", body, {"Content-Type": "application/json"})
answers = json.load(urllib.request.urlopen(request))["answers"]
print(json.dumps(answers, indent=1))

The answers use the System One shape: a noul answer is P(yes), a choice answer has the top choice and probabilities over all options, and a score answer has score (the expected level), probabilities and the legend. The confidence field is llama.cpp's normalized confidence: 0 for a uniform distribution and 1 when one option holds all the probability. Read probabilities for each option's probability.

On the example, a test-split snippet with gold labels inconsistent_style and unsafe_input, primary flaw unsafe_input and quality level 0:

Question	Gold	d1-3B Q8_0	d1-omni Q8_0
inconsistent_style	yes	P(yes) 0.99	P(yes) 0.10
unsafe_input	yes	P(yes) 0.99	P(yes) 0.61
The other 8 flaws	no	P(yes) 0.03 or less	P(yes) 0.08 or less
has_flaw	yes	P(yes) 1.00	P(yes) 0.61
primary_flaw	unsafe_input	unsafe_input (0.98)	unsafe_input (0.93)
quality	level 0	level 1 (0.51)	level 4 (0.60)

Tested with llama.cpp at commit f2cc728 (2026-10-11). The GGUF files carry lfm2.decision.type and a systemone chat template, so a llama.cpp build with decision-model support serves them without extra flags.

Ollama

Use yannelli/d1-3b-code-review-decisions-ollama. It holds the same d1-3B weights with the lfm2.decision.type key removed, because Ollama (tested with v0.40.3) refuses completion for a GGUF that carries it and its /v1/systemone endpoint does not support d1. Its ollama_d1.py sends d1's prompt for each question through /api/generate and reads the log-probabilities of the answer tokens.

ollama pull hf.co/yannelli/d1-3b-code-review-decisions-ollama:Q4_K_M    # or :Q8_0
curl -fsSLO https://huggingface.co/yannelli/d1-3b-code-review-decisions-ollama/resolve/main/code_review_tui.sh
sh code_review_tui.sh

code_review_tui.sh is a terminal UI for the model; that repo's card also shows the Python usage. The ollama/ folder here holds the same files for ollama create. Ollama sends one request per question and cannot reuse the snippet's prompt cache on this hybrid convolution and attention model, so it is about 6 times slower than llama-server.

Benchmarks

Answers are compared with the PyTorch models' answers (bf16, H100). A decision agrees when its top answer is the same (P(yes) on the same side of 0.5 for nouls). The difference is the largest absolute probability gap within a decision. Latency is wall-clock time for one snippet with all 13 questions, one request at a time.

GPU (NVIDIA L40S)

Each llama-server build answered all 1,200 test snippets (15,600 decisions) on one NVIDIA L40S, a data-center GPU with the same Ada Lovelace architecture as the RTX 4090. Scores come from the dataset's score_decisions.py. VRAM is the GPU memory in use while serving, including the CUDA context.

Build	Macro F1 @ 0.5	Accuracy	ECE	Top answer agreement	p95 diff	Max diff	p50 ms	p95 ms	VRAM
d1-3B Q8_0	0.917	0.940	0.017	99.8%	0.010	0.164	221	285	3.4 GiB
d1-3B Q4_K_M	0.913	0.939	0.016	99.2%	0.041	0.819	209	275	2.3 GiB
d1-3B BF16	0.917	0.940	0.018	99.8%	0.009	0.061	247	322	5.7 GiB
d1-3B PyTorch (reference)	0.917	0.940	0.017
d1-omni Q8_0	0.769	0.882	0.027	99.7%	0.015	0.148	140	247	1.2 GiB
d1-omni F16	0.771	0.882	0.029	100.0%	0.003	0.028	150	264	1.5 GiB
d1-omni PyTorch (reference)	0.771	0.882	0.028

Ollama on the same GPU answered 300 test snippets (3,900 decisions) per build, with every layer on the GPU:

Build	Runtime	Macro F1 @ 0.5	Accuracy	Top answer agreement	Max diff	p50 ms	p95 ms
d1-3B Q8_0	Ollama 0.40.3 + ollama_d1.py	0.930	0.944	99.9%	0.085	1,302	1,599
d1-3B Q4_K_M	Ollama 0.40.3 + ollama_d1.py	0.921	0.941	99.3%	0.508	1,396	1,604

On those 300 snippets the PyTorch model also scores 0.930 macro F1 and 0.944 accuracy, so Ollama Q8_0 matches it. Ollama is about 6 times slower than llama-server because it sends one request per question. For reference, the PyTorch model takes 30.5 ms per snippet on an H100, a faster GPU. eval/l40s/ holds every build's answers and scorer output.

CPU (16 Neoverse V3 cores)

The first 20 test snippets (260 decisions), with the same comparison.

Build	Runtime	Snippets	Top answer agreement	Median diff	p95 diff	Max diff	p50 s/snippet
d1-3B Q8_0	llama-server /v1/systemone	20	100%	0.0001	0.012	0.039	6.2
d1-3B Q4_K_M	llama-server /v1/systemone	20	98.8%	0.0004	0.063	0.422	8.9
d1-omni Q8_0	llama-server /v1/systemone	20	100%	0.0004	0.013	0.063	4.4
d1-3B Q8_0	Ollama 0.40.3 + ollama_d1.py	20	100%	0.0001	0.011	0.028	67.2
d1-3B Q4_K_M	Ollama 0.40.3 + ollama_d1.py	4	98.1%	0.0003	0.012	0.015	14.2

For comparison, the PyTorch models take 8.9 s (d1-3B) and 6.3 s (d1-omni) per snippet in fp32 on the same CPU. Apple Silicon was not measured.

Limitations

Quantization adds the differences shown in Benchmarks; Q4_K_M can flip answers close to the cutoff. The source model cards cover the models' own limitations.

License

These files are derivatives of LiquidAI/d1-3B and LiquidAI/d1-omni-600M and are released under the base models' LFM Open License v1.0, included as LICENSE. The license allows commercial use only for entities with less than $10M in annual revenue; read LICENSE for the full terms.

The training data, JS/TS Code Review Decisions, is CC BY 4.0.

Citation

@misc{yannelli2026codereviewdecisionsgguf,
  title        = {Code Review Decisions GGUF},
  author       = {Ryan Yannelli},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/yannelli/code-review-decisions-GGUF}}
}

@misc{yannelli2026jstscodereviewdecisions,
  title        = {JS/TS Code Review Decisions},
  author       = {Ryan Yannelli},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions}}
}

Maintainer

Ryan Yannelli | 10/12/2026 | GitHub: @yannelli | Hugging Face: yannelli