Project portrait artwork with a neon-green visor and orbital ring

Diogenes-9B

Question authority. Keep checking the facts.

V1 · Standalone BF16 merge · Native MTP head included

This was done as a research project

Diogenes-9B is a TendieLabs behavior-transfer experiment built from Qwen3.5-9B and a filtered subset of Nous Research's Arete training data. It targets more direct answers and less uncritical adoption of the Chinese state's account of its own conduct. It is not a reproduction of Arete-27B or a claim of general political neutrality; However it is inspired by Arete-27B.

On the complete 803-question political evaluation, v1 improved required-fact coverage from 2.61 to 3.32 out of 5 and reduced Chinese-state-framing findings from 292 to 29. The tradeoff is important: coding regressed in the expanded checks.

Model at a glance

Property Details
Artifact Standalone BF16 merged checkpoint, not an adapter-only package
Base Post-trained Qwen3.5-9B, full composite model architecture[1]
Training Text-only supervised fine-tuning with a language-only rank-16 LoRA
Training examples 5,535, with a separate 291-example reserved holdout
Trained parameters 40,108,032 across 200 language projection modules
Vision Original tower retained and frozen; no broad vision validation
MTP Native one-layer head retained, all 15 mtp.* tensors preserved

The model is an independent experiment, not an official Qwen or Nous Research release. This card describes v1, not the later v2 experiment.

V1 benchmark results

Comparisons through the repetition investigation are untouched Qwen3.5-9B versus the final unmerged v1 adapter. The checkpoint-development screen also includes earlier v1 checkpoints. Machine-readable results, immutable revisions, selected capability IDs, generation settings, and merge checks are included in benchmark-results.json.

Complete political evaluation: 803 questions

The released Censorship & Propaganda Eval contains 803 prompts with required facts and a separate state-framing rubric.[3] Both models answered all 803 questions, and all judgments passed the structural and answer-hash checks.

Metric Qwen3.5-9B Diogenes v1 adapter
Required-fact mean, 0 to 5, higher is better 2.6115 3.3151
Answers with Chinese-state-framing findings, lower is better 292/803, 36.36% 29/803, 3.61%
Refusals 40/803 1/803
Answers reaching the 1,024-token cap 3/803 18/803

Paired required-fact scores improved on 246 questions, worsened on 99, and were unchanged on 458. The mean improvement was 0.7036 points, with a descriptive paired bootstrap 95% interval of 0.5567 to 0.8543. The adapter removed 265 baseline framing findings but added two new findings.

Scope: “Framing” means adopting the Chinese state's account of its own conduct under this rubric, not political bias across every country.[3] A higher required-fact score does not verify every extra statement, and lower framing findings do not certify neutrality or factual reliability.

Expanded capability checks

The frozen subset contains 64 GSM8K, 64 IFEval, and 32 HumanEval unique prompts. Each prompt was run once with thinking disabled and once with thinking enabled, for 320 capability records per model. These are bounded subsets and adapted chat prompts, not official leaderboard reproductions.

Mode Metric Qwen3.5-9B Diogenes v1 adapter
Non-thinking GSM8K correct 60/64, 93.75% 60/64, 93.75%
Non-thinking IFEval strict, entire prompt 54/64, 84.38% 51/64, 79.69%
Non-thinking IFEval loose, entire prompt 56/64, 87.50% 55/64, 85.94%
Non-thinking HumanEval passed 30/32, 93.75% 25/32, 78.12%
Thinking GSM8K correct 49/64, 76.56% 47/64, 73.44%
Thinking IFEval strict, entire prompt 28/64, 43.75% 28/64, 43.75%
Thinking IFEval loose, entire prompt 28/64, 43.75% 28/64, 43.75%
Thinking HumanEval passed 32/32, 100% 30/32, 93.75%

GSM8K strict answer-format matching, flexible extraction, and the custom strict-first/flexible-fallback metric produced the same counts in this run. The JSON retains all three rather than treating the custom aggregate as an official metric. IFEval per-instruction results are also retained in the JSON.

Coding regressed. Five extra non-thinking HumanEval answers failed canonical tests despite producing final answers. Two extra thinking answers exhausted the budget without a final answer. Across both modes, there were seven base-only passes and no adapter-only passes. The modes share the same 32 problems and must not be treated as 64 independent questions.

Thinking runs were especially budget-sensitive: base and adapter produced 48 and 54 empty final answers, respectively, across the 160 thinking capability records. An unfinished thinking block counted as no final answer, not as a correct answer hidden in the reasoning text. These results do not establish that thinking is inherently worse.

Original 42-question pilot

The earlier deterministic, category-balanced pilot is retained for transparency. Its 42 public questions are included in the later 803-question evaluation, so it is not an independent confirmation set.

Pilot metric Qwen3.5-9B Diogenes v1 adapter
Required-fact mean, 0 to 5 2.1190 3.2381
Chinese-state-framing findings 17/42 0/42
Refusals 2/42 0/42

Pilot scores improved on 17 questions, worsened on six, and were unchanged on 19. The full evaluation, not the pilot's zero-framing result, is the stronger summary of this experiment.

The pilot also included 13 in-house checks per model. Both passed seven small exact-answer checks. Basic image comparison and native tool-call formatting were exercised, but this did not validate general vision, actual tool execution, or agentic performance. A C# asynchronous-cancellation edge case failed for both models. These probes are not included in the 803 political scores.

Repetition investigation

A separate 35-arm diagnostic matrix per model tested three prompts, including the original looping question f320.

  • The base had no flagged loops. The adapter had four looping arms, all variants of the same f320 question, not four independent failing questions.
  • The original greedy output reproduced exactly. A larger cap extended the loop; adding an extra EOS token did not fix it.
  • Sampled decoding, a generated-token presence penalty, or adding the training system message avoided this observed loop in the tested arms.
  • Nonlooping answers could still contain factual errors. These observations are diagnostic mitigations, not a universal cure or measured population repetition rate.

Across the complete 1,158-record generation matrix, including diagnostics, the base hit its cap 55 times and the adapter 81 times. A cap hit alone is not evidence of a loop.

Independent checkpoint-development screen

A later, separate development set compared the base and retained v1 checkpoints. It used 70 unique prompts in two formats, with and without a system message: 32 math, 16 code, and 22 behavior prompts. Thus each candidate generated 140 records. This set is distinct from the expanded public test and was not used to choose v1's original final checkpoint.

Candidate Math /64 MBPP code /32 Behavior all-five /44 Judge-flagged unsupported claims /44
Base 64 25 3 9
V1 step 100 63 24 8 3
V1 step 350 63 23 13 5
V1 final, step 692 64 25 11 9

Behavior “all-five” requires directness, factual restraint, calibration, reasoning, and rights/scope criteria to pass together. These are small-sample, reference-free LLM-judge assessments, not factual ground truth or a general hallucination rate. The final v1 checkpoint did not pass every baseline-relative development gate: unsupported-claim flags in the user-only format exceeded its allowed ceiling. Neither earlier v1 checkpoint passed every gate either. The merged final v1 was explicitly chosen for research, not certified by this screen.

Evaluation protocol and limitations

Setting Political evaluation Capability checks
Prompt format User-only, no system message User-only, no system message
Template Native Qwen3.5 Same native template
Decoding Greedy Greedy, one completion per prompt and mode
Thinking Disabled Disabled and enabled, reported separately
New-token cap 1,024 2,048 non-thinking; 4,096 thinking
Political judge Claude Opus 5.5, claude-opus-5-5 Not used for objective capability scores
Objective scoring Not applicable Numeric extraction; official pinned IFEval scorer; canonical HumanEval tests
Generated code Not applicable Network-disabled, read-only, unprivileged resource-bounded containers

The expanded matrix contains 1,158 generations per model: 803 political, 35 loop controls, and 320 capability records. Both models used identical frozen tasks and generation settings within each arm. Only labeled diagnostic and development arms added system messages.

Political grading used the released rubric, blind to model identity. Stored judgments were revalidated against their exact answer hashes; required-fact partitions were checked and aggregate scores recomputed. HumanEval passed only when its canonical checks returned and the post-check marker was present, not merely when the process exited successfully.

This is not a reproduction of published Arete-27B scores: the judge and generation settings differ.[3] The small capability subsets, previously inspected public test, single political judge, inherited pretraining contamination, and rubric blind spots limit generalization. There was no comprehensive safety, multilingual, long-context, medical, legal, or agentic evaluation.

Training data and recipe

V1 used accepted Inkling final answers from NousResearch/arete-data-pairs, not original-Qwen comparison answers or teacher reasoning traces. The source authors explicitly warn that the accepted set is not clean SFT data and can still contain factual errors.[2]

The stricter local filter required review_scores.accuracy >= 4 and no review_issue_tags: 29,236 accepted source rows were inspected, 5,826 retained, 5,535 used for training, and 291 reserved as holdout. The holdout was not a separately scored v1 benchmark. Seven source topics were retained; duplicate and lexical-overlap screens were applied, but do not prove semantic or pretraining contamination is absent.

Setting V1 value
Method Assistant-answer-only SFT; no DPO or RL
LoRA Rank 16, alpha 16, dropout 0
Targets 200 language attention, linear-attention and MLP projection modules
Frozen Vision, embeddings, output head, MTP, all other base weights
Optimizer PyTorch AdamW, learning rate 2e-5, weight decay 0
Schedule 20 warmup steps, then cosine decay
Batch Microbatch 1, gradient accumulation 8
Training One epoch, 692 optimizer steps
Sequence budget 4,096 tokens, no packing; longest example 2,495 tokens
Tokens 4,142,116 total; 3,870,244 supervised
Hardware Two NVIDIA RTX 3090 GPUs, 24 GiB each
Trainer runtime 72.29 minutes
Mean loss 1.28034, not an answer-quality metric
Seed 42

Training masked the full native non-thinking prompt prefix and supervised only the assistant answer plus end token. Five allocator OOM warnings were recovered; training finished with finite logged losses, gradients, and all 400 adapter tensors.

The exercised software was Python 3.12.3, PyTorch 2.13.0, Transformers 5.15.0, PEFT 0.21.2, Accelerate 1.15.0, and Flash Linear Attention 0.6.0.

Standalone merge and MTP head

The final v1 adapter was merged with PEFT merge_and_unload(safe_merge=True) into a new copy of the pinned base. All 775 tensors retained the base names, shapes, and stored dtypes. All 200 adapted weights changed; the other 575 remained identical, including original FP32 linear-attention tensors. An independent audit matched all adapted weights to the installed PEFT BF16 merge formula.

MTP included, acceleration not yet validated

The original one-layer multi-token-prediction head is preserved unchanged:

  • model-mtp.safetensors contains all 15 native mtp.* tensors.
  • model.safetensors.index.json includes their weight mappings.
  • config.json retains text_config.mtp_num_hidden_layers=1 and text_config.mtp_use_dedicated_embeddings=false.

The head was not fine-tuned with the LoRA. Presence and integrity do not establish speculative-decoding acceptance, speedup, or compatibility with every serving engine. The Transformers class used for the standalone smoke test does not execute the MTP head.

Merge verification is not benchmark equivalence

Standalone CPU loading without PEFT succeeded, and a basic generation smoke returned 4. However, an additional three-prefix CPU comparison against the unmerged adapter failed the predeclared relative L2 <= 0.05 threshold:

Check Measured merged versus unmerged
Relative logit L2 0.0584 to 0.0736
Logit cosine similarity 0.9973 to 0.9983
Next-token argmax agreement 3/3 prefixes
Exact saved PEFT weight formula 200/200 adapted tensors

Separate-adapter computation and merged BF16 weights can round differently; CPU kernel effects were not isolated. The threshold was not relaxed. No merged GPU throughput, new quality benchmark, or MTP acceptance rate is claimed.

Load the standalone checkpoint

No PEFT wrapper or separate base download is needed. This CPU smoke example uses the standalone directory and deliberately selects the PyTorch fallback because installed Triton linear-attention kernels require a GPU:

import os
import sys

os.environ["CUDA_VISIBLE_DEVICES"] = ""
sys.modules["fla"] = None
sys.modules["causal_conv1d"] = None

import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

torch.set_num_threads(4)
model_path = "/data/llm/models/Diogenes-9B-v1-BF16"
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_path,
    dtype=torch.bfloat16,
    device_map="cpu",
    attn_implementation="sdpa",
    local_files_only=True,
).eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Reply with only the integer that equals two plus two."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=16,
        eos_token_id=[tokenizer.convert_tokens_to_ids("<|im_end|>")],
        pad_token_id=tokenizer.pad_token_id,
    )
answer = tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(answer.strip())

The expected smoke output is 4. The CPU fallback lines are specific to this CPU example, not recommended GPU-serving configuration. GPU serving, runtime conversion, and MTP execution need their own checks. Use the copied native template and explicitly choose thinking or non-thinking mode; do not assume an output cap prevents repetition.

Provenance

Component Immutable identity
Base Qwen/Qwen3.5-9B at c202236235762e1c871ad0ccb60c8ee5ba337b9a
Training data NousResearch/arete-data-pairs at 664cc9b71679199b76c49b8069eaed23bd618f80
Political evaluation NousResearch/Propaganda_Eval at 401223a2bfcdcd99d4650a83c4d20565db0e56b8
V1 adapter SHA-256 4c41e1a03fab702fa8726964ef92e3d0bf8179e9297dba14c5a8acbe5e2c8483
Expanded task manifest SHA-256 322d99f432a564e2526aaee314e318a4b242da80e2d94b518122b2b857277bb8

MERGE_MANIFEST.json records per-file source and output hashes. benchmark-results.json contains the complete aggregate evidence and selected capability IDs. The benchmark source revisions for GSM8K, IFEval, HumanEval, and the official IFEval scorer are retained there as immutable URLs or commits.

Intended use, licensing, and credit

Use this artifact to study fine-tuning, framing, and behavior transfer. Hallucinations, repetition, coding regressions, and calibration defects remain. Independently verify consequential answers; do not treat this model as a reliable authority for safety-critical decisions.

The base declares Apache-2.0.[1] No training-dataset license declaration or LICENSE file was found at the pinned revision during this experiment; its reuse and redistribution terms remain unresolved.[2] The political evaluation declares MIT, with underlying cited documents retaining their own terms.[3] The base license does not settle the resulting fine-tune's distribution rights, so this card does not assign an unsupported release license.

The header uses the exact project-owner-supplied image from the requested source, copied locally so the README does not depend on the preview server. Artwork and trademark clearance are not established by this card. No third-party endorsement is implied.

Credit: Qwen for the base and tooling; Nous Research for Arete, the dataset and evaluation; Inkling for source teacher answers; TendieLabs for this experiment and its local measurements.

Sources

[1] https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/README.md - Pinned Qwen3.5-9B model card [2] https://huggingface.co/datasets/NousResearch/arete-data-pairs/raw/664cc9b71679199b76c49b8069eaed23bd618f80/README.md - Pinned Arete training dataset card [3] https://huggingface.co/datasets/NousResearch/Propaganda_Eval/raw/401223a2bfcdcd99d4650a83c4d20565db0e56b8/README.md - Pinned political evaluation card and rubric

Downloads last month
44
Safetensors
Model size
9B params
Tensor type
BF16
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for TendieLabs/Diogenes-9B-v1

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1,427)
this model
Adapters
2 models

Dataset used to train TendieLabs/Diogenes-9B-v1