SHENNONG η₯žε†œ β€” 0.5 B (small sibling)

This is not the recommended model. The benchmarked, published artifact is the 1.5 B fine-tune β†’ cptlight/shennong-1.5b-sft. This repository holds the smaller sibling, kept because two parts of the project are built on it.

What is in this repository

file size what it is
model.safetensors 1,976 MB The full model in float32. Qwen2ForCausalLM, 24 layers, hidden 896, intermediate 4864, vocab 151,936, tied embeddings (β‰ˆ494 M parameters)
tokenizer.json, tokenizer_config.json, chat_template.jinja β€” Qwen2 tokenizer and chat template

No GGUF is published in this repository. A Q4_K_M build is produced by the project's export pipeline (scripts/quantize_gguf.py) if you need one for CPU inference.

What it is used for

This model exists for two downstream jobs in the SHENNONG project:

  1. Base for the upcycled sparse mixture-of-experts. scripts/upcycle_moe.py takes this dense model, copies its FFN into 4 expert tensors, and verifies the upcycled model is numerically identical to the dense one at initialisation. That 4-expert MoE (1,435,499,392 params) is the one benchmarked as a negative result β€” it scored below the dense 1.5 B.
  2. Target of the domain adapter adapters/t2_natprod.gguf, which lives in the 1.5 B repository and drives the expert-routing demo (Track 02, natural products).

Relationship to the 1.5 B model

this repo shennong-1.5b-sft
base Qwen2.5-0.5B Qwen2.5-1.5B
parameters β‰ˆ494 M 1,543,714,304
weights on disk 1,976 MB fp32 3,087 MB fp16 + 986 MB Q4_K_M GGUF
benchmarked no β€” do not cite it for results yes β€” 49.0 % vs 37.0 % on 200 held-out questions

Evaluation

This model has not been evaluated with the project's benchmark protocol. It is published as a component, not as a result. Every number in the project β€” the 200-question benchmark, the resolution audit, the preparation-claim refutations β€” was produced with the 1.5 B model or with the non-neural pipeline, and is documented in docs/BENCHMARK.md.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("cptlight/shennong-0.5b-sft")
m = AutoModelForCausalLM.from_pretrained("cptlight/shennong-0.5b-sft", torch_dtype="float16")
ids = tok("<|user|>What is η”˜θ‰ used for?<|assistant|>", return_tensors="pt")
print(tok.decode(m.generate(**ids, max_new_tokens=40, do_sample=False)[0], skip_special_tokens=True))

Prompt format is <|user|>question<|assistant|>, stop token <|im_end|> (id 151645; the config's eos_token_id is 151643 β€” set your stop token explicitly).

Licence and provenance

Apache-2.0, inherited from Qwen/Qwen2.5-0.5B (Apache-2.0); the fine-tuning and project code are Apache-2.0. Training data derives only from permissive sources β€” CC0, public domain and CC BY β€” with sources whose licences are non-commercial (ctext.org, GRETIL) excluded. Recorded source-by-source in docs/CORPUS_PROVENANCE.md.

Intended use and limits

Research use on the history and pharmacology of traditional medicine. It is not medical advice: no dosing, no self-treatment guidance, no synthesis routes for toxins or pathogens. Its outputs are statements about a historical record, not clinical guidance, and nothing has been validated in vitro or in vivo. As a 494 M-parameter model it is weaker than the 1.5 B on every task we measured there.

Links

Downloads last month
409
Safetensors
Model size
0.5B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for cptlight/shennong-0.5b-sft

Finetuned
(858)
this model