Instructions to use cptlight/shennong-0.5b-sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use cptlight/shennong-0.5b-sft with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="cptlight/shennong-0.5b-sft") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("cptlight/shennong-0.5b-sft") model = AutoModelForCausalLM.from_pretrained("cptlight/shennong-0.5b-sft", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use cptlight/shennong-0.5b-sft with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "cptlight/shennong-0.5b-sft" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "cptlight/shennong-0.5b-sft", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/cptlight/shennong-0.5b-sft
- SGLang
How to use cptlight/shennong-0.5b-sft with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "cptlight/shennong-0.5b-sft" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "cptlight/shennong-0.5b-sft", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "cptlight/shennong-0.5b-sft" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "cptlight/shennong-0.5b-sft", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use cptlight/shennong-0.5b-sft with Docker Model Runner:
docker model run hf.co/cptlight/shennong-0.5b-sft
SHENNONG η₯ε β 0.5 B (small sibling)
This is not the recommended model. The benchmarked, published artifact is the 1.5 B fine-tune β
cptlight/shennong-1.5b-sft. This repository holds the smaller
sibling, kept because two parts of the project are built on it.
What is in this repository
| file | size | what it is |
|---|---|---|
model.safetensors |
1,976 MB | The full model in float32. Qwen2ForCausalLM, 24 layers, hidden 896, intermediate 4864, vocab 151,936, tied embeddings (β494 M parameters) |
tokenizer.json, tokenizer_config.json, chat_template.jinja |
β | Qwen2 tokenizer and chat template |
No GGUF is published in this repository. A Q4_K_M build is produced by the project's export pipeline
(scripts/quantize_gguf.py) if you need one for CPU inference.
What it is used for
This model exists for two downstream jobs in the SHENNONG project:
- Base for the upcycled sparse mixture-of-experts.
scripts/upcycle_moe.pytakes this dense model, copies its FFN into 4 expert tensors, and verifies the upcycled model is numerically identical to the dense one at initialisation. That 4-expert MoE (1,435,499,392 params) is the one benchmarked as a negative result β it scored below the dense 1.5 B. - Target of the domain adapter
adapters/t2_natprod.gguf, which lives in the 1.5 B repository and drives the expert-routing demo (Track 02, natural products).
Relationship to the 1.5 B model
| this repo | shennong-1.5b-sft |
|
|---|---|---|
| base | Qwen2.5-0.5B | Qwen2.5-1.5B |
| parameters | β494 M | 1,543,714,304 |
| weights on disk | 1,976 MB fp32 | 3,087 MB fp16 + 986 MB Q4_K_M GGUF |
| benchmarked | no β do not cite it for results | yes β 49.0 % vs 37.0 % on 200 held-out questions |
Evaluation
This model has not been evaluated with the project's benchmark protocol. It is published as a component, not
as a result. Every number in the project β the 200-question benchmark, the resolution audit, the preparation-claim
refutations β was produced with the 1.5 B model or with the non-neural pipeline, and is documented in
docs/BENCHMARK.md.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("cptlight/shennong-0.5b-sft")
m = AutoModelForCausalLM.from_pretrained("cptlight/shennong-0.5b-sft", torch_dtype="float16")
ids = tok("<|user|>What is ηθ used for?<|assistant|>", return_tensors="pt")
print(tok.decode(m.generate(**ids, max_new_tokens=40, do_sample=False)[0], skip_special_tokens=True))
Prompt format is <|user|>question<|assistant|>, stop token <|im_end|> (id 151645; the config's eos_token_id
is 151643 β set your stop token explicitly).
Licence and provenance
Apache-2.0, inherited from Qwen/Qwen2.5-0.5B (Apache-2.0); the fine-tuning and project code are Apache-2.0.
Training data derives only from permissive sources β CC0, public domain and CC BY β with sources whose licences are
non-commercial (ctext.org, GRETIL) excluded. Recorded source-by-source in
docs/CORPUS_PROVENANCE.md.
Intended use and limits
Research use on the history and pharmacology of traditional medicine. It is not medical advice: no dosing, no self-treatment guidance, no synthesis routes for toxins or pathogens. Its outputs are statements about a historical record, not clinical guidance, and nothing has been validated in vitro or in vivo. As a 494 M-parameter model it is weaker than the 1.5 B on every task we measured there.
Links
- Recommended model: https://huggingface.co/cptlight/shennong-1.5b-sft
- Live demo (CPU): https://cptlight-shennong.hf.space
- Code: https://github.com/cptlight/shennong
- Downloads last month
- 409
Model tree for cptlight/shennong-0.5b-sft
Base model
Qwen/Qwen2.5-0.5B