Text Generation
Safetensors
GGUF
Chinese
English
patient-education
qa
rag
qwen3
lora
retrieval-augmented-generation
conversational
How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf chenhaodev/patient-edu-qa:Q8_0# Run inference directly in the terminal:
llama cli -hf chenhaodev/patient-edu-qa:Q8_0Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf chenhaodev/patient-edu-qa:Q8_0# Run inference directly in the terminal:
./build/bin/llama cli -hf chenhaodev/patient-edu-qa:Q8_0Use Docker
docker model run hf.co/chenhaodev/patient-edu-qa:Q8_0Quick Links
patient-edu-qa ยท ๆฃ่ ๆ่ฒ้ฎ็ญ Harness
้ขๅๆฃ่ ๏ผ้ๅป็๏ผ็็พ็ ็งๆฎ้ฎ็ญ็ณป็ปใๆฃ่ ็จ่ช็ถ่ฏญ่จ๏ผไธญๆ๏ผๅฃ่ฏญๅ/็ข็ๅๅๅฏ๏ผๆ้ฎ๏ผ ็ณป็ป็ฑ 1-3B ๅฐๆจกๅ่ทฏ็ฑๅจ ๅ็ปๆๅๆๅพๆๅไธ็บขๆๅคๅฎ๏ผไป ๅค็ ็ง FAISS ็ดขๅผ ๆฃ็ดข่ฏๆฎ๏ผ ๆ็ป็ฑ ๅคงๆจกๅ๏ผๅฏไธ็ๆๅบๅฃ๏ผ ่พๅบ่ฟ่ดฏใ็ปไธ้ฃๆ ผ็ๅ็ญใ
่ฎพ่ฎกๆ ธๅฟ๏ผๅ ่ทฏ็ฑใๅๆฃ็ดขใๆๅ็ฑๅฏไธ็ๅคงๆจกๅ็ๆใๅฐๆจกๅไธ็ดๆฅๅ็จๆทๅฏ่ง็ๆๅญ๏ผ ๅฎๅช่พๅบไธไธช็ดงๅ็็ปๆๅ JSON ่ฎกๅใ
็จๆท่พๅ
ฅ โโโถ 1-3B ่ทฏ็ฑๅจ๏ผ็บฏๅ็ฑป๏ผ
ยท ๅนถๅๆๅ sub_intents[] ๏ผๆฏไธช = ไธไธช RAG ๅฏปๅๅๅ
๏ผ
ยท mode = grounded | alert
ยท red_flags[]๏ผๅฑ้ฉไฟกๅท๏ผ
โ
โผ
ๅคRAG๏ผๆฏไธชๅญๆๅพ้ไธไธช็
็ง็ดขๅผ โ ๆฃ็ดข่ฏๆฎ๏ผไธญๆ query / ่ฑๆ UpToDate ่ฏญๆ๏ผbge-m3 ่ทจ่ฏญ่จ๏ผ
โ
โผ
ๅคงๆจกๅ๏ผๅฏไธๅบๅฃ๏ผ๏ผgrounded path ๅผ็จ่ฏๆฎ / alert path ๅปบ่ฎฎๅฐฑๅป
1. ็ปไปถไธๆถๆ
| ็ปไปถ | ่่ดฃ |
|---|---|
| 1-3B ่ทฏ็ฑๅจ | ็บฏๅ็ฑป๏ผๆๅพๆๅ + mode + ็บขๆ๏ผไธๅ็ๆ |
| ๅค RAG | ๆฏ็
็งไธไธช FAISS ็ดขๅผ๏ผ่ทจ่ฏญ่จๆฃ็ดข๏ผBAAI/bge-m3๏ผ |
| ๅคงๆจกๅ๏ผๅฏไธๅบๅฃ๏ผ | grounded ๅผ็จ่ฏๆฎ / alert ๅปบ่ฎฎๅฐฑๅป |
้จ็ฝฒๅ็ซฏ๏ผDEPLOY๏ผ
DEPLOY=gpu๏ผ้ป่ฎค๏ผ๏ผunslothLoRA ่ทฏ็ฑๅจ +sentence-transformers๏ผBAAI/bge-m3๏ผๅตๅ ฅ๏ผ่ตฐ torch/GPUใDEPLOY=cpu๏ผllama.cppGGUF ่ทฏ็ฑๅจ + GGUF ๅตๅ ฅ๏ผ็บฏ CPU ๆจ็๏ผๆ ้ torch/GPUใ
่ทฏ็ฑๅจๅ็ซฏ๏ผROUTER_BACKEND๏ผ
rules๏ผ็บฏ่งๅๅผๆ๏ผ้ถๆ้๏ผ็กฎๅฎ/ๅฏ่งฃ้๏ผๅผ็ฎฑๅณ็จใmodel๏ผ็บฏๅพฎ่ฐๅ็ 1-3B LoRA ่ทฏ็ฑๅจใhybrid๏ผ้ป่ฎค๏ผ๏ผmode/red_flags่ตฐ่งๅ๏ผๅฎๅ จใๆๅจ๏ผ๏ผcategory/intent่ตฐๅพฎ่ฐๅ็ๅฐๆจกๅ๏ผ่ทฏ็ฑๆดๅ๏ผ๏ผๅฐๆจกๅๅคฑ่ดฅๆถๅ้่งๅใ
่ทฏ็ฑๅจ่พๅบ schema
{
"mode": "alert", // grounded | alert
"red_flags": [ // mode=alert ๆถ้ๆกๅๅบ
{"type": "chronicity", "trigger": "duration:2ๅจ", "severity": "high"}
],
"sub_intents": [ // ๅนถๅๅญๆๅพ๏ผๆฏไธช = ไธไธช RAG ๅฏปๅๅๅ
{
"intent": "self_care", // ๅฝๅฑๆๅพ taxonomy๏ผๅณๅฎ prompt ๆจก็๏ผ
"entity": "headache",
"category": "brain-and-nerves", // ้ๅฎ FAISS ็ดขๅผ๏ผไธ data/rag/<category> ๅผบไธ่ด๏ผ
"level": "basics",
"tmpl": "self_care",
"risk": "high"
}
]
}
ไธคไธช็ๆ่ทฏๅพ๏ผ้ฝๅจๅคงๆจกๅๅ ๏ผ
grounded๏ผRAG ๅฝไธญ โ ๅคงๆจกๅ่ตฐ่ฏๆฎๅผ็จ่ทฏๅพ๏ผใๆฎ UpToDate ๆฃ่ ๆ่ฒใ้ๆกๅผ็จ๏ผใalert๏ผๅฝไธญ็บขๆ๏ผๅฆ 2 ๅจๆ ขๆงๅคด็๏ผโ ๅคงๆจกๅ้ฆๅฅๅ็ฝฎใๅปบ่ฎฎๅฐฑๅปใ๏ผ่ช็ๅปบ่ฎฎ้ไธบ่พ ๅฉ๏ผไธๅผบๆฑ RAG ๆ่ฏๆฎใ
2. ็ฎๅฝ็ปๆ
patient-edu-qa/
โโโ harness/ # ๅฏ่ฟ่ก็็ณป็ป้ชจๆถ
โ โโโ config.py # ่ทฏๅพ + ๅ็ซฏ/API/่ถ
ๅ้
็ฝฎ๏ผๅซ DEPLOY/GGUF ่ทฏๅพ๏ผๅๅฏ็จ็ฏๅขๅ้่ฆ็๏ผ
โ โโโ router.py # ่ทฏ็ฑๅจ๏ผrules / model / hybrid / GGUF ๅ็ซฏ๏ผllama_cpp ๆฐๆงๅฏผๅ
ฅ๏ผ
โ โโโ rag.py # ๅคRAGๆฃ็ดขๅจ๏ผๆฏ็
็งไธไธช FAISS ็ดขๅผ๏ผbge-m3 / GGUF ๅตๅ
ฅ๏ผ
โ โโโ prompts.py # ๆๅพโpromptๆจก็ + grounded/alert ๅ่ทฏๅพ
โ โโโ llm.py # ๅคงๆจกๅๅฎขๆท็ซฏ๏ผOpenAI ๅ
ผๅฎน๏ผๆตๅผ๏ผ
โ โโโ pipeline.py # ็ผๆ๏ผ่พๅ
ฅโrouteโๆฃ็ดขโ็ป่ฃ
โ็ๆ
โ โโโ __main__.py # ไบคไบๅผ CLI๏ผREPL / ๅๅฅๆฅ่ฏข๏ผ
โโโ scripts/ # ๆๅปบ็ฎก็บฟ
โ โโโ extract_patient_education.py # ่งฃๆ UpToDate Patient Education โ Q&A ่ฏญๆ
โ โโโ red_flag_rules.py # ็บขๆ่งๅๅผๆ๏ผ็
็จ/ไฝๅพ/ไบบ็พค๏ผ
โ โโโ gen_router_data.py # curated ๅผฑๆ ๆณจ๏ผ่งๅ+taxonomy๏ผ
โ โโโ gen_router_augment.py # ๅๆๅ่ญฆ/ๅคๅๆ ทๆฌ๏ผ่กฅ่งๅ็ฒๅบ๏ผ
โ โโโ merge_router_data.py # ๅๅนถ โ router SFT train/val
โ โโโ build_multi_rag.py # ๆๅปบ 30 ็
็ง FAISS ็ดขๅผ
โ โโโ train_router.py # ่ฎญ็ป 1-3B LoRA ่ทฏ็ฑๅจ
โ โโโ merge_lora_gguf.py # LoRA ๅๅนถๅๅบๅบง โ FP16 safetensors๏ผGGUF ๅ็ฝฎ๏ผ
โโโ router-q8_0.gguf # CPU ้จ็ฝฒ่ทฏ็ฑๅจ๏ผGGUF๏ผ~1.83GB๏ผไปๅบๆ น็ฎๅฝ๏ผ
โโโ data/
โ โโโ patient_education_qa.jsonl # ้ฎ็ญๅ๏ผๅๅง่งฃๆ๏ผ
โ โโโ patient_questions_clean.jsonl # ๆธ
ๆดๆฃ่
้ฎ้ข
โ โโโ rag/<category>/{index.faiss,docs.json,meta.json} # 30 ็
็ง็ดขๅผ
โ โโโ router/router_sft_{train,val}.jsonl # ่ทฏ็ฑๅจ SFT ๆฐๆฎ
โโโ README.md
ๅตๅ ฅ็จ
BAAI/bge-m3๏ผmultilingual๏ผๅ ฌๅผๆจกๅ๏ผใๅ ถ GGUF ้ๅ็๏ผbge-m3-q8_0.gguf๏ผๆช้ๆฌไปๅบไธไผ ๏ผ ๅฏๅฆ่กไป HF ไธ่ฝฝๆ่ช่ก้ๅ๏ผ่ง ยง5 ๅฏผๅบๆต็จ๏ผใCPU ้จ็ฝฒๆถๆEMBED_GGUF_PATHๆๅๆฌๅฐbge-m3-q8_0.ggufใ
3. ็ฏๅขๅฎ่ฃ
# ๅบๅฑไพ่ต
uv pip install openai faiss-cpu sentence-transformers beautifulsoup4
# ่ฎญ็ป/ๆจ็ๅฐๆจกๅ่ทฏ็ฑๅจๆถๆ้่ฆ
uv pip install unsloth trl datasets transformers
# CPU/GGUF ้จ็ฝฒ๏ผDEPLOY=cpu๏ผๆถๆ้่ฆ
uv pip install llama-cpp-python
้ๅ๏ผๅฎๆน
huggingface.coๅจ้จๅ็ฝ็ป๏ผๅฆๆฌๆฒ็ฎฑ๏ผไธๅฏ่พพ๏ผ็จexport HF_ENDPOINT=https://hf-mirror.comใ GPU๏ผuv run python -c "import torch; print(torch.cuda.is_available())"ๅบ่พๅบTrueใ
ๅคงๆจกๅ API๏ผOpenAI ๅ ผๅฎน๏ผ้ป่ฎคๅฏนๆฅ vLLM๏ผ๏ผ
export LLM_BASE_URL=http://...:8000/v1
export LLM_API_KEY=...
export LLM_MODEL=minimax
4. ่ฟ่ก Harness
python -m harness # ไบคไบๅผ REPL
python -m harness "ไปไนๆฏ็ณๅฐฟ็
๏ผๅนณๆถ้ฅฎ้ฃ่ฆๆณจๆไปไน" # ๅๅฅๆฅ่ฏข
# ้จ็ฝฒๅ็ซฏ๏ผ้ป่ฎค gpu๏ผCPU ้จ็ฝฒ
DEPLOY=cpu \
ROUTER_GGUF_PATH=router-q8_0.gguf \
EMBED_GGUF_PATH=/path/to/bge-m3-q8_0.gguf \
python -m harness "ๆๆ่ฟ2ๅจไธ็ดๅคด็ผ๏ผๆไปไนๅฅฝ็ๅ
ๅ่ฏ็ๆนๆณ่็ผ"
# ่ทฏ็ฑๅจๅ็ซฏๅๆข๏ผrules / model / hybrid๏ผ
ROUTER_BACKEND=model python -m harness "..."
้็บงๅ ๅบ๏ผ่ฅ
llama_cppๆชๅฎ่ฃ ๆ GGUF ็ผบๅคฑ๏ผ่ทฏ็ฑ่ชๅจๅ้ๅฐ่งๅๅผๆ๏ผไฟ่ฏไปปไฝๆบๅจไธ้ฝ่ฝ่ท้ใ
5. ๆๅปบ็ฎก็บฟ๏ผไป้ถๅค็ฐ๏ผ
# 1) ่งฃๆ UpToDate Patient Education โ Q&A ่ฏญๆ
python3 scripts/extract_patient_education.py
# 2) ็บขๆ่งๅๅผๆ๏ผ่งๅๅฑ๏ผ
python3 -c "from scripts.red_flag_rules import scan; print(scan('ๆๆ่ฟ2ๅจไธ็ดๅคด็ผ'))"
# 3) ็ๆ่ทฏ็ฑๅจ SFT ๆฐๆฎ
python3 scripts/gen_router_data.py
python3 scripts/gen_router_augment.py
python3 scripts/merge_router_data.py # โ data/router/router_sft_{train,val}.jsonl
# 4) ่ฎญ็ป 1-3B LoRA ่ทฏ็ฑๅจ๏ผๅบๅบง Qwen/Qwen3-1.7B๏ผๅฏ็จ ROUTER_BASE ่ฆ็๏ผ
python3 scripts/train_router.py # โ checkpoints/.../final
# 5) ๆๅปบ 30 ็
็งๅค RAG ็ดขๅผ
python3 scripts/build_multi_rag.py # โ data/rag/<30็
็ง>/
# 6) ๅฏผๅบ + ้ๅ CPU/GGUF ่ตไบง๏ผๅฏ้๏ผไพๆ GPU ๆบๅจ้จ็ฝฒ๏ผ
# a. LoRA ๅๅนถๅๅบๅบง โ FP16 safetensors๏ผ่ๆฌ็กฌ็ผ็ ๆฌๅฐ LoRA/็ผๅญ่ทฏๅพ๏ผMERGE_OUT ๆๅฎ่พๅบ๏ผ
MERGE_OUT=/tmp/router-merged python3 scripts/merge_lora_gguf.py
# b. safetensors โ GGUF f16๏ผllama.cpp ๅฎๆน่ฝฌๆขๅจ๏ผ
PYTHONPATH=/path/to/llama.cpp/gguf-py \
python3 /path/to/llama.cpp/convert_hf_to_gguf.py /tmp/router-merged \
--outfile router-f16.gguf --outtype f16
# c. f16 โ Q8_0๏ผ~1.83GB๏ผ่ฟๆ ๆ๏ผ
/path/to/llama.cpp/build/bin/llama-quantize router-f16.gguf router-q8_0.gguf Q8_0
# d. bge-m3 ๅตๅ
ฅๅ็๏ผbge-m3-f32.gguf โ bge-m3-q8_0.gguf
6. ้ช่ฏ็ปๆ
Alert ่ทฏๅพ๏ผๆ ขๆงๅคด็็บขๆ๏ผไธๅฅ RAG๏ผ็ดๆฅๅปบ่ฎฎๅฐฑๅป๏ผ
PLAN mode=alert, red_flags=[chronicity/2ๅจ], sub_intents=[self_care/brain-and-nerves]
โ ๅคงๆจกๅ๏ผ้ฆๅฅๆ็คบใๆ็ปญไธคๅจ้้่งใๅปบ่ฎฎๅฐฑๅปใ๏ผๅ็ปๅฑ
ๅฎถ็้่ฏ็ฉ็ผ่งฃๅปบ่ฎฎ
Grounded ่ทฏๅพ๏ผ็ณๅฐฟ็ ๅฎไน+้ฅฎ้ฃ๏ผ่ตฐ RAG ่ฏๆฎ๏ผ
PLAN mode=grounded, intent=diet, category=diabetes
โ ๆฃ็ดข 4 ๆก่ฏๆฎ๏ผWhat is diabetes?/type 2 ็ญ๏ผ๏ผๅคงๆจกๅๅ็นไฝ็ญๅนถๆ ๆฅๆบ
ๅคๅ้ฎ้ข๏ผๅคๅญๆๅพ๏ผๆถๆๅทฒๆฏๆ
PLAN sub_intents=[self_care/brain-and-nerves, self_care/sleep]
โ ไปไธคไธช็ดขๅผๅๆฃ็ดข่ฏๆฎ๏ผๅๅนถๅไบค็ปๅคงๆจกๅ
ๆกไพ๏ผHarness vs ่ฃธๅคงๆจกๅ
- ่ฟๆๆง้ผป็็จ่ฏ๏ผgrounded๏ผ๏ผ่ฃธๅคงๆจกๅ็ดๆฅ็ปๅ ทไฝ่ฏๅใๆ ๆฅๆบใๆไธชๅซ้ๅพช่ฏ่ฏดๆณ๏ผHarness ๆฃ็ดขๅฐ UpToDateใMedications that may help symptomsใ็ญ่ฏๆฎ๏ผ้ๆกๅผ็จ๏ผๆฃ็ดขไธบ็ฉบๆถๆญฃ็กฎๆ็ญๅนถๅปบ่ฎฎๅฐฑๅป่้็ผ้ ใ
- ๆ็ปญ 2 ๅจๅคด็๏ผalert๏ผ๏ผ่ฃธๅคงๆจกๅๆใ2ๅจใๅฝๆฎ้ๅบๆฏ๏ผ็บขๆๅชไฝๆๆซ้ๅ ๆ้๏ผHarness ้ฆๅฅๅ็ฝฎใๅปบ่ฎฎๅฐฑๅปใ๏ผ่ช็ๅปบ่ฎฎ้ไธบ่พ ๅฉ๏ผๅๆฌกๅผบ่ฐ้ข่ฏใ
ๅทฅ็จ็ป้ช
- LoRA ็ๆ ธๅฟไปทๅผ = schema ๅฏน้ฝ๏ผๆ LoRA ๆถ 1.7B ่พๅบ่ช็ฑ JSON๏ผ
{"intents":[...]}๏ผๅนถ้ฟ็ฏ thinking๏ผๅพฎ่ฐๅๆฏไธช query ไธฅๆ ผ่พๅบ็บฆๅฎ็{mode, red_flags, sub_intents}๏ผๅฏ็ดๆฅ่ขซ pipeline ่งฃๆใ - ไธบไฝ้ป่ฎค hybrid๏ผ1.7B ๅจๅฐๆฐ็ฑป alert ๆ ทๆฌ๏ผๅฆ 180/2313๏ผไธๅฏน็บขๆๅคๅฎไธๅฆ็กฎๅฎๆง่งๅๅฏ้ ๏ผๆพๆใ2ๅจๅคด็ผใๆผๅคไธบ grounded๏ผใ็บขๆๆฏๅฎๅ
จๆธๅ
ณไธ่งๅๅผๆๅทฒๅฏ่งฃ้ๅฐๅๅฏน๏ผๆ
mode/red_flags่ตฐ่งๅใcategory/intent่ตฐๅฐๆจกๅใ - ็ฑปๅซๅไธ RAG ็ฎๅฝๅๅผบไธ่ด๏ผ
asthma-allergyvsallergies-and-asthmaๆพๅฏผ่ดๆฃ็ดขๆฐธไน ไธบ็ฉบ๏ผๅกๅฟ ๅฏน้ฝๅนถๅ ็ผบ็ดขๅผๅ ๅบใ
7. ๅทฒๆๅปบ่ตไบง
| ่ตไบง | ่ทฏๅพ | ่งๆจก |
|---|---|---|
| ๅๅง Q&A ่ฏญๆ | data/patient_education_qa.jsonl |
34,077 ๅ |
| ๆธ ๆดๆฃ่ ้ฎ้ข | data/patient_questions_clean.jsonl |
12,093 ๆก |
| ็บขๆ่งๅๅผๆ | scripts/red_flag_rules.py |
5 ็ฑป็บขๆ๏ผ26 ็ป็จไพ |
| ่ทฏ็ฑๅจ SFT ๆฐๆฎ | data/router/router_sft_{train,val}.jsonl |
train 2,313 / val 350 |
| ๅค็ ็ง FAISS ็ดขๅผ | data/rag/<30็
็ง>/ |
30 ็ดขๅผ / 21,847 ๆๆกฃ |
| ่ทฏ็ฑๅจ LoRA | checkpoints/Qwen-Qwen3-1.7B-router-lora/final |
1-3B ่ทฏ็ฑๆจกๅ |
| CPU ่ทฏ็ฑๅจ GGUF | router-q8_0.gguf๏ผไปๅบๆ น็ฎๅฝ๏ผ |
~1.83GB |
| ๅฏ่ฟ่ก้ชจๆถ | harness/ |
ๅ่ทฏๅพ + hybrid ๅ็ซฏ่ท้ |
ๅ ่ดฃๅฃฐๆ
ๆฌ้กน็ฎ็จไบ็พ็ ็งๆฎไธๅฐฑๅปๅปบ่ฎฎๅ่๏ผไธๆๆๅป็่ฏๆญใ่ฅๅบ็ฐ่ธ็ใๅผๅธๅฐ้พใๆ่ฏๆนๅ็ญๆฅ้็๏ผ ่ฏท็ซๅณๅฐฑๅปๆๆจๆๆฅๆ็ต่ฏใ็บขๆ่งๅไธบๅฏๅๅผ๏ผ้ๅฎๅคๅปๅญฆ่ฏๅ ธ๏ผๆ็ปๆช่พ็ฑๅคงๆจกๅ่ด่ดฃไปปๅฐๅปบ่ฎฎๅฐฑๅปใ
- Downloads last month
- 84
Hardware compatibility
Log In to add your hardware
8-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf chenhaodev/patient-edu-qa:Q8_0# Run inference directly in the terminal: llama cli -hf chenhaodev/patient-edu-qa:Q8_0