Text Generation
llama-cpp-python
GGUF
llama.cpp
Mixture of Experts
ssd-offload
smallthinker
expert-paging
low-ram
Instructions to use HelloSun/SmallThinker4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use HelloSun/SmallThinker4b with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="HelloSun/SmallThinker4b", filename="{{GGUF_FILE}}", )output = llm( "Once upon a time,", max_tokens=512, echo=True ) print(output)
- Notebooks
- Google Colab
- Kaggle
Commit History
8 days ago
-
v11: 方向修正(hot expert 實體放 RAM,非 page 轉換)+ b9acf138 vanilla 基線 82.1 tok/s / peak RSS 2.674 GiB
Auto Upload Agenta035243 -
v10: model card 加上完整實測數據(記憶體/速度/SSD/權重組成);修 pager expert 總量高估 7% 的 bug
Auto Upload Agentf279c31 -
v9: STATUS/README 補上 SSD 讀取能力與實測成果
Auto Upload Agent833ef4d -
v8: 修 io_probe 的 pread 計數 bug(SSD 337/667/1328/2558 MB/s)+ launcher arena 語意修正
Auto Upload Agent84fbb3b -
v7: STATUS/AGENTS 更新 — 目標達成(peak 2.56→0.60 GiB),補上完整踩坑紀錄
Auto Upload Agent370fce7 -
v6: llama_server.sh --verify 端到端通過(預算 512MiB → expert 常駐 510MiB、swap 0)
Auto Upload Agentff122a0 -
v5: 移除臨時診斷;A/B 驗證 512/256/128MB 全部 all_passed=true
Auto Upload Agentf773b24 -
v4: 修 prefetch 反轉 bug — peak RSS 2.56→0.59 GiB,輸出逐字相同,all_passed=true
Auto Upload Agenteb9c86c -
v3: 修 MAP_POPULATE 與 /proc/self/io 解析 bug;A/B 驗證工具;輸出逐字相同已證實
Auto Upload Agent1e54449 -
v2: expert 級 SSD 分頁(patches/0001)— 熱在 RAM 冷在 SSD,輸出與上游一致
Auto Upload Agentd2defb7 -
v1: 驗證通過 — llama-server 起來、chat 正常、peak RSS 4.30 GiB / swap 0
Auto Upload Agent625b02c -
v0: 啟動器骨架 + 量測工具(未驗證)
Auto Upload Agent9460178 -
init
cfaea13d