RAG (retrieval-augmented generation) is now the default chat mode on the
Space when the index loads. Pulls 83K BGE-embedded passages from the
Ghostgim/GhostLM-v0.9-experimental Models repo at startup; embeds each
user query with BAAI/bge-small-en-v1.5; takes top-4 by cosine similarity;
prepends them as 'Reference passages' in front of the question. The model
is not RAFT-trained yet so it sees retrieved context as part of the user
message, but even un-trained-for retrieval cuts the hallucination floor
the bare 81M model exhibits.
RAG load is wrapped in try/except: if the index file isn't in the Models
repo (e.g. upload still in progress), the Space falls back to bare chat
with an honest 'RAG: OFF' note in the description so the user knows
what they're getting. No silent failure modes.
Adds requirements: numpy >= 1.24, transformers >= 4.38, sentencepiece >=
0.1.99 (BGE deps). All small; no LFS budget impact on the Space.
Index files (rag/index.npy fp16 64MB + rag/chunks.jsonl 57MB + r…
Convert chat_fn to a generator so Gradio's ChatInterface shows tokens
appearing incrementally rather than waiting 15-25 s for the full reply
to materialize. New helper generate_until_end_stream is the same loop as
generate_until_end but yields the growing token list after every sampled
token; chat_fn decodes and yields the running text snapshot per Gradio's
API contract.
No extra forward-pass cost. The user sees motion within the first ~1-2 s
instead of staring at a static loading state for the full duration. As a
side effect, this also reduces peak memory pressure: Gradio holds only
the latest snapshot in flight rather than the entire response object,
and the streaming yield gives the worker more chances to release
intermediate tensors mid-generation.
fix(space): cast model weights to fp16 to keep cpu-basic worker alive
The 81M v0.9 chat checkpoint at fp32 is 324 MB of weights plus a similar
peak of forward-pass activations. On HF's cpu-basic Space (2 vCPU, shared
RAM pool with a per-worker memory budget), two consecutive generations
push the worker past its budget and the OS kills it. The Space restarts
cleanly but the user sees a client-side error after the second prompt and
has to reload.
Casting model.half() at load time on the Space halves both the weight
footprint (324 -> 162 MB) and the activation memory of each forward pass.
The cast is gated on os.environ.get('SPACE_ID') so local CPU/GPU users
still get fp32 by default. CPU-fp16 inference is slower than fp32 (no
tensor cores, no AVX-fp16) but at 81M we still expect ~15-25 s per reply,
which is acceptable for a CPU demo and far better than 'the demo errors
after two prompts'.
The Space's bundled ghostlm/ package was the v0.4-era model.py without
SwiGLU / RMSNorm support, so loading the v0.9 chat checkpoint failed at
startup with size mismatches on every blocks.*.ffn.* weight: the v0.9
checkpoint stores SwiGLU-compressed FFN weights at hidden=2048
(int(d_ff * 2/3) rounded to a multiple of 64), but the old GELU
FeedForward in the Space's model.py allocated full d_ff=3072.
Syncs five files from the GhostLM main branch:
- ghostlm/__init__.py (re-exports + version)
- ghostlm/config.py (use_rope / use_swiglu / use_rmsnorm flags)
- ghostlm/model.py (SwiGLU class, RMSNorm class, RoPE rotary embed)
- ghostlm/tokenizer.py (50264 vocab + 7 special tokens, chat roles)
- ghostlm/trainer.py (kept aligned even though Space does not train)
Switches the Space to v0.9 chat (the bench winner of the ghost-small line)
and moves the LFS-hosted weights out of the Space and into the Models repo
Ghostgim/GhostLM-v0.9-experimental. The Space pulls the 324 MB slim
checkpoint via hf_hub_download on first launch and caches it locally.
Why the move: HF's 1 GB free-Space LFS cap couldn't fit the v0.9 weights
alongside the existing v0.5 chat-v3 weights, and orphaned LFS objects from
earlier deletions were keeping usedStorage above the limit even after
super_squash_history. Hosting the weights in a Models repo (which has its
own quota and is the canonical pattern for HF demos) sidesteps the
Space-LFS budget entirely.
Bench numbers (chat-tuned, debiased text-scoring):
CTIBench MCQ full (n=2500, 2-perm): 28.9%
in-repo CTF MCQ eval (n=30): 59.2%
SecQA (n=210, external): 39.3%
free-form fact recall (n=50): 1/50 (at floor)
Same commit folds in the OOM-after-2-gens fix the chat UI was hitting on
the free…
chore: squash history to reclaim LFS storage from removed checkpoint
