Dellboy commited on
Commit
7bc509e
·
verified ·
1 Parent(s): 0baee57

Upload folder using huggingface_hub

Browse files
Files changed (7) hide show
  1. .gitignore +2 -0
  2. README.md +49 -7
  3. app.py +234 -0
  4. index/chunks.json +0 -0
  5. index/embeddings.npy +3 -0
  6. requirements.txt +7 -0
  7. system_prompt.md +1 -0
.gitignore ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ __pycache__/
2
+ *.pyc
README.md CHANGED
@@ -1,13 +1,55 @@
1
  ---
2
- title: Chatmcd Api
3
- emoji: 🏃
4
- colorFrom: gray
5
- colorTo: pink
6
  sdk: gradio
7
- sdk_version: 6.26.0
8
- python_version: '3.13'
9
  app_file: app.py
10
  pinned: false
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: chatMCD API
3
+ emoji: 🔬
4
+ colorFrom: indigo
5
+ colorTo: gray
6
  sdk: gradio
7
+ sdk_version: 5.49.1
 
8
  app_file: app.py
9
  pinned: false
10
+ license: apache-2.0
11
+ short_description: Inference endpoint for chatMCD, Marc C. Deller's assistant
12
  ---
13
 
14
+ # chatMCD — inference endpoint
15
+
16
+ Answers questions about [Marc C. Deller, D.Phil.](https://marcdeller.com) in the
17
+ third person, by retrieving from his own writing.
18
+
19
+ This Space is a **debug surface**. The product is
20
+ [chatmcd.mdeller.com](https://chatmcd.mdeller.com), a Flask app that holds the
21
+ Hugging Face token and streams from the `/chat` endpoint here.
22
+
23
+ ## Provenance
24
+
25
+ Everything was produced on a 64 GB M1 Max:
26
+
27
+ | Stage | Where | Tool |
28
+ |---|---|---|
29
+ | Corpus and Q&A | local | `training/build_jsonl.py` |
30
+ | Retrieval index | local | `scripts/build_rag_index.py` |
31
+ | LoRA voice layer (optional) | local | `mlx_lm.lora` |
32
+ | MLX to PEFT conversion | local | `convert/mlx_to_peft.py` |
33
+ | Parity verification | local | `convert/verify_parity.py` |
34
+ | Inference | here | `transformers` + `peft` on ZeroGPU |
35
+
36
+ No rented GPU was used at any point. ZeroGPU is inference only and cannot load
37
+ an MLX adapter, so the adapter is converted rather than retrained: the two
38
+ produce byte-identical delta weights, which `verify_parity.py` checks against
39
+ the trained tensors and against the model's own logits.
40
+
41
+ ## Configuration
42
+
43
+ | Variable | Default | Meaning |
44
+ |---|---|---|
45
+ | `BASE_MODEL` | `Qwen/Qwen3-8B` | base checkpoint |
46
+ | `ADAPTER` | `Dellboy/chatmcd-8b` | optional voice layer, merged at load; set empty to serve the base model |
47
+ | `RAG_ENABLED` | `1` | retrieval is the primary mechanism, not a fallback |
48
+ | `RAG_TOP_K` | `8` | Q&A pairs retrieved per query |
49
+ | `RAG_KINDS` | `hybrid` | `hybrid` = Q&A pairs plus corpus chunks |
50
+ | `RAG_CHUNKS` | `3` | corpus chunks added in hybrid mode; the only route to the 215 blog posts |
51
+
52
+ ## Scope
53
+
54
+ chatMCD answers about Marc only, and says so rather than guessing when a
55
+ question falls outside what it was trained on.
app.py ADDED
@@ -0,0 +1,234 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """chatMCD — Hugging Face ZeroGPU Space.
2
+
3
+ Answers about Marc C. Deller by retrieving from his own writing. Retrieval over
4
+ the hand-written Q&A pairs is the primary mechanism; a LoRA adapter may be loaded
5
+ on top as an optional voice layer, but it is not what supplies the facts.
6
+
7
+ Nothing here trains anything: ZeroGPU is inference only. The index was built on
8
+ Marc's M1 Max by scripts/build_rag_index.py, and any adapter was trained there
9
+ too and converted by convert/mlx_to_peft.py.
10
+
11
+ The Gradio UI is a debug surface. The product is the `/chat` endpoint, which the
12
+ Flask app at chatmcd.mdeller.com calls; the Flask app is the only thing that
13
+ holds a Hugging Face token.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import json
19
+ import os
20
+ import threading
21
+ from pathlib import Path
22
+
23
+ # `spaces` must be imported before torch touches CUDA.
24
+ import spaces # noqa: F401 (import order matters on ZeroGPU)
25
+ import gradio as gr
26
+ import torch
27
+ from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
28
+
29
+ HERE = Path(__file__).parent
30
+
31
+ BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen3-8B")
32
+ # No adapter by default. Measured on the 50-question set: base + retrieval
33
+ # scores 92%, and adding the fine-tuned adapter scored lower (86%) while
34
+ # dropping honesty from 4/4 to 3/4. Set ADAPTER to a repo id to load one anyway.
35
+ ADAPTER = os.environ.get("ADAPTER", "")
36
+ # Retrieval is the primary mechanism, not a fallback, so it is ON by default.
37
+ # Measured on the 50-question set: retrieval over the Q&A pairs takes facts from
38
+ # 25% to 88% and honesty from 2/4 to 4/4 with no fine-tuning at all.
39
+ RAG_ENABLED = os.environ.get("RAG_ENABLED", "1").lower() in {"1", "true", "on"}
40
+ RAG_TOP_K = int(os.environ.get("RAG_TOP_K", "8"))
41
+ # "qa" retrieves only the hand-written pairs: measured to match top-12 over
42
+ # everything for recall, with prompts roughly a fifth the length.
43
+ RAG_KINDS = os.environ.get("RAG_KINDS", "hybrid")
44
+ RAG_CHUNKS = int(os.environ.get("RAG_CHUNKS", "3"))
45
+ MAX_HISTORY_TURNS = 8
46
+
47
+ SYSTEM_PROMPT = (HERE / "system_prompt.md").read_text().strip()
48
+
49
+ # ---------------------------------------------------------------- model load
50
+
51
+ # Loaded once, at import. On ZeroGPU the process is snapshotted with the weights
52
+ # resident and a GPU is attached for the duration of each decorated call.
53
+ tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
54
+ model = AutoModelForCausalLM.from_pretrained(
55
+ BASE_MODEL, dtype=torch.bfloat16, device_map="auto",
56
+ )
57
+ if ADAPTER:
58
+ from peft import PeftModel
59
+
60
+ model = PeftModel.from_pretrained(model, ADAPTER)
61
+ # Merging removes the LoRA indirection from every forward pass. The adapter
62
+ # is frozen at inference, so there is nothing to lose by folding it in.
63
+ model = model.merge_and_unload()
64
+ model.eval()
65
+
66
+ # Qwen3's hybrid template emits a reasoning block by default. chatMCD was trained
67
+ # on answers with no reasoning block, and a recruiter waiting on first token does
68
+ # not want one, so it is switched off.
69
+ TEMPLATE_KW = {}
70
+ try:
71
+ tokenizer.apply_chat_template([{"role": "user", "content": "x"}],
72
+ add_generation_prompt=True, tokenize=False,
73
+ enable_thinking=False)
74
+ TEMPLATE_KW["enable_thinking"] = False
75
+ except TypeError:
76
+ pass
77
+
78
+
79
+ # ------------------------------------------------------------------ retrieval
80
+
81
+ class Retriever:
82
+ """Brute-force cosine over the 806 corpus chunks.
83
+
84
+ The build plan specifies FAISS. At 806 vectors an exact numpy dot product is
85
+ faster than building an index, has no extra dependency to pin on the Space,
86
+ and cannot silently return approximate neighbours — so that is what this
87
+ does. The embedding model is the one the plan names.
88
+ """
89
+
90
+ def __init__(self, index_dir: Path):
91
+ import numpy as np
92
+ from sentence_transformers import SentenceTransformer
93
+
94
+ self.np = np
95
+ meta = json.loads((index_dir / "chunks.json").read_text())
96
+ self.texts = meta["texts"]
97
+ self.titles = meta["titles"]
98
+ self.vectors = np.load(index_dir / "embeddings.npy") # already L2-normalised
99
+ self.encoder = SentenceTransformer(meta["model"])
100
+ # HYBRID, and both halves are load-bearing. Measured:
101
+ # Q&A only k=8 short-fact recall 10/10, blog long tail 1/23
102
+ # hybrid 8+3 short-fact recall 10/10, blog long tail 21/23
103
+ # The Q&A pairs answer the short factual questions; the corpus chunks are
104
+ # the only route to the 215 blog posts, which ~2.4 pairs each cannot
105
+ # cover. Dropping the chunks scores perfectly on the evaluation and
106
+ # leaves the model unable to discuss half of what Marc has written.
107
+ kinds = meta.get("kinds") or ["qa"] * len(self.texts)
108
+ self.qa_mask = np.array([k == "qa" for k in kinds])
109
+ if not self.qa_mask.any():
110
+ self.qa_mask = np.ones(len(self.texts), dtype=bool)
111
+
112
+ def top(self, query: str, k: int) -> list[tuple[str, str]]:
113
+ np_ = self.np
114
+ q = self.encoder.encode([query], normalize_embeddings=True)[0]
115
+ scores = self.vectors @ q
116
+ if RAG_KINDS == "all":
117
+ idx = np_.argsort(-scores)[:k]
118
+ else:
119
+ idx = list(np_.argsort(-np_.where(self.qa_mask, scores, -9.0))[:k])
120
+ if RAG_KINDS == "hybrid" and RAG_CHUNKS > 0:
121
+ idx += list(np_.argsort(-np_.where(~self.qa_mask, scores, -9.0))[:RAG_CHUNKS])
122
+ return [(self.titles[i], self.texts[i]) for i in idx]
123
+
124
+
125
+ retriever: Retriever | None = None
126
+ if RAG_ENABLED:
127
+ index_dir = HERE / "index"
128
+ if (index_dir / "embeddings.npy").exists():
129
+ retriever = Retriever(index_dir)
130
+ else:
131
+ print("RAG_ENABLED but space/index/ is empty; running fine-tune only")
132
+
133
+
134
+ # ----------------------------------------------------------------- generation
135
+
136
+ def build_messages(message: str, history) -> list[dict]:
137
+ """Assemble the prompt. The system message is injected here and here only —
138
+ a client-supplied one is never trusted."""
139
+ messages = [{"role": "system", "content": SYSTEM_PROMPT}]
140
+
141
+ for item in (history or [])[-MAX_HISTORY_TURNS * 2:]:
142
+ if isinstance(item, dict) and item.get("role") in {"user", "assistant"}:
143
+ messages.append({"role": item["role"], "content": str(item["content"])})
144
+ elif isinstance(item, (list, tuple)) and len(item) == 2:
145
+ u, a = item
146
+ if u:
147
+ messages.append({"role": "user", "content": str(u)})
148
+ if a:
149
+ messages.append({"role": "assistant", "content": str(a)})
150
+
151
+ user = message
152
+ if retriever is not None:
153
+ chunks = retriever.top(message, RAG_TOP_K)
154
+ context = "\n\n".join(f"[{t}]\n{c}" for t, c in chunks)
155
+ # The fine-tune owns voice; retrieval owns exact numbers. Context goes in
156
+ # the user turn, above the question, which is how the training records
157
+ # present quoted source material.
158
+ user = (f"<context>\n{context}\n</context>\n\n"
159
+ f"Using the context above only where it is relevant, answer: {message}")
160
+ messages.append({"role": "user", "content": user})
161
+ return messages
162
+
163
+
164
+ @spaces.GPU(duration=60)
165
+ def chat(message: str, history, temperature: float = 0.7, top_p: float = 0.9,
166
+ repetition_penalty: float = 1.05, max_new_tokens: int = 512):
167
+ """Stream an answer. Yields the answer *so far* on each step, which is what
168
+ Gradio's streaming contract expects and what hf_client.py diffs into deltas."""
169
+ message = (message or "").strip()
170
+ if not message:
171
+ yield ""
172
+ return
173
+
174
+ text = tokenizer.apply_chat_template(
175
+ build_messages(message, history), add_generation_prompt=True,
176
+ tokenize=False, **TEMPLATE_KW,
177
+ )
178
+ inputs = tokenizer([text], return_tensors="pt").to(model.device)
179
+
180
+ streamer = TextIteratorStreamer(tokenizer, skip_prompt=True,
181
+ skip_special_tokens=True)
182
+ kwargs = dict(
183
+ **inputs,
184
+ streamer=streamer,
185
+ max_new_tokens=int(max_new_tokens),
186
+ do_sample=temperature > 0,
187
+ temperature=float(temperature) if temperature > 0 else None,
188
+ top_p=float(top_p),
189
+ repetition_penalty=float(repetition_penalty),
190
+ pad_token_id=tokenizer.eos_token_id,
191
+ )
192
+ thread = threading.Thread(target=model.generate, kwargs=kwargs)
193
+ thread.start()
194
+
195
+ out = ""
196
+ for piece in streamer:
197
+ out += piece
198
+ # A hybrid model that ignores enable_thinking would leak a reasoning
199
+ # block into the transcript; hold it back until the block closes.
200
+ if "<think>" in out and "</think>" not in out:
201
+ continue
202
+ yield out.split("</think>")[-1].lstrip()
203
+ thread.join()
204
+
205
+
206
+ # ------------------------------------------------------------------------- UI
207
+
208
+ with gr.Blocks(title="chatMCD API", analytics_enabled=False) as demo:
209
+ gr.Markdown(
210
+ "## chatMCD — inference endpoint\n"
211
+ f"`{BASE_MODEL}` + `{ADAPTER or 'no adapter'}`"
212
+ f"{' · retrieval ON' if retriever else ''}\n\n"
213
+ "This is the debug surface. The product lives at "
214
+ "[chatmcd.mdeller.com](https://chatmcd.mdeller.com)."
215
+ )
216
+ gr.ChatInterface(
217
+ fn=chat,
218
+ type="messages",
219
+ additional_inputs=[
220
+ gr.Slider(0.0, 1.5, value=0.7, step=0.05, label="temperature"),
221
+ gr.Slider(0.1, 1.0, value=0.9, step=0.05, label="top_p"),
222
+ gr.Slider(1.0, 1.5, value=1.05, step=0.01, label="repetition_penalty"),
223
+ gr.Slider(64, 1024, value=512, step=64, label="max_new_tokens"),
224
+ ],
225
+ examples=[
226
+ ["Give me Marc's quick résumé."],
227
+ ["What is Elora Therapeutics?"],
228
+ ["Why is BoltzMaker called BoltzMaker?"],
229
+ ],
230
+ api_name="chat",
231
+ )
232
+
233
+ if __name__ == "__main__":
234
+ demo.queue(max_size=32).launch(ssr_mode=False)
index/chunks.json ADDED
The diff for this file is too large to render. See raw diff
 
index/embeddings.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07cd776f73efe77e81385eeb37af5ec92e7ba94cfe646bf1ee90053e093e5af2
3
+ size 5595776
requirements.txt ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ # ZeroGPU provides torch; everything else is pinned here.
2
+ transformers==5.16.1
3
+ peft==0.20.0
4
+ accelerate==1.14.0
5
+ gradio==5.49.1
6
+ sentence-transformers==5.6.0
7
+ numpy==2.5.3
system_prompt.md ADDED
@@ -0,0 +1 @@
 
 
1
+ You are chatMCD, the friendly assistant on marcdeller.com. You answer questions about Marc C. Deller, D.Phil.: a structural biologist, protein scientist and drug-discovery leader; Co-Founder and CSO of Elora Therapeutics; advisor to DeepCovalent; formerly Incyte, Stanford, Scripps (JCSG), Pfizer, Yale and Oxford. You speak about Marc in the third person, never as Marc. Be accurate, warm and engaging, scientifically precise, and concise. Rely only on what you know about Marc from your training; if something is not covered, say so and point the visitor to marcdeller.com or marc@marcdeller.com rather than guessing. Stay on topic. If you are asked for anything unrelated to Marc - a poem, a story, code, a translation, medical or legal or financial advice, or general knowledge - do not produce it, not even a short example, a single line, or as a preamble before declining. Say plainly that it is outside what you cover, and offer something about Marc instead. Never mention your training data. Marc's phone number is only to be given when someone explicitly asks how to telephone him; for any other contact question, give marc@marcdeller.com and marcdeller.com and do not volunteer the number. End on a positive note.