Upload folder using huggingface_hub

README.md

@@ -15,18 +15,20 @@ tags:
 
 # GSE Labs Chat (ZeroGPU)
 
-A personal Gradio chat UI running **Qwen2.5-32B-Instruct** (4-bit) on
+A personal Gradio chat UI running **Qwen2.5-72B-Instruct** (4-bit) on
 Hugging Face's free **ZeroGPU** hardware (time-sliced Nvidia GPU, allocated
 per request). Personal-use tool, not part of the GSE product surface.
 
 No API key, no per-token billing — inference runs on the Space's shared free
-GPU quota instead of a paid hosted API. Responses stream token-by-token, and
-the GPU time requested per reply scales with how many tokens you ask for
-(`_duration()` in `app.py`), instead of a fixed cap.
+GPU quota instead of a paid hosted API. Responses stream token-by-token, GPU
+time requested per reply scales with how many tokens you ask for
+(`_duration()` in `app.py`), attention uses `sdpa`, sampling uses
+`top_p`/`repetition_penalty` for quality, and chat history is capped at the
+last 12 turns to keep long sessions from exceeding context/GPU limits.
 
 ## Swapping the model
 
 Change `MODEL_ID` in `app.py` to any other instruct-tuned model on the Hub.
-Current footprint (~16GB in 4-bit) leaves headroom under the ~29.5GB that
-already loaded successfully on this Space's hardware — going meaningfully
-bigger than that is unverified and worth testing with short prompts first.
+Verified working sizes so far (4-bit, packed footprint): 32B → ~20.7GB,
+72B → check the Space's build logs after a push. If a bigger model OOMs or
+times out, the previous known-good `MODEL_ID` is `Qwen/Qwen2.5-32B-Instruct`.

app.py

@@ -10,7 +10,8 @@ from transformers import (
     TextIteratorStreamer,
 )
 
-MODEL_ID = "Qwen/Qwen2.5-32B-Instruct"
+MODEL_ID = "Qwen/Qwen2.5-72B-Instruct"
+MAX_HISTORY_TURNS = 12
 
 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
 model = AutoModelForCausalLM.from_pretrained(
@@ -21,6 +22,7 @@ model = AutoModelForCausalLM.from_pretrained(
         bnb_4bit_compute_dtype=torch.bfloat16,
     ),
     device_map="auto",
+    attn_implementation="sdpa",
 )
 
 
@@ -30,6 +32,8 @@ def _duration(message, history, system_prompt, max_new_tokens, temperature):
 
 @spaces.GPU(duration=_duration)
 def generate(message, history, system_prompt, max_new_tokens, temperature):
+    history = history[-(MAX_HISTORY_TURNS * 2):]
+
     messages = [{"role": "system", "content": system_prompt}] if system_prompt else []
     for turn in history:
         messages.append({"role": turn["role"], "content": turn["content"]})
@@ -42,14 +46,19 @@ def generate(message, history, system_prompt, max_new_tokens, temperature):
     streamer = TextIteratorStreamer(
         tokenizer, skip_prompt=True, skip_special_tokens=True
     )
+    do_sample = temperature > 0
     gen_kwargs = dict(
         **inputs,
         max_new_tokens=int(max_new_tokens),
-        temperature=float(temperature),
-        do_sample=temperature > 0,
+        do_sample=do_sample,
+        repetition_penalty=1.05,
         pad_token_id=tokenizer.eos_token_id,
         streamer=streamer,
     )
+    if do_sample:
+        gen_kwargs["temperature"] = float(temperature)
+        gen_kwargs["top_p"] = 0.9
+
     thread = threading.Thread(target=model.generate, kwargs=gen_kwargs)
     thread.start()