Spaces:

Leon4gr45
/

gemma-inference

Sleeping

App Files Files Community

gemma-inference

47.4 kB

Ctrl+K

Ctrl+K

1 contributor

History: 33 commits

Leon4gr45's picture

Document split KV quant + cache-reuse env vars

4963346 verified 3 months ago

__pycache__
v2.0: single-instance CPU proxy, native OpenAI tools+streaming, flash-attn+q4_0 KV, all-cores 4 months ago
.gitattributes

1.52 kB
initial commit 6 months ago
.gitignore

45 Bytes
Upload gemma-inference space 6 months ago
.hfignore

45 Bytes
Upload gemma-inference space 6 months ago
Dockerfile

2.2 kB
build: use prebuilt llama.cpp CPU binary (b9895) instead of source compile — fixes mtmd OOM hang, ~20x faster build 4 months ago
README.md

3.83 kB
Document split KV quant + cache-reuse env vars 3 months ago
app.py

18.7 kB
Split K/V cache quant (K=q8_0,V=q4_0), 32K ctx, cache-reuse, tool-call smoke test 3 months ago
requirements.txt

38 Bytes
v2.0: single-instance CPU proxy, native OpenAI tools+streaming, flash-attn+q4_0 KV, all-cores 4 months ago
test_chat.py

1.1 kB
v2.0: single-instance CPU proxy, native OpenAI tools+streaming, flash-attn+q4_0 KV, all-cores 4 months ago