Spaces:
Running on Zero
Running on Zero
File size: 4,293 Bytes
fd326d6 61e06ea fd326d6 61e06ea fd326d6 61e06ea fd326d6 61e06ea 8d482e7 61e06ea 8d482e7 61e06ea | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
title: Garfield-Naruto Encoder
emoji: π
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.27.0
app_file: app.py
pinned: false
license: mit
short_description: Hide a short message inside LLM-generated cover text.
---
# Garfield-Naruto Encoder
Hides a short message inside ordinary-looking LLM prose. Every character of the
output is an encoding slot: a secret key decides, per character position, which
letters carry a `0`, which carry a `1`, and which carry nothing.
**Decoding needs only the text and the key β no model, no GPU.** The Decode tab
runs entirely in the Space process and costs no GPU quota.
## GPU quota
This Space runs on **ZeroGPU**, where GPU time is charged to **the visitor's own
account**, not to whoever published the Space. Quota is per user, per day.
| | daily ZeroGPU quota |
|---|---|
| not identified | ~2 minutes, tracked by IP and shared |
| free account | ~5 minutes |
| PRO / Team / Enterprise | 40+ minutes, and pay-as-you-go credits past that |
**There is no sign-in button, and you do not need one.** ZeroGPU identifies you
from the `x-ip-token` header that the Hugging Face proxy injects using your
huggingface.co session, so simply opening this Space in a browser where you are
signed in is what puts the time on your own account. An app-level OAuth button
(`gr.LoginButton` / `hf_oauth`) is a separate mechanism that tells the *app* who
you are without changing your quota, so one here would only be misleading.
Two situations fall back to the shared IP pool: not being signed in to
huggingface.co, and running this Space **embedded in a frame on another site** β
browsers restrict cookies in cross-site frames, so the proxy cannot see your
session. Open the Space directly to avoid both.
## Warning
This is a research prototype. It should not be used for any privacy-critical
purpose. Measured against a model-equipped adversary the scheme is **not**
indistinguishable from ordinary model output.
## Performance
Hardware: ZeroGPU runs **NVIDIA RTX Pro 6000 Blackwell** (96 GB GDDR7, 1.79 TB/s).
`large` β the default β is half a card: **48 GB, 1Γ quota**. `xlarge` is the full
card: 96 GB, 2Γ quota. This workload never needs `xlarge`; see below.
Two optimisations, both measured:
**Top-K constraint** (`STEGO_TOPK`, default 2048). The constraint is elementwise
work over `vocab Γ max_token_chars` β 3.1M elements per step for a 128k
vocabulary. Restricting it to the highest-logit tokens cuts that ~63Γ. Measured
over 884 real steps, the legal tokens inside a 2048 window still hold **99.3%**
of the post-mask probability mass (5th pct 96.9%), with a median of 406 legal
tokens and no step ever starved. Per-step constraint cost drops 33 ms β 2.1 ms;
end-to-end **1.6Γ faster**.
**Batched sentences** (`STEGO_BATCH`, default `auto`). The CPU path regenerates a
rejected sentence sequentially, and ~82% of sampled tokens belong to attempts
that are later discarded. Those attempts are independent, so on a GPU they run as
one batch: `batch_repeat_interleave` fans the prefix out, rows sample in
parallel, `batch_select_indices` keeps the winner's KV β which also removes the
rollback/replay logic entirely. Decode is memory-*bandwidth* bound, so extra rows
are close to free until the step turns compute-bound.
**Why batch is not memory-limited.** SmolLM3-3B is 36 layers with 4 KV heads and
head_dim 128, i.e. **72 KiB of KV per token**. Batch 64 at 2048 tokens is 9 GiB
against 48 GiB available, and the weights are ~6 GB. `autosize_batch` still
computes the fit from live `mem_get_info`, but in practice the cap binds first:
best-of-N stops paying off long before the GPU saturates, and every row costs a
tokenizer decode on the CPU each step.
`STEGO_BATCH=off` forces the sequential path; a number forces that batch size
even on CPU (useful for debugging, not for speed β CPU decode is compute-bound,
so rows cost proportionally there).
Not used: `torch.compile` is unsupported on ZeroGPU, since a fresh process is
spun up per GPU task and compilation cannot be reused. Ahead-of-time compilation
via `torch.export` + `spaces.aoti_*` is the supported route and is reported at
1.3β1.8Γ, but it targets a fixed-shape forward; this generator's shapes change
per step, so it is left as future work.
|