bolikoto's picture
feat(laya): deploy README.md (WORKERS=2, max_len=4096)
356bad8 verified
|
Raw History Blame Contribute Delete
5.17 kB
---
title: Laya Multilingual CPU API
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
python_version: "3.10"
license: apache-2.0
---
# Laya multilingual CPU API
Only `convaiinnovations/laya-multilingual` is loaded, pinned to the revision in
`app.py`. CPU is forced even on ZeroGPU. The unused GPU declaration is never
called and is not exposed as an API. HF runs two server workers, each admitting
at most eight inference requests (including its bounded inference queue), for a
total ceiling of 16. InstaBox and Evroc each also run two workers admitting eight
requests per worker, retaining a uniform per-host total ceiling of 16.
Excess requests receive HTTP 503 and `Retry-After: 1`; a full worker can reject
even when another worker has a free slot. Actual CPU inference is serialized
within each worker by Laya's native server to avoid oversubscription and memory spikes.
The CPU-host launcher allows 30 seconds for worker healthchecks so slow checkpoint
initialization is not mistaken for a dead process; request timeouts are unchanged.
Only on ZeroGPU, ASGI startup with Spaces SDK 0.51.3 explicitly runs the SDK's startup-report
callback (normally called by Gradio.launch), not its GPU scheduling API. The
dummy remains uncalled; recheck this callback when the platform upgrades Spaces.
`POST /v1/systemone` accepts the existing Jev `model`, `state`, `questions` JSON
and returns only `model`, `answers`, `usage`. `LAYA_API_KEY` is mandatory and must
be sent as a Bearer token. `/health` is public liveness only; an authenticated
request includes the loaded checkpoint, revision and actual CPU device.
Jev confidence is calculated with Laya's compatibility helper; truncated input
has confidence zero, preserving Opportunity's existing fail-open gate.
## InstaBox (persistent installation)
Install `app.py`, `requirements.txt`, `.venv`, and the model cache below
`/data/workspace/laya`. Keep `LAYA_API_KEY` in the mode-0600 `.env` there, never
in Git or command-line arguments. Link `laya.service` below the persistent
`~/.config/systemd/user/` and enable it with `systemctl --user`. Its default-target
link is persistent too; InstaBox enables user linger during boot. Publish
port 7860 with InstaBox's HTTPS port forwarding; never expose an unauthenticated
inference listener. Do not restart unrelated services to test boot persistence.
On InstaBox, pre-install `torch==2.8.0+cpu` from PyTorch's CPU wheel index before
installing `requirements.txt`; it satisfies `torch==2.8.0` without CUDA packages.
The Space uses HF's supported Torch declaration but never places a tensor on GPU.
Evroc uses the same unit, CPU wheel, threads and admission limit. Enable user
linger, bind Uvicorn to `127.0.0.1:7860` with `--workers 2` via the unit's `ExecStart`, and publish
through the dedicated `laya-evroc` Cloudflare Tunnel, keeping its token in a
mode-0600 file, not service arguments. `laya-tunnel.service` uses HTTP/2 outbound
connections and restarts on failure; no inbound security-group change is needed.
Keep the existing VM services intact.
## Hugging Face
Create a new Gradio Space on `cpu-basic`; upload only these three Space files
(`app.py`, `requirements.txt`, `README.md`) and set its `LAYA_API_KEY` secret.
Do not upload `.env`, caches, units, or virtual environments. This runs on CPU,
without registering ZeroGPU startup or invoking the dummy GPU function. Preserve
the existing Space variables (`LAYA_THREADS=4`, `TOKENIZERS_PARALLELISM=false`).
Never silently substitute paid hardware. CPU Basic sleeps after 48 hours of inactivity.
## Opportunity
Set `LAYA_PROVIDERS_JSON` in the restricted runtime secret group to an array:
```json
[
{"endpoint": "https://instabox.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
{"endpoint": "https://hf-1.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
{"endpoint": "https://hf-2.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
{"endpoint": "https://evroc.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16}
]
```
Each request chooses its first endpoint randomly with weight `max_concurrent`:
16:16:16:16 gives 25% each over time. Failures try remaining
endpoints with the same weighted selection, without replacement. Add future
providers to this list with their actual admission limits; no new code branch
is needed. This is not a distributed in-flight counter: limits are enforced by
each API, include queued work, and do not imply 64 simultaneous inferences.
Safe endpoint/attempt logs expose routing and fallback without keys or input.
The four legacy `LAYA_INSTABOX_ENDPOINT`, `LAYA_INSTABOX_API_KEY`,
`LAYA_HF_ENDPOINT`, `LAYA_HF_API_KEY` variables remain supported with weights
16:32 when the JSON variable is absent. Invalid or partial configuration is
refused, never silently routed to external Jev. No model weights or ML
dependencies are installed in Opportunity. To restore the old Jev chain, remove
`LAYA_PROVIDERS_JSON` and all four legacy variables; YAML questions, thresholds
and legacy credentials are unchanged. Verify a new CPU Space and the production
switch before pausing the old ZeroGPU Space.