bolikoto's picture
perf(laya): update README.md for WORKERS=2
3984ac1 verified
|
Raw History Blame Contribute Delete
5.17 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Laya Multilingual CPU API
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
python_version: '3.10'
license: apache-2.0

Laya multilingual CPU API

Only convaiinnovations/laya-multilingual is loaded, pinned to the revision in app.py. CPU is forced even on ZeroGPU. The unused GPU declaration is never called and is not exposed as an API. HF runs two server workers, each admitting at most eight inference requests (including its bounded inference queue), for a total ceiling of 16. InstaBox and Evroc each also run two workers admitting eight requests per worker, retaining a uniform per-host total ceiling of 16. Excess requests receive HTTP 503 and Retry-After: 1; a full worker can reject even when another worker has a free slot. Actual CPU inference is serialized within each worker by Laya's native server to avoid oversubscription and memory spikes. The CPU-host launcher allows 30 seconds for worker healthchecks so slow checkpoint initialization is not mistaken for a dead process; request timeouts are unchanged.

Only on ZeroGPU, ASGI startup with Spaces SDK 0.51.3 explicitly runs the SDK's startup-report callback (normally called by Gradio.launch), not its GPU scheduling API. The dummy remains uncalled; recheck this callback when the platform upgrades Spaces.

POST /v1/systemone accepts the existing Jev model, state, questions JSON and returns only model, answers, usage. LAYA_API_KEY is mandatory and must be sent as a Bearer token. /health is public liveness only; an authenticated request includes the loaded checkpoint, revision and actual CPU device. Jev confidence is calculated with Laya's compatibility helper; truncated input has confidence zero, preserving Opportunity's existing fail-open gate.

InstaBox (persistent installation)

Install app.py, requirements.txt, .venv, and the model cache below /data/workspace/laya. Keep LAYA_API_KEY in the mode-0600 .env there, never in Git or command-line arguments. Link laya.service below the persistent ~/.config/systemd/user/ and enable it with systemctl --user. Its default-target link is persistent too; InstaBox enables user linger during boot. Publish port 7860 with InstaBox's HTTPS port forwarding; never expose an unauthenticated inference listener. Do not restart unrelated services to test boot persistence.

On InstaBox, pre-install torch==2.8.0+cpu from PyTorch's CPU wheel index before installing requirements.txt; it satisfies torch==2.8.0 without CUDA packages. The Space uses HF's supported Torch declaration but never places a tensor on GPU.

Evroc uses the same unit, CPU wheel, threads and admission limit. Enable user linger, bind Uvicorn to 127.0.0.1:7860 with --workers 2 via the unit's ExecStart, and publish through the dedicated laya-evroc Cloudflare Tunnel, keeping its token in a mode-0600 file, not service arguments. laya-tunnel.service uses HTTP/2 outbound connections and restarts on failure; no inbound security-group change is needed. Keep the existing VM services intact.

Hugging Face

Create a new Gradio Space on cpu-basic; upload only these three Space files (app.py, requirements.txt, README.md) and set its LAYA_API_KEY secret. Do not upload .env, caches, units, or virtual environments. This runs on CPU, without registering ZeroGPU startup or invoking the dummy GPU function. Preserve the existing Space variables (LAYA_THREADS=4, TOKENIZERS_PARALLELISM=false). Never silently substitute paid hardware. CPU Basic sleeps after 48 hours of inactivity.

Opportunity

Set LAYA_PROVIDERS_JSON in the restricted runtime secret group to an array:

[
  {"endpoint": "https://instabox.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
  {"endpoint": "https://hf-1.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
  {"endpoint": "https://hf-2.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16},
  {"endpoint": "https://evroc.example/v1/systemone", "api_key": "<secret>", "max_concurrent": 16}
]

Each request chooses its first endpoint randomly with weight max_concurrent: 16:16:16:16 gives 25% each over time. Failures try remaining endpoints with the same weighted selection, without replacement. Add future providers to this list with their actual admission limits; no new code branch is needed. This is not a distributed in-flight counter: limits are enforced by each API, include queued work, and do not imply 64 simultaneous inferences. Safe endpoint/attempt logs expose routing and fallback without keys or input.

The four legacy LAYA_INSTABOX_ENDPOINT, LAYA_INSTABOX_API_KEY, LAYA_HF_ENDPOINT, LAYA_HF_API_KEY variables remain supported with weights 16:32 when the JSON variable is absent. Invalid or partial configuration is refused, never silently routed to external Jev. No model weights or ML dependencies are installed in Opportunity. To restore the old Jev chain, remove LAYA_PROVIDERS_JSON and all four legacy variables; YAML questions, thresholds and legacy credentials are unchanged. Verify a new CPU Space and the production switch before pausing the old ZeroGPU Space.