Replace deprecated get_event_loop, drop the tutorial Settings docstring, and
fail CI if pyright reports type errors.
feat: landing server-status panel and embed ttft/ttfr
Aditya Kulkarni
5e96f99
Render GET /health on the landing as server status so the App tab does
not jump to raw JSON. POST /embed includes ttft_ms (enqueue to encode
start) and ttfr_ms (enqueue to embeddings ready). Keep GET /metrics as
an unused always-200 JSON alias.
feat: land nfcorpus benches, encode batch_size, and INT8-negative docs
Aditya Kulkarni
dfdc75f
Pass max_batch_size through model.encode so 128/256/512 batches are one
forward pass, not silently chunked at 32. Add file-sourced bench harness,
prepare_dataset.py, and the 24-run latency/throughput matrix including
dynamic INT8.
Document the two-regime result and the INT8 slowdown. Keep nfcorpus text
pools gitignored; they regenerate with a fixed seed.
feat: add readiness /health and structured request logging
Aditya Kulkarni
3e47cef
- /health is now a readiness probe (200/503) reporting model_loaded,
worker_alive, queue_depth, and inflight; previously a static "ok" that
inspected nothing
- DynamicBatcher exposes is_running()/queue_depth()/inflight() with an
in-flight counter tracked in submit()
- JSON logging (app/logging_config.py) with a per-request X-Request-ID
correlation id via middleware; structured access logs (method, path,
status, duration_ms); inference errors logged server-side with a
sanitized 500 returned to the client
- drop the unused request param from /embed; add LOG_LEVEL setting
- tests for readiness-503 and request-id generate/echo; README /health
+ logging docs
feat: add input validation, backpressure, and request timeouts
Aditya Kulkarni
3c0d20e
Harden the /embed serving contract:
- Validate request body (non-empty texts, max texts/request, max chars/text) -> 422
- Bound the request queue; reject with 503 when full instead of growing unbounded
- Enforce a per-request deadline; return 504 on timeout
- Map batcher errors to clean HTTP statuses; sanitize unexpected errors to 500
- Guard future resolution so a timed-out request can't crash the worker
Limits are env-configurable (MAX_TEXTS_PER_REQUEST, MAX_CHARS_PER_TEXT,
MAX_QUEUE_SIZE, REQUEST_TIMEOUT_S). Add tests for queue-full, timeout, and the
422/503/504 endpoint paths; update README failure-handling and API docs.
