README.md

metadata

title: Closed-Form RAG API
emoji: 🧮
colorFrom: gray
colorTo: blue
sdk: docker
app_port: 7860
short_description: Closed-form, LLM-free routing for agentic RAG APIs

Closed-Form Agentic RAG — Docker API

A runnable FastAPI gateway, not an LLM model checkpoint. The controller scores calibrated macro-actions in O(number of actions) time. The inference calls are made against external OpenAI-compatible chat APIs: HF Inference Providers, HF Inference Endpoints, vLLM, TGI, an appropriately configured provider gateway, etc. The Docker Space needs no GPU, even when the remote models are 8B, 32B, 70B or larger.

Scientific qualification: This is a finite contextual macro-action relaxation of the full POMDP. There is no claim of globally optimal POMDP planning or of superior accuracy on untested modern LLMs. Accuracy forecasts must be calibrated for the exact deployed model, prompt, retrieval backend and population. The earlier Qwen-0.5B training statistics should not be reused with newer models.

API surface

Endpoint	Function	Any controller LLM calls?
GET /health	Liveness	No
GET /ready	Configuration status (no secrets)	No
GET /docs	Auto-generated OpenAPI/Swagger	No
POST /v1/route	Select the best provided, forecast-feasible action	No
POST /v1/answer	Retrieval → select macro → execute 1–2 remote model calls	No extra orchestration calls

Route-only request (works without provider keys or calibration)

curl -s -X POST 'https://YOUR-SPACE.hf.space/v1/route' \
  -H "Authorization: Bearer $API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "candidates": [
      {"id":"rag-8b", "predicted_accuracy":0.78,
       "expected_cost_usd":0.0006,"latency_p95_ms":950,"llm_calls":1},
      {"id":"rag-70b", "predicted_accuracy":0.89,
       "expected_cost_usd":0.006,"latency_p95_ms":2100,"llm_calls":1},
      {"id":"iterative-70b", "predicted_accuracy":0.93,
       "expected_cost_usd":0.012,"latency_p95_ms":3800,"llm_calls":2}
    ],
    "budgets":{"max_expected_cost_usd":0.01,"deadline_ms":3000,
               "max_llm_calls":2,"price_per_usd":5,"price_per_second":0.01}
  }'

Those input accuracies, dollar amounts and latencies are illustrative request values, not benchmark results or provider price quotations. The service will return the chosen action, scores, rejected actions, and reasons.

Real RAG inference request

1. Configure both remote model backends using Space secrets/variables.
2. Configure a real search backend implementing the /search interface below.
3. Fit a calibration policy on real LLM traces from the deployed models.
4. Add the generated config/policy.json to the Space before rebuilding, or supply explicit action_forecasts in your API request for an uncalibrated, operator-estimate first deployment.

curl -s -X POST 'https://YOUR-SPACE.hf.space/v1/answer' \
  -H "Authorization: Bearer $API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"question":"Which city is the capital of Japan?",
       "budgets":{"deadline_ms":12000,"max_llm_calls":2,
                  "max_retrieval_calls":2,"max_expected_cost_usd":0.02,
                  "price_per_usd":5,"price_per_second":0.02}}'

Unlike /v1/route, /v1/answer actually calls your retriever and selected LLM(s). The response records chosen action, numerical scores, token usage and provider-reported cost estimates, measured elapsed time and call counts.

If you do not have a trained calibration table yet, send an explicit action_forecasts object in the request instead, using the action names in config/actions.json. Each value must have the same fields as the route-only candidates above. These forecasts are supplied by the operator, not learned by this deployment, and must never be reported as measured accuracy. Example:

{"action_forecasts": {
  "retrieve_fast": {"id":"retrieve_fast","predicted_accuracy":0.72,
    "expected_cost_usd":0.001,"latency_p95_ms":1500,"llm_calls":1,"retrieval_calls":0},
  "retrieve_strong": {"id":"retrieve_strong","predicted_accuracy":0.83,
    "expected_cost_usd":0.006,"latency_p95_ms":2800,"llm_calls":1,"retrieval_calls":0},
  "iterative_strong": {"id":"iterative_strong","predicted_accuracy":0.87,
    "expected_cost_usd":0.011,"latency_p95_ms":4200,"llm_calls":2,"retrieval_calls":1}
}}

These numbers are fictitious schema examples and must be replaced with real operator predictions before assessing effectiveness.

First-start safeguard

Do not configure CALIBRATION_FILE until config/policy.json actually exists. The repository intentionally does not include fabricated calibration numbers. The route-only endpoint can run without model keys or calibration; /v1/answer requires a retriever, model endpoints and either fitted calibration or request-supplied forecasts. Setting CALIBRATION_FILE=config/policy.json before creating the file prevents the application from starting.

Set up as a Hugging Face Docker Space

1. Create a new Space, choosing Docker SDK (requires appropriate account eligibility for compute Spaces). Keep the YAML header above in README.md.
2. Upload all files at the root of this repository (including service/, config/, requirements.txt, Dockerfile).
3. Set the environment variables in ENVIRONMENT.example in the Space's Settings → Variables/Secrets, especially a strong random API_TOKEN. All model API tokens belong in Secrets only.
4. Set FAST_LLM_BASE_URL and STRONG_LLM_BASE_URL to the relevant OpenAI-compatible /v1 URLs. Set FAST_LLM_MODEL and STRONG_LLM_MODEL to the model IDs supplied by those providers. Set FAST_LLM_KEY_ENV and STRONG_LLM_KEY_ENV to the names of the secret variables (e.g. HF_TOKEN).
5. Set RETRIEVAL_URL to your search service's full HTTPS URL.
6. Calibrate and upload config/policy.json (see below). Without it, the /v1/answer endpoint requires explicit action_forecasts supplied by the operator; otherwise it responds 503 rather than fake an accuracy/cost/latency forecast. /v1/route works independently.
7. Wait for the build to finish and use https://YOUR-SPACE.hf.space/docs.

HF Docker Spaces expose port 7860 by default. uvicorn listens on 0.0.0.0:7860; the model weights remain in external serving infrastructure.

Provider examples: HF Inference Providers use base URL https://router.huggingface.co/v1; dedicated HF Inference Endpoints, vLLM and TGI also expose OpenAI-compatible chat completions. Replace model IDs with models actually supported by your provider. The gateway does not download models, so it can route to large modern LLMs without expanding Space memory.

Pricing: Set model input/output prices explicitly in the environment; prices are provider-specific and change. Set RETRIEVAL_USD_PER_CALL explicitly (including 0 when the retriever is free) to track retrieval costs alongside model charges. Monetary ceilings are forecast ceilings, not hard spend caps; actual per-request charges can differ.

Retrieval service contract

The Space sends POST $RETRIEVAL_URL with JSON:

{"query":"What is the capital of Japan?","top_k":5}

Your retriever replies:

{"documents":[
  {"id":"doc-101","title":"Tokyo","text":"Tokyo is the capital of Japan.","score":0.91},
  {"id":"doc-102","title":"Japan","text":"Japan is in East Asia.","score":0.42}
]}

Production implementations can adapt an existing Qdrant, Elasticsearch, OpenSearch, Azure AI Search, Vespa, or other search service to this contract. Use an external index for corpus sizes in the millions; do not load a large vector index inside the CPU Space.

Security: The retriever URL and inference URLs are server-controlled environment variables; clients cannot force arbitrary URLs. The Space API requires bearer authentication. Provider secrets are not emitted to clients. Use a separate API gateway with distributed rate limiting, audit logging, and quotas for public multi-tenant production deployments.

Fit the policy on real LLM outcomes

The included fit_policy.py expects one row per question and action, with all actions tested in each calibration question, from a calibration dataset separate from the evaluation set.

bucket,action,correct,cost_usd,latency_ms
c0_g0,retrieve_fast,1,0.0007,750
c0_g0,retrieve_strong,1,0.0051,1750
c0_g0,iterative_strong,1,0.0101,3200

The numbers in this CSV snippet are placeholders illustrating the schema, not experimental observations. Do not use them as actual calibration.

Collect actual calibration outcomes from your running API using a training-only JSONL dataset, one question and answers array per row:

python collect_calibration.py \
  --api https://YOUR-SPACE.hf.space \
  --questions calibration_train.jsonl \
  --out calibration_traces.csv \
  --raw-jsonl calibration_raw.jsonl --concurrency 4

The collector calls all three actions for every calibration question, using actual model inference and returning the observed prediction, costs, tokens, and latency. It explicitly forces each action and uses placeholder action_forecasts only to bypass missing calibration while collecting data; these placeholder forecasts are not used as empirical accuracy results. The collector reports failed and unpriced model or retrieval calls. All calibration actions must succeed and report priced usage before fitting a dollar-aware policy. Set RETRIEVAL_USD_PER_CALL explicitly.

Then fit and upload:

python fit_policy.py --traces calibration_traces.csv --output config/policy.json

For benchmarking, keep calibration questions entirely separate from the 500-question evaluation set.

The empirical-Bayes estimator is

[ \widehat p(z,a)=\frac{S_{z,a}+\kappa\widehat p_0(a)}{n_z+\kappa}, \qquad S_a=\widehat p(z,a)-\lambda\widehat c_a-\mu\widehat t_a. ]

The code uses per-action mean full-workflow cost and empirical p95 post-preflight latency from the actual models and retriever traces. This avoids double-counting initial retrieval time when making the routing decision; it reproduces the lightweight macro-policy family from the reported experiments. It is not a globally optimal constrained POMDP solver.

Call budgets: max_llm_calls and max_retrieval_calls are enforced before downstream calls. Deadlines: both predicted p95 feasibility screening and an actual wall-clock timeout are applied. Cancelled HTTP requests may still incur provider charges. Model-token pricing is computed only if provider usage metadata and configured prices are both present; unknown usage is disclosed.

Scale and limitations

• API controller: O(number of candidate actions) and no generative LLM calls for routing. Numerical scoring happens on CPU.
• Async httpx calls with connection pooling; MAX_INFLIGHT_REQUESTS bounds simultaneous answers per process. Default admission capacity: 64.
• Large model families (8B, 32B, 70B, 120B+) are supported as external OpenAI-compatible endpoints, not hosted on the Space CPU.
• Throughput and end-to-end latency depend on retriever and model-serving provisioned capacity, network latency and rate limits. The supplied concurrency number is a configuration limit, not a measured QPS claim.
• Horizontal scale requires replicated paid Spaces or a conventional container deployment behind a load balancer; per-replica semaphores are not distributed quotas.
• Model-call pricing and latency forecasts need periodic recalibration; old Qwen-0.5B calibration cannot be assumed valid for a 70B model.
• No SSE streaming, persistent memory, multi-tenant identity, or guaranteed hard-dollar expenditure is claimed in this release.

Run locally and test

python -m pip install -r requirements.txt pytest
export API_TOKEN=test-only-secret
python -m pytest -q
python bench_controller.py --requests 50000 --actions 20
uvicorn service.api:app --host 0.0.0.0 --port 7860

To run the local test retriever:

uvicorn examples.retriever_stub:app --host 127.0.0.1 --port 7861

The stub contains only four public example documents; it is not a production-scale vector search engine.

Docker sanity checks

docker build -t closedform-rag-api .
docker run --rm -p 7860:7860 \
  -e API_TOKEN=test-only-secret closedform-rag-api
curl http://127.0.0.1:7860/health

Artifact and research integrity

The API is deployable infrastructure, not an ESWA benchmark in itself. No commercial LLM accuracy, price, or latency is invented by this repository. For a journal evaluation, run this exact API with real 8B/70B providers and capture paired question-level traces with configured per-provider token prices.