curatorbot / docs /model-providers.md
ignatius's picture
Raise the default model output limit to 16000 tokens
de2ecdc
|
Raw History Blame Contribute Delete
2.79 kB

Choosing the language model

CuratorBot sends one request per interview to a language model, which drafts the summary, chapters, subject headings, journeys and donor checks. The validators then check every quote against the transcript, whichever model wrote it.

Each institution chooses where its testimony goes. An administrator sets this once with environment variables (Space secrets and variables on Hugging Face, .env locally). Nothing is chosen per user.

CURATORBOT_LLM_PROVIDER Key Default model Use it when
openrouter (default) OPENROUTER_API_KEY anthropic/claude-sonnet-5.5, falling back to openai/gpt-4.1-mini One account reaches many models; good for demos and comparisons
anthropic ANTHROPIC_API_KEY claude-sonnet-5-5 The institution has an Anthropic contract
openai OPENAI_API_KEY gpt-4.1-mini The institution has an OpenAI contract
compatible CURATORBOT_LLM_API_KEY (if needed) set CURATORBOT_LLM_MODEL Testimony must not leave the institution: its own gateway, or open models served with vLLM or Ollama

Other settings:

  • CURATORBOT_LLM_MODEL: use a different model from the provider's catalogue.
  • CURATORBOT_LLM_FALLBACK_MODEL: tried once if the main model fails for a reason another model could fix (an outage, a rate limit, an unreadable answer). It is not tried for account problems such as a wrong key or no credit. Set it to an empty value to switch the fallback off. Briefs record which model actually answered.
  • CURATORBOT_LLM_BASE_URL: the address of a compatible endpoint, for example http://localhost:11434/v1 for Ollama.
  • CURATORBOT_LLM_JSON_MODE=0: for local servers that reject the JSON-mode option.
  • CURATORBOT_LLM_MAX_TOKENS (default 16000): the longest answer the model may write. A full interview's brief needs about 6,000; if an answer is cut off, the run falls back to the second model, which the brief records.

/api/health shows the active provider, model and fallback.

Data protection

  • OpenRouter forwards each request to the company running the chosen model (Anthropic, OpenAI, or a cloud host of either). In the OpenRouter account's privacy settings, allow only providers that neither train on nor retain prompts before sending pre-accession material.
  • Direct contracts (Anthropic, OpenAI) fall under the institution's own terms with that company.
  • compatible with a local server keeps testimony on the institution's own machines. Quality depends on the open model chosen; run the back-test (docs/real-data-runbook.md) before relying on it.

When a model call fails, curators see a plain sentence ("Live analysis is unavailable right now: …"); the technical detail goes to the server log only.