Spaces:
Build error
Build error
Commit History
-
ci: make RAG evaluation job pass; fix promptfoo config
Shubham Pawar +1: Claude Sonnet 5The evaluate job failed at "Run Promptfoo evaluation". Causes, all pre-existing: - provider id `gemini-2.5-flash` is not a valid promptfoo id -> use `google:gemini-2.5-flash` (bare ids are treated as a file path) - `apiKey: ${GOOGLE_API_KEY}` was passed literally (promptfoo does not expand ${...} here) -> drop it; the google provider reads GOOGLE_API_KEY from the env - `llm-rubric` asserts defaulted to an OpenAI grader (needs OPENAI_API_KEY) -> point defaultTest.options.provider at Gemini - `- type: relevance` is not a real assertion type -> use llm-rubric - `promptfoo eval` exits non-zero on any failing case, hard-failing the PR for a non-deterministic, key-dependent check CI changes: - eval step is `continue-on-error` and skips cleanly when GOOGLE_API_KEY is not configured (it is an informational gate, like the mypy step) - upload / comment steps are guarded on the results file existing - comment script handles both old and new promptfoo JSON shapes Co-Authored-By: Clau…c16b2ba
-
fix: fix linting issues
a6df84a
-
feat: add Promptfoo evaluation framework
- Add promptfooconfig.yaml with test cases - Add golden_tests.yaml with expected Q&A - Add runner.py Python wrapper - Add evaluation documentationd9d0a60