davidpomerenke commited on
Commit
bec2f46
·
verified ·
1 Parent(s): 691e6c2

Upload from GitHub Actions: unblock workflow: materialize gcloud creds on runner; lazy-init translate client

Browse files

The nightly workflow has been silently failing since at least Dec 2025
because the eval pipeline depends on gcloud_credentials.json for the
Google Translate baseline but the file is gitignored and there was no
GHA secret/step to materialize it on the runner. Also, evals/models.py
called translate.Client() at module import time, so any import of the
module (including backend.py, smoke tests, any contributor without GCP
locally) blew up with DefaultCredentialsError.

.github/workflows/nightly-evals.yml
- Add a "Write Google Cloud credentials" step that decodes the
GCLOUD_CREDENTIALS_JSON secret into gcloud_credentials.json at
the repo root + validates the file parses as JSON (fail-fast).
- Set GOOGLE_APPLICATION_CREDENTIALS=gcloud_credentials.json in
the "Run evaluations" env so the google-auth lookup finds it.

evals/models.py
- Replace module-level `translate_client = translate.Client()` with
a lazy factory `_get_translate_client()` that defers instantiation
until first real translation call. Import-time is now tolerant of
missing creds; only the eval task that actually calls Google
Translate will trip the credentials check.

.github/workflows/nightly-evals.yml CHANGED
@@ -23,11 +23,20 @@ jobs:
23
  curl -LsSf https://astral.sh/uv/install.sh | sh
24
  # Use the `dev` dependency group defined in pyproject.toml
25
  uv sync --frozen --group dev
26
-
 
 
 
 
 
 
 
 
27
  - name: Run evaluations
28
  env:
29
  OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
30
  HUGGINGFACE_ACCESS_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
 
31
  N_SENTENCES: 10
32
  # Keep these aligned with defaults in evals/main.py for comparability
33
  N_LANGUAGES: 1000
 
23
  curl -LsSf https://astral.sh/uv/install.sh | sh
24
  # Use the `dev` dependency group defined in pyproject.toml
25
  uv sync --frozen --group dev
26
+
27
+ - name: Write Google Cloud credentials
28
+ env:
29
+ GCLOUD_CREDENTIALS_JSON: ${{ secrets.GCLOUD_CREDENTIALS_JSON }}
30
+ run: |
31
+ printf '%s' "$GCLOUD_CREDENTIALS_JSON" > gcloud_credentials.json
32
+ # Fail fast if the secret was empty/malformed.
33
+ python -c "import json; json.load(open('gcloud_credentials.json'))"
34
+
35
  - name: Run evaluations
36
  env:
37
  OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
38
  HUGGINGFACE_ACCESS_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
39
+ GOOGLE_APPLICATION_CREDENTIALS: gcloud_credentials.json
40
  N_SENTENCES: 10
41
  # Keep these aligned with defaults in evals/main.py for comparability
42
  N_LANGUAGES: 1000
evals/models.py CHANGED
@@ -306,17 +306,28 @@ async def complete(**kwargs) -> str | None:
306
  return response.choices[0].message.content.strip()
307
 
308
 
309
- translate_client = translate.Client()
 
 
 
 
 
 
 
 
 
 
 
310
 
311
 
312
  def get_google_supported_languages():
313
- return [l["language"] for l in translate_client.get_languages()]
314
 
315
 
316
  @cache
317
  async def translate_google(text, source_language, target_language):
318
  async with google_rate_limit:
319
- response = translate_client.translate(
320
  text, source_language=source_language, target_language=target_language
321
  )
322
  return response["translatedText"]
 
306
  return response.choices[0].message.content.strip()
307
 
308
 
309
+ # Lazy-init: building translate.Client() at import time tries to read
310
+ # GOOGLE_APPLICATION_CREDENTIALS and crashes the whole module if creds are
311
+ # missing — even for callers that never translate (backend, smoke tests).
312
+ # Defer until first use so import works without GCP creds.
313
+ _translate_client = None
314
+
315
+
316
+ def _get_translate_client():
317
+ global _translate_client
318
+ if _translate_client is None:
319
+ _translate_client = translate.Client()
320
+ return _translate_client
321
 
322
 
323
  def get_google_supported_languages():
324
+ return [l["language"] for l in _get_translate_client().get_languages()]
325
 
326
 
327
  @cache
328
  async def translate_google(text, source_language, target_language):
329
  async with google_rate_limit:
330
+ response = _get_translate_client().translate(
331
  text, source_language=source_language, target_language=target_language
332
  )
333
  return response["translatedText"]