Spaces:
Running
Running
| # OpenOperator reliability release | |
| ## Source and architecture | |
| Work branch: `jules-17409741302335967131-509964cf`; starting commit: | |
| `c993b33b0392cb36aea54522fb95846f5e60b8a0`. Continue existing PR #27 against | |
| `jules-4586832932011214207-6b36d6d5`; do not bypass that integration stack by | |
| merging directly to `main`. | |
| At inspection, PR #27 and draft PR #28 were open. PR #28's source was | |
| `2f072b132a7e46bd6fb06c3e15a8b6583036885f`. Its `scripts/dspy_gepa.py` and | |
| `agents/prompt-improver/` tree exactly match this branch. No second implementation | |
| or merge was introduced. All pre-existing branch commits are retained. | |
| The running Space's actual image/source revision is **unverified**: public HF | |
| metadata/runtime requests were denied by the cloud network proxy. GitHub Actions | |
| history is not proof of the running image. The latest recorded snapshot workflow | |
| on `main` (`aaab39da5a757668099ca97470a6c1046a22f465`) was cancelled and the | |
| concurrent legacy sync failed (runs 33669631891 and 33669632020). Reconcile the | |
| live `/health` SHA and Space build history with the integration stack before | |
| release; do not reset this branch to a guessed production revision. | |
| Architecture: | |
| - `tools/search_engine.py` calls `helpers/searxng.py` through the existing runtime | |
| bridge. The helper selects bounded search backends and the tool retains its | |
| normal `Response(..., break_loop=False)` contract. | |
| - The chat naming monologue-end extension schedules plugin-local background work | |
| on AgentContext's persistent loop. Utility requests pass through the existing | |
| model configuration/LiteLLM pipeline; titles use existing chat persistence and | |
| state-dirty notifications. | |
| - Spynel's skill dispatcher uses the shared deterministic resolver, then the | |
| existing authenticated internal API/goal delegate or trusted HTTPS gateway. | |
| Existing idempotency, revisions, attention states and adaptive polling remain. | |
| - The Space image synchronizes `/git/agent-zero` to `/a0`, excluding `/a0/usr`, | |
| then runs the framework as UID 1000 on port 7860. DSPy stays in `/opt/venv-dspy`. | |
| ## Confirmed defects and repairs | |
| ### Search | |
| Previously `_search` posted to fixed `localhost:55510` without explicit timeouts, | |
| status validation or exception containment. `SearchEngine.searxng_search` awaited | |
| it without catching failures; checking whether the returned value was an | |
| Exception never handled a raised `ClientConnectorError`. | |
| The previous Space entrypoint bypassed Supervisor and never launched SearXNG. | |
| The Space image now verifies the base image's SearXNG source and dedicated venv, | |
| installs pinned Gunicorn there, and makes that installation readable by uid 1000. | |
| A missing installation fails the image build explicitly. On every Space startup, | |
| `huggingface-services.py` launches one Gunicorn worker with four threads on | |
| `127.0.0.1:55510`, with JSON search enabled and a fresh secret in private temporary | |
| settings. It does not expose SearXNG publicly, require root/Redis/Supervisor, change | |
| `/a0/usr`, or install packages at runtime. Installation belongs to the HF image | |
| build; normal restarts only start the installed services. | |
| The launcher probes `/config` for a valid engine registry for up to 30 seconds | |
| before starting the UI. If readiness fails, the UI still starts with the existing | |
| search fallbacks. SearXNG exits are logged and retried after five seconds. Container | |
| termination stops both process groups; UI exit stops SearXNG and preserves the UI | |
| exit status. Set `A0_SEARCH_BACKEND=auto` (or unset it) to use restored SearXNG; | |
| a previously configured `tinyfish` backend continues to skip it. | |
| Real local child-process tests cover readiness, failed startup, restart after | |
| exit, UI failure and joint SIGTERM cleanup. The actual SearXNG image, upstream | |
| search results and live memory usage remain unverified: no Docker build or HF | |
| deployment was performed. After an authorized release, check Space logs for | |
| `SearXNG ready`, request `http://127.0.0.1:55510/config` from inside the container, | |
| and exercise a real search through OpenOperator. The public Space URL remains | |
| the UI/API endpoint, not the search backend. | |
| `A0_SEARCH_BACKEND=auto` tries SearXNG, then DuckDuckGo HTML. When | |
| `TINYFISH_API_KEY` is set, it instead tries SearXNG, Tinyfish, then DuckDuckGo. | |
| Explicit `searxng` or `duckduckgo` also falls back to Tinyfish when configured; | |
| `tinyfish` selects it directly. Without the key, existing behavior is preserved. `A0_SEARXNG_URL` overrides the endpoint. | |
| `A0_SEARCH_TIMEOUT_SECONDS` defaults to 15 seconds per backend (1–60), with a | |
| 3-second connection cap. Each failed provider consumes its own bounded timeout. | |
| Use `duckduckgo` on a Space intentionally lacking SearXNG to avoid probing it. | |
| Both adapters preserve titles, source URLs and snippets. HTTP errors, malformed | |
| responses, DNS/connection failures and timeouts return safe reason codes and no | |
| fabricated results. Legitimate empty results succeed; challenge pages do not. | |
| Cancellation propagates and closes HTTP sessions. Tinyfish is opt-in through | |
| `TINYFISH_API_KEY`, supplied by the operator in the HF Space environment. | |
| Tinyfish uses GET `https://api.search.tinyfish.ai?query=…` for search and POST | |
| `https://api.fetch.tinyfish.ai` with `{"urls":[url],"format":"markdown"}` for | |
| HTTPS document content, authenticated with `X-API-Key`. These contracts match the | |
| [official REST client](https://github.com/tinyfish-io/tinyfish-web-agent-integrations/blob/main/hermes/tinyfish_hermes/rest_client.py). | |
| Fetch invokes Tinyfish immediately after the first native fetch failure, before | |
| retry delays. This means immediate fallback invocation after failure, not zero | |
| network latency. Native success and empty search results do not call Tinyfish. | |
| HTTP/file documents, unsafe URLs, size-limit rejections, user intervention and | |
| cancellation do not trigger remote fetching. Public URL validation applies to | |
| requested and returned HTTPS URLs. Responses are bounded, TLS is verified, | |
| provider redirects are disabled, and errors omit credentials and response bodies. | |
| No additional package or inference service is needed. Live Tinyfish requests | |
| remain untested because no API key is configured in this environment. | |
| The pinned `duckduckgo-search==6.1.12` client hardcodes `verify=False`; this repair | |
| does **not** use that client. The HTML adapter uses default verified aiohttp TLS, | |
| environment proxy support and no redirects. No results obtained through the | |
| unverified client were retained or used. Production HTTPS search remains untested. | |
| ### Naming | |
| Commit `45654ad8` changed the browser rename modal's request timeout to **60,000 | |
| milliseconds**, not the provider timeout. It cannot by itself explain a backend | |
| automatic-naming regression. `background=True` in `Agent.call_utility_model` | |
| bypasses the rate-limiter callback; it does not create or retain background work. | |
| Confirmed lifecycle defects: the extension did not retain tasks or bound the | |
| entire utility operation, discarded once-mode results after any later user | |
| message, and allowed always-mode results to overwrite manual titles. These | |
| conditions are reproduced by the regression tests. Cancellation/garbage | |
| collection or an unreachable live provider are not claimed as confirmed causes | |
| of the specific production incident. | |
| The plugin now retains one task per context, suppresses duplicate requests for | |
| an unchanged sequence, supersedes pending always-mode requests, and lets | |
| once-mode finish after later messages. Identity, sequence, title and revision | |
| checks reject stale results. Manual saves and automatic compare-and-save are | |
| serialized; context deletion cannot interleave with automatic persistence. | |
| `automatic_naming_timeout_seconds` is scoped through the existing project/profile | |
| configuration and UI, defaults to 45 seconds, and is clamped to 1–300. The budget | |
| covers provider retries; this plugin adds no retries and changes no global model | |
| timeout. Failures preserve titles and report safe diagnostic categories. Manual | |
| retry remains available. Successful updates use existing persistence/sidebar | |
| notification paths. The actual provider preset, authorization and timeout/retry | |
| configuration of the live Space remain unverified. | |
| ### Spynel | |
| The previous dispatcher recognized only explicit tokens and collapsed all | |
| whitespace, including multiline/code payloads. It now also recognizes explicit | |
| natural-language delegation against the active profile registry and trusted | |
| `SPYNEL_SDK_*_URL`/`SPYNEL_AGENT_*_URL` registrations. Names, titles, normalized | |
| multiword forms and conditional convenience aliases are supported. No profiles | |
| or directories are created from user text. | |
| The standalone resolver returns `dispatch`, `local` or `abstain`, a validated | |
| target, canonical workspace path/candidates, recognition method, ambiguity and | |
| the unchanged original message. Workspace candidates must exist beneath | |
| `/a0/usr/workdir` after symlink resolution. Multiple targets or directories | |
| abstain. Explicit profile+SDK gateway syntax keeps its precedence. Quotes, | |
| examples, code blocks, URLs, partial names and filenames cannot select agents. | |
| Natural dispatch preserves the original instruction and appends a canonical | |
| workspace annotation when available; explicit dispatch removes only route tokens. | |
| ```bash | |
| python scripts/spynel_resolver.py --resolve-only \ | |
| 'Have the reviewer inspect nodepad' | |
| ``` | |
| `--project` selects project-scoped discovery in the CLI; `SPYNEL_PROJECT` does so | |
| in the dispatcher. `--workspace-root` is a read-only CLI testing override. | |
| The final routing architecture has no classifier interface, endpoint integration, | |
| model fallback or extra inference dependency. Explicit user tokens have highest | |
| precedence. Deterministic matching resolves registered names and aliases; the | |
| execution gate `prepare_route` validates files, folder references and configured | |
| aliases before submission. The existing Spynel main interaction may select an | |
| existing profile/SDK with `--profile` or `--sdk` and an existing `--workspace`; | |
| these proposals are validated through the same gate and cannot override an | |
| explicit, unknown or ambiguous target. No second model call is made for routing. | |
| The read-only CLI accepts the same options. Legacy `resolve` retains the original | |
| recognition corpus; only `prepare_route` authorizes actual dispatch. | |
| Folder aliases can be existing confined symlinks or trusted | |
| `SPYNEL_WORKSPACE_ALIASES` JSON mappings. Canonical workspace and file annotations | |
| are delivered with the original multiline task. Missing files, escaping aliases, | |
| and ambiguous directories abstain. The accompanying profile on a `/sdk` gateway | |
| is also validated; a model-generated identifier cannot bypass registry checks. | |
| ### Local experiment reconciliation and persistent overrides | |
| The supplied experiment summary reported 18/20 classifications, an inconsistent | |
| table count and SDK outcomes conditional on registration. In this cloud machine, | |
| all three experiment sources are absent: `/a0/usr/agents/spynel/`, | |
| `/tmp/spynel_patch.diff`, and `/a0/usr/workdir/spynel_test_cases.md`. A row-by-row | |
| comparison, reproduction of that score, and patch merge were **not** possible. | |
| No local patch is claimed as merged. The useful edge-case requirements from the | |
| supplied report were independently encoded in tests: ordinary URLs, registered | |
| versus unregistered SDKs, real files/folders, canonical aliases, quotes/source | |
| code, actual worker dispatch and multiline preservation. | |
| The tracked bundled role/specifics and skill now describe the same routing | |
| protocol, replacing the stale instruction to keep every token-free request | |
| local. A core `_00_spynel_profile_guard.py` hook runs before the main model, and | |
| the shared dispatcher performs the same read-only active-profile audit. Divergent | |
| user, project or plugin routing prompts, context instructions, extension code or | |
| skill files block routing with an explicit reconciliation message. Identical | |
| copies and metadata-only title/avatar edits are allowed. No user files are | |
| changed or deleted. This intentionally prevents a retained experimental profile | |
| from silently taking precedence after redeployment. | |
| ```bash | |
| python scripts/spynel_resolver.py --check-profile | |
| # For a scoped profile, also supply --project <existing-project>. | |
| ``` | |
| A blocked audit exits nonzero. Before a future release, an authorized operator | |
| must back up, review and explicitly reconcile any listed persistent overrides; | |
| the cloud test cannot prove that the live Space has no such overrides. No | |
| migration or removal of `/a0/usr/agents/spynel` has been performed. The amendment | |
| leaves NeuralMind, SearXNG/search and Parallel MCP requirements unchanged. | |
| The labeled deterministic corpus includes explicit/natural routes, aliases, | |
| nested workspaces, collisions, ambiguity, quoted/code/URL examples, unknown | |
| targets, and multilingual abstention/explicit routes. The original 62 labeled | |
| recognition cases are retained; extra tests exercise the final execution gate. | |
| Initial confusion cases (filename suffixes, partial aliases, multiple targets) | |
| were corrected; all labeled cases now pass. | |
| ### Release control | |
| Both HF workflows previously triggered on the same main commit with the same | |
| cancel-in-progress group. That allowed the history mirror to cancel the clean | |
| snapshot deployment. Only `deploy-huggingface-space.yml` now uploads; its manual | |
| runs are guarded to `refs/heads/main`. The legacy workflow is manual and prints | |
| an explanation without accessing credentials or uploading. | |
| Snapshots write `openoperator-build.json` with the GitHub `source_sha`, using the | |
| existing `/health` provenance contract. CI includes the new reliability suites. | |
| Two stale test fixtures were repaired: the HTTP-auth test app now registers the | |
| index endpoint present in the real app and checks decoded redirect semantics; | |
| the sidebar assertion accepts the existing multiline API call formatting. | |
| No authentication/CSRF implementation or assertions were disabled. | |
| ## Validation and resources | |
| 441 selected tests passed, zero failed/skipped. The suite includes the Tinyfish | |
| REST contract, immediate fallback, document dispatch, cancellation, URL validation | |
| and credential-safe failure tests. This includes search HTTP | |
| failures through real loopback requests and the ordinary tool response; naming | |
| success, delay, errors, cancellation, manual races, context deletion, scoped | |
| configuration, concurrent chats and persistence; the **real Agent monologue and | |
| response tool completing while utility naming remains pending**; deterministic | |
| routing, final selection validation, profile-override guards, real local authenticated | |
| API/AgentContext/response-tool/polling dispatch, A2A, goal delegation, authentication/CSRF, | |
| profiles, HF defaults, Colab, Repository Manager and DSPy/GEPA offline contracts. | |
| Full deployment/image tests and live external agent/model calls were not run. | |
| The exact selected pytest command is in the PR validation description. The | |
| original 62 recognition rows and existing dispatch regressions are preserved; | |
| obsolete classifier-specific tests are replaced by no-classifier/no-network and | |
| registry-validation regressions. Four authenticated local worker cases exercise | |
| actual profile selection, native curl, task submission, execution and polling | |
| with controlled model responses; 9 rejection cases create no worker and a | |
| separate wrong-key case verifies authentication. Ruff | |
| checks for changed reliability modules/tests, compile checks, shell syntax and | |
| `git diff --check` pass. Local UI startup returned HTTP 200 for `/` and | |
| `{"status":"ok", "sha": ...}` from `/health` on loopback. A real unavailable | |
| search returned structured failure without an escaping traceback. | |
| The Space adds pinned Gunicorn in the existing SearXNG venv and one private | |
| SearXNG worker with four threads. Its actual RAM cost still needs image/runtime | |
| measurement. No database or public port is added. Search | |
| adds bounded HTTP fallbacks plus modest HTML parsing per request; no per-query | |
| process is spawned. Naming retains at most one task per chat (default lifetime | |
| 45 seconds) and a brief persistence lock. Production CPU/RAM were not measured; | |
| no production performance numbers are claimed. Startup launches private SearXNG but adds no new model load. DSPy/GEPA job ownership and its separate environment are unchanged. | |
| The local backend/UI development environment uses a CPU-only Python virtualenv. | |
| The full optional document stack's initial installation backtracked into an old | |
| scikit-learn source build requiring unavailable clang. A backend subset, Whisper, | |
| LiteParse and test tools installed successfully. Optional document ingestion/TTS | |
| were not validated. Browser provisioning is blocked by `cdn.playwright.dev` | |
| network policy; offline embedding preload has no cached model. UI and selected | |
| backend tests work despite these optional-feature limits. Fresh full-image | |
| package resolution remains part of the later authorized build acceptance. | |
| ## Future deployment acceptance (not executed) | |
| 1. Verify the live source SHA and preserve the current image reference and a | |
| recoverable `/a0/usr` backup using the operator's existing backup procedure. | |
| Review PR #27/#28 overlap and the integration stack; do not duplicate GEPA. | |
| 2. Review and merge through the intended integration branch. Authorize the | |
| eventual main-branch deployment separately: merging to `main` still triggers | |
| the single snapshot publisher and HF builds the image. | |
| 3. Verify the HF build contains existing runtime dependencies and model/browser | |
| artifacts at image-build time. Keep non-root execution, only port 7860, | |
| `A0_CLOUDFLARE_DISABLED=true`, `PIP_NO_INDEX=1`, `/a0`, `/git/agent-zero` and | |
| the `/a0/usr` rsync exclusion. No Dockerfile/entrypoint changes are required | |
| by these repairs. Do not install dependencies during Space startup. | |
| 4. From an authorized operator environment, run the existing smoke script with | |
| credentials supplied securely by existing environment bindings: | |
| ```bash | |
| export EXPECTED_OPENOPERATOR_SHA='<approved merged GitHub SHA>' | |
| python scripts/smoke_hf_openoperator.py | |
| ``` | |
| Verify `/health` reports that SHA, authenticated API/A2A `message/send` and | |
| `tasks/get` work, malformed calls fail cleanly, and blocking mode remains. | |
| Run a real remote-agent and internal Spynel request, `/goal`, Colab health, | |
| and a small existing DSPy/GEPA validation job. The offline tests do not prove | |
| those external services or credentials are usable. | |
| 5. Exercise Search Engine with a real query. With `A0_SEARCH_BACKEND=auto`, | |
| intentionally unavailable SearXNG must fall back or report structured | |
| unavailability; with `duckduckgo`, SearXNG is intentionally unused. If a remote | |
| SearXNG is configured, verify its reachability and TLS independently. With | |
| `TINYFISH_API_KEY` configured, verify search and public HTTPS document fallback | |
| against Tinyfish after an intentional native failure; never put the key in logs. | |
| 6. Start a new chat: verify automatic title persistence and sidebar updates while | |
| the main model remains responsive. Rename it manually while a delayed title | |
| is pending; verify the manual title wins. Test once/always/disabled settings. | |
| 7. Run the read-only Spynel profile audit with the framework interpreter before | |
| using Spynel, including any active project scope. If blocked, preserve and | |
| explicitly reconcile the reported persistent overrides. Create a harmless test directory through the usual workspace workflow, then | |
| use the read-only resolver with natural and explicit routes. Test a symlink | |
| escape and ambiguous targets; neither may be dispatched automatically. | |
| 8. Check logs, memory, only one public listener, preserved profiles/model settings, | |
| persistent working directories and existing secrets before accepting release. | |
| ## Rollback | |
| After separate authorization, restore the recorded previous application image | |
| or redeploy the prior reviewed source snapshot through the one authoritative | |
| publisher. Do not use the legacy history mirror, force-push a production branch, | |
| or run destructive synchronization against user state. Keep `/a0/usr` and | |
| existing secrets/model settings in place; verify `/health`, auth/A2A, user data | |
| and the prior functionality after rollback. These repairs introduce no database | |
| migration. Revert the individual repair commits on an integration branch for a | |
| source-level rollback and review before a later main deployment. | |
| ## Boundaries honored | |
| No local Docker build; no HF upload, Space restart, rebuild or workflow trigger; | |
| no production credential changes; no production-branch push; no public listener | |
| added; no `/a0/usr` reset. This is a code-review candidate with pending live | |
| source reconciliation and deployment acceptance, not a deployed Space. | |