# Failure Injection & Graceful Degradation

The suite exercises happy paths in the modules' own test files. This page maps
the complementary set: what MiniSearch does when a dependency misbehaves. Every
row is a Vitest case with the failure injected through a mocked `fetch`, a mocked
module boundary, or an injected circuit breaker, so no external service is
needed.

## Degradation Matrix

| Failure injected | Expected degradation | Where it is pinned |
| --- | --- | --- |
| SearXNG answers 500 once, then recovers | Retried with exponential backoff, results returned | `server/webSearchService.test.ts` › retry logic |
| SearXNG answers 500 on every attempt | Retry cycle exhausts (4 requests) and costs a single circuit-breaker failure | `server/webSearchService.test.ts` › graceful degradation |
| SearXNG answers 200 with no results and a transiently unresponsive engine (e.g. a timeout) once, then recovers | Retried with the same backoff, results returned, not counted as a failure | `server/webSearchService.test.ts` › retry logic |
| SearXNG answers 200 with no results and a non-suspending engine error (e.g. server API error) on every attempt | Retry cycle exhausts (4 requests), counts one failure, and `fetchSearXNG` throws with the engine reasons rather than an empty set, so the case takes the provider-down path below | `server/webSearchService.test.ts` › retry logic, graceful degradation |
| SearXNG answers 200 with no results and every unresponsive engine under a long suspension (CAPTCHA, rate limit) | Fails fast on the first attempt (1 request, no backoff), counts one failure, and throws the same error: a suspension outlasts the whole backoff by hours to a day | `server/webSearchService.test.ts` › retry logic |
| SearXNG fails five cycles in a row | Circuit opens; further searches short-circuit without calling the upstream | `server/webSearchService.test.ts` › graceful degradation |
| SearXNG recovers after the reset timeout | One healthy response closes the circuit again | `server/webSearchService.test.ts` › graceful degradation |
| SearXNG's circuit is open while `/healthz` still answers OK | `webSearchServiceStatus` reports `unhealthy`: every search is failing, so the liveness probe alone must not report health | `server/webSearchService.test.ts` › getWebSearchServiceStatus |
| A search is lost to unresponsive engines, then SearXNG answers the next one of that type | `webSearchServiceStatus` reports `degraded`, then `healthy` again; the per-engine failure counts stay | `server/webSearchService.test.ts` › getWebSearchServiceStatus |
| A text search is lost to unresponsive engines and the image search the client fires next answers | Still `degraded`, and `searches.degradedSearchTypes` names the pool: the two go out to different engine pools, so answering one says nothing about the other | `server/webSearchService.test.ts` › getWebSearchServiceStatus |
| The circuit reaches its reset timeout with no search since | `degraded`, not `healthy`: half-open is a timer, not a proven recovery | `server/webSearchService.test.ts` › getWebSearchServiceStatus |
| SearXNG answers 200 with a non-JSON body | `fetchSearXNG` throws and the endpoint answers HTTP 502 | `server/webSearchService.test.ts` › graceful degradation |
| SearXNG returns results that are all unusable | Empty result set, no throw | `server/webSearchService.test.ts` › graceful degradation |
| Provider down vs. genuinely zero results | Provider down: `fetchSearXNG` throws, the endpoint answers HTTP 502, and the client search state becomes `failed`; genuinely zero results: HTTP 200 with `[]`, state stays `completed` and the no-results alert renders | `server/webSearchService.test.ts` and `server/searchEndpointServerHook.test.ts` › graceful degradation |
| SearXNG is down | `/search/text` and `/search/images` answer HTTP 502 with a JSON error | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Reranker is not ready | Results served in SearXNG order, HTTP 200 | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Reranking throws mid-request | Results served in SearXNG order, HTTP 200 | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Reranker is not ready on an image search | Images still served, with the thumbnail URLs as SearXNG sent them | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Reranking throws on an image search | Images still served, with the thumbnail URLs as SearXNG sent them | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Reranker returns a URL absent from the result set | That image is dropped | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| SearXNG is down and the text-search fallback answers | The fallback's results are ranked and served as HTTP 200 on the normal text path, `searchesServedByFallback` is incremented, and no search duration is recorded | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG is down and the fallback answers with nothing usable | HTTP 200 with `[]`, so the client shows the no-results alert instead of the unavailable one, and `searchesServedByFallback` is still incremented | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG is down and the fallback fails too | HTTP 502 with a JSON error, `searchesFailedOnFallback` is incremented, and the log line carries the fallback's status code but never the query | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG fails with too little of the search deadline left | The fallback is never consulted, `searchesFailedOnFallback` is incremented, and `/search/text` answers HTTP 502 rather than a result the client would never receive | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG is down while the fallback is switched off | HTTP 502; the fallback is never consulted and neither fallback counter moves | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG is down on an image search while the fallback is on | HTTP 502; the fallback is text-only, so it is never consulted and neither fallback counter moves | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| SearXNG answers while the fallback is on | HTTP 200 from SearXNG's results; the fallback is never consulted and neither fallback counter moves | `server/searchEndpointServerHook.test.ts` › graceful degradation › text search fallback |
| The first fallback provider fails and the next one answers | The cascade moves on to the next provider within what is left of the shared budget, and that provider's results are served; the failure is logged with the provider name and a status code, never the query | `server/fallbackSearchService.test.ts` › the cascade |
| A fallback provider answers HTTP 429 | That provider is paused until its `Retry-After` (60 s when missing or invalid, at most an hour), the next provider serves the search, and later searches skip the paused one without a request until the pause ends | `server/fallbackSearchService.test.ts` › pausing a rate-limited provider |
| Every fallback provider is paused | No provider is called, the cascade throws, and `/search/text` answers HTTP 502 as for a fallback that failed | `server/fallbackSearchService.test.ts` › pausing a rate-limited provider |
| Thumbnail host never answers | Request aborted after the timeout, `/thumbnail` answers HTTP 502 and the tile shows the host name | `server/thumbnailEndpointServerHook.test.ts` |
| DNS lookup for a thumbnail never settles | Bounded by the same deadline as the fetch, `/thumbnail` answers HTTP 502 | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail URL resolves into a private range | Refused before any request, `/thumbnail` answers HTTP 403 | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail redirects into a private range | Redirect not followed, HTTP 403 | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail redirect carries a malformed Location | Not followed, HTTP 502, no unhandled rejection | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail upstream answers with a non-raster or an empty body (SVG included) | `/thumbnail` answers HTTP 502, the tile shows the host name | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail body exceeds the byte cap | Body truncated at the cap and served | `server/thumbnailEndpointServerHook.test.ts` |
| Thumbnail upstream refuses once, then recovers | The failure is not cached; the next tile load fetches again and serves | `server/thumbnailEndpointServerHook.test.ts` |
| `/thumbnail` budget is spent | HTTP 429 for the tiles only; the search budget is a separate limiter | `server/handleTokenVerification.test.ts`, `server/thumbnailEndpointServerHook.test.ts` |
| A tile's `/thumbnail` request fails in the browser | That tile shows the host name, the rest of the grid is untouched | `client/components/Search/Results/Graphical/ImageResultsList.test.tsx` |
| SearXNG returns an image result with no thumbnail URL | The result is dropped, so the grid never shows a host-name tile for it | `server/webSearchService.test.ts` › graceful degradation |
| Search returns nothing to the endpoint | HTTP 200 with `[]`, not an error | `server/searchEndpointServerHook.test.ts` › graceful degradation |
| Text search returns nothing to the client | Keyword-only query retried as a fallback | `client/modules/textGeneration.degradation.test.ts` |
| Keyword fallback also returns nothing | Text search state stays `completed` with the no-results alert | `client/modules/textGeneration.degradation.test.ts` |
| SearXNG is down on the client | Text search state becomes `failed`, keyword fallback does not fire | `client/modules/textGeneration.degradation.test.ts` |
| Client search fails after a previous search populated the LLM channel | The grounding channel is cleared, so the AI can't ground on the previous query's results; when image search is on and it answers, the channel is repopulated with the (image)-tagged titles and URLs instead | `client/modules/textGeneration.degradation.test.ts` |
| Text search fails and the image search answers | The answer waits for the image results and grounds on their (image)-tagged titles and URLs, and the failed-state alert says so | `client/modules/textGeneration.degradation.test.ts` › search degradation |
| Live search fails but a stale cached result exists for the same query | The cached result is served flagged stale and the UI shows the "Showing cached results" banner | `client/modules/search.test.ts` › Stale Result Fallback |
| Result page host resolves into a private range | Page skipped, no request is made | `server/pageContentService.test.ts` › fetchPageContents |
| Result page redirects into a private range | Redirect not followed, page skipped | `server/pageContentService.test.ts` › fetchPageContents |
| Result page errors, is not a document, or yields no text | That page is skipped, the others still return | `server/pageContentService.test.ts` › fetchPageContents |
| Result host refuses three reads in a row | The host is skipped without a request for the cooling window, and the skip is counted as `skippedByBreaker` | `server/pageContentService.test.ts` › host circuit breaker |
| The window passes on a skipped host | One read probes the host; a success reopens it, a refusal boxes it again | `server/pageContentService.test.ts` › host circuit breaker |
| A skipped host appears twice in one batch after the window | One probe is sent, the other read is skipped | `server/pageContentService.test.ts` › host circuit breaker |
| Result page body never ends | Reading stops at the byte cap | `server/pageContentService.test.ts` › fetchPageContents |
| `/page-content` errors or never answers | Answer falls back to snippets, search still completes | `client/modules/pageContent.test.ts`, `client/modules/textGeneration.degradation.test.ts` › page content grounding |
| `/inference` upstream is not configured | HTTP 500 with a JSON error | `server/internalApiEndpointServerHook.test.ts` › environment configuration |
| `/inference` cannot list upstream models | HTTP 500 with a JSON error | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| Upstream model fails but another one is available | Retried on the next model, answer still streams | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| Every model fails before a single token is sent | HTTP 503 with a JSON error | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| Upstream stream ends without a finish part | Treated as a failure: retried, then `Stream ended unexpectedly` as the last error | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| Upstream fails after content was already sent | No retry, SSE error frame, the partial answer stands | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| Response is already unwritable when streaming would start | No headers set, the upstream is never called | `server/internalApiEndpointServerHook.test.ts` › streaming path |
| `/inference` answers 503 | Generation state becomes `failed`, nothing persisted | `client/modules/textGeneration.degradation.test.ts` |
| Generation interrupted mid-stream | Partial answer preserved, state stays `interrupted` | `client/modules/textGeneration.degradation.test.ts` |
| The local voice catalogue cannot be fetched | The local engine is skipped and the answer is read by an OS voice | `client/modules/textToSpeech.test.ts` › falls back when the voice catalogue cannot be reached |
| No local voice matches the language | Same, with a log entry naming the reason | `client/modules/textToSpeech.test.ts` › falls back when no local voice matches the language |
| The synthesis worker fails before anything is audible | The answer is read by an OS voice instead | `client/modules/textToSpeech.test.ts` › falls back to the system voice when the local engine cannot load, falls back when the worker fails and no chunk was ever audible |
| Every synthesized chunk fails to play (blocked autoplay) | Treated as a local-engine failure, so an OS voice reads the answer rather than the user getting silence | `client/modules/textToSpeech.test.ts` › falls back when every synthesized chunk fails to play |
| The browser provides no `speechSynthesis` at all and the local engine failed | Playback resolves and returns to idle with a log entry, instead of an unhandled rejection | `client/modules/textToSpeech.test.ts` › resolves and logs when the local engine failed and there are no OS voices |
| A caller streams an oversized body to `/api/validate-access-key` | Refused with `413` mid-upload and the socket dropped; the answer carries `Connection: close`, so the caller's next request opens a fresh connection instead of a dead one | `server/validateAccessKeyServerHook.socket.test.ts` › refuses an oversized body with 413 without breaking the next request |
| Two follow-up question generations overlap | The older completion writes nothing, so the flag stays on until the newest finishes and the input keeps the newest question | `client/components/AiResponse/ChatInterface.test.tsx` › keeps the follow-up question flag on until the newest call finishes |
| `ChatInterface` unmounts while a follow-up question is in flight | Both generation flags are cleared and the orphaned completion writes nothing into the next mount | `client/components/AiResponse/ChatInterface.test.tsx` › leaves both generation flags off after unmounting mid-generation |
| A follow-up question generation rejects after a newer one started | The failed call blanks nothing, so the newer question still lands and the flag stays on until it does | `client/components/AiResponse/ChatInterface.test.tsx` › blanks nothing when an orphaned call rejects after a newer one started |
| `ChatInterface` unmounts while a regenerate is still awaiting its response | The orphaned regenerate leaves the next mount's response flag alone | `client/components/AiResponse/ChatInterface.test.tsx` › leaves the next mount's response flag alone when an orphaned regenerate settles |
| `ChatInterface` unmounts while a send is still awaiting its response | The orphaned send asks for no follow-up question and leaves the next mount's response flag alone | `client/components/AiResponse/ChatInterface.test.tsx` › starts no follow-up question for a send that settles after unmount, leaves the next mount's response flag alone when an orphaned send settles |
| The dictation model cannot load in the browser | While the on-device model is preferred, the local engine is abandoned for the browser's `SpeechRecognition`, and the button hides when that is missing too | `client/modules/speechToText.test.ts` › falls back to the browser recognizer when the local engine fails |
| The browser recognizer fails while the on-device model is off | The session fails outright; there is no reverse fallback, so the disabled model is never downloaded behind the user's back | `client/modules/speechToText.test.ts` › does not fall back to the wasm engine when the recognizer fails |
| Microphone permission is denied | A notification says so instead of the press failing quietly, and no fallback asks again | `client/components/DictationButton.test.tsx` › shows a notification when microphone permission is denied, `client/modules/speechToText.test.ts` › reports a denied microphone instead of falling back |
| The dictation engine fails while the microphone permission prompt is open | The load fails rather than resolving into a dead transcriber, so the browser recognizer takes over | `client/modules/speechToText.test.ts` › fails the load when the engine dies while permission is pending, fails the load when the worker dies uncaught while permission is pending |
| The browser recognizer is denied after the session started | Reported through the error callback, since the promise it would have rejected has already resolved | `client/modules/speechToText.test.ts` › reports a denial that arrives after the session resolved |
| The browser recognizer ends on its own after silence | The session ends and the button leaves its recording state, with no notification, because nothing failed | `client/modules/speechToText.test.ts` › ends the session quietly when the recognizer stops on its own, `client/components/DictationButton.test.tsx` › leaves the recording state quietly when the engine ends on its own |
| The dictation engine fails after it loaded | A notification says so and the session stops, instead of the UI listening forever with the microphone open | `client/modules/speechToText.test.ts` › reports a worker failure that happens after the engine loaded, `client/components/DictationButton.test.tsx` › notifies and stops when the engine fails after it loaded |
| The search form unmounts while dictation is recording | The microphone track, the AudioContext and the worker are all released | `client/components/DictationButton.test.tsx` › releases the microphone when the button unmounts mid-recording |
| A chat response starts generating while dictation is recording in the follow-up box | The session is stopped instead of splicing its transcript into a field that has gone read-only and is about to be cleared | `client/components/AiResponse/ChatInputArea.test.tsx` › stops a running session when a response starts generating |
| An upstream dictation model file is unavailable | `/dictation-models/` answers a 502 rather than caching a partial file | `server/dictationModelServerHook.test.ts` › returns a 502 when the upstream download fails |

## Adding a Row

Keep each case next to the module it covers, reusing that file's mocks and
harness: a `describe("graceful degradation")` block where the file has no
failure-shaped block yet, the existing one where it does (the `/inference` rows
live under `environment configuration` and `streaming path`), a sibling
`*.degradation.test.ts` when the case needs a harness of its own (the client
rows drive `searchAndRespond` through a fake pubSub store), or a sibling
`*.socket.test.ts` when only a real connection can show the failure (the
`/api/validate-access-key` body cap drops the socket mid-upload, and a mocked
`req`/`res` has no pooled connection to lose). A row earns
its place when it fails for the right reason: mutate the degradation branch in
the module (delete the `catch`, the fallback, the interrupt check, or the
response header the case depends on) and confirm the case goes red before
committing it.

## Related Topics

- **Development Commands**: `docs/development-commands.md` - Running the suite
- **Reranking**: `docs/reranking.md` - Reranker lifecycle and readiness
- **Page Content**: `docs/page-content.md` - Reading result pages and its failure behavior
- **AI Integration**: `docs/ai-integration.md` - Inference backends
- **Overview**: `docs/overview.md` - Search and inference data flow