docs/failure-injection.md

Failure Injection & Graceful Degradation

The suite exercises happy paths in the modules' own test files. This page maps the complementary set: what MiniSearch does when a dependency misbehaves. Every row is a Vitest case with the failure injected through a mocked fetch, a mocked module boundary, or an injected circuit breaker, so no external service is needed.

Degradation Matrix

Failure injected	Expected degradation	Where it is pinned
SearXNG answers 500 once, then recovers	Retried with exponential backoff, results returned	server/webSearchService.test.ts › retry logic
SearXNG answers 500 on every attempt	Retry cycle exhausts (4 requests) and costs a single circuit-breaker failure	server/webSearchService.test.ts › graceful degradation
SearXNG answers 200 with no results and a transiently unresponsive engine (e.g. a timeout) once, then recovers	Retried with the same backoff, results returned, not counted as a failure	server/webSearchService.test.ts › retry logic
SearXNG answers 200 with no results and a non-suspending engine error (e.g. server API error) on every attempt	Retry cycle exhausts (4 requests), counts one failure, and fetchSearXNG throws with the engine reasons rather than an empty set, so the case takes the provider-down path below	server/webSearchService.test.ts › retry logic, graceful degradation
SearXNG answers 200 with no results and every unresponsive engine under a long suspension (CAPTCHA, rate limit)	Fails fast on the first attempt (1 request, no backoff), counts one failure, and throws the same error: a suspension outlasts the whole backoff by hours to a day	server/webSearchService.test.ts › retry logic
SearXNG fails five cycles in a row	Circuit opens; further searches short-circuit without calling the upstream	server/webSearchService.test.ts › graceful degradation
SearXNG recovers after the reset timeout	One healthy response closes the circuit again	server/webSearchService.test.ts › graceful degradation
SearXNG's circuit is open while /healthz still answers OK	webSearchServiceStatus reports unhealthy: every search is failing, so the liveness probe alone must not report health	server/webSearchService.test.ts › getWebSearchServiceStatus
A search is lost to unresponsive engines, then SearXNG answers the next one of that type	webSearchServiceStatus reports degraded, then healthy again; the per-engine failure counts stay	server/webSearchService.test.ts › getWebSearchServiceStatus
A text search is lost to unresponsive engines and the image search the client fires next answers	Still degraded, and searches.degradedSearchTypes names the pool: the two go out to different engine pools, so answering one says nothing about the other	server/webSearchService.test.ts › getWebSearchServiceStatus
The circuit reaches its reset timeout with no search since	degraded, not healthy: half-open is a timer, not a proven recovery	server/webSearchService.test.ts › getWebSearchServiceStatus
SearXNG answers 200 with a non-JSON body	fetchSearXNG throws and the endpoint answers HTTP 502	server/webSearchService.test.ts › graceful degradation
SearXNG returns results that are all unusable	Empty result set, no throw	server/webSearchService.test.ts › graceful degradation
Provider down vs. genuinely zero results	Provider down: fetchSearXNG throws, the endpoint answers HTTP 502, and the client search state becomes failed; genuinely zero results: HTTP 200 with [], state stays completed and the no-results alert renders	server/webSearchService.test.ts and server/searchEndpointServerHook.test.ts › graceful degradation
SearXNG is down	/search/text and /search/images answer HTTP 502 with a JSON error	server/searchEndpointServerHook.test.ts › graceful degradation
Reranker is not ready	Results served in SearXNG order, HTTP 200	server/searchEndpointServerHook.test.ts › graceful degradation
Reranking throws mid-request	Results served in SearXNG order, HTTP 200	server/searchEndpointServerHook.test.ts › graceful degradation
Reranker is not ready on an image search	Images still served, with the thumbnail URLs as SearXNG sent them	server/searchEndpointServerHook.test.ts › graceful degradation
Reranking throws on an image search	Images still served, with the thumbnail URLs as SearXNG sent them	server/searchEndpointServerHook.test.ts › graceful degradation
Reranker returns a URL absent from the result set	That image is dropped	server/searchEndpointServerHook.test.ts › graceful degradation
SearXNG is down and the text-search fallback answers	The fallback's results are ranked and served as HTTP 200 on the normal text path, searchesServedByFallback is incremented, and no search duration is recorded	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG is down and the fallback answers with nothing usable	HTTP 200 with [], so the client shows the no-results alert instead of the unavailable one, and searchesServedByFallback is still incremented	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG is down and the fallback fails too	HTTP 502 with a JSON error, searchesFailedOnFallback is incremented, and the log line carries the fallback's status code but never the query	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG fails with too little of the search deadline left	The fallback is never consulted, searchesFailedOnFallback is incremented, and /search/text answers HTTP 502 rather than a result the client would never receive	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG is down while the fallback is switched off	HTTP 502; the fallback is never consulted and neither fallback counter moves	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG is down on an image search while the fallback is on	HTTP 502; the fallback is text-only, so it is never consulted and neither fallback counter moves	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
SearXNG answers while the fallback is on	HTTP 200 from SearXNG's results; the fallback is never consulted and neither fallback counter moves	server/searchEndpointServerHook.test.ts › graceful degradation › text search fallback
The first fallback provider fails and the next one answers	The cascade moves on to the next provider within what is left of the shared budget, and that provider's results are served; the failure is logged with the provider name and a status code, never the query	server/fallbackSearchService.test.ts › the cascade
A fallback provider answers HTTP 429	That provider is paused until its Retry-After (60 s when missing or invalid, at most an hour), the next provider serves the search, and later searches skip the paused one without a request until the pause ends	server/fallbackSearchService.test.ts › pausing a rate-limited provider
Every fallback provider is paused	No provider is called, the cascade throws, and /search/text answers HTTP 502 as for a fallback that failed	server/fallbackSearchService.test.ts › pausing a rate-limited provider
Thumbnail host never answers	Request aborted after the timeout, /thumbnail answers HTTP 502 and the tile shows the host name	server/thumbnailEndpointServerHook.test.ts
DNS lookup for a thumbnail never settles	Bounded by the same deadline as the fetch, /thumbnail answers HTTP 502	server/thumbnailEndpointServerHook.test.ts
Thumbnail URL resolves into a private range	Refused before any request, /thumbnail answers HTTP 403	server/thumbnailEndpointServerHook.test.ts
Thumbnail redirects into a private range	Redirect not followed, HTTP 403	server/thumbnailEndpointServerHook.test.ts
Thumbnail redirect carries a malformed Location	Not followed, HTTP 502, no unhandled rejection	server/thumbnailEndpointServerHook.test.ts
Thumbnail upstream answers with a non-raster or an empty body (SVG included)	/thumbnail answers HTTP 502, the tile shows the host name	server/thumbnailEndpointServerHook.test.ts
Thumbnail body exceeds the byte cap	Body truncated at the cap and served	server/thumbnailEndpointServerHook.test.ts
Thumbnail upstream refuses once, then recovers	The failure is not cached; the next tile load fetches again and serves	server/thumbnailEndpointServerHook.test.ts
/thumbnail budget is spent	HTTP 429 for the tiles only; the search budget is a separate limiter	server/handleTokenVerification.test.ts, server/thumbnailEndpointServerHook.test.ts
A tile's /thumbnail request fails in the browser	That tile shows the host name, the rest of the grid is untouched	client/components/Search/Results/Graphical/ImageResultsList.test.tsx
SearXNG returns an image result with no thumbnail URL	The result is dropped, so the grid never shows a host-name tile for it	server/webSearchService.test.ts › graceful degradation
Search returns nothing to the endpoint	HTTP 200 with [], not an error	server/searchEndpointServerHook.test.ts › graceful degradation
Text search returns nothing to the client	Keyword-only query retried as a fallback	client/modules/textGeneration.degradation.test.ts
Keyword fallback also returns nothing	Text search state stays completed with the no-results alert	client/modules/textGeneration.degradation.test.ts
SearXNG is down on the client	Text search state becomes failed, keyword fallback does not fire	client/modules/textGeneration.degradation.test.ts
Client search fails after a previous search populated the LLM channel	The grounding channel is cleared, so the AI can't ground on the previous query's results; when image search is on and it answers, the channel is repopulated with the (image)-tagged titles and URLs instead	client/modules/textGeneration.degradation.test.ts
Text search fails and the image search answers	The answer waits for the image results and grounds on their (image)-tagged titles and URLs, and the failed-state alert says so	client/modules/textGeneration.degradation.test.ts › search degradation
Live search fails but a stale cached result exists for the same query	The cached result is served flagged stale and the UI shows the "Showing cached results" banner	client/modules/search.test.ts › Stale Result Fallback
Result page host resolves into a private range	Page skipped, no request is made	server/pageContentService.test.ts › fetchPageContents
Result page redirects into a private range	Redirect not followed, page skipped	server/pageContentService.test.ts › fetchPageContents
Result page errors, is not a document, or yields no text	That page is skipped, the others still return	server/pageContentService.test.ts › fetchPageContents
Result host refuses three reads in a row	The host is skipped without a request for the cooling window, and the skip is counted as skippedByBreaker	server/pageContentService.test.ts › host circuit breaker
The window passes on a skipped host	One read probes the host; a success reopens it, a refusal boxes it again	server/pageContentService.test.ts › host circuit breaker
A skipped host appears twice in one batch after the window	One probe is sent, the other read is skipped	server/pageContentService.test.ts › host circuit breaker
Result page body never ends	Reading stops at the byte cap	server/pageContentService.test.ts › fetchPageContents
/page-content errors or never answers	Answer falls back to snippets, search still completes	client/modules/pageContent.test.ts, client/modules/textGeneration.degradation.test.ts › page content grounding
/inference upstream is not configured	HTTP 500 with a JSON error	server/internalApiEndpointServerHook.test.ts › environment configuration
/inference cannot list upstream models	HTTP 500 with a JSON error	server/internalApiEndpointServerHook.test.ts › streaming path
Upstream model fails but another one is available	Retried on the next model, answer still streams	server/internalApiEndpointServerHook.test.ts › streaming path
Every model fails before a single token is sent	HTTP 503 with a JSON error	server/internalApiEndpointServerHook.test.ts › streaming path
Upstream stream ends without a finish part	Treated as a failure: retried, then Stream ended unexpectedly as the last error	server/internalApiEndpointServerHook.test.ts › streaming path
Upstream fails after content was already sent	No retry, SSE error frame, the partial answer stands	server/internalApiEndpointServerHook.test.ts › streaming path
Response is already unwritable when streaming would start	No headers set, the upstream is never called	server/internalApiEndpointServerHook.test.ts › streaming path
/inference answers 503	Generation state becomes failed, nothing persisted	client/modules/textGeneration.degradation.test.ts
Generation interrupted mid-stream	Partial answer preserved, state stays interrupted	client/modules/textGeneration.degradation.test.ts
The local voice catalogue cannot be fetched	The local engine is skipped and the answer is read by an OS voice	client/modules/textToSpeech.test.ts › falls back when the voice catalogue cannot be reached
No local voice matches the language	Same, with a log entry naming the reason	client/modules/textToSpeech.test.ts › falls back when no local voice matches the language
The synthesis worker fails before anything is audible	The answer is read by an OS voice instead	client/modules/textToSpeech.test.ts › falls back to the system voice when the local engine cannot load, falls back when the worker fails and no chunk was ever audible
Every synthesized chunk fails to play (blocked autoplay)	Treated as a local-engine failure, so an OS voice reads the answer rather than the user getting silence	client/modules/textToSpeech.test.ts › falls back when every synthesized chunk fails to play
The browser provides no speechSynthesis at all and the local engine failed	Playback resolves and returns to idle with a log entry, instead of an unhandled rejection	client/modules/textToSpeech.test.ts › resolves and logs when the local engine failed and there are no OS voices
A caller streams an oversized body to /api/validate-access-key	Refused with 413 mid-upload and the socket dropped; the answer carries Connection: close, so the caller's next request opens a fresh connection instead of a dead one	server/validateAccessKeyServerHook.socket.test.ts › refuses an oversized body with 413 without breaking the next request
Two follow-up question generations overlap	The older completion writes nothing, so the flag stays on until the newest finishes and the input keeps the newest question	client/components/AiResponse/ChatInterface.test.tsx › keeps the follow-up question flag on until the newest call finishes
ChatInterface unmounts while a follow-up question is in flight	Both generation flags are cleared and the orphaned completion writes nothing into the next mount	client/components/AiResponse/ChatInterface.test.tsx › leaves both generation flags off after unmounting mid-generation
A follow-up question generation rejects after a newer one started	The failed call blanks nothing, so the newer question still lands and the flag stays on until it does	client/components/AiResponse/ChatInterface.test.tsx › blanks nothing when an orphaned call rejects after a newer one started
ChatInterface unmounts while a regenerate is still awaiting its response	The orphaned regenerate leaves the next mount's response flag alone	client/components/AiResponse/ChatInterface.test.tsx › leaves the next mount's response flag alone when an orphaned regenerate settles
ChatInterface unmounts while a send is still awaiting its response	The orphaned send asks for no follow-up question and leaves the next mount's response flag alone	client/components/AiResponse/ChatInterface.test.tsx › starts no follow-up question for a send that settles after unmount, leaves the next mount's response flag alone when an orphaned send settles
The dictation model cannot load in the browser	While the on-device model is preferred, the local engine is abandoned for the browser's SpeechRecognition, and the button hides when that is missing too	client/modules/speechToText.test.ts › falls back to the browser recognizer when the local engine fails
The browser recognizer fails while the on-device model is off	The session fails outright; there is no reverse fallback, so the disabled model is never downloaded behind the user's back	client/modules/speechToText.test.ts › does not fall back to the wasm engine when the recognizer fails
Microphone permission is denied	A notification says so instead of the press failing quietly, and no fallback asks again	client/components/DictationButton.test.tsx › shows a notification when microphone permission is denied, client/modules/speechToText.test.ts › reports a denied microphone instead of falling back
The dictation engine fails while the microphone permission prompt is open	The load fails rather than resolving into a dead transcriber, so the browser recognizer takes over	client/modules/speechToText.test.ts › fails the load when the engine dies while permission is pending, fails the load when the worker dies uncaught while permission is pending
The browser recognizer is denied after the session started	Reported through the error callback, since the promise it would have rejected has already resolved	client/modules/speechToText.test.ts › reports a denial that arrives after the session resolved
The browser recognizer ends on its own after silence	The session ends and the button leaves its recording state, with no notification, because nothing failed	client/modules/speechToText.test.ts › ends the session quietly when the recognizer stops on its own, client/components/DictationButton.test.tsx › leaves the recording state quietly when the engine ends on its own
The dictation engine fails after it loaded	A notification says so and the session stops, instead of the UI listening forever with the microphone open	client/modules/speechToText.test.ts › reports a worker failure that happens after the engine loaded, client/components/DictationButton.test.tsx › notifies and stops when the engine fails after it loaded
The search form unmounts while dictation is recording	The microphone track, the AudioContext and the worker are all released	client/components/DictationButton.test.tsx › releases the microphone when the button unmounts mid-recording
A chat response starts generating while dictation is recording in the follow-up box	The session is stopped instead of splicing its transcript into a field that has gone read-only and is about to be cleared	client/components/AiResponse/ChatInputArea.test.tsx › stops a running session when a response starts generating
An upstream dictation model file is unavailable	/dictation-models/ answers a 502 rather than caching a partial file	server/dictationModelServerHook.test.ts › returns a 502 when the upstream download fails

Adding a Row

Keep each case next to the module it covers, reusing that file's mocks and harness: a describe("graceful degradation") block where the file has no failure-shaped block yet, the existing one where it does (the /inference rows live under environment configuration and streaming path), a sibling *.degradation.test.ts when the case needs a harness of its own (the client rows drive searchAndRespond through a fake pubSub store), or a sibling *.socket.test.ts when only a real connection can show the failure (the /api/validate-access-key body cap drops the socket mid-upload, and a mocked req/res has no pooled connection to lose). A row earns its place when it fails for the right reason: mutate the degradation branch in the module (delete the catch, the fallback, the interrupt check, or the response header the case depends on) and confirm the case goes red before committing it.

Related Topics

• Development Commands: docs/development-commands.md - Running the suite
• Reranking: docs/reranking.md - Reranker lifecycle and readiness
• Page Content: docs/page-content.md - Reading result pages and its failure behavior
• AI Integration: docs/ai-integration.md - Inference backends
• Overview: docs/overview.md - Search and inference data flow