Spaces:
Running
Running
| # Observability | |
| Telemetry is gated on `RA_MCP_OTEL_ENABLED=true`. When enabled, traces, metrics, and logs are exported via OTLP. | |
| --- | |
| ## Environment Variables | |
| | Variable | Default | Description | | |
| |----------|---------|-------------| | |
| | `RA_MCP_OTEL_ENABLED` | `false` | Master switch | | |
| | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | Collector endpoint | | |
| | `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | `grpc` or `http/protobuf` | | |
| | `OTEL_SERVICE_NAME` | `ra-mcp` | Service name (`service.name`) | | |
| | `ENVIRONMENT` | _(unset)_ | `deployment.environment.name` resource attribute (falls back to `DEPLOYMENT_ENVIRONMENT`) | | |
| | `RA_MCP_OTEL_LOG_BRIDGE` | `true` | Bridge Python logging to OTel | | |
| The SDK sets `service.name`, `service.version`, a per-process | |
| `service.instance.id`, and — when `ENVIRONMENT` is set — | |
| `deployment.environment.name` as resource attributes; `OTEL_RESOURCE_ATTRIBUTES` | |
| is merged on top. | |
| `service.instance.id` defaults to a fresh UUID per process so replicas are | |
| distinguishable (the chart autoscales to several pods). To pin a **stable** | |
| per-pod id, set it from the Kubernetes downward API: | |
| ```yaml | |
| env: | |
| - name: POD_NAME | |
| valueFrom: { fieldRef: { fieldPath: metadata.name } } | |
| - name: OTEL_RESOURCE_ATTRIBUTES | |
| value: "service.instance.id=$(POD_NAME)" | |
| ``` | |
| ## Trace Tree | |
| Every MCP tool call produces a trace spanning from the protocol layer down to HTTP: | |
| ``` mermaid | |
| graph TD | |
| A["tools/call search_transcribed\n<i>FastMCP — automatic</i>"] | |
| B["delegate search_transcribed\n<i>FastMCP — composed server</i>"] | |
| C["tools/call search_transcribed\n<i>FastMCP — provider</i>"] | |
| D["SearchOperations.search\n<i>domain layer</i>"] | |
| E["SearchClient.search\n<i>API client</i>"] | |
| F["HTTP GET\n<i>HTTP client</i>"] | |
| A --> B --> C --> D --> E --> F | |
| ``` | |
| ## Instrumented Components | |
| | Component | Tracer name | Spans | Metrics | | |
| |-----------|-------------|-------|---------| | |
| | HTTP client | `ra_mcp.http_client` | `HTTP GET` | request count, error count, retry count, duration, response size | | |
| | Search client | `ra_mcp.search.client` | `SearchClient.search` | — | | |
| | Search ops | `ra_mcp.search_operations` | `SearchOperations.search` | `ra_mcp.search.requests`, `ra_mcp.search.results` | | |
| | Browse ops | `ra_mcp.browse_operations` | `BrowseOperations.browse_document`, `BrowseOperations._fetch_page_contexts` | `ra_mcp.browse.requests`, `ra_mcp.browse.pages`, `ra_mcp.browse.empty_pages` | | |
| | ALTO client | `ra_mcp.alto_client` | `ALTOClient.fetch_content` | `ra_mcp.alto.fetches` | | |
| | IIIF client | `ra_mcp.iiif_client` | `IIIFClient.get_collection`, `IIIFClient.fetch_manifest` | — | | |
| | OAI-PMH client | `ra_mcp.oai_pmh_client` | `OAIPMHClient.get_metadata`, `OAIPMHClient.extract_manifest_id` | `ra_mcp.oai_pmh.fetches` | | |
| | Viewer fetchers | fastmcp tracer (`fetchers.py`) | `fetch_text_layer` | — | | |
| | LanceDB spine | `ra_mcp.lancedb` | `search <table>` (CLIENT) — all 13 datasets + PDF-guide search | `ra_mcp.lancedb.queries`, `ra_mcp.lancedb.errors`, `ra_mcp.lancedb.query.duration`, `ra_mcp.lancedb.results` | | |
| | TORA geocoding | `ra_mcp.tora.client` | `tora sparql` (CLIENT) | — | | |
| | PDF cache | `ra_mcp.pdf.cache` | `fetch pdf range` (CLIENT), `prefetch pdf` (INTERNAL) | — | | |
| | CLI commands | `ra_mcp.cli.*` | `cli.search`, `cli.browse` | — | | |
| FastMCP adds automatic spans for all `tools/call` and `resources/read` operations. | |
| Handlers enrich the current tool-call span with behavioural intent (search terms | |
| via `db.query.text`, `view.reference_code`, `pdf.url`, `geocode.place`) and mark | |
| it `ERROR` via `mark_span_error` when they return an error string instead of | |
| raising — see [Behavioural analytics](analytics.md) for the product-question map. | |
| Custom metrics all declare a UCUM unit (`s`, `By`, or an annotation such as | |
| `{request}` / `{fetch}` / `{page}`); metric dimensions are kept low-cardinality | |
| (method, status, result enums), while per-request identifiers (URLs, PIDs, | |
| reference codes, keywords, session ids) live on spans only. | |
| ## Error recording | |
| Failures set `span.set_status(ERROR, "{ErrorClass}: …")` and record `error.type` | |
| + a structured `exception.*` log record via `record_span_exception()` — the OTel | |
| Span Event API (`record_exception`) is deprecated, and the log record carries the | |
| active `trace_id`/`span_id` through the log bridge. The stacktrace is logged once | |
| per exception even as it unwinds through several layers (the counter isn't | |
| inflated). Retried-then-succeeded requests stay `UNSET`. | |
| A tool handler that catches an exception (or fails input validation) and returns | |
| an error **string** instead of raising calls `mark_span_error(message, | |
| error_type=…)` first, so the FastMCP `tools/call` span goes `ERROR` — otherwise a | |
| normal return leaves it green and the tool failure rate reads zero. | |
| ## Verify Telemetry | |
| ```bash | |
| # Start Jaeger, exercise tools, check trace tree | |
| dagger call verify-telemetry --source=. | |
| ``` | |