avoigt1121 commited on
Commit
bd144d6
·
2 Parent(s): 7c9f08cbaa2008

merge: feat/ctgov-processed-date — trials answers show the date ClinicalTrials.gov processed the data (dataTimestamp via one direct /api/v2/version call, ADR-0001 exception approved 2026-11-12) and state our changes; addendum pillow count 40

Browse files
docs/adr/ADR-0001-knowledge-specialist-contract.md CHANGED
@@ -81,6 +81,28 @@ two weeks). The fleet's contract has to sit between ToolUniverse and the orchest
81
  or limit becomes a structured `source_errors` entry, never a retry loop that outlasts the orchestrator's call
82
  limit.
83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  ## Knowledge is annotation, never analysis
85
 
86
  - The specialist computes nothing across sources: no combined scores, no rankings of its own, no p-values. When a
 
81
  or limit becomes a structured `source_errors` entry, never a retry loop that outlasts the orchestrator's call
82
  limit.
83
 
84
+ ## Exception: direct calls (2026-11-12)
85
+
86
+ **Decided 2026-11-12 by the project lead (option A, "direct call").** Decision 7 makes every outbound call an
87
+ allow-listed ToolUniverse tool. One fact no ToolUniverse 1.5.6 tool returns is the date ClinicalTrials.gov processed
88
+ its data, which the ClinicalTrials.gov Terms and Conditions ask a redistributor to show (with a statement of any
89
+ changes): every ClinicalTrials.gov tool rebuilds a study from its record keeping only the start and completion dates,
90
+ and none wraps `/api/v2/version`, whose `dataTimestamp` is that date. So the Space makes one direct call,
91
+ `ctgov_data_timestamp` (`GET https://clinicaltrials.gov/api/v2/version`), once per `trials_for_genes` answer.
92
+
93
+ The exception is bounded, and tests hold each bound (`tests/test_direct_calls.py`):
94
+
95
+ - A direct call is a named entry in `src/direct_calls.py`, the only module that opens a URL itself. It takes no
96
+ arguments, so no identifier or other input reaches it.
97
+ - Its host is already declared for its source in `config/sources.yaml`; this one adds no host.
98
+ - It runs through `ToolClient` like a tool: our cache with the source's TTL (24 h), the source's rate bucket, the
99
+ deadline, a pinned schema (`src/schemas/ctgov_data_timestamp.schema.json`), and cassette replay in tests.
100
+ - Adding another direct call needs the same kind of decision; it is not a config change.
101
+
102
+ What it shows: every `trials_for_genes` payload with an answer carries `source_terms` and ends its narrative with
103
+ "Data from ClinicalTrials.gov, processed by ClinicalTrials.gov on <date>." plus the statement of what this Space
104
+ changes (`src/ctgov_terms.py`). If the date cannot be read the notice says so and why, and the studies stay.
105
+
106
  ## Knowledge is annotation, never analysis
107
 
108
  - The specialist computes nothing across sources: no combined scores, no rankings of its own, no p-values. When a
docs/security/addendum-knowledge-specialist.md CHANGED
@@ -45,6 +45,12 @@ loaded tool's configuration and code in the installed package and fails on any u
45
  8 hosts in all; 6 are called by a Phase 1 endpoint. Source of truth: `config/sources.yaml`, checked against the installed ToolUniverse by `tests/test_sources_hosts.py`.
46
  <!-- host-table:end -->
47
 
 
 
 
 
 
 
48
  **What leaves the Space.** Only public identifiers: gene symbols (HGNC), gene ids (Ensembl), protein entry numbers (UniProt),
49
  EFO/MONDO/Orphanet disease ids, NCT ids, and a disease name as a bounded string (at most 80 characters; letters,
50
  digits, spaces, hyphens, commas, apostrophes), at most 100 per call. Free text, sample identifiers, numbers
@@ -93,7 +99,7 @@ It holds no secret and no model key, and it keeps counts and timings only, never
93
  is share-alike, so a redistributed table built from ChEMBL rows must carry the same licence [`config/sources.yaml`].
94
  - **Development.** The same enforced pre-push gate as the other repositories: pip-audit over the full 119-package
95
  lock, bandit, gitleaks over the tree and full history, and an SBOM [`scripts/security_scan.sh`]. One accepted
96
- finding set, shared with the rest of the fleet, is documented and expires on 2026-12-18: 39 pillow 12.3.0
97
  advisories that gradio's own version constraint prevents fixing, none reachable here because the Space never
98
  decodes an image a caller supplies [`security/ACCEPTED-FINDINGS.md` §1; `security/accepted-vulns.txt`].
99
  - **The vendor's own posture.** ToolUniverse states that it operates no servers and collects no telemetry; a lookup's
 
45
  8 hosts in all; 6 are called by a Phase 1 endpoint. Source of truth: `config/sources.yaml`, checked against the installed ToolUniverse by `tests/test_sources_hosts.py`.
46
  <!-- host-table:end -->
47
 
48
+ **One direct call (ADR-0001 exception, 2026-11-12).** Besides the ToolUniverse tools, the Space makes one call of
49
+ its own: `GET https://clinicaltrials.gov/api/v2/version`, to read the date ClinicalTrials.gov processed its data,
50
+ which its terms ask every trials answer to show. It reaches a host already in the table, sends nothing (no
51
+ arguments), and goes through the same cache, rate limit, deadline and schema check as the tools
52
+ [`src/direct_calls.py`; `tests/test_direct_calls.py`].
53
+
54
  **What leaves the Space.** Only public identifiers: gene symbols (HGNC), gene ids (Ensembl), protein entry numbers (UniProt),
55
  EFO/MONDO/Orphanet disease ids, NCT ids, and a disease name as a bounded string (at most 80 characters; letters,
56
  digits, spaces, hyphens, commas, apostrophes), at most 100 per call. Free text, sample identifiers, numbers
 
99
  is share-alike, so a redistributed table built from ChEMBL rows must carry the same licence [`config/sources.yaml`].
100
  - **Development.** The same enforced pre-push gate as the other repositories: pip-audit over the full 119-package
101
  lock, bandit, gitleaks over the tree and full history, and an SBOM [`scripts/security_scan.sh`]. One accepted
102
+ finding set, shared with the rest of the fleet, is documented and expires on 2026-12-18: 40 pillow 12.3.0
103
  advisories that gradio's own version constraint prevents fixing, none reachable here because the Space never
104
  decodes an image a caller supplies [`security/ACCEPTED-FINDINGS.md` §1; `security/accepted-vulns.txt`].
105
  - **The vendor's own posture.** ToolUniverse states that it operates no servers and collects no telemetry; a lookup's
scripts/record_fixtures.py CHANGED
@@ -25,7 +25,7 @@ from pathlib import Path
25
  ROOT = Path(__file__).resolve().parent.parent
26
  sys.path.insert(0, str(ROOT))
27
 
28
- from src import config, schemas
29
  from src.cache import ResponseCache
30
  from src.endpoints import ENDPOINTS, _common
31
  from src.tu_client import (
@@ -133,7 +133,10 @@ class Recorder:
133
  self.calls: list[dict] = []
134
 
135
  def __call__(self, tool: str, args: dict):
136
- response = self.tu.run_one_function({"name": tool, "arguments": args}, use_cache=False)
 
 
 
137
  self.calls.append({"tool": tool, "args": dict(args), "response": response})
138
  return response
139
 
 
25
  ROOT = Path(__file__).resolve().parent.parent
26
  sys.path.insert(0, str(ROOT))
27
 
28
+ from src import config, direct_calls, schemas
29
  from src.cache import ResponseCache
30
  from src.endpoints import ENDPOINTS, _common
31
  from src.tu_client import (
 
133
  self.calls: list[dict] = []
134
 
135
  def __call__(self, tool: str, args: dict):
136
+ if direct_calls.is_direct(tool):
137
+ response = direct_calls.run(tool, args, timeout=30)
138
+ else:
139
+ response = self.tu.run_one_function({"name": tool, "arguments": args}, use_cache=False)
140
  self.calls.append({"tool": tool, "args": dict(args), "response": response})
141
  return response
142
 
src/config.py CHANGED
@@ -136,6 +136,10 @@ def tools_for_endpoint(endpoint: str) -> tuple[str, ...]:
136
 
137
 
138
  def source_of(tool: str) -> str:
 
 
 
 
139
  try:
140
  return allowlist()["tools"][tool]["source"]
141
  except KeyError:
 
136
 
137
 
138
  def source_of(tool: str) -> str:
139
+ from src.direct_calls import DIRECT_CALLS # the ADR-0001 direct-call exception
140
+
141
+ if tool in DIRECT_CALLS:
142
+ return DIRECT_CALLS[tool].source
143
  try:
144
  return allowlist()["tools"][tool]["source"]
145
  except KeyError:
src/ctgov_terms.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ClinicalTrials.gov redistribution notice: the date its data were processed, and our changes.
2
+
3
+ The ClinicalTrials.gov Terms and Conditions (config/sources.yaml, checked 2026-10-27) ask anyone
4
+ who redistributes the data to show the date ClinicalTrials.gov processed it and to state any
5
+ changes made. The date is ``dataTimestamp`` from ClinicalTrials.gov's own ``/api/v2/version``
6
+ response (src/direct_calls.py), never our fetch time or today's date. Every payload with
7
+ ClinicalTrials.gov results carries the notice as a structured ``source_terms`` entry AND as the
8
+ last sentence of its narrative, because the narrative is what the UI summary shows and what the
9
+ orchestrator relays. The wording is deterministic text, never generated.
10
+ """
11
+
12
+ from __future__ import annotations
13
+
14
+ from typing import Any
15
+
16
+ SOURCE = "clinicaltrials"
17
+ DATE_TOOL = "ctgov_data_timestamp"
18
+ # The fields the row builder in src/endpoints/trials.py keeps, in words
19
+ # (tests/test_ctgov_terms.py checks this list against the rows).
20
+ SHOWN_FIELDS = (
21
+ "NCT number",
22
+ "title",
23
+ "overall status",
24
+ "phases",
25
+ "study type",
26
+ "conditions",
27
+ "up to five intervention names",
28
+ "lead sponsor",
29
+ "start date",
30
+ "completion date",
31
+ )
32
+ CHANGES = (
33
+ "This Space shows selected fields of each matching study record ("
34
+ + ", ".join(SHOWN_FIELDS)
35
+ + ") and lists only studies whose intervention name matches the gene; it does not edit "
36
+ "the values. Records change: the current version of each is at "
37
+ "clinicaltrials.gov/study/<NCT number>."
38
+ )
39
+
40
+
41
+ def notice(data_timestamp: str | None, problem: str | None = None) -> dict[str, Any]:
42
+ """The structured notice. ``data_timestamp`` is ClinicalTrials.gov's ``dataTimestamp``."""
43
+ processed = data_timestamp[:10] if data_timestamp else None
44
+ if processed:
45
+ lead = f"Data from ClinicalTrials.gov, processed by ClinicalTrials.gov on {processed}."
46
+ else:
47
+ lead = (
48
+ "Data from ClinicalTrials.gov; the date ClinicalTrials.gov processed it could not "
49
+ f"be read ({problem or 'not reported'})."
50
+ )
51
+ return {
52
+ "source": SOURCE,
53
+ "data_processed_on": processed,
54
+ "data_timestamp": data_timestamp,
55
+ "date_from": "ClinicalTrials.gov /api/v2/version dataTimestamp",
56
+ "changes": CHANGES,
57
+ "sentence": f"{lead} {CHANGES}",
58
+ }
src/direct_calls.py ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Direct source calls — the ADR-0001 exception for what ToolUniverse cannot return.
2
+
3
+ ADR-0001 says this Space wraps an allow-listed ToolUniverse subset. One fact no ToolUniverse
4
+ 1.5.6 tool returns is the date ClinicalTrials.gov processed its data, which the
5
+ ClinicalTrials.gov Terms and Conditions ask a redistributor to show: every ClinicalTrials.gov
6
+ tool rebuilds each study from its record and keeps only the start and completion dates, and none
7
+ wraps ``/api/v2/version``, whose ``dataTimestamp`` is that date. The project lead approved one
8
+ direct call for it on 2026-11-12 (option A; recorded in
9
+ docs/adr/ADR-0001-knowledge-specialist-contract.md, "Exception: direct calls").
10
+
11
+ A direct call is not a back door around the client. It is a named entry here, it goes through
12
+ :class:`src.tu_client.ToolClient` exactly like a ToolUniverse tool (our cache with the source's
13
+ TTL, the source's rate bucket, the deadline, a pinned schema under ``src/schemas/``, cassette
14
+ replay in tests), its URL's host must already be declared for its source in
15
+ ``config/sources.yaml`` (tests/test_direct_calls.py), and it takes no arguments, so nothing a
16
+ caller types reaches the URL. This module imports nothing from ``src``.
17
+ """
18
+
19
+ from __future__ import annotations
20
+
21
+ import json
22
+ import urllib.request
23
+ from dataclasses import dataclass
24
+ from typing import Any
25
+
26
+
27
+ @dataclass(frozen=True)
28
+ class DirectCall:
29
+ source: str
30
+ url: str
31
+ why: str
32
+
33
+
34
+ DIRECT_CALLS: dict[str, DirectCall] = {
35
+ "ctgov_data_timestamp": DirectCall(
36
+ source="clinicaltrials",
37
+ url="https://clinicaltrials.gov/api/v2/version",
38
+ why=(
39
+ "dataTimestamp = the date ClinicalTrials.gov processed its data, shown on every "
40
+ "trials answer as its terms ask; no ToolUniverse 1.5.6 tool returns it "
41
+ "(project lead, 2026-11-12)"
42
+ ),
43
+ ),
44
+ }
45
+
46
+
47
+ def is_direct(tool: str) -> bool:
48
+ return tool in DIRECT_CALLS
49
+
50
+
51
+ def run(tool: str, args: dict[str, Any], timeout: float) -> dict[str, Any]:
52
+ """GET the call's fixed URL; the response in ToolUniverse's envelope, so the client treats
53
+ it like any tool's. Errors propagate: the client turns them into ``source_unavailable``."""
54
+ call = DIRECT_CALLS[tool]
55
+ if args:
56
+ raise ValueError(f"{tool} takes no arguments")
57
+ request = urllib.request.Request(
58
+ call.url, headers={"Accept": "application/json", "User-Agent": "pdac-knowledge-agent"}
59
+ )
60
+ with urllib.request.urlopen(request, timeout=timeout) as response:
61
+ data = json.loads(response.read().decode("utf-8"))
62
+ return {"status": "success", "data": data, "metadata": {"source": call.url}}
src/endpoints/trials.py CHANGED
@@ -6,13 +6,19 @@ name (Phase 0 finding 5: "CD40" gives the one pancreas-specific mitazalimab tria
6
  API query also returns two solid-tumour basket trials). That rewrite is helpful but silent, so
7
  every payload states it: ``query_semantics`` in words, and ``executed_queries`` with the exact
8
  query ClinicalTrials.gov ran, taken from the response itself and pinned by a recorded fixture.
 
 
 
 
 
 
9
  """
10
 
11
  from __future__ import annotations
12
 
13
  from typing import Any
14
 
15
- from src import payloads
16
  from src.endpoints._common import Budget, Collector, according_to, get_client, stamped
17
  from src.identifiers import parse_disease, parse_gene_list
18
 
@@ -110,6 +116,10 @@ def trials_for_genes(genes: str = "", condition: str = "", status: str = "") ->
110
  "ClinicalTrials.gov did not answer; nothing was looked up.",
111
  source_errors=out.source_errors,
112
  )
 
 
 
 
113
  lead = according_to(["clinicaltrials"], out.retrieved)
114
  kind = "of any status" if wanted == "ANY" else wanted
115
  clauses = []
@@ -136,6 +146,8 @@ def trials_for_genes(genes: str = "", condition: str = "", status: str = "") ->
136
  + "; ".join(f"{e['source']} {e['reason']}" for e in out.source_errors)
137
  + "."
138
  )
 
 
139
  return payloads.ok(
140
  ENDPOINT,
141
  query={"genes": symbols, "condition": condition_name, "status": wanted},
@@ -146,4 +158,19 @@ def trials_for_genes(genes: str = "", condition: str = "", status: str = "") ->
146
  source_errors=out.source_errors,
147
  query_semantics=QUERY_SEMANTICS,
148
  executed_queries=executed,
 
149
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  API query also returns two solid-tumour basket trials). That rewrite is helpful but silent, so
7
  every payload states it: ``query_semantics`` in words, and ``executed_queries`` with the exact
8
  query ClinicalTrials.gov ran, taken from the response itself and pinned by a recorded fixture.
9
+
10
+ ClinicalTrials.gov's terms ask a redistributor to show the date ClinicalTrials.gov processed the
11
+ data and to state any changes: once a search has answered, one more call reads that date
12
+ (``dataTimestamp``, src/direct_calls.py) and the payload carries the notice of
13
+ :mod:`src.ctgov_terms` in ``source_terms`` and as the narrative's last sentence. Each row also
14
+ carries ``ctgov_data_processed_on``, so a relay that shows only rows keeps the date.
15
  """
16
 
17
  from __future__ import annotations
18
 
19
  from typing import Any
20
 
21
+ from src import ctgov_terms, payloads
22
  from src.endpoints._common import Budget, Collector, according_to, get_client, stamped
23
  from src.identifiers import parse_disease, parse_gene_list
24
 
 
116
  "ClinicalTrials.gov did not answer; nothing was looked up.",
117
  source_errors=out.source_errors,
118
  )
119
+ terms = _terms_notice(client, budget, out) if per_gene else None
120
+ if terms is not None:
121
+ for row in out.rows:
122
+ row["ctgov_data_processed_on"] = terms["data_processed_on"]
123
  lead = according_to(["clinicaltrials"], out.retrieved)
124
  kind = "of any status" if wanted == "ANY" else wanted
125
  clauses = []
 
146
  + "; ".join(f"{e['source']} {e['reason']}" for e in out.source_errors)
147
  + "."
148
  )
149
+ if terms is not None:
150
+ sentences.append(terms["sentence"])
151
  return payloads.ok(
152
  ENDPOINT,
153
  query={"genes": symbols, "condition": condition_name, "status": wanted},
 
158
  source_errors=out.source_errors,
159
  query_semantics=QUERY_SEMANTICS,
160
  executed_queries=executed,
161
+ source_terms=[terms] if terms is not None else [],
162
  )
163
+
164
+
165
+ def _terms_notice(client, budget: Budget, out: Collector) -> dict[str, Any]:
166
+ """The ClinicalTrials.gov notice, with the processed date read from ClinicalTrials.gov.
167
+
168
+ A failure here never hides the studies already found: the notice says the date could not be
169
+ read and why, and the failure is not a ``source_errors`` entry (the search itself answered).
170
+ """
171
+ if budget.spent:
172
+ return ctgov_terms.notice(None, "the call budget ran out before it could be read")
173
+ result, err = out.attempt(client, budget, ctgov_terms.DATE_TOOL, {})
174
+ if err is not None:
175
+ return ctgov_terms.notice(None, f"{err.reason}: {err.detail}")
176
+ return ctgov_terms.notice(result.data["dataTimestamp"])
src/schemas/ctgov_data_timestamp.schema.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$comment": "Pinned 2026-11-12 for the direct call src/direct_calls.py::ctgov_data_timestamp (GET clinicaltrials.gov/api/v2/version; ADR-0001 exception approved 2026-11-12). dataTimestamp is the date ClinicalTrials.gov processed its data, shown on every trials answer; it must start with an ISO date so the shown date is never guessed.",
4
+ "type": "object",
5
+ "required": [
6
+ "dataTimestamp"
7
+ ],
8
+ "properties": {
9
+ "dataTimestamp": {
10
+ "type": "string",
11
+ "pattern": "^\\d{4}-\\d{2}-\\d{2}"
12
+ },
13
+ "apiVersion": {
14
+ "type": "string"
15
+ }
16
+ }
17
+ }
src/tu_client.py CHANGED
@@ -25,7 +25,9 @@ ADR-0001 §7, enforced in code rather than by convention:
25
  ``tooluniverse_version``. A bump is its own lane (doctor, re-recorded fixtures, schema tests,
26
  security scan) and never changes an endpoint's name or payload shape.
27
 
28
- :class:`ToolClient` is what the endpoints call. It returns a :class:`SourceResult` or raises a
 
 
29
  :class:`SourceError` whose ``reason`` is one of the three payload source-error reasons, so a
30
  timeout, a limit or a drifted response is always structured, never a guess.
31
  """
@@ -43,7 +45,7 @@ from collections.abc import Callable, Mapping
43
  from dataclasses import dataclass
44
  from typing import Any
45
 
46
- from src import config, schemas
47
  from src.cache import ResponseCache
48
  from src.ratelimit import RateLimiters
49
 
@@ -285,6 +287,9 @@ class ToolClient:
285
  return self._version
286
 
287
  def _default_runner(self, tool: str, args: dict[str, Any]) -> Any:
 
 
 
288
  with _build_lock:
289
  if self._tu is None:
290
  self._tu = build_tooluniverse()
@@ -299,8 +304,9 @@ class ToolClient:
299
  deadline: float | None = None,
300
  ) -> SourceResult:
301
  """``deadline`` can shorten the source's ``timeout_seconds`` for this call, never extend it."""
302
- if tool not in config.endpoint_tool_names():
303
- # A programming error, not a user input: endpoints only ever name loaded tools.
 
304
  raise ValueError(f"{tool!r} is not an endpoint tool")
305
  source = config.source_of(tool)
306
  cfg = config.source(source)
 
25
  ``tooluniverse_version``. A bump is its own lane (doctor, re-recorded fixtures, schema tests,
26
  security scan) and never changes an endpoint's name or payload shape.
27
 
28
+ :class:`ToolClient` is what the endpoints call. It also runs the few named direct calls of
29
+ :mod:`src.direct_calls` (the ADR-0001 exception for facts no ToolUniverse tool returns) through
30
+ the same cache, rate limit, deadline and schema check. It returns a :class:`SourceResult` or raises a
31
  :class:`SourceError` whose ``reason`` is one of the three payload source-error reasons, so a
32
  timeout, a limit or a drifted response is always structured, never a guess.
33
  """
 
45
  from dataclasses import dataclass
46
  from typing import Any
47
 
48
+ from src import config, direct_calls, schemas
49
  from src.cache import ResponseCache
50
  from src.ratelimit import RateLimiters
51
 
 
287
  return self._version
288
 
289
  def _default_runner(self, tool: str, args: dict[str, Any]) -> Any:
290
+ if direct_calls.is_direct(tool):
291
+ timeout = config.source(config.source_of(tool))["timeout_seconds"]
292
+ return direct_calls.run(tool, args, timeout)
293
  with _build_lock:
294
  if self._tu is None:
295
  self._tu = build_tooluniverse()
 
304
  deadline: float | None = None,
305
  ) -> SourceResult:
306
  """``deadline`` can shorten the source's ``timeout_seconds`` for this call, never extend it."""
307
+ if tool not in config.endpoint_tool_names() and not direct_calls.is_direct(tool):
308
+ # A programming error, not a user input: endpoints only ever name loaded tools (or a
309
+ # named direct call, src/direct_calls.py).
310
  raise ValueError(f"{tool!r} is not an endpoint tool")
311
  source = config.source_of(tool)
312
  cfg = config.source(source)
tests/fixtures/cassettes/q5_trials_cd40.json CHANGED
@@ -9,8 +9,8 @@
9
  ""
10
  ],
11
  "tooluniverse_version": "1.5.6",
12
- "recorded_on": "2026-10-21T04:42:55Z",
13
- "recorded_at_epoch": 1792557775.678,
14
  "recorded_with": "scripts/record_fixtures.py",
15
  "payload_status_when_recorded": "ok",
16
  "licences": {
@@ -85,6 +85,20 @@
85
  "operation": "search"
86
  }
87
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  }
89
  ]
90
  }
 
9
  ""
10
  ],
11
  "tooluniverse_version": "1.5.6",
12
+ "recorded_on": "2026-11-12T16:59:22Z",
13
+ "recorded_at_epoch": 1794502761.98,
14
  "recorded_with": "scripts/record_fixtures.py",
15
  "payload_status_when_recorded": "ok",
16
  "licences": {
 
85
  "operation": "search"
86
  }
87
  }
88
+ },
89
+ {
90
+ "tool": "ctgov_data_timestamp",
91
+ "args": {},
92
+ "response": {
93
+ "status": "success",
94
+ "data": {
95
+ "apiVersion": "2.0.5",
96
+ "dataTimestamp": "2026-11-12T10:00:05"
97
+ },
98
+ "metadata": {
99
+ "source": "https://clinicaltrials.gov/api/v2/version"
100
+ }
101
+ }
102
  }
103
  ]
104
  }
tests/test_ctgov_terms.py ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ClinicalTrials.gov terms: every trials answer shows the date ClinicalTrials.gov processed the
2
+ data (its own ``dataTimestamp``, never our fetch time or today's date) and states our changes.
3
+
4
+ The date comes from the direct ``/api/v2/version`` call (ADR-0001 exception, 2026-11-12). The
5
+ recorded CD40 cassette carries that response; the synthetic cases use a date far from the clock,
6
+ so a value taken from anywhere but the response would show.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ import datetime
12
+ import json
13
+
14
+ import pytest
15
+
16
+ from src import ctgov_terms
17
+ from src.endpoints import _common
18
+ from src.endpoints.trials import trials_for_genes
19
+ from src.ratelimit import RateLimiters
20
+ from src.tu_client import ToolClient
21
+ from tests.conftest import CASSETTES, FakeRunner
22
+
23
+ CT = "ClinicalTrials_search_studies"
24
+
25
+
26
+ def _cassette_timestamp() -> str:
27
+ calls = json.loads((CASSETTES / "q5_trials_cd40.json").read_text())["calls"]
28
+ [call] = [c for c in calls if c["tool"] == ctgov_terms.DATE_TOOL]
29
+ return call["response"]["data"]["dataTimestamp"]
30
+
31
+
32
+ def test_recorded_cd40_answer_shows_the_processed_date_and_the_changes(replay):
33
+ payload = replay("q5_trials_cd40")
34
+ processed = _cassette_timestamp()[:10]
35
+ [terms] = payload["source_terms"]
36
+ assert terms["source"] == "clinicaltrials"
37
+ assert terms["data_processed_on"] == processed
38
+ assert terms["data_timestamp"] == _cassette_timestamp()
39
+ expected = f"Data from ClinicalTrials.gov, processed by ClinicalTrials.gov on {processed}."
40
+ # The narrative is what the UI summary shows and the orchestrator relays.
41
+ assert payload["narrative"].endswith(f"{expected} {ctgov_terms.CHANGES}")
42
+ assert all(r["ctgov_data_processed_on"] == processed for r in payload["rows"])
43
+
44
+
45
+ def _study(nct="NCT00000001"):
46
+ return {
47
+ "nct_id": nct,
48
+ "brief_title": "A trial",
49
+ "status": "RECRUITING",
50
+ "phases": ["PHASE1"],
51
+ "study_type": "INTERVENTIONAL",
52
+ "conditions": ["Pancreatic Cancer"],
53
+ "interventions": ["drug"],
54
+ "sponsor": "Someone",
55
+ "start_date": "2024-01-01",
56
+ "completion_date": "2030-01",
57
+ }
58
+
59
+
60
+ def _search(studies):
61
+ return {
62
+ "status": "success",
63
+ "data": {
64
+ "studies": studies,
65
+ "total_count": len(studies),
66
+ "executed_query": {"query.intr": "x"},
67
+ },
68
+ }
69
+
70
+
71
+ @pytest.fixture()
72
+ def client_with(memory_cache, clock):
73
+ def make(responses):
74
+ runner = FakeRunner(responses)
75
+ client = ToolClient(
76
+ runner=runner,
77
+ cache=memory_cache,
78
+ limiters=RateLimiters(clock=clock, sleep=clock.sleep),
79
+ clock=clock,
80
+ tooluniverse_version="1.5.6",
81
+ )
82
+ _common.set_client(client)
83
+ return runner
84
+
85
+ yield make
86
+ _common.set_client(None)
87
+
88
+
89
+ def test_the_date_is_the_responses_not_the_clocks(client_with, clock):
90
+ client_with(
91
+ {
92
+ CT: _search([_study()]),
93
+ ctgov_terms.DATE_TOOL: {
94
+ "status": "success",
95
+ "data": {"apiVersion": "2.0.5", "dataTimestamp": "2019-03-04T10:00:05"},
96
+ },
97
+ }
98
+ )
99
+ payload = trials_for_genes("CD40")
100
+ today = datetime.datetime.fromtimestamp(clock(), datetime.UTC).date().isoformat()
101
+ assert today != "2019-03-04"
102
+ assert payload["source_terms"][0]["data_processed_on"] == "2019-03-04"
103
+ assert "processed by ClinicalTrials.gov on 2019-03-04." in payload["narrative"]
104
+ assert payload["rows"][0]["ctgov_data_processed_on"] == "2019-03-04"
105
+
106
+
107
+ def test_the_date_is_read_once_per_answer(client_with):
108
+ runner = client_with(
109
+ {
110
+ CT: _search([_study()]),
111
+ ctgov_terms.DATE_TOOL: {"status": "success", "data": {"dataTimestamp": "2026-11-12"}},
112
+ }
113
+ )
114
+ trials_for_genes("CD40,KRAS")
115
+ assert [t for t, _ in runner.calls].count(ctgov_terms.DATE_TOOL) == 1
116
+
117
+
118
+ def test_an_unreadable_date_is_said_and_the_studies_stay(client_with):
119
+ client_with({CT: _search([_study()]), ctgov_terms.DATE_TOOL: OSError("down")})
120
+ payload = trials_for_genes("CD40")
121
+ assert payload["status"] == "ok" and len(payload["rows"]) == 1
122
+ [terms] = payload["source_terms"]
123
+ assert terms["data_processed_on"] is None
124
+ assert "the date ClinicalTrials.gov processed it could not be read" in payload["narrative"]
125
+ assert ctgov_terms.CHANGES in payload["narrative"]
126
+ assert payload["source_errors"] == [] # the search answered; only the date is missing
127
+
128
+
129
+ def test_a_drifted_date_response_is_not_guessed(client_with):
130
+ client_with(
131
+ {
132
+ CT: _search([_study()]),
133
+ ctgov_terms.DATE_TOOL: {"status": "success", "data": {"dataTimestamp": "yesterday"}},
134
+ }
135
+ )
136
+ terms = trials_for_genes("CD40")["source_terms"][0]
137
+ assert terms["data_processed_on"] is None
138
+
139
+
140
+ def test_a_failed_search_reads_no_date(client_with):
141
+ runner = client_with({CT: OSError("down"), ctgov_terms.DATE_TOOL: AssertionError("unused")})
142
+ payload = trials_for_genes("CD40")
143
+ assert payload["status"] == "error"
144
+ assert ctgov_terms.DATE_TOOL not in [t for t, _ in runner.calls]
145
+
146
+
147
+ def test_zero_studies_still_carries_the_notice(client_with):
148
+ client_with(
149
+ {
150
+ CT: _search([]),
151
+ ctgov_terms.DATE_TOOL: {"status": "success", "data": {"dataTimestamp": "2026-11-12"}},
152
+ }
153
+ )
154
+ payload = trials_for_genes("CD40")
155
+ assert payload["source_terms"][0]["data_processed_on"] == "2026-11-12"
156
+
157
+
158
+ def test_the_changes_statement_names_every_field_the_rows_keep(replay):
159
+ payload = replay("q5_trials_cd40")
160
+ kept = {
161
+ "nct_id": "NCT number",
162
+ "title": "title",
163
+ "status": "overall status",
164
+ "phases": "phases",
165
+ "study_type": "study type",
166
+ "conditions": "conditions",
167
+ "interventions": "up to five intervention names",
168
+ "sponsor": "lead sponsor",
169
+ "start_date": "start date",
170
+ "completion_date": "completion date",
171
+ }
172
+ row = payload["rows"][0]
173
+ assert set(kept) <= set(row)
174
+ assert list(kept.values()) == list(ctgov_terms.SHOWN_FIELDS)
175
+ # A new study field in the row builder must be named in the statement too.
176
+ study_fields = {"gene", "ctgov_data_processed_on", *kept}
177
+ from src.provenance import PROVENANCE_FIELDS
178
+
179
+ assert set(row) - set(PROVENANCE_FIELDS) - {"provenance"} <= study_fields
tests/test_direct_calls.py ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The ADR-0001 direct-call exception stays narrow (approved by the project lead, 2026-11-12).
2
+
3
+ A direct call is a named entry in src/direct_calls.py, takes no arguments, reaches only a host
4
+ already declared for its source, has a pinned schema and goes through ToolClient. Nothing else in
5
+ the Space opens a URL itself.
6
+ """
7
+
8
+ from __future__ import annotations
9
+
10
+ import re
11
+ from pathlib import Path
12
+
13
+ import pytest
14
+
15
+ from src import config, schemas
16
+ from src.direct_calls import DIRECT_CALLS, run
17
+ from src.tu_client import ToolClient
18
+ from tests.conftest import FakeRunner
19
+
20
+ ROOT = Path(__file__).resolve().parent.parent
21
+
22
+
23
+ def test_only_the_approved_direct_call_exists():
24
+ assert set(DIRECT_CALLS) == {"ctgov_data_timestamp"}
25
+
26
+
27
+ @pytest.mark.parametrize("tool", sorted(DIRECT_CALLS))
28
+ def test_direct_call_host_is_declared_for_its_source(tool):
29
+ call = DIRECT_CALLS[tool]
30
+ host = re.match(r"https://([^/]+)/", call.url).group(1)
31
+ assert host in config.source(call.source)["hosts"]
32
+ assert config.source_of(tool) == call.source
33
+
34
+
35
+ @pytest.mark.parametrize("tool", sorted(DIRECT_CALLS))
36
+ def test_direct_call_has_a_pinned_schema_and_is_not_a_tooluniverse_name(tool):
37
+ assert tool in schemas.pinned_tools()
38
+ assert tool not in config.allowed_tool_names()
39
+
40
+
41
+ def test_direct_calls_take_no_arguments():
42
+ with pytest.raises(ValueError):
43
+ run("ctgov_data_timestamp", {"q": "x"}, timeout=1)
44
+
45
+
46
+ def test_direct_call_goes_through_cache_and_schema(memory_cache, clock):
47
+ runner = FakeRunner(
48
+ {"ctgov_data_timestamp": {"status": "success", "data": {"dataTimestamp": "2026-11-12"}}}
49
+ )
50
+ client = ToolClient(
51
+ runner=runner, cache=memory_cache, clock=clock, tooluniverse_version="1.5.6"
52
+ )
53
+ first = client.call("ctgov_data_timestamp", {})
54
+ second = client.call("ctgov_data_timestamp", {})
55
+ assert (first.cache, second.cache) == ("miss", "hit")
56
+ assert first.source == "clinicaltrials"
57
+ assert len(runner.calls) == 1
58
+
59
+
60
+ def test_no_other_module_opens_a_url_itself():
61
+ pattern = re.compile(
62
+ r"^\s*(import|from)\s+(urllib\.request|requests|httpx|http\.client)\b", re.MULTILINE
63
+ )
64
+ offenders = [
65
+ str(p.relative_to(ROOT))
66
+ for folder in ("src", "scripts")
67
+ for p in (ROOT / folder).rglob("*.py")
68
+ if pattern.search(p.read_text(encoding="utf-8"))
69
+ ]
70
+ offenders += [
71
+ p.name for p in ROOT.glob("*.py") if pattern.search(p.read_text(encoding="utf-8"))
72
+ ]
73
+ assert offenders == ["src/direct_calls.py"]
tests/test_output_schemas.py CHANGED
@@ -110,9 +110,13 @@ def test_every_recorded_response_validates(scenario):
110
  schemas.validate(call["tool"], data)
111
 
112
 
113
- def test_every_pinned_schema_belongs_to_an_endpoint_tool():
114
- assert schemas.pinned_tools() <= config.endpoint_tool_names()
115
 
 
116
 
117
- def test_every_endpoint_tool_has_a_pinned_schema():
118
- assert schemas.pinned_tools() == config.endpoint_tool_names()
 
 
 
 
110
  schemas.validate(call["tool"], data)
111
 
112
 
113
+ def test_every_pinned_schema_belongs_to_an_endpoint_tool_or_a_direct_call():
114
+ from src.direct_calls import DIRECT_CALLS
115
 
116
+ assert schemas.pinned_tools() <= config.endpoint_tool_names() | set(DIRECT_CALLS)
117
 
118
+
119
+ def test_every_endpoint_tool_and_direct_call_has_a_pinned_schema():
120
+ from src.direct_calls import DIRECT_CALLS
121
+
122
+ assert schemas.pinned_tools() == config.endpoint_tool_names() | set(DIRECT_CALLS)