nikeshn commited on
Commit
e83d126
Β·
verified Β·
1 Parent(s): 396115e

Upload 7 files

Browse files
Files changed (4) hide show
  1. README.md +20 -7
  2. app.py +40 -8
  3. pipeline.py +206 -18
  4. pure_client.py +242 -2
README.md CHANGED
@@ -11,19 +11,25 @@ short_description: Natural-language search over KU's Khazna (Pure) repository
11
 
12
  # Khazna Research Assistant
13
 
14
- Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, researcher profiles, departments/centres and datasets**, and writes a grounded answer with `[n]` citations that link to result cards.
15
 
16
  ## What each tab / intent searches
17
 
18
  | Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent |
19
  |---|---|---|---|
20
  | Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ |
21
- | Profiles | `expert_search` (who works on X), `author_search` (named person) | `/persons/search` + `/research-outputs/search` (KU authors aggregated), `GET /persons/{uuid}` | /en/persons/ |
 
22
  | Departments | `org_search` | `/organizations/search` | /en/organisations/ |
23
- | Datasets | `dataset_search` | `/data-sets/search` (**add `/data-sets/*` to the API key**) | /en/datasets/ |
 
 
 
24
  | Everything | planner decides | any of the above | β€” |
25
 
26
- Expert finding combines two signals: Pure's profile search, plus the KU authors of the top-ranked publications for the topic. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card.
 
 
27
 
28
  ## Pipeline
29
 
@@ -84,7 +90,7 @@ Live page: **https://library.ku.ac.ae/khaznabot**
84
  3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`.
85
 
86
  What the block adds on top of the LibBee model:
87
- - **Scope chips** (Everything, Publications, Profiles, Departments, Datasets), sent to the Space as `scope`. They swipe on phones.
88
  - **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card.
89
  - **Keywords sent to Khazna** panel, plus a large "Search Khazna for: …" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
90
  - **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?".
@@ -92,8 +98,15 @@ What the block adds on top of the LibBee model:
92
 
93
  `/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking.
94
 
 
 
95
  **Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design.
96
 
 
 
 
 
 
97
  ## Endpoints
98
 
99
  | Method | Path | Purpose |
@@ -101,12 +114,12 @@ What the block adds on top of the LibBee model:
101
  | GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) |
102
  | GET | `/test` | the Space's own test chat (widget) |
103
  | GET | `/widget.js` | embeddable widget |
104
- | POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k}` β†’ answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / person / organisation / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` |
105
  | POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) |
106
  | GET | `/config` | public frontend config: announcement, maintenance mode |
107
  | GET | `/health` | backend status |
108
  | GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) |
109
- | GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| persons \| organizations \| data-sets (admin) |
110
 
111
  The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source.
112
 
 
11
 
12
  # Khazna Research Assistant
13
 
14
+ Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, student theses, researcher profiles, departments/centres, research projects, equipment and collaborations**, and writes a grounded answer with `[n]` citations that link to result cards.
15
 
16
  ## What each tab / intent searches
17
 
18
  | Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent |
19
  |---|---|---|---|
20
  | Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ |
21
+ | Theses | `thesis_search` | `/student-theses/search` (needs `/student-theses/*` on the key), merged with thesis-type research outputs; falls back to research outputs alone on 403 | /en/studentTheses/ |
22
+ | Profiles | `expert_search` (who works on / supervises X), `author_search` (named person) | `/persons/search` + `/research-outputs/search` (KU authors aggregated) + `/student-theses/search` (supervisors), `GET /persons/{uuid}` | /en/persons/ |
23
  | Departments | `org_search` | `/organizations/search` | /en/organisations/ |
24
+ | Projects | `project_search` | `/projects/search` (funding **amounts are never shown**) | /en/projects/ |
25
+ | Equipment | `equipment_search` | `/equipment/search` (contact **names only**, no emails) | /en/equipments/ |
26
+ | Collaborations | `collab_search` | external organisations on co-authors of the topic's top KU publications, names via `GET /external-organizations/{uuid}` | β€” |
27
+ | Datasets (no tab) | `dataset_search` | `/data-sets/search` if the Pure API offers it (KU's key currently has no such endpoint) β†’ falls back to dataset-type research outputs | /en/datasets/ |
28
  | Everything | planner decides | any of the above | β€” |
29
 
30
+ **Concept expansion:** for topic, thesis, expert, project and collaboration searches, one Pure thesaurus concept from `/concepts/search` that shares a word with the query is added as an extra search variant. Set `USE_CONCEPTS=false` to switch it off.
31
+
32
+ Expert finding combines three signals: Pure's profile search, the KU authors of the top-ranked publications for the topic, and the supervisors of matching student theses. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card.
33
 
34
  ## Pipeline
35
 
 
90
  3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`.
91
 
92
  What the block adds on top of the LibBee model:
93
+ - **Scope chips** (Everything, Publications, Theses, Profiles, Departments, Projects, Equipment, Collaborations), sent to the Space as `scope`. They swipe on phones.
94
  - **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card.
95
  - **Keywords sent to Khazna** panel, plus a large "Search Khazna for: …" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
96
  - **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?".
 
98
 
99
  `/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking.
100
 
101
+ **Deep links:** `https://library.ku.ac.ae/khaznabot?ask=<question>&scope=<all|publications|theses|persons|organisations|projects|equipment|collaborations|datasets>` asks the question on load, then removes it from the address bar. `libguides/khazna-portal-searchbox.html` is a small search box for the Khazna portal (plain HTML with inline styles, no JS) that uses this link.
102
+
103
  **Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design.
104
 
105
+ ## Pure API key: endpoints the bot uses
106
+
107
+ Used: `/research-outputs/*`, `/student-theses/*`, `/persons/*`, `/organizations/*`, `/external-organizations/*`, `/projects/*`, `/equipment/*`, `/concepts/*`.
108
+ Not used, so they can be removed from the key: `/users/*`, `/roles/*`, `/changes/*`, `/applications/*`, `/awards/*`, `/journals/*`, `/publishers/*`, `/classification-schemes/*`, `/thesauri/*`, `/author-collaborations/*`, `/external-persons/*`, `/courses/*`, `/activities/*`, `/events/*`, `/pressmedia/*`, `/prizes/*`.
109
+
110
  ## Endpoints
111
 
112
  | Method | Path | Purpose |
 
114
  | GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) |
115
  | GET | `/test` | the Space's own test chat (widget) |
116
  | GET | `/widget.js` | embeddable widget |
117
+ | POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k}` β†’ answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / thesis / person / organisation / project / equipment / collaborator / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` |
118
  | POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) |
119
  | GET | `/config` | public frontend config: announcement, maintenance mode |
120
  | GET | `/health` | backend status |
121
  | GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) |
122
+ | GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| student-theses \| persons \| organizations \| projects \| equipment \| external-organizations \| data-sets (admin) |
123
 
124
  The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source.
125
 
app.py CHANGED
@@ -49,9 +49,12 @@ FRONTEND_URL = os.environ.get("FRONTEND_URL", "https://library.ku.ac.ae/khaznabo
49
  RATE_LIMIT = int(os.environ.get("RATE_LIMIT_PER_MIN", "12"))
50
  DB_PATH = "/data/khazna_analytics.db" if os.path.isdir("/data") else "khazna_analytics.db"
51
  HERE = os.path.dirname(os.path.abspath(__file__))
52
- SCOPES = ("all", "publications", "persons", "organisations", "datasets")
53
- SCOPE_PORTAL_PATH = {"publications": "publications", "persons": "persons", "organisations": "organisations", "datasets": "datasets"}
54
- INTENT_SCOPE = {"topic_search": "publications", "author_search": "publications", "count": "publications",
 
 
 
55
  "expert_search": "persons", "org_search": "organisations", "dataset_search": "datasets"}
56
 
57
  pure: PureClient | None = None
@@ -161,15 +164,28 @@ def _require_admin(pw: str | None):
161
  def _public_result(o: dict, n: int) -> dict:
162
  kind = o.get("kind", "publication")
163
  if kind == "person":
164
- keep = ("kind", "uuid", "name", "job_titles", "organisations", "keywords", "orcid", "portal_url", "evidence", "matching_outputs")
 
165
  elif kind == "organisation":
166
  keep = ("kind", "uuid", "name", "type", "parents", "portal_url")
 
 
 
 
 
 
167
  else:
168
  keep = ("kind", "uuid", "title", "subtitle", "type", "year", "date", "authors", "ku_authors", "venue",
169
  "doi", "doi_url", "open_access", "file_available", "portal_url", "keywords")
170
  out = {k: o.get(k) for k in keep}
 
 
 
 
 
 
171
  out["n"] = n
172
- if kind in ("publication", "dataset"):
173
  out["abstract"] = (o.get("abstract") or "")[:400]
174
  return out
175
 
@@ -199,9 +215,23 @@ def _followups(plan, retrieval: dict | None) -> list[dict]:
199
  return out
200
  if plan.intent in ("topic_search", "count"):
201
  out += [{"label": "Who at KU works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
202
- {"label": "Related datasets", "question": f"Datasets on {kw}", "scope": "datasets"}]
 
 
203
  if not plan.year_from:
204
  out.append({"label": f"Only since {year - 3}", "question": f"KU publications on {kw} since {year - 3}", "scope": "publications"})
 
 
 
 
 
 
 
 
 
 
 
 
205
  elif plan.intent == "expert_search":
206
  out += [{"label": "Show the publications", "question": f"KU publications on {kw}", "scope": "publications"},
207
  {"label": "Which departments or centres?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"}]
@@ -211,7 +241,7 @@ def _followups(plan, retrieval: dict | None) -> list[dict]:
211
  elif plan.intent == "dataset_search":
212
  out += [{"label": "Publications on this", "question": f"KU publications on {kw}", "scope": "publications"},
213
  {"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
214
- return out[:3]
215
 
216
 
217
  # ───────────────────────────── routes ───────────────────────────────
@@ -376,7 +406,9 @@ def admin_stats(x_admin_password: str | None = Header(default=None)):
376
 
377
 
378
  DEBUG_ENDPOINTS = {"research-outputs": "normalize_research_output", "persons": "normalize_person",
379
- "organizations": "normalize_organization", "data-sets": "normalize_dataset"}
 
 
380
 
381
 
382
  @app.get("/debug/pure")
 
49
  RATE_LIMIT = int(os.environ.get("RATE_LIMIT_PER_MIN", "12"))
50
  DB_PATH = "/data/khazna_analytics.db" if os.path.isdir("/data") else "khazna_analytics.db"
51
  HERE = os.path.dirname(os.path.abspath(__file__))
52
+ SCOPES = ("all", "publications", "theses", "persons", "organisations", "projects", "equipment", "collaborations", "datasets")
53
+ SCOPE_PORTAL_PATH = {"publications": "publications", "theses": "studentTheses", "persons": "persons",
54
+ "organisations": "organisations", "projects": "projects", "equipment": "equipments",
55
+ "collaborations": "publications", "datasets": "datasets"}
56
+ INTENT_SCOPE = {"thesis_search": "theses", "project_search": "projects", "equipment_search": "equipment",
57
+ "collab_search": "collaborations", "topic_search": "publications", "author_search": "publications", "count": "publications",
58
  "expert_search": "persons", "org_search": "organisations", "dataset_search": "datasets"}
59
 
60
  pure: PureClient | None = None
 
164
  def _public_result(o: dict, n: int) -> dict:
165
  kind = o.get("kind", "publication")
166
  if kind == "person":
167
+ keep = ("kind", "uuid", "name", "job_titles", "organisations", "keywords", "orcid", "portal_url", "evidence", "matching_outputs",
168
+ "supervised", "supervised_theses")
169
  elif kind == "organisation":
170
  keep = ("kind", "uuid", "name", "type", "parents", "portal_url")
171
+ elif kind == "collaborator":
172
+ keep = ("kind", "uuid", "name", "type", "country", "joint_outputs", "examples", "portal_url")
173
+ elif kind == "project":
174
+ keep = ("kind", "uuid", "title", "subtitle", "type", "year", "period", "status", "venue", "funders", "keywords", "portal_url")
175
+ elif kind == "equipment":
176
+ keep = ("kind", "uuid", "title", "type", "category", "venue", "location", "keywords", "portal_url")
177
  else:
178
  keep = ("kind", "uuid", "title", "subtitle", "type", "year", "date", "authors", "ku_authors", "venue",
179
  "doi", "doi_url", "open_access", "file_available", "portal_url", "keywords")
180
  out = {k: o.get(k) for k in keep}
181
+ if kind == "thesis":
182
+ out["supervisors"] = [{"name": x.get("name"), "uuid": x.get("uuid")} for x in (o.get("supervisors") or [])]
183
+ if kind == "project":
184
+ out["participants"] = [{"name": x.get("name"), "role": x.get("role")} for x in (o.get("participants") or [])]
185
+ if kind == "equipment":
186
+ out["contacts"] = [{"name": x.get("name")} for x in (o.get("contacts") or [])] # names only
187
  out["n"] = n
188
+ if kind in ("publication", "dataset", "thesis", "project", "equipment"):
189
  out["abstract"] = (o.get("abstract") or "")[:400]
190
  return out
191
 
 
215
  return out
216
  if plan.intent in ("topic_search", "count"):
217
  out += [{"label": "Who at KU works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
218
+ {"label": "KU theses on this", "question": f"KU theses on {kw}", "scope": "theses"},
219
+ {"label": "Related projects", "question": f"KU research projects on {kw}", "scope": "projects"},
220
+ {"label": "Who does KU collaborate with?", "question": f"Which organisations does KU collaborate with on {kw}?", "scope": "collaborations"}]
221
  if not plan.year_from:
222
  out.append({"label": f"Only since {year - 3}", "question": f"KU publications on {kw} since {year - 3}", "scope": "publications"})
223
+ elif plan.intent == "project_search":
224
+ out += [{"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
225
+ {"label": "Publications from this area", "question": f"KU publications on {kw}", "scope": "publications"}]
226
+ elif plan.intent == "equipment_search":
227
+ out += [{"label": "Which departments use this?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"},
228
+ {"label": "Experts in this area", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
229
+ elif plan.intent == "collab_search":
230
+ out += [{"label": "The joint publications", "question": f"KU publications on {kw}", "scope": "publications"},
231
+ {"label": "KU experts in this area", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
232
+ elif plan.intent == "thesis_search":
233
+ out += [{"label": "Who supervises in this area?", "question": f"Who at KU supervises theses on {kw}?", "scope": "persons"},
234
+ {"label": "Related publications", "question": f"KU publications on {kw}", "scope": "publications"}]
235
  elif plan.intent == "expert_search":
236
  out += [{"label": "Show the publications", "question": f"KU publications on {kw}", "scope": "publications"},
237
  {"label": "Which departments or centres?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"}]
 
241
  elif plan.intent == "dataset_search":
242
  out += [{"label": "Publications on this", "question": f"KU publications on {kw}", "scope": "publications"},
243
  {"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
244
+ return out[:4]
245
 
246
 
247
  # ───────────────────────────── routes ───────────────────────────────
 
406
 
407
 
408
  DEBUG_ENDPOINTS = {"research-outputs": "normalize_research_output", "persons": "normalize_person",
409
+ "organizations": "normalize_organization", "data-sets": "normalize_dataset",
410
+ "student-theses": "normalize_student_thesis", "projects": "normalize_project",
411
+ "equipment": "normalize_equipment", "external-organizations": "normalize_external_organization"}
412
 
413
 
414
  @app.get("/debug/pure")
pipeline.py CHANGED
@@ -45,10 +45,15 @@ OutputType = Literal["article", "conference", "book", "chapter", "thesis", "data
45
 
46
  class QueryPlan(BaseModel):
47
  intent: Literal["topic_search", "author_search", "expert_search", "org_search", "dataset_search",
 
48
  "count", "repository_help", "social", "out_of_scope"] = Field(
49
  description="topic_search: find KU publications on a subject; author_search: profile and/or outputs of a NAMED "
50
  "KU researcher; expert_search: WHO at KU works on a topic (find experts/supervisors/collaborators); "
51
  "org_search: KU colleges, departments, research centres/labs; dataset_search: research datasets; "
 
 
 
 
52
  "count: how many outputs match; repository_help: questions ABOUT Khazna (deposit, theses submission, "
53
  "what is Khazna); social: greetings/thanks; out_of_scope: anything else."
54
  )
@@ -76,7 +81,10 @@ Rules:
76
  - Keyword queries are ALWAYS in English (Khazna metadata is English), even if the user writes in Arabic.
77
  - Strip filler ("can you find", "papers about", "KU", "recent", "research on"). Keep domain terms and named entities.
78
  - Years: "recent"/"latest" => sort=newest, and year_from = {year}-3. "since 2022" => year_from=2022. "in 2024" => both 2024.
79
- - Theses/dissertations => output_types=["thesis"]. "conference papers" => ["conference"]. "journal articles" => ["article"].
 
 
 
80
  - Follow-ups ("only 2024 ones", "more by her", "what about desalination?") must be resolved using the conversation history.
81
  - For author_search put the name in author_name; put any topic words in keyword_queries (may be empty).
82
  - Today's year is {year}.
@@ -84,7 +92,11 @@ Rules:
84
  Examples:
85
  "Find KU papers on machine learning for predicting corrosion in pipelines"
86
  => topic_search, keyword_queries=["machine learning corrosion pipelines","corrosion prediction neural network"], broad_query="corrosion"
87
- "Latest theses on solar desalination" => topic_search, ["solar desalination","solar-driven water desalination"], broad_query="desalination", output_types=["thesis"], sort=newest, year_from={y3}
 
 
 
 
88
  "What has Dr. Jane Smith published on graphene?" => author_search, author_name="Jane Smith", keyword_queries=["graphene"]
89
  "How many KU publications on COVID-19?" => count, ["COVID-19","SARS-CoV-2"]
90
  "How do I deposit my thesis in Khazna?" => repository_help
@@ -101,7 +113,11 @@ _STOP = set("""a an the of in on for to and or with by from about into at as is
101
  what which who whom how why when where can could would should will shall may might do does did have has had any some
102
  me my i we our you your find show give get list search look papers paper articles article research publications publication
103
  studies study work works ku khalifa university recent latest new please related regarding
104
- published publish since before after between during year years ku's theses thesis dissertation dissertations""".split())
 
 
 
 
105
 
106
 
107
  def _heuristic_plan(question: str, year: int) -> QueryPlan:
@@ -112,7 +128,17 @@ def _heuristic_plan(question: str, year: int) -> QueryPlan:
112
  yrs = [int(y) for y in re.findall(r"\b(19[89]\d|20[0-4]\d)\b", question)]
113
  recent = bool(re.search(r"\b(recent|latest|new)\b", question, re.I))
114
  intent = "topic_search"
115
- if re.search(r"\b(dataset|datasets|data set)\b", question, re.I):
 
 
 
 
 
 
 
 
 
 
116
  intent = "dataset_search"
117
  elif re.search(r"\b(who (works|researches|is working)|experts?|supervisors?|researchers? (in|on|working))\b", question, re.I):
118
  intent = "expert_search"
@@ -127,18 +153,23 @@ def _heuristic_plan(question: str, year: int) -> QueryPlan:
127
  broad_query=" ".join(words[:2]),
128
  year_from=min(yrs) if yrs else (year - 3 if recent else None),
129
  year_to=max(yrs) if len(yrs) > 1 else None,
130
- output_types=["thesis"] if re.search(r"\b(thesis|theses|dissertation)", question, re.I) else [],
131
  sort="newest" if recent else "relevance",
132
  )
133
 
134
 
135
  SCOPE_INTENTS = {
136
- "publications": {"topic_search", "author_search", "count"},
 
 
 
 
137
  "persons": {"expert_search", "author_search"},
138
  "organisations": {"org_search"},
139
  "datasets": {"dataset_search"},
140
  }
141
- _SCOPE_DEFAULT = {"publications": "topic_search", "persons": "expert_search",
 
142
  "organisations": "org_search", "datasets": "dataset_search"}
143
 
144
 
@@ -153,6 +184,8 @@ def apply_scope(plan: QueryPlan, scope: str) -> QueryPlan:
153
 
154
  async def plan_query(question: str, history: list[dict], model: str, year: int, scope: str = "all") -> QueryPlan:
155
  plan = await _plan_query(question, history, model, year)
 
 
156
  return apply_scope(plan, scope)
157
 
158
 
@@ -171,7 +204,8 @@ async def _plan_query(question: str, history: list[dict], model: str, year: int)
171
  return _heuristic_plan(question, year)
172
  # Guardrails on the LLM's output
173
  plan.keyword_queries = [q.strip() for q in plan.keyword_queries if q and q.strip()][:3]
174
- if plan.intent in ("topic_search", "count", "expert_search", "org_search", "dataset_search") and not plan.keyword_queries:
 
175
  plan.keyword_queries = _heuristic_plan(question, year).keyword_queries
176
  return plan
177
 
@@ -317,11 +351,13 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
317
  q0 = plan.keyword_queries[0] if plan.keyword_queries else plan.broad_query
318
  pubs_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "year_from": plan.year_from}))
319
  people_task = pure.search_persons(q0, size=10)
320
- pubs, people = await asyncio.gather(pubs_task, people_task, return_exceptions=True)
 
321
  if isinstance(pubs, Exception):
322
  raise pubs
323
  if isinstance(people, Exception):
324
  people = []
 
325
 
326
  score: Counter = Counter()
327
  evidence: dict[str, list[dict]] = {}
@@ -333,6 +369,13 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
333
  score[a["uuid"]] += 1.0 / (3 + rank_i)
334
  evidence.setdefault(a["uuid"], []).append(o)
335
  names[a["uuid"]] = a["name"]
 
 
 
 
 
 
 
336
  profile_by_uuid = {p["uuid"]: p for p in people}
337
  for r, p in enumerate(people):
338
  score[p["uuid"]] += 0.25 / (1 + r)
@@ -348,11 +391,14 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
348
  p = profile_by_uuid.get(u)
349
  if p is None: # non-public profile β†’ skip rather than expose
350
  continue
351
- ev = evidence.get(u, [])
352
  results.append({**p, "evidence": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ev[:3]],
353
- "matching_outputs": len(ev), "_score": round(score[u], 4)})
 
 
354
  await pure.fill_org_names(results)
355
- trace = pubs["trace"] + [{"endpoint": "persons/search", "query": q0, "hits": len(people)}]
 
356
  return {"results": results, "unfiltered": len(results), "total": len(score), "person": None, "trace": trace}
357
 
358
 
@@ -371,25 +417,143 @@ async def _retrieve_orgs(pure: PureClient, plan: QueryPlan) -> dict:
371
  return {"results": results, "unfiltered": len(results), "total": len(results), "person": None, "trace": trace}
372
 
373
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
374
  async def retrieve(pure: PureClient, plan: QueryPlan) -> dict:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
375
  if plan.intent == "expert_search":
376
  return await _retrieve_experts(pure, plan)
377
  if plan.intent == "org_search":
378
  return await _retrieve_orgs(pure, plan)
379
  if plan.intent == "dataset_search":
380
- return await _retrieve_outputs(pure, plan, search=pure.search_datasets)
381
  return await _retrieve_outputs(pure, plan)
382
 
383
 
384
  # ─────────────────────────── synthesis ──────────────────────────────
385
  SYNTH_SYSTEM = """You are the Khazna Research Assistant for Khalifa University (Abu Dhabi, UAE).
386
  You answer questions about KU research using ONLY the Khazna (Elsevier Pure) records provided below.
387
- Records can be publications, datasets, researcher profiles or organisational units (colleges, departments, centres).
 
388
 
389
  Rules:
390
  - Ground every claim in the records. Cite records inline as [n] using their numbers. Never invent titles, authors, years, findings or DOIs.
391
  - Open with a one-sentence direct answer (e.g. how many relevant outputs and the main themes).
392
- - For researcher records: say who seems most relevant and why, based on their listed matching outputs and expertise keywords.
 
 
 
 
393
  Do not rank people by seniority or make claims about availability; suggest contacting them via their Khazna profile.
394
  - Then group the most relevant records by theme or approach in 2-4 short paragraphs or "- " bullets, naming key KU authors and years.
395
  - If some records are only loosely related, say so and do not stretch them. If none really answer the question, say that plainly
@@ -410,20 +574,44 @@ def _format_records(outs: list[dict]) -> str:
410
  f"[{i}] RESEARCHER: {o['name']}\n"
411
  f" Role: {', '.join(o.get('job_titles') or []) or '-'} | Unit: {', '.join(x['name'] for x in o.get('organisations', []) if x.get('name')) or '-'}\n"
412
  f" Expertise keywords: {', '.join(o.get('keywords', [])[:8]) or '-'}\n"
413
- f" Matching KU outputs on this topic: {o.get('matching_outputs', 0)}. Examples: {ev or '-'}"
 
 
414
  )
415
  continue
416
  if kind == "organisation":
417
  lines.append(f"[{i}] ORGANISATIONAL UNIT: {o['name']} | type: {o.get('type') or '-'} | part of: {', '.join(o.get('parents', [])) or '-'}")
418
  continue
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
419
  authors = ", ".join(o["authors"][:6]) + (" et al." if len(o["authors"]) > 6 else "")
420
  ku = ", ".join(a["name"] for a in o["ku_authors"][:6])
421
  lines.append(
422
- f"[{i}] {'DATASET: ' if kind == 'dataset' else ''}{o['title']}{(': ' + o['subtitle']) if o['subtitle'] else ''}\n"
423
  f" {o['type'] or 'Output'} | {o['year'] or 'n.d.'} | {o['venue'] or '-'}\n"
424
  f" Authors: {authors or '-'}\n"
425
  f" KU authors: {ku or '-'}\n"
426
- f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
 
427
  f" Abstract: {o['abstract'][:700] or '(no abstract)'}"
428
  )
429
  return "\n\n".join(lines)
 
45
 
46
  class QueryPlan(BaseModel):
47
  intent: Literal["topic_search", "author_search", "expert_search", "org_search", "dataset_search",
48
+ "thesis_search", "project_search", "equipment_search", "collab_search",
49
  "count", "repository_help", "social", "out_of_scope"] = Field(
50
  description="topic_search: find KU publications on a subject; author_search: profile and/or outputs of a NAMED "
51
  "KU researcher; expert_search: WHO at KU works on a topic (find experts/supervisors/collaborators); "
52
  "org_search: KU colleges, departments, research centres/labs; dataset_search: research datasets; "
53
+ "thesis_search: KU student theses and dissertations (PhD/Master's), including who supervised them; "
54
+ "project_search: KU research projects and grants (funded projects, who leads them, funders); "
55
+ "equipment_search: KU research equipment, instruments and facilities; "
56
+ "collab_search: which external universities/organisations KU collaborates with on a topic; "
57
  "count: how many outputs match; repository_help: questions ABOUT Khazna (deposit, theses submission, "
58
  "what is Khazna); social: greetings/thanks; out_of_scope: anything else."
59
  )
 
81
  - Keyword queries are ALWAYS in English (Khazna metadata is English), even if the user writes in Arabic.
82
  - Strip filler ("can you find", "papers about", "KU", "recent", "research on"). Keep domain terms and named entities.
83
  - Years: "recent"/"latest" => sort=newest, and year_from = {year}-3. "since 2022" => year_from=2022. "in 2024" => both 2024.
84
+ - Theses/dissertations (PhD, doctoral, Master's, student theses) => intent thesis_search. "conference papers" => output_types ["conference"]. "journal articles" => ["article"].
85
+ - "Find a supervisor for X" or "who supervises theses on X" => expert_search.
86
+ - Research projects, grants, funded research => project_search. Instruments, equipment, facilities, labs with a device => equipment_search.
87
+ - "Who does KU collaborate with", partner universities/institutions, international collaboration => collab_search.
88
  - Follow-ups ("only 2024 ones", "more by her", "what about desalination?") must be resolved using the conversation history.
89
  - For author_search put the name in author_name; put any topic words in keyword_queries (may be empty).
90
  - Today's year is {year}.
 
92
  Examples:
93
  "Find KU papers on machine learning for predicting corrosion in pipelines"
94
  => topic_search, keyword_queries=["machine learning corrosion pipelines","corrosion prediction neural network"], broad_query="corrosion"
95
+ "Latest theses on solar desalination" => thesis_search, ["solar desalination","solar-driven water desalination"], broad_query="desalination", sort=newest, year_from={y3}
96
+ "PhD dissertations supervised by Jane Smith" => thesis_search, keyword_queries=["Jane Smith"]
97
+ "What funded projects does KU have on water security?" => project_search, ["water security","water resources"], broad_query="water"
98
+ "Which KU labs have an electron microscope?" => equipment_search, ["electron microscope","SEM TEM microscopy"], broad_query="microscope"
99
+ "Which universities does KU collaborate with on 5G?" => collab_search, ["5G","5G wireless networks"], broad_query="wireless"
100
  "What has Dr. Jane Smith published on graphene?" => author_search, author_name="Jane Smith", keyword_queries=["graphene"]
101
  "How many KU publications on COVID-19?" => count, ["COVID-19","SARS-CoV-2"]
102
  "How do I deposit my thesis in Khazna?" => repository_help
 
113
  what which who whom how why when where can could would should will shall may might do does did have has had any some
114
  me my i we our you your find show give get list search look papers paper articles article research publications publication
115
  studies study work works ku khalifa university recent latest new please related regarding
116
+ published publish since before after between during year years ku's theses thesis dissertation dissertations
117
+ supervise supervises supervised supervisor supervisors supervision advisor advisors expert experts researcher researchers
118
+ people person phd doctoral master masters master's project projects grant grants funded funding
119
+ equipment instrument instruments facility facilities labs lab have has collaborate collaborates collaboration
120
+ collaborations collaborating partner partners universities university institutions does""".split())
121
 
122
 
123
  def _heuristic_plan(question: str, year: int) -> QueryPlan:
 
128
  yrs = [int(y) for y in re.findall(r"\b(19[89]\d|20[0-4]\d)\b", question)]
129
  recent = bool(re.search(r"\b(recent|latest|new)\b", question, re.I))
130
  intent = "topic_search"
131
+ if re.search(r"\b(collaborat\w*|partner (universities|institutions)|co-?author(ed|s)? with)\b", question, re.I):
132
+ intent = "collab_search"
133
+ elif re.search(r"\b(projects?|grants?|funded|funding)\b", question, re.I):
134
+ intent = "project_search"
135
+ elif re.search(r"\b(equipment|instruments?|facilit(y|ies)|microscopes?|spectromet\w*|machines?)\b", question, re.I):
136
+ intent = "equipment_search"
137
+ elif re.search(r"\b(supervis\w*|advisors?)\b", question, re.I) and not re.search(r"\bsupervised by\b", question, re.I):
138
+ intent = "expert_search"
139
+ elif re.search(r"\b(thesis|theses|dissertations?)\b", question, re.I) and not re.search(r"\b(deposit|submit|upload)\b", question, re.I):
140
+ intent = "thesis_search"
141
+ elif re.search(r"\b(dataset|datasets|data set)\b", question, re.I):
142
  intent = "dataset_search"
143
  elif re.search(r"\b(who (works|researches|is working)|experts?|supervisors?|researchers? (in|on|working))\b", question, re.I):
144
  intent = "expert_search"
 
153
  broad_query=" ".join(words[:2]),
154
  year_from=min(yrs) if yrs else (year - 3 if recent else None),
155
  year_to=max(yrs) if len(yrs) > 1 else None,
156
+ output_types=[],
157
  sort="newest" if recent else "relevance",
158
  )
159
 
160
 
161
  SCOPE_INTENTS = {
162
+ "publications": {"topic_search", "author_search", "count", "thesis_search"},
163
+ "theses": {"thesis_search"},
164
+ "projects": {"project_search"},
165
+ "equipment": {"equipment_search"},
166
+ "collaborations": {"collab_search"},
167
  "persons": {"expert_search", "author_search"},
168
  "organisations": {"org_search"},
169
  "datasets": {"dataset_search"},
170
  }
171
+ _SCOPE_DEFAULT = {"publications": "topic_search", "theses": "thesis_search", "persons": "expert_search",
172
+ "projects": "project_search", "equipment": "equipment_search", "collaborations": "collab_search",
173
  "organisations": "org_search", "datasets": "dataset_search"}
174
 
175
 
 
184
 
185
  async def plan_query(question: str, history: list[dict], model: str, year: int, scope: str = "all") -> QueryPlan:
186
  plan = await _plan_query(question, history, model, year)
187
+ if plan.intent == "topic_search" and plan.output_types == ["thesis"]: # older planner habit
188
+ plan.intent, plan.output_types = "thesis_search", []
189
  return apply_scope(plan, scope)
190
 
191
 
 
204
  return _heuristic_plan(question, year)
205
  # Guardrails on the LLM's output
206
  plan.keyword_queries = [q.strip() for q in plan.keyword_queries if q and q.strip()][:3]
207
+ if plan.intent in ("topic_search", "count", "expert_search", "org_search", "dataset_search", "thesis_search",
208
+ "project_search", "equipment_search", "collab_search") and not plan.keyword_queries:
209
  plan.keyword_queries = _heuristic_plan(question, year).keyword_queries
210
  return plan
211
 
 
351
  q0 = plan.keyword_queries[0] if plan.keyword_queries else plan.broad_query
352
  pubs_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "year_from": plan.year_from}))
353
  people_task = pure.search_persons(q0, size=10)
354
+ theses_task = pure.search_student_theses(q0, size=25)
355
+ pubs, people, theses = await asyncio.gather(pubs_task, people_task, theses_task, return_exceptions=True)
356
  if isinstance(pubs, Exception):
357
  raise pubs
358
  if isinstance(people, Exception):
359
  people = []
360
+ theses = [] if isinstance(theses, Exception) else apply_filters(rank([theses[1]], plan), plan.model_copy(update={"output_types": []}))
361
 
362
  score: Counter = Counter()
363
  evidence: dict[str, list[dict]] = {}
 
369
  score[a["uuid"]] += 1.0 / (3 + rank_i)
370
  evidence.setdefault(a["uuid"], []).append(o)
371
  names[a["uuid"]] = a["name"]
372
+ supervised: dict[str, list[dict]] = {}
373
+ for rank_i, t in enumerate(theses[:30]): # supervising theses on the topic is strong evidence for "find a supervisor"
374
+ for sup in t.get("supervisors", []):
375
+ if not sup.get("uuid"):
376
+ continue
377
+ score[sup["uuid"]] += 0.8 / (3 + rank_i)
378
+ supervised.setdefault(sup["uuid"], []).append(t)
379
  profile_by_uuid = {p["uuid"]: p for p in people}
380
  for r, p in enumerate(people):
381
  score[p["uuid"]] += 0.25 / (1 + r)
 
391
  p = profile_by_uuid.get(u)
392
  if p is None: # non-public profile β†’ skip rather than expose
393
  continue
394
+ ev, sv = evidence.get(u, []), supervised.get(u, [])
395
  results.append({**p, "evidence": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ev[:3]],
396
+ "matching_outputs": len(ev),
397
+ "supervised": [{"title": t["title"], "year": t["year"], "portal_url": t["portal_url"]} for t in sv[:3]],
398
+ "supervised_theses": len(sv), "_score": round(score[u], 4)})
399
  await pure.fill_org_names(results)
400
+ trace = pubs["trace"] + [{"endpoint": "persons/search", "query": q0, "hits": len(people)},
401
+ {"endpoint": "student-theses/search", "query": q0, "hits": len(theses)}]
402
  return {"results": results, "unfiltered": len(results), "total": len(score), "person": None, "trace": trace}
403
 
404
 
 
417
  return {"results": results, "unfiltered": len(results), "total": len(results), "person": None, "trace": trace}
418
 
419
 
420
+ def _title_key(o: dict) -> str:
421
+ return re.sub(r"[^a-z0-9]", "", o["title"].lower())[:80]
422
+
423
+
424
+ async def _retrieve_theses(pure: PureClient, plan: QueryPlan) -> dict:
425
+ """Student theses live in their own Pure content type (/student-theses), but some
426
+ theses are also registered as research outputs. Search both and merge, student
427
+ theses first; fall back to research outputs alone if the key lacks /student-theses/*."""
428
+ plan = plan.model_copy(update={"output_types": []})
429
+ st_task = _retrieve_outputs(pure, plan, search=pure.search_student_theses)
430
+ ro_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "output_types": ["thesis"]}))
431
+ st, ro = await asyncio.gather(st_task, ro_task, return_exceptions=True)
432
+ if isinstance(st, Exception) and isinstance(ro, Exception):
433
+ raise st
434
+ if isinstance(st, Exception):
435
+ ro["trace"].insert(0, {"endpoint": "student-theses/search", "error": "unavailable"})
436
+ return ro
437
+ if isinstance(ro, Exception):
438
+ return st
439
+ seen = {_title_key(o) for o in st["results"]}
440
+ merged = st["results"] + [o for o in ro["results"] if _title_key(o) not in seen]
441
+ if plan.sort == "newest":
442
+ merged.sort(key=lambda o: (o.get("year") or 0, o.get("date") or ""), reverse=True)
443
+ return {"results": merged, "unfiltered": st["unfiltered"] + ro["unfiltered"], "total": st["total"] + ro["total"],
444
+ "person": None, "trace": st["trace"] + ro["trace"]}
445
+
446
+
447
+ USE_CONCEPTS = os.environ.get("USE_CONCEPTS", "true").lower() != "false"
448
+
449
+
450
+ async def _expand_with_concepts(pure: PureClient, plan: QueryPlan) -> dict | None:
451
+ """Add one Pure thesaurus concept as an extra search variant (e.g. 'LLM' β†’ 'Large Language Models').
452
+ Only concepts sharing a word with the query are used, to keep expansion on-topic."""
453
+ if not USE_CONCEPTS or not plan.keyword_queries or len(plan.keyword_queries) >= 4:
454
+ return None
455
+ q0 = plan.keyword_queries[0]
456
+ try:
457
+ names = await pure.search_concepts(q0, size=5)
458
+ except Exception:
459
+ return None
460
+ existing = {q.lower() for q in plan.keyword_queries}
461
+ qtok = set(_tokens(q0))
462
+ for name in names:
463
+ if name.lower() not in existing and len(name) <= 60 and qtok & set(_tokens(name)):
464
+ plan.keyword_queries.append(name)
465
+ return {"endpoint": "concepts/search", "concept_for": q0, "added": name}
466
+ return None
467
+
468
+
469
+ async def _retrieve_collaborations(pure: PureClient, plan: QueryPlan, max_orgs: int = 10) -> dict:
470
+ """External organisations most often co-authoring KU's top publications on the topic."""
471
+ pubs = await _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search"}))
472
+ score: Counter = Counter()
473
+ examples: dict[str, list[dict]] = {}
474
+ ref_names: dict[str, str] = {}
475
+ for rank_i, o in enumerate(pubs["results"][:50]):
476
+ for org in o.get("external_orgs", []):
477
+ score[org["uuid"]] += 1.0 / (3 + rank_i)
478
+ examples.setdefault(org["uuid"], []).append(o)
479
+ if org.get("name"):
480
+ ref_names[org["uuid"]] = org["name"]
481
+ top = [u for u, _ in score.most_common(max_orgs * 2)]
482
+ fetched = await asyncio.gather(*(pure.get_external_organization(u) for u in top), return_exceptions=True)
483
+ results = []
484
+ for u, f in zip(top, fetched):
485
+ org = f if isinstance(f, dict) else {"kind": "collaborator", "uuid": u, "name": ref_names.get(u, ""), "country": "", "type": "", "portal_url": ""}
486
+ if not org.get("name") or re.search(r"khalifa university", org["name"], re.I):
487
+ continue
488
+ ex = examples.get(u, [])
489
+ results.append({**org, "title": org["name"], "joint_outputs": len(ex),
490
+ "examples": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ex[:3]],
491
+ "_score": round(score[u], 4)})
492
+ if len(results) >= max_orgs:
493
+ break
494
+ trace = pubs["trace"] + [{"endpoint": "external-organizations", "hits": len(results)}]
495
+ return {"results": results, "unfiltered": len(score), "total": len(score), "person": None, "trace": trace}
496
+
497
+
498
+ async def _retrieve_datasets(pure: PureClient, plan: QueryPlan) -> dict:
499
+ """/data-sets when the Pure API offers it; otherwise dataset-type research outputs."""
500
+ try:
501
+ r = await _retrieve_outputs(pure, plan, search=pure.search_datasets)
502
+ if r["results"]:
503
+ return r
504
+ except PureError as e:
505
+ if e.status not in (403, 404):
506
+ raise
507
+ r = await _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "output_types": ["dataset"]}))
508
+ r["trace"].insert(0, {"endpoint": "data-sets", "note": "not available; used dataset-type research outputs"})
509
+ return r
510
+
511
+
512
  async def retrieve(pure: PureClient, plan: QueryPlan) -> dict:
513
+ concept = None
514
+ if plan.intent in ("topic_search", "thesis_search", "expert_search", "project_search", "collab_search", "count"):
515
+ concept = await _expand_with_concepts(pure, plan)
516
+ result = await _route(pure, plan)
517
+ if concept:
518
+ result["trace"].insert(0, concept)
519
+ return result
520
+
521
+
522
+ async def _route(pure: PureClient, plan: QueryPlan) -> dict:
523
+ if plan.intent == "project_search":
524
+ return await _retrieve_outputs(pure, plan.model_copy(update={"output_types": [], "open_access_only": False}),
525
+ search=pure.search_projects)
526
+ if plan.intent == "equipment_search":
527
+ return await _retrieve_outputs(pure, plan.model_copy(update={"output_types": [], "open_access_only": False,
528
+ "year_from": None, "year_to": None}),
529
+ search=pure.search_equipment)
530
+ if plan.intent == "collab_search":
531
+ return await _retrieve_collaborations(pure, plan)
532
+ if plan.intent == "thesis_search":
533
+ return await _retrieve_theses(pure, plan)
534
  if plan.intent == "expert_search":
535
  return await _retrieve_experts(pure, plan)
536
  if plan.intent == "org_search":
537
  return await _retrieve_orgs(pure, plan)
538
  if plan.intent == "dataset_search":
539
+ return await _retrieve_datasets(pure, plan)
540
  return await _retrieve_outputs(pure, plan)
541
 
542
 
543
  # ─────────────────────────── synthesis ──────────────────────────────
544
  SYNTH_SYSTEM = """You are the Khazna Research Assistant for Khalifa University (Abu Dhabi, UAE).
545
  You answer questions about KU research using ONLY the Khazna (Elsevier Pure) records provided below.
546
+ Records can be publications, student theses, datasets, research projects, equipment/facilities, researcher profiles,
547
+ organisational units (colleges, departments, centres) or external collaborating organisations.
548
 
549
  Rules:
550
  - Ground every claim in the records. Cite records inline as [n] using their numbers. Never invent titles, authors, years, findings or DOIs.
551
  - Open with a one-sentence direct answer (e.g. how many relevant outputs and the main themes).
552
+ - For researcher records: say who seems most relevant and why, based on their matching outputs, theses they supervised and expertise keywords.
553
+ - For theses: mention the degree level, year, student and supervisors where given.
554
+ - For projects: mention the period, status, KU participants and funders where given. Never state funding amounts.
555
+ - For equipment: say what it is and which unit manages it; suggest contacting the listed contact person through Khazna.
556
+ - For collaborators: rank by the number of joint KU outputs listed, and cite those outputs' records.
557
  Do not rank people by seniority or make claims about availability; suggest contacting them via their Khazna profile.
558
  - Then group the most relevant records by theme or approach in 2-4 short paragraphs or "- " bullets, naming key KU authors and years.
559
  - If some records are only loosely related, say so and do not stretch them. If none really answer the question, say that plainly
 
574
  f"[{i}] RESEARCHER: {o['name']}\n"
575
  f" Role: {', '.join(o.get('job_titles') or []) or '-'} | Unit: {', '.join(x['name'] for x in o.get('organisations', []) if x.get('name')) or '-'}\n"
576
  f" Expertise keywords: {', '.join(o.get('keywords', [])[:8]) or '-'}\n"
577
+ f" Matching KU outputs on this topic: {o.get('matching_outputs', 0)}. Examples: {ev or '-'}\n"
578
+ f" Supervised KU theses on this topic: {o.get('supervised_theses', 0)}. Examples: "
579
+ + ("; ".join(f"{t['title']} ({t['year'] or 'n.d.'})" for t in o.get("supervised", [])) or "-")
580
  )
581
  continue
582
  if kind == "organisation":
583
  lines.append(f"[{i}] ORGANISATIONAL UNIT: {o['name']} | type: {o.get('type') or '-'} | part of: {', '.join(o.get('parents', [])) or '-'}")
584
  continue
585
+ if kind == "collaborator":
586
+ ex = "; ".join(f"{e['title']} ({e['year'] or 'n.d.'})" for e in o.get("examples", []))
587
+ lines.append(f"[{i}] EXTERNAL COLLABORATOR: {o['name']} | country: {o.get('country') or '-'} | type: {o.get('type') or '-'}\n"
588
+ f" Joint KU outputs on this topic: {o.get('joint_outputs', 0)}. Examples: {ex or '-'}")
589
+ continue
590
+ if kind == "project":
591
+ lines.append(
592
+ f"[{i}] PROJECT: {o['title']}{(' (' + o['subtitle'] + ')') if o['subtitle'] else ''}\n"
593
+ f" {o['type']} | period: {o.get('period') or '-'} | status: {o.get('status') or '-'} | managed by: {o['venue'] or '-'}\n"
594
+ f" KU participants: {', '.join(p['name'] + (' (' + p['role'] + ')' if p.get('role') else '') for p in o.get('participants', [])) or '-'}\n"
595
+ f" Funders: {', '.join(o.get('funders', [])) or '-'}\n"
596
+ f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
597
+ f" Description: {o['abstract'][:700] or '(none)'}")
598
+ continue
599
+ if kind == "equipment":
600
+ lines.append(
601
+ f"[{i}] EQUIPMENT: {o['title']} | {o['type']} | managed by: {o['venue'] or '-'} | location: {o.get('location') or '-'}\n"
602
+ f" Contact: {', '.join(c['name'] for c in o.get('contacts', [])) or '-'}\n"
603
+ f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
604
+ f" Description: {o['abstract'][:700] or '(none)'}")
605
+ continue
606
  authors = ", ".join(o["authors"][:6]) + (" et al." if len(o["authors"]) > 6 else "")
607
  ku = ", ".join(a["name"] for a in o["ku_authors"][:6])
608
  lines.append(
609
+ f"[{i}] {'DATASET: ' if kind == 'dataset' else 'THESIS: ' if kind == 'thesis' else ''}{o['title']}{(': ' + o['subtitle']) if o['subtitle'] else ''}\n"
610
  f" {o['type'] or 'Output'} | {o['year'] or 'n.d.'} | {o['venue'] or '-'}\n"
611
  f" Authors: {authors or '-'}\n"
612
  f" KU authors: {ku or '-'}\n"
613
+ + (f" Supervisors: {', '.join(x['name'] for x in o.get('supervisors', [])) or '-'}\n" if kind == "thesis" else "")
614
+ + f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
615
  f" Abstract: {o['abstract'][:700] or '(no abstract)'}"
616
  )
617
  return "\n\n".join(lines)
pure_client.py CHANGED
@@ -8,6 +8,11 @@ Endpoints used (all read-only, match the API key's access definition):
8
  GET /persons/{uuid}/research-outputs (falls back to name search if not available)
9
  POST /organizations/search GET /organizations/{uuid}
10
  POST /data-sets/search (needs /data-sets/* added to the API key)
 
 
 
 
 
11
 
12
  Auth: header `api-key: <PURE_API_KEY>`.
13
 
@@ -117,9 +122,12 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
117
  year = pdate.get("year")
118
  date = "-".join(str(p) for p in (year, pdate.get("month"), pdate.get("day")) if p)
119
 
120
- # Contributors
121
- authors, ku_authors = [], []
122
  for c in item.get("contributors") or item.get("personAssociations") or []:
 
 
 
123
  n = _person_name(c.get("name"))
124
  if not n:
125
  continue
@@ -127,6 +135,10 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
127
  disc = c.get("typeDiscriminator", "")
128
  if "Internal" in disc or (c.get("person") or {}).get("uuid"):
129
  ku_authors.append({"name": n, "uuid": (c.get("person") or {}).get("uuid")})
 
 
 
 
130
 
131
  keywords = _keywords(item)
132
 
@@ -169,6 +181,7 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
169
  "open_access": open_access,
170
  "file_available": file_available,
171
  "peer_reviewed": item.get("peerReview"),
 
172
  "portal_url": item.get("portalUrl") or (f"{portal_url}/en/publications/{uuid}" if uuid else ""),
173
  }
174
 
@@ -268,6 +281,202 @@ def normalize_dataset(item: dict, portal_url: str) -> dict:
268
  }
269
 
270
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
271
  # ───────────────────────────── client ───────────────────────────────
272
  class PureClient:
273
  def __init__(
@@ -394,6 +603,37 @@ class PureClient:
394
  for o in p.get("organisations", []):
395
  o["name"] = o.get("name") or names.get(o["uuid"], "")
396
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
397
  # -- datasets ---------------------------------------------------
398
  async def search_datasets(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
399
  data = await self.request("POST", "/data-sets/search", json={"searchString": search_string, "size": size, "offset": offset})
 
8
  GET /persons/{uuid}/research-outputs (falls back to name search if not available)
9
  POST /organizations/search GET /organizations/{uuid}
10
  POST /data-sets/search (needs /data-sets/* added to the API key)
11
+ POST /student-theses/search (needs /student-theses/* added to the API key)
12
+ POST /projects/search GET /persons/{uuid} for participants
13
+ POST /equipment/search
14
+ POST /concepts/search (thesaurus terms used for query expansion)
15
+ GET /external-organizations/{uuid} (collaborator names, cached)
16
 
17
  Auth: header `api-key: <PURE_API_KEY>`.
18
 
 
122
  year = pdate.get("year")
123
  date = "-".join(str(p) for p in (year, pdate.get("month"), pdate.get("day")) if p)
124
 
125
+ # Contributors (+ the external organisations they are affiliated with, for collaboration analysis)
126
+ authors, ku_authors, external_orgs = [], [], []
127
  for c in item.get("contributors") or item.get("personAssociations") or []:
128
+ for ref in c.get("externalOrganizations") or c.get("externalOrganisations") or []:
129
+ if isinstance(ref, dict) and ref.get("uuid"):
130
+ external_orgs.append({"uuid": ref["uuid"], "name": _loc(ref.get("name"))})
131
  n = _person_name(c.get("name"))
132
  if not n:
133
  continue
 
135
  disc = c.get("typeDiscriminator", "")
136
  if "Internal" in disc or (c.get("person") or {}).get("uuid"):
137
  ku_authors.append({"name": n, "uuid": (c.get("person") or {}).get("uuid")})
138
+ for ref in item.get("externalOrganizations") or []:
139
+ if isinstance(ref, dict) and ref.get("uuid"):
140
+ external_orgs.append({"uuid": ref["uuid"], "name": _loc(ref.get("name"))})
141
+ external_orgs = list({o["uuid"]: o for o in external_orgs}.values())
142
 
143
  keywords = _keywords(item)
144
 
 
181
  "open_access": open_access,
182
  "file_available": file_available,
183
  "peer_reviewed": item.get("peerReview"),
184
+ "external_orgs": external_orgs,
185
  "portal_url": item.get("portalUrl") or (f"{portal_url}/en/publications/{uuid}" if uuid else ""),
186
  }
187
 
 
281
  }
282
 
283
 
284
+ def _date_parts(v: Any) -> tuple[int | None, str]:
285
+ """Pure dates arrive as {"year":..,"month":..,"day":..} or as "YYYY-MM-DD" strings."""
286
+ if isinstance(v, dict):
287
+ y = v.get("year")
288
+ parts = [str(p) for p in (y, v.get("month"), v.get("day")) if p]
289
+ return (int(y) if str(y or "").isdigit() else None), "-".join(parts)
290
+ if isinstance(v, str):
291
+ m = re.match(r"(\d{4})(?:-(\d{2}))?(?:-(\d{2}))?", v)
292
+ if m:
293
+ return int(m.group(1)), "-".join(x for x in m.groups() if x)
294
+ return None, ""
295
+
296
+
297
+ def normalize_student_thesis(item: dict, portal_url: str) -> dict:
298
+ uuid = item.get("uuid", "")
299
+ year, date = None, ""
300
+ for key in ("awardDate", "publicationDate", "submissionDate"):
301
+ year, date = _date_parts(item.get(key))
302
+ if year:
303
+ break
304
+ if not year:
305
+ for st in item.get("publicationStatuses") or []:
306
+ year, date = _date_parts(st.get("publicationDate"))
307
+ if year:
308
+ break
309
+
310
+ authors, ku_authors, supervisors = [], [], []
311
+ for c in item.get("contributors") or item.get("personAssociations") or []:
312
+ n = _person_name(c.get("name"))
313
+ if not n:
314
+ continue
315
+ role = _loc(c.get("role")).lower() or _uri_tail(c.get("role")).lower()
316
+ entry = {"name": n, "uuid": (c.get("person") or {}).get("uuid")}
317
+ if "supervis" in role or "advisor" in role:
318
+ supervisors.append(entry)
319
+ continue
320
+ authors.append(n)
321
+ if "Internal" in c.get("typeDiscriminator", "") or entry["uuid"]:
322
+ ku_authors.append(entry)
323
+ for sup in item.get("supervisors") or []: # some Pure versions list supervisors separately
324
+ n = _person_name(sup.get("name"))
325
+ if n and n not in {x["name"] for x in supervisors}:
326
+ supervisors.append({"name": n, "uuid": (sup.get("person") or {}).get("uuid")})
327
+
328
+ institution = ""
329
+ for inst in item.get("awardingInstitutions") or []:
330
+ institution = (_loc((inst.get("organization") or inst.get("externalOrganization") or {}).get("name"))
331
+ or _loc(inst.get("name")) or _loc(inst))
332
+ if institution:
333
+ break
334
+
335
+ doi = ""
336
+ file_available, open_access = bool(item.get("documents")), False
337
+ for ev in item.get("electronicVersions") or []:
338
+ if not doi and ev.get("doi"):
339
+ doi = _norm_doi(ev["doi"])
340
+ if _uri_tail(ev.get("accessType")).lower() in ("open", "free"):
341
+ open_access = True
342
+ if "File" in ev.get("typeDiscriminator", "") or ev.get("file"):
343
+ file_available = True
344
+
345
+ return {
346
+ "kind": "thesis",
347
+ "uuid": uuid,
348
+ "title": _clean_text(_loc(item.get("title"))) or "Untitled thesis",
349
+ "subtitle": _clean_text(_loc(item.get("subTitle"))),
350
+ "type": _loc(item.get("type")) or "Thesis",
351
+ "type_key": "thesis",
352
+ "category": "",
353
+ "year": year,
354
+ "date": date,
355
+ "status": "",
356
+ "authors": authors,
357
+ "ku_authors": ku_authors,
358
+ "supervisors": supervisors[:4],
359
+ "abstract": _clean_text(_loc(item.get("abstract")), limit=1200),
360
+ "keywords": _keywords(item),
361
+ "venue": _clean_text(institution),
362
+ "doi": doi,
363
+ "doi_url": f"https://doi.org/{doi}" if doi else "",
364
+ "open_access": open_access,
365
+ "file_available": file_available,
366
+ "peer_reviewed": None,
367
+ "portal_url": item.get("portalUrl") or (f"{portal_url}/en/studentTheses/{uuid}" if uuid else ""),
368
+ }
369
+
370
+
371
+ def _descriptions(item: dict, limit: int = 1200) -> str:
372
+ for d in item.get("descriptions") or []:
373
+ text = _loc(d.get("value") if isinstance(d, dict) else d)
374
+ if text:
375
+ return _clean_text(text, limit=limit)
376
+ return _clean_text(_loc(item.get("description")) or _loc(item.get("abstract")), limit=limit)
377
+
378
+
379
+ def _people(item: dict, *keys: str) -> list[dict]:
380
+ out = []
381
+ for key in keys:
382
+ for c in item.get(key) or []:
383
+ if not isinstance(c, dict):
384
+ continue
385
+ n = _person_name(c.get("name")) or _person_name((c.get("person") or {}).get("name"))
386
+ if n:
387
+ out.append({"name": n, "uuid": (c.get("person") or {}).get("uuid") or c.get("uuid"),
388
+ "role": _loc(c.get("role"))})
389
+ return list({(p["uuid"] or p["name"]): p for p in out}.values())
390
+
391
+
392
+ def normalize_project(item: dict, portal_url: str) -> dict:
393
+ """Project records. Funding amounts are deliberately NOT copied (often confidential)."""
394
+ uuid = item.get("uuid", "")
395
+ period = item.get("period") or {}
396
+ start_y, start = _date_parts(period.get("startDate") or item.get("startDate"))
397
+ end_y, end = _date_parts(period.get("endDate") or item.get("endDate"))
398
+ participants = _people(item, "participants", "contributors", "personAssociations")
399
+ funders = []
400
+ for f in item.get("fundings") or item.get("fundingDetails") or []:
401
+ if not isinstance(f, dict):
402
+ continue
403
+ name = (_loc((f.get("funder") or f.get("fundingOrganization") or f.get("externalOrganization") or {}).get("name"))
404
+ or _loc(f.get("funderName")))
405
+ if name:
406
+ funders.append(name)
407
+ managing = _loc((item.get("managingOrganization") or item.get("managingOrganisationalUnit") or {}).get("name"))
408
+ status = _loc(item.get("status")) or _loc(item.get("workflow"))
409
+ return {
410
+ "kind": "project",
411
+ "uuid": uuid,
412
+ "title": _clean_text(_loc(item.get("title"))) or "Untitled project",
413
+ "subtitle": _clean_text(_loc(item.get("shortTitle")) or _loc(item.get("acronym"))),
414
+ "type": _loc(item.get("type")) or "Project",
415
+ "type_key": "project",
416
+ "category": "",
417
+ "year": start_y,
418
+ "end_year": end_y,
419
+ "date": start,
420
+ "period": " – ".join(x for x in (start, end) if x),
421
+ "status": status,
422
+ "authors": [p["name"] for p in participants],
423
+ "ku_authors": [p for p in participants if p.get("uuid")],
424
+ "participants": participants[:8],
425
+ "funders": list(dict.fromkeys(funders))[:4],
426
+ "abstract": _descriptions(item),
427
+ "keywords": _keywords(item),
428
+ "venue": _clean_text(managing),
429
+ "doi": "", "doi_url": "", "open_access": False, "file_available": False, "peer_reviewed": None,
430
+ "portal_url": item.get("portalUrl") or (f"{portal_url}/en/projects/{uuid}" if uuid else ""),
431
+ }
432
+
433
+
434
+ def normalize_equipment(item: dict, portal_url: str) -> dict:
435
+ """Equipment / facilities. Contact persons are names only; emails and phones are never copied."""
436
+ uuid = item.get("uuid", "")
437
+ contacts = _people(item, "contactPersons", "contacts", "personAssociations")
438
+ orgs = [_loc((o or {}).get("name")) for o in (item.get("organizations") or item.get("organisationalUnits") or [])]
439
+ managing = _loc((item.get("managingOrganization") or {}).get("name")) or next((o for o in orgs if o), "")
440
+ location = ""
441
+ for loc in item.get("locations") or item.get("addresses") or []:
442
+ location = _loc(loc.get("name") if isinstance(loc, dict) and "name" in loc else loc) or _loc((loc or {}).get("building"))
443
+ if location:
444
+ break
445
+ return {
446
+ "kind": "equipment",
447
+ "uuid": uuid,
448
+ "title": _clean_text(_loc(item.get("title")) or _loc(item.get("name"))) or "Unnamed equipment",
449
+ "subtitle": "",
450
+ "type": _loc(item.get("type")) or "Equipment",
451
+ "type_key": "equipment",
452
+ "category": _loc(item.get("category")),
453
+ "year": None, "date": "", "status": "",
454
+ "authors": [], "ku_authors": [],
455
+ "contacts": contacts[:4],
456
+ "abstract": _descriptions(item),
457
+ "keywords": _keywords(item),
458
+ "venue": _clean_text(managing),
459
+ "location": _clean_text(location),
460
+ "doi": "", "doi_url": "", "open_access": False, "file_available": False, "peer_reviewed": None,
461
+ "portal_url": item.get("portalUrl") or (f"{portal_url}/en/equipments/{uuid}" if uuid else ""),
462
+ }
463
+
464
+
465
+ def normalize_external_organization(item: dict, portal_url: str) -> dict:
466
+ addr = item.get("address") or {}
467
+ if isinstance(addr, list):
468
+ addr = addr[0] if addr else {}
469
+ return {
470
+ "kind": "collaborator",
471
+ "uuid": item.get("uuid", ""),
472
+ "name": _loc(item.get("name")),
473
+ "title": _loc(item.get("name")),
474
+ "type": _loc(item.get("type")),
475
+ "country": _loc(addr.get("country")) or _loc(item.get("country")),
476
+ "portal_url": item.get("portalUrl") or "",
477
+ }
478
+
479
+
480
  # ───────────────────────────── client ───────────────────────────────
481
  class PureClient:
482
  def __init__(
 
603
  for o in p.get("organisations", []):
604
  o["name"] = o.get("name") or names.get(o["uuid"], "")
605
 
606
+ # -- student theses ---------------------------------------------
607
+ async def search_student_theses(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
608
+ data = await self.request("POST", "/student-theses/search", json={"searchString": search_string, "size": size, "offset": offset})
609
+ items = self._filter(data.get("items") or [])
610
+ return int(data.get("count") or 0), [normalize_student_thesis(i, self.portal_url) for i in items]
611
+
612
+ # -- projects / equipment ---------------------------------------
613
+ async def search_projects(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
614
+ data = await self.request("POST", "/projects/search", json={"searchString": search_string, "size": size, "offset": offset})
615
+ items = self._filter(data.get("items") or [])
616
+ return int(data.get("count") or 0), [normalize_project(i, self.portal_url) for i in items]
617
+
618
+ async def search_equipment(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
619
+ data = await self.request("POST", "/equipment/search", json={"searchString": search_string, "size": size, "offset": offset})
620
+ items = self._filter(data.get("items") or [])
621
+ return int(data.get("count") or 0), [normalize_equipment(i, self.portal_url) for i in items]
622
+
623
+ # -- concepts (thesaurus) ---------------------------------------
624
+ async def search_concepts(self, search_string: str, size: int = 5) -> list[str]:
625
+ data = await self.request("POST", "/concepts/search", json={"searchString": search_string, "size": size, "offset": 0})
626
+ names = [_loc(i.get("name") or i.get("term") or i.get("title")) for i in data.get("items") or []]
627
+ return [n for n in names if n]
628
+
629
+ # -- external organisations (collaborators) ---------------------
630
+ async def get_external_organization(self, uuid: str) -> dict | None:
631
+ try:
632
+ item = await self.request("GET", f"/external-organizations/{uuid}")
633
+ except PureError:
634
+ return None
635
+ return normalize_external_organization(item, self.portal_url)
636
+
637
  # -- datasets ---------------------------------------------------
638
  async def search_datasets(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
639
  data = await self.request("POST", "/data-sets/search", json={"searchString": search_string, "size": size, "offset": offset})