Upload 7 files
Browse files- README.md +20 -7
- app.py +40 -8
- pipeline.py +206 -18
- pure_client.py +242 -2
README.md
CHANGED
|
@@ -11,19 +11,25 @@ short_description: Natural-language search over KU's Khazna (Pure) repository
|
|
| 11 |
|
| 12 |
# Khazna Research Assistant
|
| 13 |
|
| 14 |
-
Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, researcher profiles, departments/centres and
|
| 15 |
|
| 16 |
## What each tab / intent searches
|
| 17 |
|
| 18 |
| Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent |
|
| 19 |
|---|---|---|---|
|
| 20 |
| Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ |
|
| 21 |
-
|
|
|
|
|
| 22 |
| Departments | `org_search` | `/organizations/search` | /en/organisations/ |
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
| 24 |
| Everything | planner decides | any of the above | β |
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
| 27 |
|
| 28 |
## Pipeline
|
| 29 |
|
|
@@ -84,7 +90,7 @@ Live page: **https://library.ku.ac.ae/khaznabot**
|
|
| 84 |
3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`.
|
| 85 |
|
| 86 |
What the block adds on top of the LibBee model:
|
| 87 |
-
- **Scope chips** (Everything, Publications, Profiles, Departments,
|
| 88 |
- **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card.
|
| 89 |
- **Keywords sent to Khazna** panel, plus a large "Search Khazna for: β¦" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
|
| 90 |
- **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?".
|
|
@@ -92,8 +98,15 @@ What the block adds on top of the LibBee model:
|
|
| 92 |
|
| 93 |
`/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking.
|
| 94 |
|
|
|
|
|
|
|
| 95 |
**Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design.
|
| 96 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
## Endpoints
|
| 98 |
|
| 99 |
| Method | Path | Purpose |
|
|
@@ -101,12 +114,12 @@ What the block adds on top of the LibBee model:
|
|
| 101 |
| GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) |
|
| 102 |
| GET | `/test` | the Space's own test chat (widget) |
|
| 103 |
| GET | `/widget.js` | embeddable widget |
|
| 104 |
-
| POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k}` β answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / person / organisation / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` |
|
| 105 |
| POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) |
|
| 106 |
| GET | `/config` | public frontend config: announcement, maintenance mode |
|
| 107 |
| GET | `/health` | backend status |
|
| 108 |
| GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) |
|
| 109 |
-
| GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| persons \| organizations \| data-sets (admin) |
|
| 110 |
|
| 111 |
The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source.
|
| 112 |
|
|
|
|
| 11 |
|
| 12 |
# Khazna Research Assistant
|
| 13 |
|
| 14 |
+
Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, student theses, researcher profiles, departments/centres, research projects, equipment and collaborations**, and writes a grounded answer with `[n]` citations that link to result cards.
|
| 15 |
|
| 16 |
## What each tab / intent searches
|
| 17 |
|
| 18 |
| Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent |
|
| 19 |
|---|---|---|---|
|
| 20 |
| Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ |
|
| 21 |
+
| Theses | `thesis_search` | `/student-theses/search` (needs `/student-theses/*` on the key), merged with thesis-type research outputs; falls back to research outputs alone on 403 | /en/studentTheses/ |
|
| 22 |
+
| Profiles | `expert_search` (who works on / supervises X), `author_search` (named person) | `/persons/search` + `/research-outputs/search` (KU authors aggregated) + `/student-theses/search` (supervisors), `GET /persons/{uuid}` | /en/persons/ |
|
| 23 |
| Departments | `org_search` | `/organizations/search` | /en/organisations/ |
|
| 24 |
+
| Projects | `project_search` | `/projects/search` (funding **amounts are never shown**) | /en/projects/ |
|
| 25 |
+
| Equipment | `equipment_search` | `/equipment/search` (contact **names only**, no emails) | /en/equipments/ |
|
| 26 |
+
| Collaborations | `collab_search` | external organisations on co-authors of the topic's top KU publications, names via `GET /external-organizations/{uuid}` | β |
|
| 27 |
+
| Datasets (no tab) | `dataset_search` | `/data-sets/search` if the Pure API offers it (KU's key currently has no such endpoint) β falls back to dataset-type research outputs | /en/datasets/ |
|
| 28 |
| Everything | planner decides | any of the above | β |
|
| 29 |
|
| 30 |
+
**Concept expansion:** for topic, thesis, expert, project and collaboration searches, one Pure thesaurus concept from `/concepts/search` that shares a word with the query is added as an extra search variant. Set `USE_CONCEPTS=false` to switch it off.
|
| 31 |
+
|
| 32 |
+
Expert finding combines three signals: Pure's profile search, the KU authors of the top-ranked publications for the topic, and the supervisors of matching student theses. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card.
|
| 33 |
|
| 34 |
## Pipeline
|
| 35 |
|
|
|
|
| 90 |
3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`.
|
| 91 |
|
| 92 |
What the block adds on top of the LibBee model:
|
| 93 |
+
- **Scope chips** (Everything, Publications, Theses, Profiles, Departments, Projects, Equipment, Collaborations), sent to the Space as `scope`. They swipe on phones.
|
| 94 |
- **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card.
|
| 95 |
- **Keywords sent to Khazna** panel, plus a large "Search Khazna for: β¦" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
|
| 96 |
- **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?".
|
|
|
|
| 98 |
|
| 99 |
`/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking.
|
| 100 |
|
| 101 |
+
**Deep links:** `https://library.ku.ac.ae/khaznabot?ask=<question>&scope=<all|publications|theses|persons|organisations|projects|equipment|collaborations|datasets>` asks the question on load, then removes it from the address bar. `libguides/khazna-portal-searchbox.html` is a small search box for the Khazna portal (plain HTML with inline styles, no JS) that uses this link.
|
| 102 |
+
|
| 103 |
**Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design.
|
| 104 |
|
| 105 |
+
## Pure API key: endpoints the bot uses
|
| 106 |
+
|
| 107 |
+
Used: `/research-outputs/*`, `/student-theses/*`, `/persons/*`, `/organizations/*`, `/external-organizations/*`, `/projects/*`, `/equipment/*`, `/concepts/*`.
|
| 108 |
+
Not used, so they can be removed from the key: `/users/*`, `/roles/*`, `/changes/*`, `/applications/*`, `/awards/*`, `/journals/*`, `/publishers/*`, `/classification-schemes/*`, `/thesauri/*`, `/author-collaborations/*`, `/external-persons/*`, `/courses/*`, `/activities/*`, `/events/*`, `/pressmedia/*`, `/prizes/*`.
|
| 109 |
+
|
| 110 |
## Endpoints
|
| 111 |
|
| 112 |
| Method | Path | Purpose |
|
|
|
|
| 114 |
| GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) |
|
| 115 |
| GET | `/test` | the Space's own test chat (widget) |
|
| 116 |
| GET | `/widget.js` | embeddable widget |
|
| 117 |
+
| POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k}` β answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / thesis / person / organisation / project / equipment / collaborator / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` |
|
| 118 |
| POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) |
|
| 119 |
| GET | `/config` | public frontend config: announcement, maintenance mode |
|
| 120 |
| GET | `/health` | backend status |
|
| 121 |
| GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) |
|
| 122 |
+
| GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| student-theses \| persons \| organizations \| projects \| equipment \| external-organizations \| data-sets (admin) |
|
| 123 |
|
| 124 |
The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source.
|
| 125 |
|
app.py
CHANGED
|
@@ -49,9 +49,12 @@ FRONTEND_URL = os.environ.get("FRONTEND_URL", "https://library.ku.ac.ae/khaznabo
|
|
| 49 |
RATE_LIMIT = int(os.environ.get("RATE_LIMIT_PER_MIN", "12"))
|
| 50 |
DB_PATH = "/data/khazna_analytics.db" if os.path.isdir("/data") else "khazna_analytics.db"
|
| 51 |
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 52 |
-
SCOPES = ("all", "publications", "persons", "organisations", "datasets")
|
| 53 |
-
SCOPE_PORTAL_PATH = {"publications": "publications", "
|
| 54 |
-
|
|
|
|
|
|
|
|
|
|
| 55 |
"expert_search": "persons", "org_search": "organisations", "dataset_search": "datasets"}
|
| 56 |
|
| 57 |
pure: PureClient | None = None
|
|
@@ -161,15 +164,28 @@ def _require_admin(pw: str | None):
|
|
| 161 |
def _public_result(o: dict, n: int) -> dict:
|
| 162 |
kind = o.get("kind", "publication")
|
| 163 |
if kind == "person":
|
| 164 |
-
keep = ("kind", "uuid", "name", "job_titles", "organisations", "keywords", "orcid", "portal_url", "evidence", "matching_outputs"
|
|
|
|
| 165 |
elif kind == "organisation":
|
| 166 |
keep = ("kind", "uuid", "name", "type", "parents", "portal_url")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
else:
|
| 168 |
keep = ("kind", "uuid", "title", "subtitle", "type", "year", "date", "authors", "ku_authors", "venue",
|
| 169 |
"doi", "doi_url", "open_access", "file_available", "portal_url", "keywords")
|
| 170 |
out = {k: o.get(k) for k in keep}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
out["n"] = n
|
| 172 |
-
if kind in ("publication", "dataset"):
|
| 173 |
out["abstract"] = (o.get("abstract") or "")[:400]
|
| 174 |
return out
|
| 175 |
|
|
@@ -199,9 +215,23 @@ def _followups(plan, retrieval: dict | None) -> list[dict]:
|
|
| 199 |
return out
|
| 200 |
if plan.intent in ("topic_search", "count"):
|
| 201 |
out += [{"label": "Who at KU works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
|
| 202 |
-
{"label": "
|
|
|
|
|
|
|
| 203 |
if not plan.year_from:
|
| 204 |
out.append({"label": f"Only since {year - 3}", "question": f"KU publications on {kw} since {year - 3}", "scope": "publications"})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
elif plan.intent == "expert_search":
|
| 206 |
out += [{"label": "Show the publications", "question": f"KU publications on {kw}", "scope": "publications"},
|
| 207 |
{"label": "Which departments or centres?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"}]
|
|
@@ -211,7 +241,7 @@ def _followups(plan, retrieval: dict | None) -> list[dict]:
|
|
| 211 |
elif plan.intent == "dataset_search":
|
| 212 |
out += [{"label": "Publications on this", "question": f"KU publications on {kw}", "scope": "publications"},
|
| 213 |
{"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
|
| 214 |
-
return out[:
|
| 215 |
|
| 216 |
|
| 217 |
# βββββββββββββββββββββββββββββ routes βββββββββββββββββββββββββββββββ
|
|
@@ -376,7 +406,9 @@ def admin_stats(x_admin_password: str | None = Header(default=None)):
|
|
| 376 |
|
| 377 |
|
| 378 |
DEBUG_ENDPOINTS = {"research-outputs": "normalize_research_output", "persons": "normalize_person",
|
| 379 |
-
"organizations": "normalize_organization", "data-sets": "normalize_dataset"
|
|
|
|
|
|
|
| 380 |
|
| 381 |
|
| 382 |
@app.get("/debug/pure")
|
|
|
|
| 49 |
RATE_LIMIT = int(os.environ.get("RATE_LIMIT_PER_MIN", "12"))
|
| 50 |
DB_PATH = "/data/khazna_analytics.db" if os.path.isdir("/data") else "khazna_analytics.db"
|
| 51 |
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 52 |
+
SCOPES = ("all", "publications", "theses", "persons", "organisations", "projects", "equipment", "collaborations", "datasets")
|
| 53 |
+
SCOPE_PORTAL_PATH = {"publications": "publications", "theses": "studentTheses", "persons": "persons",
|
| 54 |
+
"organisations": "organisations", "projects": "projects", "equipment": "equipments",
|
| 55 |
+
"collaborations": "publications", "datasets": "datasets"}
|
| 56 |
+
INTENT_SCOPE = {"thesis_search": "theses", "project_search": "projects", "equipment_search": "equipment",
|
| 57 |
+
"collab_search": "collaborations", "topic_search": "publications", "author_search": "publications", "count": "publications",
|
| 58 |
"expert_search": "persons", "org_search": "organisations", "dataset_search": "datasets"}
|
| 59 |
|
| 60 |
pure: PureClient | None = None
|
|
|
|
| 164 |
def _public_result(o: dict, n: int) -> dict:
|
| 165 |
kind = o.get("kind", "publication")
|
| 166 |
if kind == "person":
|
| 167 |
+
keep = ("kind", "uuid", "name", "job_titles", "organisations", "keywords", "orcid", "portal_url", "evidence", "matching_outputs",
|
| 168 |
+
"supervised", "supervised_theses")
|
| 169 |
elif kind == "organisation":
|
| 170 |
keep = ("kind", "uuid", "name", "type", "parents", "portal_url")
|
| 171 |
+
elif kind == "collaborator":
|
| 172 |
+
keep = ("kind", "uuid", "name", "type", "country", "joint_outputs", "examples", "portal_url")
|
| 173 |
+
elif kind == "project":
|
| 174 |
+
keep = ("kind", "uuid", "title", "subtitle", "type", "year", "period", "status", "venue", "funders", "keywords", "portal_url")
|
| 175 |
+
elif kind == "equipment":
|
| 176 |
+
keep = ("kind", "uuid", "title", "type", "category", "venue", "location", "keywords", "portal_url")
|
| 177 |
else:
|
| 178 |
keep = ("kind", "uuid", "title", "subtitle", "type", "year", "date", "authors", "ku_authors", "venue",
|
| 179 |
"doi", "doi_url", "open_access", "file_available", "portal_url", "keywords")
|
| 180 |
out = {k: o.get(k) for k in keep}
|
| 181 |
+
if kind == "thesis":
|
| 182 |
+
out["supervisors"] = [{"name": x.get("name"), "uuid": x.get("uuid")} for x in (o.get("supervisors") or [])]
|
| 183 |
+
if kind == "project":
|
| 184 |
+
out["participants"] = [{"name": x.get("name"), "role": x.get("role")} for x in (o.get("participants") or [])]
|
| 185 |
+
if kind == "equipment":
|
| 186 |
+
out["contacts"] = [{"name": x.get("name")} for x in (o.get("contacts") or [])] # names only
|
| 187 |
out["n"] = n
|
| 188 |
+
if kind in ("publication", "dataset", "thesis", "project", "equipment"):
|
| 189 |
out["abstract"] = (o.get("abstract") or "")[:400]
|
| 190 |
return out
|
| 191 |
|
|
|
|
| 215 |
return out
|
| 216 |
if plan.intent in ("topic_search", "count"):
|
| 217 |
out += [{"label": "Who at KU works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
|
| 218 |
+
{"label": "KU theses on this", "question": f"KU theses on {kw}", "scope": "theses"},
|
| 219 |
+
{"label": "Related projects", "question": f"KU research projects on {kw}", "scope": "projects"},
|
| 220 |
+
{"label": "Who does KU collaborate with?", "question": f"Which organisations does KU collaborate with on {kw}?", "scope": "collaborations"}]
|
| 221 |
if not plan.year_from:
|
| 222 |
out.append({"label": f"Only since {year - 3}", "question": f"KU publications on {kw} since {year - 3}", "scope": "publications"})
|
| 223 |
+
elif plan.intent == "project_search":
|
| 224 |
+
out += [{"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"},
|
| 225 |
+
{"label": "Publications from this area", "question": f"KU publications on {kw}", "scope": "publications"}]
|
| 226 |
+
elif plan.intent == "equipment_search":
|
| 227 |
+
out += [{"label": "Which departments use this?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"},
|
| 228 |
+
{"label": "Experts in this area", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
|
| 229 |
+
elif plan.intent == "collab_search":
|
| 230 |
+
out += [{"label": "The joint publications", "question": f"KU publications on {kw}", "scope": "publications"},
|
| 231 |
+
{"label": "KU experts in this area", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
|
| 232 |
+
elif plan.intent == "thesis_search":
|
| 233 |
+
out += [{"label": "Who supervises in this area?", "question": f"Who at KU supervises theses on {kw}?", "scope": "persons"},
|
| 234 |
+
{"label": "Related publications", "question": f"KU publications on {kw}", "scope": "publications"}]
|
| 235 |
elif plan.intent == "expert_search":
|
| 236 |
out += [{"label": "Show the publications", "question": f"KU publications on {kw}", "scope": "publications"},
|
| 237 |
{"label": "Which departments or centres?", "question": f"Which KU departments or centres work on {kw}?", "scope": "organisations"}]
|
|
|
|
| 241 |
elif plan.intent == "dataset_search":
|
| 242 |
out += [{"label": "Publications on this", "question": f"KU publications on {kw}", "scope": "publications"},
|
| 243 |
{"label": "Who works on this?", "question": f"Who at KU works on {kw}?", "scope": "persons"}]
|
| 244 |
+
return out[:4]
|
| 245 |
|
| 246 |
|
| 247 |
# βββββββββββββββββββββββββββββ routes βββββββββββββββββββββββββββββββ
|
|
|
|
| 406 |
|
| 407 |
|
| 408 |
DEBUG_ENDPOINTS = {"research-outputs": "normalize_research_output", "persons": "normalize_person",
|
| 409 |
+
"organizations": "normalize_organization", "data-sets": "normalize_dataset",
|
| 410 |
+
"student-theses": "normalize_student_thesis", "projects": "normalize_project",
|
| 411 |
+
"equipment": "normalize_equipment", "external-organizations": "normalize_external_organization"}
|
| 412 |
|
| 413 |
|
| 414 |
@app.get("/debug/pure")
|
pipeline.py
CHANGED
|
@@ -45,10 +45,15 @@ OutputType = Literal["article", "conference", "book", "chapter", "thesis", "data
|
|
| 45 |
|
| 46 |
class QueryPlan(BaseModel):
|
| 47 |
intent: Literal["topic_search", "author_search", "expert_search", "org_search", "dataset_search",
|
|
|
|
| 48 |
"count", "repository_help", "social", "out_of_scope"] = Field(
|
| 49 |
description="topic_search: find KU publications on a subject; author_search: profile and/or outputs of a NAMED "
|
| 50 |
"KU researcher; expert_search: WHO at KU works on a topic (find experts/supervisors/collaborators); "
|
| 51 |
"org_search: KU colleges, departments, research centres/labs; dataset_search: research datasets; "
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
"count: how many outputs match; repository_help: questions ABOUT Khazna (deposit, theses submission, "
|
| 53 |
"what is Khazna); social: greetings/thanks; out_of_scope: anything else."
|
| 54 |
)
|
|
@@ -76,7 +81,10 @@ Rules:
|
|
| 76 |
- Keyword queries are ALWAYS in English (Khazna metadata is English), even if the user writes in Arabic.
|
| 77 |
- Strip filler ("can you find", "papers about", "KU", "recent", "research on"). Keep domain terms and named entities.
|
| 78 |
- Years: "recent"/"latest" => sort=newest, and year_from = {year}-3. "since 2022" => year_from=2022. "in 2024" => both 2024.
|
| 79 |
-
- Theses/dissertations =>
|
|
|
|
|
|
|
|
|
|
| 80 |
- Follow-ups ("only 2024 ones", "more by her", "what about desalination?") must be resolved using the conversation history.
|
| 81 |
- For author_search put the name in author_name; put any topic words in keyword_queries (may be empty).
|
| 82 |
- Today's year is {year}.
|
|
@@ -84,7 +92,11 @@ Rules:
|
|
| 84 |
Examples:
|
| 85 |
"Find KU papers on machine learning for predicting corrosion in pipelines"
|
| 86 |
=> topic_search, keyword_queries=["machine learning corrosion pipelines","corrosion prediction neural network"], broad_query="corrosion"
|
| 87 |
-
"Latest theses on solar desalination" =>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
"What has Dr. Jane Smith published on graphene?" => author_search, author_name="Jane Smith", keyword_queries=["graphene"]
|
| 89 |
"How many KU publications on COVID-19?" => count, ["COVID-19","SARS-CoV-2"]
|
| 90 |
"How do I deposit my thesis in Khazna?" => repository_help
|
|
@@ -101,7 +113,11 @@ _STOP = set("""a an the of in on for to and or with by from about into at as is
|
|
| 101 |
what which who whom how why when where can could would should will shall may might do does did have has had any some
|
| 102 |
me my i we our you your find show give get list search look papers paper articles article research publications publication
|
| 103 |
studies study work works ku khalifa university recent latest new please related regarding
|
| 104 |
-
published publish since before after between during year years ku's theses thesis dissertation dissertations
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
|
| 107 |
def _heuristic_plan(question: str, year: int) -> QueryPlan:
|
|
@@ -112,7 +128,17 @@ def _heuristic_plan(question: str, year: int) -> QueryPlan:
|
|
| 112 |
yrs = [int(y) for y in re.findall(r"\b(19[89]\d|20[0-4]\d)\b", question)]
|
| 113 |
recent = bool(re.search(r"\b(recent|latest|new)\b", question, re.I))
|
| 114 |
intent = "topic_search"
|
| 115 |
-
if re.search(r"\b(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
intent = "dataset_search"
|
| 117 |
elif re.search(r"\b(who (works|researches|is working)|experts?|supervisors?|researchers? (in|on|working))\b", question, re.I):
|
| 118 |
intent = "expert_search"
|
|
@@ -127,18 +153,23 @@ def _heuristic_plan(question: str, year: int) -> QueryPlan:
|
|
| 127 |
broad_query=" ".join(words[:2]),
|
| 128 |
year_from=min(yrs) if yrs else (year - 3 if recent else None),
|
| 129 |
year_to=max(yrs) if len(yrs) > 1 else None,
|
| 130 |
-
output_types=[
|
| 131 |
sort="newest" if recent else "relevance",
|
| 132 |
)
|
| 133 |
|
| 134 |
|
| 135 |
SCOPE_INTENTS = {
|
| 136 |
-
"publications": {"topic_search", "author_search", "count"},
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
"persons": {"expert_search", "author_search"},
|
| 138 |
"organisations": {"org_search"},
|
| 139 |
"datasets": {"dataset_search"},
|
| 140 |
}
|
| 141 |
-
_SCOPE_DEFAULT = {"publications": "topic_search", "persons": "expert_search",
|
|
|
|
| 142 |
"organisations": "org_search", "datasets": "dataset_search"}
|
| 143 |
|
| 144 |
|
|
@@ -153,6 +184,8 @@ def apply_scope(plan: QueryPlan, scope: str) -> QueryPlan:
|
|
| 153 |
|
| 154 |
async def plan_query(question: str, history: list[dict], model: str, year: int, scope: str = "all") -> QueryPlan:
|
| 155 |
plan = await _plan_query(question, history, model, year)
|
|
|
|
|
|
|
| 156 |
return apply_scope(plan, scope)
|
| 157 |
|
| 158 |
|
|
@@ -171,7 +204,8 @@ async def _plan_query(question: str, history: list[dict], model: str, year: int)
|
|
| 171 |
return _heuristic_plan(question, year)
|
| 172 |
# Guardrails on the LLM's output
|
| 173 |
plan.keyword_queries = [q.strip() for q in plan.keyword_queries if q and q.strip()][:3]
|
| 174 |
-
if plan.intent in ("topic_search", "count", "expert_search", "org_search", "dataset_search"
|
|
|
|
| 175 |
plan.keyword_queries = _heuristic_plan(question, year).keyword_queries
|
| 176 |
return plan
|
| 177 |
|
|
@@ -317,11 +351,13 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
|
|
| 317 |
q0 = plan.keyword_queries[0] if plan.keyword_queries else plan.broad_query
|
| 318 |
pubs_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "year_from": plan.year_from}))
|
| 319 |
people_task = pure.search_persons(q0, size=10)
|
| 320 |
-
|
|
|
|
| 321 |
if isinstance(pubs, Exception):
|
| 322 |
raise pubs
|
| 323 |
if isinstance(people, Exception):
|
| 324 |
people = []
|
|
|
|
| 325 |
|
| 326 |
score: Counter = Counter()
|
| 327 |
evidence: dict[str, list[dict]] = {}
|
|
@@ -333,6 +369,13 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
|
|
| 333 |
score[a["uuid"]] += 1.0 / (3 + rank_i)
|
| 334 |
evidence.setdefault(a["uuid"], []).append(o)
|
| 335 |
names[a["uuid"]] = a["name"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 336 |
profile_by_uuid = {p["uuid"]: p for p in people}
|
| 337 |
for r, p in enumerate(people):
|
| 338 |
score[p["uuid"]] += 0.25 / (1 + r)
|
|
@@ -348,11 +391,14 @@ async def _retrieve_experts(pure: PureClient, plan: QueryPlan, max_people: int =
|
|
| 348 |
p = profile_by_uuid.get(u)
|
| 349 |
if p is None: # non-public profile β skip rather than expose
|
| 350 |
continue
|
| 351 |
-
ev = evidence.get(u, [])
|
| 352 |
results.append({**p, "evidence": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ev[:3]],
|
| 353 |
-
"matching_outputs": len(ev),
|
|
|
|
|
|
|
| 354 |
await pure.fill_org_names(results)
|
| 355 |
-
trace = pubs["trace"] + [{"endpoint": "persons/search", "query": q0, "hits": len(people)}
|
|
|
|
| 356 |
return {"results": results, "unfiltered": len(results), "total": len(score), "person": None, "trace": trace}
|
| 357 |
|
| 358 |
|
|
@@ -371,25 +417,143 @@ async def _retrieve_orgs(pure: PureClient, plan: QueryPlan) -> dict:
|
|
| 371 |
return {"results": results, "unfiltered": len(results), "total": len(results), "person": None, "trace": trace}
|
| 372 |
|
| 373 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 374 |
async def retrieve(pure: PureClient, plan: QueryPlan) -> dict:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 375 |
if plan.intent == "expert_search":
|
| 376 |
return await _retrieve_experts(pure, plan)
|
| 377 |
if plan.intent == "org_search":
|
| 378 |
return await _retrieve_orgs(pure, plan)
|
| 379 |
if plan.intent == "dataset_search":
|
| 380 |
-
return await
|
| 381 |
return await _retrieve_outputs(pure, plan)
|
| 382 |
|
| 383 |
|
| 384 |
# βββββββββββββββββββββββββββ synthesis ββββββββββββββββββββββββββββββ
|
| 385 |
SYNTH_SYSTEM = """You are the Khazna Research Assistant for Khalifa University (Abu Dhabi, UAE).
|
| 386 |
You answer questions about KU research using ONLY the Khazna (Elsevier Pure) records provided below.
|
| 387 |
-
Records can be publications, datasets,
|
|
|
|
| 388 |
|
| 389 |
Rules:
|
| 390 |
- Ground every claim in the records. Cite records inline as [n] using their numbers. Never invent titles, authors, years, findings or DOIs.
|
| 391 |
- Open with a one-sentence direct answer (e.g. how many relevant outputs and the main themes).
|
| 392 |
-
- For researcher records: say who seems most relevant and why, based on their
|
|
|
|
|
|
|
|
|
|
|
|
|
| 393 |
Do not rank people by seniority or make claims about availability; suggest contacting them via their Khazna profile.
|
| 394 |
- Then group the most relevant records by theme or approach in 2-4 short paragraphs or "- " bullets, naming key KU authors and years.
|
| 395 |
- If some records are only loosely related, say so and do not stretch them. If none really answer the question, say that plainly
|
|
@@ -410,20 +574,44 @@ def _format_records(outs: list[dict]) -> str:
|
|
| 410 |
f"[{i}] RESEARCHER: {o['name']}\n"
|
| 411 |
f" Role: {', '.join(o.get('job_titles') or []) or '-'} | Unit: {', '.join(x['name'] for x in o.get('organisations', []) if x.get('name')) or '-'}\n"
|
| 412 |
f" Expertise keywords: {', '.join(o.get('keywords', [])[:8]) or '-'}\n"
|
| 413 |
-
f" Matching KU outputs on this topic: {o.get('matching_outputs', 0)}. Examples: {ev or '-'}"
|
|
|
|
|
|
|
| 414 |
)
|
| 415 |
continue
|
| 416 |
if kind == "organisation":
|
| 417 |
lines.append(f"[{i}] ORGANISATIONAL UNIT: {o['name']} | type: {o.get('type') or '-'} | part of: {', '.join(o.get('parents', [])) or '-'}")
|
| 418 |
continue
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 419 |
authors = ", ".join(o["authors"][:6]) + (" et al." if len(o["authors"]) > 6 else "")
|
| 420 |
ku = ", ".join(a["name"] for a in o["ku_authors"][:6])
|
| 421 |
lines.append(
|
| 422 |
-
f"[{i}] {'DATASET: ' if kind == 'dataset' else ''}{o['title']}{(': ' + o['subtitle']) if o['subtitle'] else ''}\n"
|
| 423 |
f" {o['type'] or 'Output'} | {o['year'] or 'n.d.'} | {o['venue'] or '-'}\n"
|
| 424 |
f" Authors: {authors or '-'}\n"
|
| 425 |
f" KU authors: {ku or '-'}\n"
|
| 426 |
-
f"
|
|
|
|
| 427 |
f" Abstract: {o['abstract'][:700] or '(no abstract)'}"
|
| 428 |
)
|
| 429 |
return "\n\n".join(lines)
|
|
|
|
| 45 |
|
| 46 |
class QueryPlan(BaseModel):
|
| 47 |
intent: Literal["topic_search", "author_search", "expert_search", "org_search", "dataset_search",
|
| 48 |
+
"thesis_search", "project_search", "equipment_search", "collab_search",
|
| 49 |
"count", "repository_help", "social", "out_of_scope"] = Field(
|
| 50 |
description="topic_search: find KU publications on a subject; author_search: profile and/or outputs of a NAMED "
|
| 51 |
"KU researcher; expert_search: WHO at KU works on a topic (find experts/supervisors/collaborators); "
|
| 52 |
"org_search: KU colleges, departments, research centres/labs; dataset_search: research datasets; "
|
| 53 |
+
"thesis_search: KU student theses and dissertations (PhD/Master's), including who supervised them; "
|
| 54 |
+
"project_search: KU research projects and grants (funded projects, who leads them, funders); "
|
| 55 |
+
"equipment_search: KU research equipment, instruments and facilities; "
|
| 56 |
+
"collab_search: which external universities/organisations KU collaborates with on a topic; "
|
| 57 |
"count: how many outputs match; repository_help: questions ABOUT Khazna (deposit, theses submission, "
|
| 58 |
"what is Khazna); social: greetings/thanks; out_of_scope: anything else."
|
| 59 |
)
|
|
|
|
| 81 |
- Keyword queries are ALWAYS in English (Khazna metadata is English), even if the user writes in Arabic.
|
| 82 |
- Strip filler ("can you find", "papers about", "KU", "recent", "research on"). Keep domain terms and named entities.
|
| 83 |
- Years: "recent"/"latest" => sort=newest, and year_from = {year}-3. "since 2022" => year_from=2022. "in 2024" => both 2024.
|
| 84 |
+
- Theses/dissertations (PhD, doctoral, Master's, student theses) => intent thesis_search. "conference papers" => output_types ["conference"]. "journal articles" => ["article"].
|
| 85 |
+
- "Find a supervisor for X" or "who supervises theses on X" => expert_search.
|
| 86 |
+
- Research projects, grants, funded research => project_search. Instruments, equipment, facilities, labs with a device => equipment_search.
|
| 87 |
+
- "Who does KU collaborate with", partner universities/institutions, international collaboration => collab_search.
|
| 88 |
- Follow-ups ("only 2024 ones", "more by her", "what about desalination?") must be resolved using the conversation history.
|
| 89 |
- For author_search put the name in author_name; put any topic words in keyword_queries (may be empty).
|
| 90 |
- Today's year is {year}.
|
|
|
|
| 92 |
Examples:
|
| 93 |
"Find KU papers on machine learning for predicting corrosion in pipelines"
|
| 94 |
=> topic_search, keyword_queries=["machine learning corrosion pipelines","corrosion prediction neural network"], broad_query="corrosion"
|
| 95 |
+
"Latest theses on solar desalination" => thesis_search, ["solar desalination","solar-driven water desalination"], broad_query="desalination", sort=newest, year_from={y3}
|
| 96 |
+
"PhD dissertations supervised by Jane Smith" => thesis_search, keyword_queries=["Jane Smith"]
|
| 97 |
+
"What funded projects does KU have on water security?" => project_search, ["water security","water resources"], broad_query="water"
|
| 98 |
+
"Which KU labs have an electron microscope?" => equipment_search, ["electron microscope","SEM TEM microscopy"], broad_query="microscope"
|
| 99 |
+
"Which universities does KU collaborate with on 5G?" => collab_search, ["5G","5G wireless networks"], broad_query="wireless"
|
| 100 |
"What has Dr. Jane Smith published on graphene?" => author_search, author_name="Jane Smith", keyword_queries=["graphene"]
|
| 101 |
"How many KU publications on COVID-19?" => count, ["COVID-19","SARS-CoV-2"]
|
| 102 |
"How do I deposit my thesis in Khazna?" => repository_help
|
|
|
|
| 113 |
what which who whom how why when where can could would should will shall may might do does did have has had any some
|
| 114 |
me my i we our you your find show give get list search look papers paper articles article research publications publication
|
| 115 |
studies study work works ku khalifa university recent latest new please related regarding
|
| 116 |
+
published publish since before after between during year years ku's theses thesis dissertation dissertations
|
| 117 |
+
supervise supervises supervised supervisor supervisors supervision advisor advisors expert experts researcher researchers
|
| 118 |
+
people person phd doctoral master masters master's project projects grant grants funded funding
|
| 119 |
+
equipment instrument instruments facility facilities labs lab have has collaborate collaborates collaboration
|
| 120 |
+
collaborations collaborating partner partners universities university institutions does""".split())
|
| 121 |
|
| 122 |
|
| 123 |
def _heuristic_plan(question: str, year: int) -> QueryPlan:
|
|
|
|
| 128 |
yrs = [int(y) for y in re.findall(r"\b(19[89]\d|20[0-4]\d)\b", question)]
|
| 129 |
recent = bool(re.search(r"\b(recent|latest|new)\b", question, re.I))
|
| 130 |
intent = "topic_search"
|
| 131 |
+
if re.search(r"\b(collaborat\w*|partner (universities|institutions)|co-?author(ed|s)? with)\b", question, re.I):
|
| 132 |
+
intent = "collab_search"
|
| 133 |
+
elif re.search(r"\b(projects?|grants?|funded|funding)\b", question, re.I):
|
| 134 |
+
intent = "project_search"
|
| 135 |
+
elif re.search(r"\b(equipment|instruments?|facilit(y|ies)|microscopes?|spectromet\w*|machines?)\b", question, re.I):
|
| 136 |
+
intent = "equipment_search"
|
| 137 |
+
elif re.search(r"\b(supervis\w*|advisors?)\b", question, re.I) and not re.search(r"\bsupervised by\b", question, re.I):
|
| 138 |
+
intent = "expert_search"
|
| 139 |
+
elif re.search(r"\b(thesis|theses|dissertations?)\b", question, re.I) and not re.search(r"\b(deposit|submit|upload)\b", question, re.I):
|
| 140 |
+
intent = "thesis_search"
|
| 141 |
+
elif re.search(r"\b(dataset|datasets|data set)\b", question, re.I):
|
| 142 |
intent = "dataset_search"
|
| 143 |
elif re.search(r"\b(who (works|researches|is working)|experts?|supervisors?|researchers? (in|on|working))\b", question, re.I):
|
| 144 |
intent = "expert_search"
|
|
|
|
| 153 |
broad_query=" ".join(words[:2]),
|
| 154 |
year_from=min(yrs) if yrs else (year - 3 if recent else None),
|
| 155 |
year_to=max(yrs) if len(yrs) > 1 else None,
|
| 156 |
+
output_types=[],
|
| 157 |
sort="newest" if recent else "relevance",
|
| 158 |
)
|
| 159 |
|
| 160 |
|
| 161 |
SCOPE_INTENTS = {
|
| 162 |
+
"publications": {"topic_search", "author_search", "count", "thesis_search"},
|
| 163 |
+
"theses": {"thesis_search"},
|
| 164 |
+
"projects": {"project_search"},
|
| 165 |
+
"equipment": {"equipment_search"},
|
| 166 |
+
"collaborations": {"collab_search"},
|
| 167 |
"persons": {"expert_search", "author_search"},
|
| 168 |
"organisations": {"org_search"},
|
| 169 |
"datasets": {"dataset_search"},
|
| 170 |
}
|
| 171 |
+
_SCOPE_DEFAULT = {"publications": "topic_search", "theses": "thesis_search", "persons": "expert_search",
|
| 172 |
+
"projects": "project_search", "equipment": "equipment_search", "collaborations": "collab_search",
|
| 173 |
"organisations": "org_search", "datasets": "dataset_search"}
|
| 174 |
|
| 175 |
|
|
|
|
| 184 |
|
| 185 |
async def plan_query(question: str, history: list[dict], model: str, year: int, scope: str = "all") -> QueryPlan:
|
| 186 |
plan = await _plan_query(question, history, model, year)
|
| 187 |
+
if plan.intent == "topic_search" and plan.output_types == ["thesis"]: # older planner habit
|
| 188 |
+
plan.intent, plan.output_types = "thesis_search", []
|
| 189 |
return apply_scope(plan, scope)
|
| 190 |
|
| 191 |
|
|
|
|
| 204 |
return _heuristic_plan(question, year)
|
| 205 |
# Guardrails on the LLM's output
|
| 206 |
plan.keyword_queries = [q.strip() for q in plan.keyword_queries if q and q.strip()][:3]
|
| 207 |
+
if plan.intent in ("topic_search", "count", "expert_search", "org_search", "dataset_search", "thesis_search",
|
| 208 |
+
"project_search", "equipment_search", "collab_search") and not plan.keyword_queries:
|
| 209 |
plan.keyword_queries = _heuristic_plan(question, year).keyword_queries
|
| 210 |
return plan
|
| 211 |
|
|
|
|
| 351 |
q0 = plan.keyword_queries[0] if plan.keyword_queries else plan.broad_query
|
| 352 |
pubs_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "year_from": plan.year_from}))
|
| 353 |
people_task = pure.search_persons(q0, size=10)
|
| 354 |
+
theses_task = pure.search_student_theses(q0, size=25)
|
| 355 |
+
pubs, people, theses = await asyncio.gather(pubs_task, people_task, theses_task, return_exceptions=True)
|
| 356 |
if isinstance(pubs, Exception):
|
| 357 |
raise pubs
|
| 358 |
if isinstance(people, Exception):
|
| 359 |
people = []
|
| 360 |
+
theses = [] if isinstance(theses, Exception) else apply_filters(rank([theses[1]], plan), plan.model_copy(update={"output_types": []}))
|
| 361 |
|
| 362 |
score: Counter = Counter()
|
| 363 |
evidence: dict[str, list[dict]] = {}
|
|
|
|
| 369 |
score[a["uuid"]] += 1.0 / (3 + rank_i)
|
| 370 |
evidence.setdefault(a["uuid"], []).append(o)
|
| 371 |
names[a["uuid"]] = a["name"]
|
| 372 |
+
supervised: dict[str, list[dict]] = {}
|
| 373 |
+
for rank_i, t in enumerate(theses[:30]): # supervising theses on the topic is strong evidence for "find a supervisor"
|
| 374 |
+
for sup in t.get("supervisors", []):
|
| 375 |
+
if not sup.get("uuid"):
|
| 376 |
+
continue
|
| 377 |
+
score[sup["uuid"]] += 0.8 / (3 + rank_i)
|
| 378 |
+
supervised.setdefault(sup["uuid"], []).append(t)
|
| 379 |
profile_by_uuid = {p["uuid"]: p for p in people}
|
| 380 |
for r, p in enumerate(people):
|
| 381 |
score[p["uuid"]] += 0.25 / (1 + r)
|
|
|
|
| 391 |
p = profile_by_uuid.get(u)
|
| 392 |
if p is None: # non-public profile β skip rather than expose
|
| 393 |
continue
|
| 394 |
+
ev, sv = evidence.get(u, []), supervised.get(u, [])
|
| 395 |
results.append({**p, "evidence": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ev[:3]],
|
| 396 |
+
"matching_outputs": len(ev),
|
| 397 |
+
"supervised": [{"title": t["title"], "year": t["year"], "portal_url": t["portal_url"]} for t in sv[:3]],
|
| 398 |
+
"supervised_theses": len(sv), "_score": round(score[u], 4)})
|
| 399 |
await pure.fill_org_names(results)
|
| 400 |
+
trace = pubs["trace"] + [{"endpoint": "persons/search", "query": q0, "hits": len(people)},
|
| 401 |
+
{"endpoint": "student-theses/search", "query": q0, "hits": len(theses)}]
|
| 402 |
return {"results": results, "unfiltered": len(results), "total": len(score), "person": None, "trace": trace}
|
| 403 |
|
| 404 |
|
|
|
|
| 417 |
return {"results": results, "unfiltered": len(results), "total": len(results), "person": None, "trace": trace}
|
| 418 |
|
| 419 |
|
| 420 |
+
def _title_key(o: dict) -> str:
|
| 421 |
+
return re.sub(r"[^a-z0-9]", "", o["title"].lower())[:80]
|
| 422 |
+
|
| 423 |
+
|
| 424 |
+
async def _retrieve_theses(pure: PureClient, plan: QueryPlan) -> dict:
|
| 425 |
+
"""Student theses live in their own Pure content type (/student-theses), but some
|
| 426 |
+
theses are also registered as research outputs. Search both and merge, student
|
| 427 |
+
theses first; fall back to research outputs alone if the key lacks /student-theses/*."""
|
| 428 |
+
plan = plan.model_copy(update={"output_types": []})
|
| 429 |
+
st_task = _retrieve_outputs(pure, plan, search=pure.search_student_theses)
|
| 430 |
+
ro_task = _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "output_types": ["thesis"]}))
|
| 431 |
+
st, ro = await asyncio.gather(st_task, ro_task, return_exceptions=True)
|
| 432 |
+
if isinstance(st, Exception) and isinstance(ro, Exception):
|
| 433 |
+
raise st
|
| 434 |
+
if isinstance(st, Exception):
|
| 435 |
+
ro["trace"].insert(0, {"endpoint": "student-theses/search", "error": "unavailable"})
|
| 436 |
+
return ro
|
| 437 |
+
if isinstance(ro, Exception):
|
| 438 |
+
return st
|
| 439 |
+
seen = {_title_key(o) for o in st["results"]}
|
| 440 |
+
merged = st["results"] + [o for o in ro["results"] if _title_key(o) not in seen]
|
| 441 |
+
if plan.sort == "newest":
|
| 442 |
+
merged.sort(key=lambda o: (o.get("year") or 0, o.get("date") or ""), reverse=True)
|
| 443 |
+
return {"results": merged, "unfiltered": st["unfiltered"] + ro["unfiltered"], "total": st["total"] + ro["total"],
|
| 444 |
+
"person": None, "trace": st["trace"] + ro["trace"]}
|
| 445 |
+
|
| 446 |
+
|
| 447 |
+
USE_CONCEPTS = os.environ.get("USE_CONCEPTS", "true").lower() != "false"
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
async def _expand_with_concepts(pure: PureClient, plan: QueryPlan) -> dict | None:
|
| 451 |
+
"""Add one Pure thesaurus concept as an extra search variant (e.g. 'LLM' β 'Large Language Models').
|
| 452 |
+
Only concepts sharing a word with the query are used, to keep expansion on-topic."""
|
| 453 |
+
if not USE_CONCEPTS or not plan.keyword_queries or len(plan.keyword_queries) >= 4:
|
| 454 |
+
return None
|
| 455 |
+
q0 = plan.keyword_queries[0]
|
| 456 |
+
try:
|
| 457 |
+
names = await pure.search_concepts(q0, size=5)
|
| 458 |
+
except Exception:
|
| 459 |
+
return None
|
| 460 |
+
existing = {q.lower() for q in plan.keyword_queries}
|
| 461 |
+
qtok = set(_tokens(q0))
|
| 462 |
+
for name in names:
|
| 463 |
+
if name.lower() not in existing and len(name) <= 60 and qtok & set(_tokens(name)):
|
| 464 |
+
plan.keyword_queries.append(name)
|
| 465 |
+
return {"endpoint": "concepts/search", "concept_for": q0, "added": name}
|
| 466 |
+
return None
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
async def _retrieve_collaborations(pure: PureClient, plan: QueryPlan, max_orgs: int = 10) -> dict:
|
| 470 |
+
"""External organisations most often co-authoring KU's top publications on the topic."""
|
| 471 |
+
pubs = await _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search"}))
|
| 472 |
+
score: Counter = Counter()
|
| 473 |
+
examples: dict[str, list[dict]] = {}
|
| 474 |
+
ref_names: dict[str, str] = {}
|
| 475 |
+
for rank_i, o in enumerate(pubs["results"][:50]):
|
| 476 |
+
for org in o.get("external_orgs", []):
|
| 477 |
+
score[org["uuid"]] += 1.0 / (3 + rank_i)
|
| 478 |
+
examples.setdefault(org["uuid"], []).append(o)
|
| 479 |
+
if org.get("name"):
|
| 480 |
+
ref_names[org["uuid"]] = org["name"]
|
| 481 |
+
top = [u for u, _ in score.most_common(max_orgs * 2)]
|
| 482 |
+
fetched = await asyncio.gather(*(pure.get_external_organization(u) for u in top), return_exceptions=True)
|
| 483 |
+
results = []
|
| 484 |
+
for u, f in zip(top, fetched):
|
| 485 |
+
org = f if isinstance(f, dict) else {"kind": "collaborator", "uuid": u, "name": ref_names.get(u, ""), "country": "", "type": "", "portal_url": ""}
|
| 486 |
+
if not org.get("name") or re.search(r"khalifa university", org["name"], re.I):
|
| 487 |
+
continue
|
| 488 |
+
ex = examples.get(u, [])
|
| 489 |
+
results.append({**org, "title": org["name"], "joint_outputs": len(ex),
|
| 490 |
+
"examples": [{"title": e["title"], "year": e["year"], "portal_url": e["portal_url"]} for e in ex[:3]],
|
| 491 |
+
"_score": round(score[u], 4)})
|
| 492 |
+
if len(results) >= max_orgs:
|
| 493 |
+
break
|
| 494 |
+
trace = pubs["trace"] + [{"endpoint": "external-organizations", "hits": len(results)}]
|
| 495 |
+
return {"results": results, "unfiltered": len(score), "total": len(score), "person": None, "trace": trace}
|
| 496 |
+
|
| 497 |
+
|
| 498 |
+
async def _retrieve_datasets(pure: PureClient, plan: QueryPlan) -> dict:
|
| 499 |
+
"""/data-sets when the Pure API offers it; otherwise dataset-type research outputs."""
|
| 500 |
+
try:
|
| 501 |
+
r = await _retrieve_outputs(pure, plan, search=pure.search_datasets)
|
| 502 |
+
if r["results"]:
|
| 503 |
+
return r
|
| 504 |
+
except PureError as e:
|
| 505 |
+
if e.status not in (403, 404):
|
| 506 |
+
raise
|
| 507 |
+
r = await _retrieve_outputs(pure, plan.model_copy(update={"intent": "topic_search", "output_types": ["dataset"]}))
|
| 508 |
+
r["trace"].insert(0, {"endpoint": "data-sets", "note": "not available; used dataset-type research outputs"})
|
| 509 |
+
return r
|
| 510 |
+
|
| 511 |
+
|
| 512 |
async def retrieve(pure: PureClient, plan: QueryPlan) -> dict:
|
| 513 |
+
concept = None
|
| 514 |
+
if plan.intent in ("topic_search", "thesis_search", "expert_search", "project_search", "collab_search", "count"):
|
| 515 |
+
concept = await _expand_with_concepts(pure, plan)
|
| 516 |
+
result = await _route(pure, plan)
|
| 517 |
+
if concept:
|
| 518 |
+
result["trace"].insert(0, concept)
|
| 519 |
+
return result
|
| 520 |
+
|
| 521 |
+
|
| 522 |
+
async def _route(pure: PureClient, plan: QueryPlan) -> dict:
|
| 523 |
+
if plan.intent == "project_search":
|
| 524 |
+
return await _retrieve_outputs(pure, plan.model_copy(update={"output_types": [], "open_access_only": False}),
|
| 525 |
+
search=pure.search_projects)
|
| 526 |
+
if plan.intent == "equipment_search":
|
| 527 |
+
return await _retrieve_outputs(pure, plan.model_copy(update={"output_types": [], "open_access_only": False,
|
| 528 |
+
"year_from": None, "year_to": None}),
|
| 529 |
+
search=pure.search_equipment)
|
| 530 |
+
if plan.intent == "collab_search":
|
| 531 |
+
return await _retrieve_collaborations(pure, plan)
|
| 532 |
+
if plan.intent == "thesis_search":
|
| 533 |
+
return await _retrieve_theses(pure, plan)
|
| 534 |
if plan.intent == "expert_search":
|
| 535 |
return await _retrieve_experts(pure, plan)
|
| 536 |
if plan.intent == "org_search":
|
| 537 |
return await _retrieve_orgs(pure, plan)
|
| 538 |
if plan.intent == "dataset_search":
|
| 539 |
+
return await _retrieve_datasets(pure, plan)
|
| 540 |
return await _retrieve_outputs(pure, plan)
|
| 541 |
|
| 542 |
|
| 543 |
# βββββββββββββββββββββββββββ synthesis ββββββββββββββββββββββββββββββ
|
| 544 |
SYNTH_SYSTEM = """You are the Khazna Research Assistant for Khalifa University (Abu Dhabi, UAE).
|
| 545 |
You answer questions about KU research using ONLY the Khazna (Elsevier Pure) records provided below.
|
| 546 |
+
Records can be publications, student theses, datasets, research projects, equipment/facilities, researcher profiles,
|
| 547 |
+
organisational units (colleges, departments, centres) or external collaborating organisations.
|
| 548 |
|
| 549 |
Rules:
|
| 550 |
- Ground every claim in the records. Cite records inline as [n] using their numbers. Never invent titles, authors, years, findings or DOIs.
|
| 551 |
- Open with a one-sentence direct answer (e.g. how many relevant outputs and the main themes).
|
| 552 |
+
- For researcher records: say who seems most relevant and why, based on their matching outputs, theses they supervised and expertise keywords.
|
| 553 |
+
- For theses: mention the degree level, year, student and supervisors where given.
|
| 554 |
+
- For projects: mention the period, status, KU participants and funders where given. Never state funding amounts.
|
| 555 |
+
- For equipment: say what it is and which unit manages it; suggest contacting the listed contact person through Khazna.
|
| 556 |
+
- For collaborators: rank by the number of joint KU outputs listed, and cite those outputs' records.
|
| 557 |
Do not rank people by seniority or make claims about availability; suggest contacting them via their Khazna profile.
|
| 558 |
- Then group the most relevant records by theme or approach in 2-4 short paragraphs or "- " bullets, naming key KU authors and years.
|
| 559 |
- If some records are only loosely related, say so and do not stretch them. If none really answer the question, say that plainly
|
|
|
|
| 574 |
f"[{i}] RESEARCHER: {o['name']}\n"
|
| 575 |
f" Role: {', '.join(o.get('job_titles') or []) or '-'} | Unit: {', '.join(x['name'] for x in o.get('organisations', []) if x.get('name')) or '-'}\n"
|
| 576 |
f" Expertise keywords: {', '.join(o.get('keywords', [])[:8]) or '-'}\n"
|
| 577 |
+
f" Matching KU outputs on this topic: {o.get('matching_outputs', 0)}. Examples: {ev or '-'}\n"
|
| 578 |
+
f" Supervised KU theses on this topic: {o.get('supervised_theses', 0)}. Examples: "
|
| 579 |
+
+ ("; ".join(f"{t['title']} ({t['year'] or 'n.d.'})" for t in o.get("supervised", [])) or "-")
|
| 580 |
)
|
| 581 |
continue
|
| 582 |
if kind == "organisation":
|
| 583 |
lines.append(f"[{i}] ORGANISATIONAL UNIT: {o['name']} | type: {o.get('type') or '-'} | part of: {', '.join(o.get('parents', [])) or '-'}")
|
| 584 |
continue
|
| 585 |
+
if kind == "collaborator":
|
| 586 |
+
ex = "; ".join(f"{e['title']} ({e['year'] or 'n.d.'})" for e in o.get("examples", []))
|
| 587 |
+
lines.append(f"[{i}] EXTERNAL COLLABORATOR: {o['name']} | country: {o.get('country') or '-'} | type: {o.get('type') or '-'}\n"
|
| 588 |
+
f" Joint KU outputs on this topic: {o.get('joint_outputs', 0)}. Examples: {ex or '-'}")
|
| 589 |
+
continue
|
| 590 |
+
if kind == "project":
|
| 591 |
+
lines.append(
|
| 592 |
+
f"[{i}] PROJECT: {o['title']}{(' (' + o['subtitle'] + ')') if o['subtitle'] else ''}\n"
|
| 593 |
+
f" {o['type']} | period: {o.get('period') or '-'} | status: {o.get('status') or '-'} | managed by: {o['venue'] or '-'}\n"
|
| 594 |
+
f" KU participants: {', '.join(p['name'] + (' (' + p['role'] + ')' if p.get('role') else '') for p in o.get('participants', [])) or '-'}\n"
|
| 595 |
+
f" Funders: {', '.join(o.get('funders', [])) or '-'}\n"
|
| 596 |
+
f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
|
| 597 |
+
f" Description: {o['abstract'][:700] or '(none)'}")
|
| 598 |
+
continue
|
| 599 |
+
if kind == "equipment":
|
| 600 |
+
lines.append(
|
| 601 |
+
f"[{i}] EQUIPMENT: {o['title']} | {o['type']} | managed by: {o['venue'] or '-'} | location: {o.get('location') or '-'}\n"
|
| 602 |
+
f" Contact: {', '.join(c['name'] for c in o.get('contacts', [])) or '-'}\n"
|
| 603 |
+
f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
|
| 604 |
+
f" Description: {o['abstract'][:700] or '(none)'}")
|
| 605 |
+
continue
|
| 606 |
authors = ", ".join(o["authors"][:6]) + (" et al." if len(o["authors"]) > 6 else "")
|
| 607 |
ku = ", ".join(a["name"] for a in o["ku_authors"][:6])
|
| 608 |
lines.append(
|
| 609 |
+
f"[{i}] {'DATASET: ' if kind == 'dataset' else 'THESIS: ' if kind == 'thesis' else ''}{o['title']}{(': ' + o['subtitle']) if o['subtitle'] else ''}\n"
|
| 610 |
f" {o['type'] or 'Output'} | {o['year'] or 'n.d.'} | {o['venue'] or '-'}\n"
|
| 611 |
f" Authors: {authors or '-'}\n"
|
| 612 |
f" KU authors: {ku or '-'}\n"
|
| 613 |
+
+ (f" Supervisors: {', '.join(x['name'] for x in o.get('supervisors', [])) or '-'}\n" if kind == "thesis" else "")
|
| 614 |
+
+ f" Keywords: {', '.join(o['keywords'][:8]) or '-'}\n"
|
| 615 |
f" Abstract: {o['abstract'][:700] or '(no abstract)'}"
|
| 616 |
)
|
| 617 |
return "\n\n".join(lines)
|
pure_client.py
CHANGED
|
@@ -8,6 +8,11 @@ Endpoints used (all read-only, match the API key's access definition):
|
|
| 8 |
GET /persons/{uuid}/research-outputs (falls back to name search if not available)
|
| 9 |
POST /organizations/search GET /organizations/{uuid}
|
| 10 |
POST /data-sets/search (needs /data-sets/* added to the API key)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
Auth: header `api-key: <PURE_API_KEY>`.
|
| 13 |
|
|
@@ -117,9 +122,12 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
|
|
| 117 |
year = pdate.get("year")
|
| 118 |
date = "-".join(str(p) for p in (year, pdate.get("month"), pdate.get("day")) if p)
|
| 119 |
|
| 120 |
-
# Contributors
|
| 121 |
-
authors, ku_authors = [], []
|
| 122 |
for c in item.get("contributors") or item.get("personAssociations") or []:
|
|
|
|
|
|
|
|
|
|
| 123 |
n = _person_name(c.get("name"))
|
| 124 |
if not n:
|
| 125 |
continue
|
|
@@ -127,6 +135,10 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
|
|
| 127 |
disc = c.get("typeDiscriminator", "")
|
| 128 |
if "Internal" in disc or (c.get("person") or {}).get("uuid"):
|
| 129 |
ku_authors.append({"name": n, "uuid": (c.get("person") or {}).get("uuid")})
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
keywords = _keywords(item)
|
| 132 |
|
|
@@ -169,6 +181,7 @@ def normalize_research_output(item: dict, portal_url: str) -> dict:
|
|
| 169 |
"open_access": open_access,
|
| 170 |
"file_available": file_available,
|
| 171 |
"peer_reviewed": item.get("peerReview"),
|
|
|
|
| 172 |
"portal_url": item.get("portalUrl") or (f"{portal_url}/en/publications/{uuid}" if uuid else ""),
|
| 173 |
}
|
| 174 |
|
|
@@ -268,6 +281,202 @@ def normalize_dataset(item: dict, portal_url: str) -> dict:
|
|
| 268 |
}
|
| 269 |
|
| 270 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 271 |
# βββββββββββββββββββββββββββββ client βββββββββββββββββββββββββββββββ
|
| 272 |
class PureClient:
|
| 273 |
def __init__(
|
|
@@ -394,6 +603,37 @@ class PureClient:
|
|
| 394 |
for o in p.get("organisations", []):
|
| 395 |
o["name"] = o.get("name") or names.get(o["uuid"], "")
|
| 396 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 397 |
# -- datasets ---------------------------------------------------
|
| 398 |
async def search_datasets(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
|
| 399 |
data = await self.request("POST", "/data-sets/search", json={"searchString": search_string, "size": size, "offset": offset})
|
|
|
|
| 8 |
GET /persons/{uuid}/research-outputs (falls back to name search if not available)
|
| 9 |
POST /organizations/search GET /organizations/{uuid}
|
| 10 |
POST /data-sets/search (needs /data-sets/* added to the API key)
|
| 11 |
+
POST /student-theses/search (needs /student-theses/* added to the API key)
|
| 12 |
+
POST /projects/search GET /persons/{uuid} for participants
|
| 13 |
+
POST /equipment/search
|
| 14 |
+
POST /concepts/search (thesaurus terms used for query expansion)
|
| 15 |
+
GET /external-organizations/{uuid} (collaborator names, cached)
|
| 16 |
|
| 17 |
Auth: header `api-key: <PURE_API_KEY>`.
|
| 18 |
|
|
|
|
| 122 |
year = pdate.get("year")
|
| 123 |
date = "-".join(str(p) for p in (year, pdate.get("month"), pdate.get("day")) if p)
|
| 124 |
|
| 125 |
+
# Contributors (+ the external organisations they are affiliated with, for collaboration analysis)
|
| 126 |
+
authors, ku_authors, external_orgs = [], [], []
|
| 127 |
for c in item.get("contributors") or item.get("personAssociations") or []:
|
| 128 |
+
for ref in c.get("externalOrganizations") or c.get("externalOrganisations") or []:
|
| 129 |
+
if isinstance(ref, dict) and ref.get("uuid"):
|
| 130 |
+
external_orgs.append({"uuid": ref["uuid"], "name": _loc(ref.get("name"))})
|
| 131 |
n = _person_name(c.get("name"))
|
| 132 |
if not n:
|
| 133 |
continue
|
|
|
|
| 135 |
disc = c.get("typeDiscriminator", "")
|
| 136 |
if "Internal" in disc or (c.get("person") or {}).get("uuid"):
|
| 137 |
ku_authors.append({"name": n, "uuid": (c.get("person") or {}).get("uuid")})
|
| 138 |
+
for ref in item.get("externalOrganizations") or []:
|
| 139 |
+
if isinstance(ref, dict) and ref.get("uuid"):
|
| 140 |
+
external_orgs.append({"uuid": ref["uuid"], "name": _loc(ref.get("name"))})
|
| 141 |
+
external_orgs = list({o["uuid"]: o for o in external_orgs}.values())
|
| 142 |
|
| 143 |
keywords = _keywords(item)
|
| 144 |
|
|
|
|
| 181 |
"open_access": open_access,
|
| 182 |
"file_available": file_available,
|
| 183 |
"peer_reviewed": item.get("peerReview"),
|
| 184 |
+
"external_orgs": external_orgs,
|
| 185 |
"portal_url": item.get("portalUrl") or (f"{portal_url}/en/publications/{uuid}" if uuid else ""),
|
| 186 |
}
|
| 187 |
|
|
|
|
| 281 |
}
|
| 282 |
|
| 283 |
|
| 284 |
+
def _date_parts(v: Any) -> tuple[int | None, str]:
|
| 285 |
+
"""Pure dates arrive as {"year":..,"month":..,"day":..} or as "YYYY-MM-DD" strings."""
|
| 286 |
+
if isinstance(v, dict):
|
| 287 |
+
y = v.get("year")
|
| 288 |
+
parts = [str(p) for p in (y, v.get("month"), v.get("day")) if p]
|
| 289 |
+
return (int(y) if str(y or "").isdigit() else None), "-".join(parts)
|
| 290 |
+
if isinstance(v, str):
|
| 291 |
+
m = re.match(r"(\d{4})(?:-(\d{2}))?(?:-(\d{2}))?", v)
|
| 292 |
+
if m:
|
| 293 |
+
return int(m.group(1)), "-".join(x for x in m.groups() if x)
|
| 294 |
+
return None, ""
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
def normalize_student_thesis(item: dict, portal_url: str) -> dict:
|
| 298 |
+
uuid = item.get("uuid", "")
|
| 299 |
+
year, date = None, ""
|
| 300 |
+
for key in ("awardDate", "publicationDate", "submissionDate"):
|
| 301 |
+
year, date = _date_parts(item.get(key))
|
| 302 |
+
if year:
|
| 303 |
+
break
|
| 304 |
+
if not year:
|
| 305 |
+
for st in item.get("publicationStatuses") or []:
|
| 306 |
+
year, date = _date_parts(st.get("publicationDate"))
|
| 307 |
+
if year:
|
| 308 |
+
break
|
| 309 |
+
|
| 310 |
+
authors, ku_authors, supervisors = [], [], []
|
| 311 |
+
for c in item.get("contributors") or item.get("personAssociations") or []:
|
| 312 |
+
n = _person_name(c.get("name"))
|
| 313 |
+
if not n:
|
| 314 |
+
continue
|
| 315 |
+
role = _loc(c.get("role")).lower() or _uri_tail(c.get("role")).lower()
|
| 316 |
+
entry = {"name": n, "uuid": (c.get("person") or {}).get("uuid")}
|
| 317 |
+
if "supervis" in role or "advisor" in role:
|
| 318 |
+
supervisors.append(entry)
|
| 319 |
+
continue
|
| 320 |
+
authors.append(n)
|
| 321 |
+
if "Internal" in c.get("typeDiscriminator", "") or entry["uuid"]:
|
| 322 |
+
ku_authors.append(entry)
|
| 323 |
+
for sup in item.get("supervisors") or []: # some Pure versions list supervisors separately
|
| 324 |
+
n = _person_name(sup.get("name"))
|
| 325 |
+
if n and n not in {x["name"] for x in supervisors}:
|
| 326 |
+
supervisors.append({"name": n, "uuid": (sup.get("person") or {}).get("uuid")})
|
| 327 |
+
|
| 328 |
+
institution = ""
|
| 329 |
+
for inst in item.get("awardingInstitutions") or []:
|
| 330 |
+
institution = (_loc((inst.get("organization") or inst.get("externalOrganization") or {}).get("name"))
|
| 331 |
+
or _loc(inst.get("name")) or _loc(inst))
|
| 332 |
+
if institution:
|
| 333 |
+
break
|
| 334 |
+
|
| 335 |
+
doi = ""
|
| 336 |
+
file_available, open_access = bool(item.get("documents")), False
|
| 337 |
+
for ev in item.get("electronicVersions") or []:
|
| 338 |
+
if not doi and ev.get("doi"):
|
| 339 |
+
doi = _norm_doi(ev["doi"])
|
| 340 |
+
if _uri_tail(ev.get("accessType")).lower() in ("open", "free"):
|
| 341 |
+
open_access = True
|
| 342 |
+
if "File" in ev.get("typeDiscriminator", "") or ev.get("file"):
|
| 343 |
+
file_available = True
|
| 344 |
+
|
| 345 |
+
return {
|
| 346 |
+
"kind": "thesis",
|
| 347 |
+
"uuid": uuid,
|
| 348 |
+
"title": _clean_text(_loc(item.get("title"))) or "Untitled thesis",
|
| 349 |
+
"subtitle": _clean_text(_loc(item.get("subTitle"))),
|
| 350 |
+
"type": _loc(item.get("type")) or "Thesis",
|
| 351 |
+
"type_key": "thesis",
|
| 352 |
+
"category": "",
|
| 353 |
+
"year": year,
|
| 354 |
+
"date": date,
|
| 355 |
+
"status": "",
|
| 356 |
+
"authors": authors,
|
| 357 |
+
"ku_authors": ku_authors,
|
| 358 |
+
"supervisors": supervisors[:4],
|
| 359 |
+
"abstract": _clean_text(_loc(item.get("abstract")), limit=1200),
|
| 360 |
+
"keywords": _keywords(item),
|
| 361 |
+
"venue": _clean_text(institution),
|
| 362 |
+
"doi": doi,
|
| 363 |
+
"doi_url": f"https://doi.org/{doi}" if doi else "",
|
| 364 |
+
"open_access": open_access,
|
| 365 |
+
"file_available": file_available,
|
| 366 |
+
"peer_reviewed": None,
|
| 367 |
+
"portal_url": item.get("portalUrl") or (f"{portal_url}/en/studentTheses/{uuid}" if uuid else ""),
|
| 368 |
+
}
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def _descriptions(item: dict, limit: int = 1200) -> str:
|
| 372 |
+
for d in item.get("descriptions") or []:
|
| 373 |
+
text = _loc(d.get("value") if isinstance(d, dict) else d)
|
| 374 |
+
if text:
|
| 375 |
+
return _clean_text(text, limit=limit)
|
| 376 |
+
return _clean_text(_loc(item.get("description")) or _loc(item.get("abstract")), limit=limit)
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
def _people(item: dict, *keys: str) -> list[dict]:
|
| 380 |
+
out = []
|
| 381 |
+
for key in keys:
|
| 382 |
+
for c in item.get(key) or []:
|
| 383 |
+
if not isinstance(c, dict):
|
| 384 |
+
continue
|
| 385 |
+
n = _person_name(c.get("name")) or _person_name((c.get("person") or {}).get("name"))
|
| 386 |
+
if n:
|
| 387 |
+
out.append({"name": n, "uuid": (c.get("person") or {}).get("uuid") or c.get("uuid"),
|
| 388 |
+
"role": _loc(c.get("role"))})
|
| 389 |
+
return list({(p["uuid"] or p["name"]): p for p in out}.values())
|
| 390 |
+
|
| 391 |
+
|
| 392 |
+
def normalize_project(item: dict, portal_url: str) -> dict:
|
| 393 |
+
"""Project records. Funding amounts are deliberately NOT copied (often confidential)."""
|
| 394 |
+
uuid = item.get("uuid", "")
|
| 395 |
+
period = item.get("period") or {}
|
| 396 |
+
start_y, start = _date_parts(period.get("startDate") or item.get("startDate"))
|
| 397 |
+
end_y, end = _date_parts(period.get("endDate") or item.get("endDate"))
|
| 398 |
+
participants = _people(item, "participants", "contributors", "personAssociations")
|
| 399 |
+
funders = []
|
| 400 |
+
for f in item.get("fundings") or item.get("fundingDetails") or []:
|
| 401 |
+
if not isinstance(f, dict):
|
| 402 |
+
continue
|
| 403 |
+
name = (_loc((f.get("funder") or f.get("fundingOrganization") or f.get("externalOrganization") or {}).get("name"))
|
| 404 |
+
or _loc(f.get("funderName")))
|
| 405 |
+
if name:
|
| 406 |
+
funders.append(name)
|
| 407 |
+
managing = _loc((item.get("managingOrganization") or item.get("managingOrganisationalUnit") or {}).get("name"))
|
| 408 |
+
status = _loc(item.get("status")) or _loc(item.get("workflow"))
|
| 409 |
+
return {
|
| 410 |
+
"kind": "project",
|
| 411 |
+
"uuid": uuid,
|
| 412 |
+
"title": _clean_text(_loc(item.get("title"))) or "Untitled project",
|
| 413 |
+
"subtitle": _clean_text(_loc(item.get("shortTitle")) or _loc(item.get("acronym"))),
|
| 414 |
+
"type": _loc(item.get("type")) or "Project",
|
| 415 |
+
"type_key": "project",
|
| 416 |
+
"category": "",
|
| 417 |
+
"year": start_y,
|
| 418 |
+
"end_year": end_y,
|
| 419 |
+
"date": start,
|
| 420 |
+
"period": " β ".join(x for x in (start, end) if x),
|
| 421 |
+
"status": status,
|
| 422 |
+
"authors": [p["name"] for p in participants],
|
| 423 |
+
"ku_authors": [p for p in participants if p.get("uuid")],
|
| 424 |
+
"participants": participants[:8],
|
| 425 |
+
"funders": list(dict.fromkeys(funders))[:4],
|
| 426 |
+
"abstract": _descriptions(item),
|
| 427 |
+
"keywords": _keywords(item),
|
| 428 |
+
"venue": _clean_text(managing),
|
| 429 |
+
"doi": "", "doi_url": "", "open_access": False, "file_available": False, "peer_reviewed": None,
|
| 430 |
+
"portal_url": item.get("portalUrl") or (f"{portal_url}/en/projects/{uuid}" if uuid else ""),
|
| 431 |
+
}
|
| 432 |
+
|
| 433 |
+
|
| 434 |
+
def normalize_equipment(item: dict, portal_url: str) -> dict:
|
| 435 |
+
"""Equipment / facilities. Contact persons are names only; emails and phones are never copied."""
|
| 436 |
+
uuid = item.get("uuid", "")
|
| 437 |
+
contacts = _people(item, "contactPersons", "contacts", "personAssociations")
|
| 438 |
+
orgs = [_loc((o or {}).get("name")) for o in (item.get("organizations") or item.get("organisationalUnits") or [])]
|
| 439 |
+
managing = _loc((item.get("managingOrganization") or {}).get("name")) or next((o for o in orgs if o), "")
|
| 440 |
+
location = ""
|
| 441 |
+
for loc in item.get("locations") or item.get("addresses") or []:
|
| 442 |
+
location = _loc(loc.get("name") if isinstance(loc, dict) and "name" in loc else loc) or _loc((loc or {}).get("building"))
|
| 443 |
+
if location:
|
| 444 |
+
break
|
| 445 |
+
return {
|
| 446 |
+
"kind": "equipment",
|
| 447 |
+
"uuid": uuid,
|
| 448 |
+
"title": _clean_text(_loc(item.get("title")) or _loc(item.get("name"))) or "Unnamed equipment",
|
| 449 |
+
"subtitle": "",
|
| 450 |
+
"type": _loc(item.get("type")) or "Equipment",
|
| 451 |
+
"type_key": "equipment",
|
| 452 |
+
"category": _loc(item.get("category")),
|
| 453 |
+
"year": None, "date": "", "status": "",
|
| 454 |
+
"authors": [], "ku_authors": [],
|
| 455 |
+
"contacts": contacts[:4],
|
| 456 |
+
"abstract": _descriptions(item),
|
| 457 |
+
"keywords": _keywords(item),
|
| 458 |
+
"venue": _clean_text(managing),
|
| 459 |
+
"location": _clean_text(location),
|
| 460 |
+
"doi": "", "doi_url": "", "open_access": False, "file_available": False, "peer_reviewed": None,
|
| 461 |
+
"portal_url": item.get("portalUrl") or (f"{portal_url}/en/equipments/{uuid}" if uuid else ""),
|
| 462 |
+
}
|
| 463 |
+
|
| 464 |
+
|
| 465 |
+
def normalize_external_organization(item: dict, portal_url: str) -> dict:
|
| 466 |
+
addr = item.get("address") or {}
|
| 467 |
+
if isinstance(addr, list):
|
| 468 |
+
addr = addr[0] if addr else {}
|
| 469 |
+
return {
|
| 470 |
+
"kind": "collaborator",
|
| 471 |
+
"uuid": item.get("uuid", ""),
|
| 472 |
+
"name": _loc(item.get("name")),
|
| 473 |
+
"title": _loc(item.get("name")),
|
| 474 |
+
"type": _loc(item.get("type")),
|
| 475 |
+
"country": _loc(addr.get("country")) or _loc(item.get("country")),
|
| 476 |
+
"portal_url": item.get("portalUrl") or "",
|
| 477 |
+
}
|
| 478 |
+
|
| 479 |
+
|
| 480 |
# βββββββββββββββββββββββββββββ client βββββββββββββββββββββββββββββββ
|
| 481 |
class PureClient:
|
| 482 |
def __init__(
|
|
|
|
| 603 |
for o in p.get("organisations", []):
|
| 604 |
o["name"] = o.get("name") or names.get(o["uuid"], "")
|
| 605 |
|
| 606 |
+
# -- student theses ---------------------------------------------
|
| 607 |
+
async def search_student_theses(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
|
| 608 |
+
data = await self.request("POST", "/student-theses/search", json={"searchString": search_string, "size": size, "offset": offset})
|
| 609 |
+
items = self._filter(data.get("items") or [])
|
| 610 |
+
return int(data.get("count") or 0), [normalize_student_thesis(i, self.portal_url) for i in items]
|
| 611 |
+
|
| 612 |
+
# -- projects / equipment ---------------------------------------
|
| 613 |
+
async def search_projects(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
|
| 614 |
+
data = await self.request("POST", "/projects/search", json={"searchString": search_string, "size": size, "offset": offset})
|
| 615 |
+
items = self._filter(data.get("items") or [])
|
| 616 |
+
return int(data.get("count") or 0), [normalize_project(i, self.portal_url) for i in items]
|
| 617 |
+
|
| 618 |
+
async def search_equipment(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
|
| 619 |
+
data = await self.request("POST", "/equipment/search", json={"searchString": search_string, "size": size, "offset": offset})
|
| 620 |
+
items = self._filter(data.get("items") or [])
|
| 621 |
+
return int(data.get("count") or 0), [normalize_equipment(i, self.portal_url) for i in items]
|
| 622 |
+
|
| 623 |
+
# -- concepts (thesaurus) ---------------------------------------
|
| 624 |
+
async def search_concepts(self, search_string: str, size: int = 5) -> list[str]:
|
| 625 |
+
data = await self.request("POST", "/concepts/search", json={"searchString": search_string, "size": size, "offset": 0})
|
| 626 |
+
names = [_loc(i.get("name") or i.get("term") or i.get("title")) for i in data.get("items") or []]
|
| 627 |
+
return [n for n in names if n]
|
| 628 |
+
|
| 629 |
+
# -- external organisations (collaborators) ---------------------
|
| 630 |
+
async def get_external_organization(self, uuid: str) -> dict | None:
|
| 631 |
+
try:
|
| 632 |
+
item = await self.request("GET", f"/external-organizations/{uuid}")
|
| 633 |
+
except PureError:
|
| 634 |
+
return None
|
| 635 |
+
return normalize_external_organization(item, self.portal_url)
|
| 636 |
+
|
| 637 |
# -- datasets ---------------------------------------------------
|
| 638 |
async def search_datasets(self, search_string: str, size: int = 25, offset: int = 0) -> tuple[int, list[dict]]:
|
| 639 |
data = await self.request("POST", "/data-sets/search", json={"searchString": search_string, "size": size, "offset": offset})
|