--- title: Khazna AI Assistant emoji: ๐Ÿ“š colorFrom: blue colorTo: yellow sdk: docker app_port: 7860 pinned: false short_description: Natural-language search over KU's Khazna (Pure) repository --- # Khazna AI Assistant Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, student theses, researcher profiles, departments/centres, research projects, equipment and collaborations**, and writes a grounded answer with `[n]` citations that link to result cards. ## What each tab / intent searches | Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent | |---|---|---|---| | Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ | | Theses | `thesis_search` | `/student-theses/search` (needs `/student-theses/*` on the key), merged with thesis-type research outputs; falls back to research outputs alone on 403 | /en/studentTheses/ | | Profiles | `expert_search` (who works on / supervises X), `author_search` (named person) | `/persons/search` + `/research-outputs/search` (KU authors aggregated) + `/student-theses/search` (supervisors), `GET /persons/{uuid}` | /en/persons/ | | Departments | `org_search` | `/organizations/search` | /en/organisations/ | | Projects | `project_search` | `/projects/search` (funding **amounts are never shown**) | /en/projects/ | | Equipment | `equipment_search` | `/equipment/search` (contact **names only**, no emails) | /en/equipments/ | | Collaborations | `collab_search` | external organisations on co-authors of the topic's top KU publications, names via `GET /external-organizations/{uuid}` | โ€” | | Datasets (no tab) | `dataset_search` | `/data-sets/search` if the Pure API offers it (KU's key currently has no such endpoint) โ†’ falls back to dataset-type research outputs | /en/datasets/ | | Everything | planner decides | any of the above | โ€” | **Concept expansion:** for topic, thesis, expert, project and collaboration searches, one Pure thesaurus concept from `/concepts/search` that shares a word with the query is added as an extra search variant. Set `USE_CONCEPTS=false` to switch it off. **Who is listed comes from the Khazna portal:** for a named researcher, the bot reads the portal's own "Find Profiles" search (`/en/persons/?search=โ€ฆ&community=false`). That gives current, public, KU-only profiles, the same list people see on Khazna. The API adds each person's uuid, role and unit. Someone the portal lists but the API can't match is still shown and explored by name (`person_name`). If the portal can't be reached, the bot falls back to `/persons/search`. Khazna is behind Cloudflare, which may block server requests for portal pages; `/health` โ†’ `portal_people_status` shows the result. The fallback can only drop former staff if the API key's **access definition** exposes persons' `staffOrganizationAssociations`. On KU's key it currently doesn't (only 2 of 26 Kumars return them), so **expose that field** in Pure โ†’ API keys โ†’ Access definition. Set `USE_PORTAL_PEOPLE=false` to always use the API. Tests use a real saved portal page (`tests/fixtures/portal_persons_kumar.html`, 8 Kumars). **Current KU staff only (API fallback):** `/persons` never includes external persons. By default the bot also drops anyone without a current staff appointment, meaning a `staffOrganizationAssociations` entry with no `period.endDate` or one in the future. That leaves out former staff and student-only records. KU's Pure does not return appointment data for every current staff member, so a person with no appointment data is kept. Only an ended appointment drops someone. When a named researcher has no current KU profile, the bot only returns publications that have a **KU** author of that name, never ones that match an external co-author. `GET /debug/persons?q=Kumar` (admin) lists every person Pure returned, their appointments, and why each was shown or dropped. Set `PURE_ACTIVE_PERSONS_ONLY=false` to list everyone. A researcher the user explicitly picks is always shown. Khazna people-search links add `community=false`, so external Pure Community profiles are excluded. **Name disambiguation:** for a named researcher, `/persons/search` returns up to 50 people, kept only if every typed name word equals one of their name words (first, middle or last; prefix matches only when nothing matches exactly). One match, or one exact full-name match, goes straight to that profile. Two or more return `intent: "author_clarify"` with up to `MAX_NAME_CANDIDATES` (default 12) cards marked `choose: true`; the first 12 are shown, with the rest in `more_results` behind **Show more**. The frontend then resends the original question with `person_uuid`, which bypasses the name search entirely. Expert finding combines three signals: Pure's profile search, the KU authors of the top-ranked publications for the topic, and the supervisors of matching student theses. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card. ## Pipeline ``` question + history โ”‚ โ”œโ”€ 1. PLAN (LLM, structured output โ†’ QueryPlan) โ”‚ intent ยท 1-3 English keyword variants ยท broad fallback ยท author ยท years ยท types ยท OA ยท sort โ”‚ โ”œโ”€ 2. RETRIEVE (Pure API, parallel) โ”‚ topic โ†’ POST /research-outputs/search ร— variants โ”‚ author โ†’ POST /persons/search โ†’ GET /persons/{uuid}/research-outputs โ”‚ (falls back to a name search filtered on contributor UUID) โ”‚ 0 hits โ†’ retry once with broad_query โ”‚ drop non-public records (visibility โ‰  FREE, confidential) โ”‚ โ”œโ”€ 3. RANK + FILTER reciprocal-rank fusion across variants + title/keyword/abstract term overlap; โ”‚ year / type / OA filters applied client-side โ”‚ โ””โ”€ 4. SYNTHESIZE (LLM) answer grounded in the top-K records, cited [n], same language as the question ``` That's two LLM calls per search turn. Social, repository-help and out-of-scope questions are answered without calling Pure. ## Setup on Hugging Face 1. Push these files to the Docker Space `nikeshn/kulibrary` (https://huggingface.co/spaces/nikeshn/kulibrary, served at https://nikeshn-kulibrary.hf.space). 2. **Settings โ†’ Variables and secrets โ†’ New secret**: | Name | Value | |---|---| | `PURE_API_KEY` | your Pure API key (**secret** โ€” the Space is public) | | `OPENAI_API_KEY` | OpenAI key | | `ANTHROPIC_API_KEY` | optional, enables Claude Haiku | | `ADMIN_PASSWORD` | protects `/admin/stats` and `/debug/pure` | 3. Optional variables: `PURE_BASE_URL` (default `https://khazna.ku.ac.ae/ws/api`), `PURE_PORTAL_URL`, `PURE_PUBLIC_ONLY` (default `true`), `PURE_ACTIVE_PERSONS_ONLY` (default `true`), `TOP_K` (default `8`), `ALLOWED_ORIGINS` = `https://library.ku.ac.ae` (where the LibGuides block lives), `RATE_LIMIT_PER_MIN` (default `12` per IP), `FRONTEND_URL` (default `https://library.ku.ac.ae/khaznabot`: the Space URL redirects there; the Space's own test chat is at `/test`). 4. Optional: enable persistent storage so `/data/khazna_analytics.db` survives restarts. ## First-run check The field mapping follows the Pure API's JSON shapes, but Pure versions differ slightly. After the first deploy, run: ```bash curl -H "X-Admin-Password: $ADMIN_PASSWORD" "https://.hf.space/debug/pure?q=hydrogen" ``` Compare `raw_first_item` with `normalized`. Title, authors, year, DOI and `portal_url` should all be filled in. If one is empty, adjust `normalize_research_output()` in `pure_client.py`. ## LibGuides frontend (full block, LibBee model) Live page: **https://library.ku.ac.ae/khaznabot** `libguides/khazna-libguides-block.html` is a complete, paste-ready frontend built on the same model as the LibBee block: same KU header, composer, popular searches, status pill, progress and cancel, answer tools, `/config` announcements and footer cards. All of its HTML, CSS and JS lives in LibGuides; it only calls the Space for data. 1. In a guide, add a **Media / Widget** asset (or an HTML/source box, the same as LibBee) and paste the whole file. 2. To change the backend, edit one line in the script's CONFIG section: `var API_URL = "https://nikeshn-kulibrary.hf.space";` (for example, to route through the LibBee Cloudflare gateway). 3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`. What the block adds on top of the LibBee model: - **Scope chips** (Everything, Publications, Theses, Profiles, Departments, Projects, Equipment, Collaborations), sent to the Space as `scope`. They swipe on phones. - **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card. - **Keywords sent to Khazna** panel, plus a large "Search Khazna for: โ€ฆ" button: the Khazna versions of LibBee's Boolean panel and PRIMO button. - **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?". - Styles are scoped to `#ku-khazna-page` / `.kz-` and the keyframes are renamed, so LibBee and Khazna can share a page. `/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking. **Deep links:** `https://library.ku.ac.ae/khaznabot?ask=&scope=` asks the question on load, then removes it from the address bar. `libguides/khazna-portal-searchbox.html` is a small search box for the Khazna portal (plain HTML with inline styles, no JS) that uses this link. **Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design. ## Pure API key: endpoints the bot uses Used: `/research-outputs/*`, `/student-theses/*`, `/persons/*`, `/organizations/*`, `/external-organizations/*`, `/projects/*`, `/equipment/*`, `/concepts/*`. Not used, so they can be removed from the key: `/users/*`, `/roles/*`, `/changes/*`, `/applications/*`, `/awards/*`, `/journals/*`, `/publishers/*`, `/classification-schemes/*`, `/thesauri/*`, `/author-collaborations/*`, `/external-persons/*`, `/courses/*`, `/activities/*`, `/events/*`, `/pressmedia/*`, `/prizes/*`. ## Endpoints | Method | Path | Purpose | |---|---|---| | GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) | | GET | `/test` | the Space's own test chat (widget) | | GET | `/widget.js` | embeddable widget | | POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k, person_uuid?}` โ†’ answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / thesis / person / organisation / project / equipment / collaborator / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` | | POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) | | GET | `/config` | public frontend config: announcement, maintenance mode | | GET | `/health` | backend status | | GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) | | GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| student-theses \| persons \| organizations \| projects \| equipment \| external-organizations \| data-sets (admin) | The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source. ## Tests ```bash pip install -r requirements.txt pytest && pytest -q tests ``` These run against a mock Pure API and need no keys.