kulibrary / README.md
nikeshn's picture
Upload 7 files
aaaab82 verified
|
Raw History Blame Contribute Delete
15.1 kB
---
title: Khazna AI Assistant
emoji: πŸ“š
colorFrom: blue
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
short_description: Natural-language search over KU's Khazna (Pure) repository
---
# Khazna AI Assistant
Turns a natural-language question into keyword searches on **Khazna** (KU's Elsevier Pure repository), searching **publications, student theses, researcher profiles, departments/centres, research projects, equipment and collaborations**, and writes a grounded answer with `[n]` citations that link to result cards.
## What each tab / intent searches
| Tab (`scope`) | Intent(s) | Pure endpoints | Khazna portal equivalent |
|---|---|---|---|
| Publications | `topic_search`, `author_search`, `count` | `/research-outputs/search`, `/persons/search` | /en/publications/ |
| Theses | `thesis_search` | `/student-theses/search` (needs `/student-theses/*` on the key), merged with thesis-type research outputs; falls back to research outputs alone on 403 | /en/studentTheses/ |
| Profiles | `expert_search` (who works on / supervises X), `author_search` (named person) | `/persons/search` + `/research-outputs/search` (KU authors aggregated) + `/student-theses/search` (supervisors), `GET /persons/{uuid}` | /en/persons/ |
| Departments | `org_search` | `/organizations/search` | /en/organisations/ |
| Projects | `project_search` | `/projects/search` (funding **amounts are never shown**) | /en/projects/ |
| Equipment | `equipment_search` | `/equipment/search` (contact **names only**, no emails) | /en/equipments/ |
| Collaborations | `collab_search` | external organisations on co-authors of the topic's top KU publications, names via `GET /external-organizations/{uuid}` | β€” |
| Datasets (no tab) | `dataset_search` | `/data-sets/search` if the Pure API offers it (KU's key currently has no such endpoint) β†’ falls back to dataset-type research outputs | /en/datasets/ |
| Everything | planner decides | any of the above | β€” |
**Concept expansion:** for topic, thesis, expert, project and collaboration searches, one Pure thesaurus concept from `/concepts/search` that shares a word with the query is added as an extra search variant. Set `USE_CONCEPTS=false` to switch it off.
**Who is listed comes from the Khazna portal:** for a named researcher, the bot reads the portal's own "Find Profiles" search (`/en/persons/?search=<name>…&community=false`). That gives current, public, KU-only profiles, the same list people see on Khazna. The API adds each person's uuid, role and unit. Someone the portal lists but the API can't match is still shown and explored by name (`person_name`). If the portal can't be reached, the bot falls back to `/persons/search`. Khazna is behind Cloudflare, which may block server requests for portal pages; `/health` β†’ `portal_people_status` shows the result. The fallback can only drop former staff if the API key's **access definition** exposes persons' `staffOrganizationAssociations`. On KU's key it currently doesn't (only 2 of 26 Kumars return them), so **expose that field** in Pure β†’ API keys β†’ Access definition. Set `USE_PORTAL_PEOPLE=false` to always use the API. Tests use a real saved portal page (`tests/fixtures/portal_persons_kumar.html`, 8 Kumars).
**Current KU staff only (API fallback):** `/persons` never includes external persons. By default the bot also drops anyone without a current staff appointment, meaning a `staffOrganizationAssociations` entry with no `period.endDate` or one in the future. That leaves out former staff and student-only records. KU's Pure does not return appointment data for every current staff member, so a person with no appointment data is kept. Only an ended appointment drops someone. When a named researcher has no current KU profile, the bot only returns publications that have a **KU** author of that name, never ones that match an external co-author. `GET /debug/persons?q=Kumar` (admin) lists every person Pure returned, their appointments, and why each was shown or dropped. Set `PURE_ACTIVE_PERSONS_ONLY=false` to list everyone. A researcher the user explicitly picks is always shown. Khazna people-search links add `community=false`, so external Pure Community profiles are excluded.
**Name disambiguation:** for a named researcher, `/persons/search` returns up to 50 people, kept only if every typed name word equals one of their name words (first, middle or last; prefix matches only when nothing matches exactly). One match, or one exact full-name match, goes straight to that profile. Two or more return `intent: "author_clarify"` with up to `MAX_NAME_CANDIDATES` (default 12) cards marked `choose: true`; the first 12 are shown, with the rest in `more_results` behind **Show more**. The frontend then resends the original question with `person_uuid`, which bypasses the name search entirely.
Expert finding combines three signals: Pure's profile search, the KU authors of the top-ranked publications for the topic, and the supervisors of matching student theses. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card.
## Pipeline
```
question + history
β”‚
β”œβ”€ 1. PLAN (LLM, structured output β†’ QueryPlan)
β”‚ intent Β· 1-3 English keyword variants Β· broad fallback Β· author Β· years Β· types Β· OA Β· sort
β”‚
β”œβ”€ 2. RETRIEVE (Pure API + vector index, parallel)
β”‚ topic β†’ POST /research-outputs/search Γ— variants
β”‚ + vector index (meaning-based + BM25) β†’ top 15 UUIDs β†’ GET /research-outputs/{uuid}
β”‚ author β†’ POST /persons/search β†’ GET /persons/{uuid}/research-outputs
β”‚ (falls back to a name search filtered on contributor UUID)
β”‚ 0 hits β†’ retry once with broad_query
β”‚ drop non-public records (visibility β‰  FREE, confidential)
β”‚
β”œβ”€ 3. RANK + FILTER reciprocal-rank fusion across variants + title/keyword/abstract term overlap;
β”‚ year / type / OA filters applied client-side
β”‚
└─ 4. SYNTHESIZE (LLM) answer grounded in the top-K records, cited [n], same language as the question
```
That's two LLM calls per search turn. Social, repository-help and out-of-scope questions are answered without calling Pure.
## Vector index (combined with the Pure API search)
Pure's search is keyword-only: a record is missed when the user's words differ from the title/abstract, and Arabic
questions depend entirely on the planner's translation. The vector index adds meaning-based search over **research
outputs** (titles, keywords, subjects, SDGs and abstracts of all ~46k records), and both run on every publication search:
- **When:** topic and count questions, plus the publication searches behind expert finding and collaborations.
Author/name searches, theses, projects, equipment, departments and datasets stay Pure-only (theses can get their own index later).
- **How:** `khazna_index.py` runs multilingual-e5-small (dense) + BM25 on the question, merged by reciprocal-rank fusion,
with the year filter applied first. The top 15 UUIDs (`INDEX_TOP`) are fetched live from Pure, so cards keep the real
portal link, KU author IDs and the public-visibility check (non-public or deleted records are dropped). They then join
the Pure keyword variants in `pipeline.rank()` with weight `INDEX_WEIGHT` (1.5). A DOI in the question is looked up exactly.
- **Response:** the trace gets `{"endpoint": "vector-index", "source": "index", ...}`, `/agent` adds `index_as_of`, and
publication cards can carry `department`, `oa_url`, `oa_label` (free full-text link from the harvester, if built with it).
- **Safe by default:** the index loads in the background after startup; until it is ready, or if it fails, or with
`USE_VECTOR_INDEX=false`, the bot is Pure-only exactly as before. `/health` β†’ `vector_index` shows the status.
**Build and publish the index** (on a laptop, not in the Space; about 30-60 min on CPU, a few minutes on a GPU/Apple chip):
```bash
pip install pandas openpyxl pyarrow numpy bm25s PyStemmer sentence-transformers huggingface_hub
hf auth login
python scripts/build_khazna_index.py --export PURE_for_Claude.xlsx --repo nikeshn/khazna-index \
[--manifest manifest.csv] # optional: harvester's open-access links
```
The export needs UUID, Pure ID, Title and Abstract (plus keywords, ASJC, SDGs, Person, organisational units, journal,
year, type, DOI). The dataset is private and holds about 120 MB (records.parquet, dense.npy, bm25/, build_info.json,
duplicate_dois.csv). Then set on the Space: secret `HF_TOKEN` (read access) and variable
`KHAZNA_INDEX_REPO=nikeshn/khazna-index`, and restart or call
`curl -X POST -H "X-Admin-Password: $ADMIN_PASSWORD" https://nikeshn-kulibrary.hf.space/admin/reload-index`.
Rebuild after each new Pure export; newer records are still found by the live Pure search in the meantime.
## Setup on Hugging Face
1. Push these files to the Docker Space `nikeshn/kulibrary` (https://huggingface.co/spaces/nikeshn/kulibrary, served at https://nikeshn-kulibrary.hf.space).
2. **Settings β†’ Variables and secrets β†’ New secret**:
| Name | Value |
|---|---|
| `PURE_API_KEY` | your Pure API key (**secret** β€” the Space is public) |
| `OPENAI_API_KEY` | OpenAI key |
| `ANTHROPIC_API_KEY` | optional, enables Claude Haiku |
| `ADMIN_PASSWORD` | protects `/admin/stats` and `/debug/pure` |
3. Optional variables: `PURE_BASE_URL` (default `https://khazna.ku.ac.ae/ws/api`), `PURE_PORTAL_URL`, `PURE_PUBLIC_ONLY` (default `true`), `PURE_ACTIVE_PERSONS_ONLY` (default `true`), `TOP_K` (default `8`),
`ALLOWED_ORIGINS` = `https://library.ku.ac.ae` (where the LibGuides block lives), `RATE_LIMIT_PER_MIN` (default `12` per IP),
`FRONTEND_URL` (default `https://library.ku.ac.ae/khaznabot`: the Space URL redirects there; the Space's own test chat is at `/test`).
4. Optional: enable persistent storage so `/data/khazna_analytics.db` survives restarts.
## First-run check
The field mapping follows the Pure API's JSON shapes, but Pure versions differ slightly. After the first deploy, run:
```bash
curl -H "X-Admin-Password: $ADMIN_PASSWORD" "https://<space>.hf.space/debug/pure?q=hydrogen"
```
Compare `raw_first_item` with `normalized`. Title, authors, year, DOI and `portal_url` should all be filled in. If one is empty, adjust `normalize_research_output()` in `pure_client.py`.
## LibGuides frontend (full block, LibBee model)
Live page: **https://library.ku.ac.ae/khaznabot**
`libguides/khazna-libguides-block.html` is a complete, paste-ready frontend built on the same model as the LibBee block: same KU header, composer, popular searches, status pill, progress and cancel, answer tools, `/config` announcements and footer cards. All of its HTML, CSS and JS lives in LibGuides; it only calls the Space for data.
1. In a guide, add a **Media / Widget** asset (or an HTML/source box, the same as LibBee) and paste the whole file.
2. To change the backend, edit one line in the script's CONFIG section: `var API_URL = "https://nikeshn-kulibrary.hf.space";` (for example, to route through the LibBee Cloudflare gateway).
3. On the Space, set `ALLOWED_ORIGINS` to `https://library.ku.ac.ae`.
What the block adds on top of the LibBee model:
- **Scope chips** (Everything, Publications, Theses, Profiles, Departments, Projects, Equipment, Collaborations), sent to the Space as `scope`. They swipe on phones.
- **Record cards** rendered from `search_results` (publication, person, organisation, dataset). `[n]` citations in the answer jump to the matching card.
- **Keywords sent to Khazna** panel, plus a large "Search Khazna for: …" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
- **Follow-up chips** from `follow_up_suggestions` that switch scope, e.g. "Who at KU works on this?".
- Styles are scoped to `#ku-khazna-page` / `.kz-` and the keyframes are renamed, so LibBee and Khazna can share a page.
`/config` (environment variables on the Space): `ANNOUNCEMENT` shows a banner; `MAINTENANCE_MODE=true` (with optional `MAINTENANCE_MESSAGE`) disables asking.
**Deep links:** `https://library.ku.ac.ae/khaznabot?ask=<question>&scope=<all|publications|theses|persons|organisations|projects|equipment|collaborations|datasets>` asks the question on load, then removes it from the address bar. `libguides/khazna-portal-searchbox.html` is a small search box for the Khazna portal (plain HTML with inline styles, no JS) that uses this link.
**Alternative:** `libguides/embed-script.html` is a two-line embed that loads `widget.js` from the Space. It updates automatically on every push, but it is not the LibBee design.
## Pure API key: endpoints the bot uses
Used: `/research-outputs/*`, `/student-theses/*`, `/persons/*`, `/organizations/*`, `/external-organizations/*`, `/projects/*`, `/equipment/*`, `/concepts/*`.
Not used, so they can be removed from the key: `/users/*`, `/roles/*`, `/changes/*`, `/applications/*`, `/awards/*`, `/journals/*`, `/publishers/*`, `/classification-schemes/*`, `/thesauri/*`, `/author-collaborations/*`, `/external-persons/*`, `/courses/*`, `/activities/*`, `/events/*`, `/pressmedia/*`, `/prizes/*`.
## Endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | `/` | redirects to `FRONTEND_URL` (https://library.ku.ac.ae/khaznabot) |
| GET | `/test` | the Space's own test chat (widget) |
| GET | `/widget.js` | embeddable widget |
| POST | `/agent` | `{question, history, model: "gpt"\|"claude", scope, top_k, person_uuid?}` β†’ answer, `request_id` (also an `X-Request-ID` header), `search_results` (each has `kind`: publication / thesis / person / organisation / project / equipment / collaborator / dataset), `person`, `trace`, `plan`, `follow_up_suggestions` |
| POST | `/feedback` | `{request_id, feedback: "up"\|"down", session_id}` (LibGuides block) or `{question, rating}` (widget) |
| GET | `/config` | public frontend config: announcement, maintenance mode |
| GET | `/health` | backend status |
| GET | `/admin/stats` | queries, zero-result queries, feedback (header `X-Admin-Password`) |
| POST | `/admin/reload-index` | download + load the newest vector index from `KHAZNA_INDEX_REPO` (admin) |
| GET | `/debug/pure?q=&endpoint=` | raw vs normalised record; `endpoint` = research-outputs \| student-theses \| persons \| organizations \| projects \| equipment \| external-organizations \| data-sets (admin) |
The `/agent` response uses the same `answer` and `search_results` keys as LibBee's `/agent`, so the LibBee frontend can call it as a "Khazna" source.
## Tests
```bash
pip install -r requirements.txt pytest && pytest -q tests
```
These run against a mock Pure API and need no keys. `tests/test_vector_index.py` builds a real index from the mock
records with the build script (a stand-in replaces only the embedding model) and checks the combined search.