kulibrary / README.md
nikeshn's picture
Upload 7 files
f912809 verified
|
Raw History Blame Contribute Delete
11.9 kB
metadata
title: Khazna AI Assistant
emoji: πŸ“š
colorFrom: blue
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
short_description: Natural-language search over KU's Khazna (Pure) repository

Khazna AI Assistant

Turns a natural-language question into keyword searches on Khazna (KU's Elsevier Pure repository), searching publications, student theses, researcher profiles, departments/centres, research projects, equipment and collaborations, and writes a grounded answer with [n] citations that link to result cards.

What each tab / intent searches

Tab (scope) Intent(s) Pure endpoints Khazna portal equivalent
Publications topic_search, author_search, count /research-outputs/search, /persons/search /en/publications/
Theses thesis_search /student-theses/search (needs /student-theses/* on the key), merged with thesis-type research outputs; falls back to research outputs alone on 403 /en/studentTheses/
Profiles expert_search (who works on / supervises X), author_search (named person) /persons/search + /research-outputs/search (KU authors aggregated) + /student-theses/search (supervisors), GET /persons/{uuid} /en/persons/
Departments org_search /organizations/search /en/organisations/
Projects project_search /projects/search (funding amounts are never shown) /en/projects/
Equipment equipment_search /equipment/search (contact names only, no emails) /en/equipments/
Collaborations collab_search external organisations on co-authors of the topic's top KU publications, names via GET /external-organizations/{uuid} β€”
Datasets (no tab) dataset_search /data-sets/search if the Pure API offers it (KU's key currently has no such endpoint) β†’ falls back to dataset-type research outputs /en/datasets/
Everything planner decides any of the above β€”

Concept expansion: for topic, thesis, expert, project and collaboration searches, one Pure thesaurus concept from /concepts/search that shares a word with the query is added as an extra search variant. Set USE_CONCEPTS=false to switch it off.

Who is listed comes from the Khazna portal: for a named researcher, the bot reads the portal's own "Find Profiles" search (/en/persons/?search=<name>…&community=false). That gives current, public, KU-only profiles, the same list people see on Khazna. The API adds each person's uuid, role and unit. Someone the portal lists but the API can't match is still shown and explored by name (person_name). If the portal can't be reached, the bot falls back to /persons/search. Khazna is behind Cloudflare, which may block server requests for portal pages; /health β†’ portal_people_status shows the result. The fallback can only drop former staff if the API key's access definition exposes persons' staffOrganizationAssociations. On KU's key it currently doesn't (only 2 of 26 Kumars return them), so expose that field in Pure β†’ API keys β†’ Access definition. Set USE_PORTAL_PEOPLE=false to always use the API. Tests use a real saved portal page (tests/fixtures/portal_persons_kumar.html, 8 Kumars).

Current KU staff only (API fallback): /persons never includes external persons. By default the bot also drops anyone without a current staff appointment, meaning a staffOrganizationAssociations entry with no period.endDate or one in the future. That leaves out former staff and student-only records. KU's Pure does not return appointment data for every current staff member, so a person with no appointment data is kept. Only an ended appointment drops someone. When a named researcher has no current KU profile, the bot only returns publications that have a KU author of that name, never ones that match an external co-author. GET /debug/persons?q=Kumar (admin) lists every person Pure returned, their appointments, and why each was shown or dropped. Set PURE_ACTIVE_PERSONS_ONLY=false to list everyone. A researcher the user explicitly picks is always shown. Khazna people-search links add community=false, so external Pure Community profiles are excluded.

Name disambiguation: for a named researcher, /persons/search returns up to 50 people, kept only if every typed name word equals one of their name words (first, middle or last; prefix matches only when nothing matches exactly). One match, or one exact full-name match, goes straight to that profile. Two or more return intent: "author_clarify" with up to MAX_NAME_CANDIDATES (default 12) cards marked choose: true; the first 12 are shown, with the rest in more_results behind Show more. The frontend then resends the original question with person_uuid, which bypasses the name search entirely.

Expert finding combines three signals: Pure's profile search, the KU authors of the top-ranked publications for the topic, and the supervisors of matching student theses. Profile search alone misses active researchers with sparse profiles. The publications also become the "matching outputs" evidence on each profile card.

Pipeline

question + history
  β”‚
  β”œβ”€ 1. PLAN      (LLM, structured output β†’ QueryPlan)
  β”‚     intent Β· 1-3 English keyword variants Β· broad fallback Β· author Β· years Β· types Β· OA Β· sort
  β”‚
  β”œβ”€ 2. RETRIEVE  (Pure API, parallel)
  β”‚     topic  β†’ POST /research-outputs/search  Γ— variants
  β”‚     author β†’ POST /persons/search β†’ GET /persons/{uuid}/research-outputs
  β”‚              (falls back to a name search filtered on contributor UUID)
  β”‚     0 hits β†’ retry once with broad_query
  β”‚     drop non-public records (visibility β‰  FREE, confidential)
  β”‚
  β”œβ”€ 3. RANK + FILTER   reciprocal-rank fusion across variants + title/keyword/abstract term overlap;
  β”‚                     year / type / OA filters applied client-side
  β”‚
  └─ 4. SYNTHESIZE (LLM) answer grounded in the top-K records, cited [n], same language as the question

That's two LLM calls per search turn. Social, repository-help and out-of-scope questions are answered without calling Pure.

Setup on Hugging Face

  1. Push these files to the Docker Space nikeshn/kulibrary (https://huggingface.co/spaces/nikeshn/kulibrary, served at https://nikeshn-kulibrary.hf.space).
  2. Settings β†’ Variables and secrets β†’ New secret:
    Name Value
    PURE_API_KEY your Pure API key (secret β€” the Space is public)
    OPENAI_API_KEY OpenAI key
    ANTHROPIC_API_KEY optional, enables Claude Haiku
    ADMIN_PASSWORD protects /admin/stats and /debug/pure
  3. Optional variables: PURE_BASE_URL (default https://khazna.ku.ac.ae/ws/api), PURE_PORTAL_URL, PURE_PUBLIC_ONLY (default true), PURE_ACTIVE_PERSONS_ONLY (default true), TOP_K (default 8), ALLOWED_ORIGINS = https://library.ku.ac.ae (where the LibGuides block lives), RATE_LIMIT_PER_MIN (default 12 per IP), FRONTEND_URL (default https://library.ku.ac.ae/khaznabot: the Space URL redirects there; the Space's own test chat is at /test).
  4. Optional: enable persistent storage so /data/khazna_analytics.db survives restarts.

First-run check

The field mapping follows the Pure API's JSON shapes, but Pure versions differ slightly. After the first deploy, run:

curl -H "X-Admin-Password: $ADMIN_PASSWORD" "https://<space>.hf.space/debug/pure?q=hydrogen"

Compare raw_first_item with normalized. Title, authors, year, DOI and portal_url should all be filled in. If one is empty, adjust normalize_research_output() in pure_client.py.

LibGuides frontend (full block, LibBee model)

Live page: https://library.ku.ac.ae/khaznabot

libguides/khazna-libguides-block.html is a complete, paste-ready frontend built on the same model as the LibBee block: same KU header, composer, popular searches, status pill, progress and cancel, answer tools, /config announcements and footer cards. All of its HTML, CSS and JS lives in LibGuides; it only calls the Space for data.

  1. In a guide, add a Media / Widget asset (or an HTML/source box, the same as LibBee) and paste the whole file.
  2. To change the backend, edit one line in the script's CONFIG section: var API_URL = "https://nikeshn-kulibrary.hf.space"; (for example, to route through the LibBee Cloudflare gateway).
  3. On the Space, set ALLOWED_ORIGINS to https://library.ku.ac.ae.

What the block adds on top of the LibBee model:

  • Scope chips (Everything, Publications, Theses, Profiles, Departments, Projects, Equipment, Collaborations), sent to the Space as scope. They swipe on phones.
  • Record cards rendered from search_results (publication, person, organisation, dataset). [n] citations in the answer jump to the matching card.
  • Keywords sent to Khazna panel, plus a large "Search Khazna for: …" button: the Khazna versions of LibBee's Boolean panel and PRIMO button.
  • Follow-up chips from follow_up_suggestions that switch scope, e.g. "Who at KU works on this?".
  • Styles are scoped to #ku-khazna-page / .kz- and the keyframes are renamed, so LibBee and Khazna can share a page.

/config (environment variables on the Space): ANNOUNCEMENT shows a banner; MAINTENANCE_MODE=true (with optional MAINTENANCE_MESSAGE) disables asking.

Deep links: https://library.ku.ac.ae/khaznabot?ask=<question>&scope=<all|publications|theses|persons|organisations|projects|equipment|collaborations|datasets> asks the question on load, then removes it from the address bar. libguides/khazna-portal-searchbox.html is a small search box for the Khazna portal (plain HTML with inline styles, no JS) that uses this link.

Alternative: libguides/embed-script.html is a two-line embed that loads widget.js from the Space. It updates automatically on every push, but it is not the LibBee design.

Pure API key: endpoints the bot uses

Used: /research-outputs/*, /student-theses/*, /persons/*, /organizations/*, /external-organizations/*, /projects/*, /equipment/*, /concepts/*. Not used, so they can be removed from the key: /users/*, /roles/*, /changes/*, /applications/*, /awards/*, /journals/*, /publishers/*, /classification-schemes/*, /thesauri/*, /author-collaborations/*, /external-persons/*, /courses/*, /activities/*, /events/*, /pressmedia/*, /prizes/*.

Endpoints

Method Path Purpose
GET / redirects to FRONTEND_URL (https://library.ku.ac.ae/khaznabot)
GET /test the Space's own test chat (widget)
GET /widget.js embeddable widget
POST /agent {question, history, model: "gpt"|"claude", scope, top_k, person_uuid?} β†’ answer, request_id (also an X-Request-ID header), search_results (each has kind: publication / thesis / person / organisation / project / equipment / collaborator / dataset), person, trace, plan, follow_up_suggestions
POST /feedback {request_id, feedback: "up"|"down", session_id} (LibGuides block) or {question, rating} (widget)
GET /config public frontend config: announcement, maintenance mode
GET /health backend status
GET /admin/stats queries, zero-result queries, feedback (header X-Admin-Password)
GET /debug/pure?q=&endpoint= raw vs normalised record; endpoint = research-outputs | student-theses | persons | organizations | projects | equipment | external-organizations | data-sets (admin)

The /agent response uses the same answer and search_results keys as LibBee's /agent, so the LibBee frontend can call it as a "Khazna" source.

Tests

pip install -r requirements.txt pytest && pytest -q tests

These run against a mock Pure API and need no keys.