Hugging Face's logo

Spaces:
HalcyonSolutions
/
KGQA-Statistics-Lab
Running

App Files Community
KGQA-Statistics-Lab / README.md
muttsurini's picture
muttsurini
Initial Push
8a34897 verified
preview code
|
Raw History Blame Contribute Delete
16.3 kB
metadata 
title: KGQA Statistics Lab
emoji: 📊
colorFrom: indigo
colorTo: blue
sdk: static
pinned: false
short_description: KG, QA, and joint KGQA statistics in the browser
datasets:
  - HalcyonSolutions/Kinship
  - HalcyonSolutions/MQuAKE-ST

KGQA Statistics Lab

This static Space computes three complementary views entirely in the browser.

  1. KG analysis

    • Graph size, connectivity, density, degree statistics, and relation/entity summaries.
    • Optional clustering coefficient, eigenvector centrality, and legacy path/hierarchy calculations.

  2. QA analysis

    • QA row counts, hop distribution, split distribution, answer multiplicity, paraphrase availability, and reference-path annotation summaries.

  3. Joint KGQA analysis

    • Exact fixed-horizon random-walk answer calibration over uniform labeled actions and unique outgoing neighbors.
    • Optional explicit SELF_LOOP action at each node to allow remaining at the current entity for one traversal step.
    • Optional Undirected random walk to add virtual reverse traversal actions, useful for MetaQA. This changes only the joint random-walk action space, not the underlying graph metrics.
    • Optional expansion of grouped question variants stored in Question-Paraphrased when canonical records have a Question-Number.
    • Probability of reaching a released answer and of executing an annotated reference path at a fixed horizon.
    • Reachable-endpoint counts from the source entity.
    • Released or reconstructable reference-path structural summaries when available. PQ/PQL Multi-Paths takes precedence over Paths.

Built-in sources

  • Kinship

    • Graph: kg/orig/triplets.txt
    • QA: qa/kinship_qa_nhop.csv

  • MQuAKE-ST

    • Graph: kg/triplets.txt
    • QA Single Answer: qa/single_answers/qa_nhop.csv
    • QA Multi Answer: qa/multi_answers/qa_nhop.csv
    • Relation labels: metadata/relation_data.csv
    • Entity titles: metadata/node_data.csv (shown as Title (QID) in charts and tables)

Google Cloud Storage JSON API (v11)

Kinship and MQuAKE-ST support Hugging Face file downloads or public GCS dataset ZIP archives. MetaQA (encoded) is provided only via GCS. The built-in dataset provider selector applies to both graph and QA selection and is saved in lightweight presets. Existing v10.1/v10.2 JSON presets that lack a provider field default to Hugging Face on import.

Archives and paths:

Dataset Published archive Expected root QA file
Kinship Kinship/kinship_dataset.zip kinship_hinton/ qa/kinship_qa_nhop.csv
MQuAKE-ST MQuAKE_ST/mquake_st_dataset.zip mquake_st_dataset/ qa/single_answers/qa_nhop.csv or qa/multi_answers/qa_nhop.csv
MetaQA MetaQA/metaqa_dataset.zip metaqa_dataset/ qa/metaqa_nhop.csv

The download URL uses Google Cloud Storage's JSON API:

https://storage.googleapis.com/download/storage/v1/b/halcyon_data/o/multihop_ds%2Fdatasets%2FKinship%2Fkinship_dataset.zip?alt=media

Google documents that JSON API requests are CORS-accessible independently of the bucket CORS settings. Public object-read permission is still required. No bucket configuration change or authentication is added to the Space. Downloaded ZIPs are decoded in a Web Worker with locally bundled JSZip. The same archive is fetched once per browser tab session, even across repeated analyses, while the session cache is retained. Hugging Face dataset files and their metadata are also reused. Use Clear cached datasets to release downloads and fetch fresh copies on the next analysis. The cache uses in-memory storage only and is discarded when the page is closed or reloaded, but not when an analysis is canceled. Public archives may be substantially larger than individual Hugging Face files, requiring more browser memory. The GCS endpoint and public object accessibility still need to be tested from the deployed Space. Switching back to Hugging Face remains available for Kinship/MQuAKE-ST.

Source: Google Cloud Storage CORS and JSON API download endpoint.

Graph and QA compatibility

The selected graph restricts which QA collections are available for joint KGQA analysis:

Graph source Available QA sources
Kinship Kinship QA, No QA
MQuAKE-ST MQuAKE-ST Single Answer, MQuAKE-ST Multi Answer, No QA
MetaQA (encoded) MetaQA QA, No QA
Custom graph Custom QA, No QA
No graph Kinship QA, MQuAKE-ST Single Answer, MQuAKE-ST Multi Answer, MetaQA QA, Custom QA, No QA

Incompatible QA options remain visible but disabled. If a graph change makes the current QA source incompatible, the Lab automatically selects the matching QA source. An intentional No QA selection remains unchanged. No Graph supports QA-only analysis with any QA collection, while No QA supports graph-only analysis with any graph. Selecting both No Graph and No QA is not a runnable analysis.

Lightweight analysis presets

Use Export preset and Import preset to save or restore a setup. The Save / Restore Setup panel appears above 01 / Knowledge Graph for quick access.

  • A preset is a small .kgqa-preset.json file containing the selected graph and QA sources, download provider, options, dashboard preferences, column mappings, and references to custom input files. The data files themselves are not included.
  • For local files selected normally, browsers disclose the filename but not the absolute filesystem path. The preset therefore stores a filename reference (and file size). A folder-selected file may also expose a relative path. A local absolute path cannot be read or automatically reopened by a normal browser page.
  • On import, analysis settings are restored immediately. Any required local file that has not already been selected must be reconnected, using Select folder to reconnect or the individual file inputs. The Lab matches filename and byte size, and uses relative paths when available. Multiple ambiguous matches are not selected automatically.
  • Metadata column selections are restored once the corresponding metadata file is reconnected. The Lab clearly identifies outstanding files and does not claim the setup is ready until those references are resolved.
  • Presets do not include results. Files never leave your browser as part of the import/export workflow. Do not rename or move referenced files without reconnecting them.
  • Existing v10 presets in .zip format are not directly compatible with the lightweight .json preset importer. Export a new preset after upgrading.

MetaQA

MetaQA uses the encoded THESEUS graph, QA collection, and metadata files. It has no released reference navigation paths. Path-dependent measures that require those annotations are unavailable. For random-walk comparisons, enable Undirected random walk if the desired evaluation action space contains reverse edges.

Custom inputs

Graph

  • Triplets file with three columns: head, relation, tail.
  • Explicit Entity list / Node labels is one optional input. It accepts one ID per line, or a node metadata CSV such as QID,Title, EID,Name, or EntityID,Label. For metadata CSVs, choose which column is the entity ID and which is the display title. By default, the file is used for labels only and does not add graph nodes. Enable Include metadata-only entities (isolated nodes) if the selected file intentionally defines additional graph nodes.
  • Optional relation-label file. Files such as Property,Title or RID,Title are supported with independent ID and display-label column choices.
  • For plain ID lists, no title is available, so nodes display their entity IDs. The ID-column selection is applied to chart labels. When the opt-in isolated-node setting is enabled, it also determines which metadata IDs are included in the graph analysis.

QA

Recommended columns:

  • Question
  • Source-Entity
  • Answer-Entity
  • Hops

Optional columns:

  • SplitLabel
  • Question-Paraphrased
  • Paths
  • Multi-Paths / Multi-Paths-Label (PQ/PQL released grouped paths)
  • Multi-Paths-Key (complete relation chains, aligned by group)
  • Path-Key
  • Path-Count (released path count)
  • Graph-Answer-Entity / Graph-Answer (graph-consistent alternatives, not released gold)
  • Graph-Multi-Paths / Graph-Multi-Paths-Label / Graph-Path-Count (graph-relative paths)
  • Question-Number
  • Question-Family-ID / Question-Family-Size (PQ/PQL explicit structural family)

Notes

  • All processing happens client-side.
  • By default, graph entities are exactly the head and tail IDs observed in triplets. Entity metadata does not change graph population or statistics. This is important for filtered PQ/PQL graphs whose metadata may list nodes from a much larger source graph.
  • Opting in to metadata-only nodes changes graph size, zero-degree counts, degree distributions, weak components, and density. Use the opt-in only for graph definitions that intentionally include those isolated nodes.
  • PQ/PQL Multi-Paths contains complete paths grouped by relation chain. The Lab flattens and deduplicates released entity paths for reference-state analysis while preserving Path-Count separately. It does not merge Graph-Multi-Paths into the released evidence set.
  • The joint report requires both a graph and a QA collection.
  • The split filter is applied to the QA rows before reporting.
  • For joint random-walk calibration, dead ends are absorbing.

Dashboard metric views

The top dashboard lets you choose between Instance-micro and Family-macro whenever the QA data provides family identifiers and aggregation is available.

  • Instance-micro gives each evaluated QA instance equal weight.
  • Family-macro averages instances within a structural family, then averages those family values equally.
  • Answer (released) evaluates Answer-Entity, the reference-answer set.
  • Graph-Answer (graph-consistent) evaluates Graph-Answer-Entity, when available. These alternatives are derived from the graph and are not independently verified as gold natural-language answers.

The selectors affect top summary cards only. Detailed reports and JSON exports keep the underlying statistics unchanged and include both views. The QA split defaults to Test only. If a QA collection lacks a SplitLabel column, all of its rows are included.

Random-walk settings

  • Use the same graph, benchmark split, and step horizon when comparing results across experiments.
  • Explicit SELF_LOOP actions allow the random walker to stay at its current entity and consume one step.
  • Undirected random walk adds a reverse traversal for each unique directed triple. Original and reverse actions remain distinct when both occur in the graph. Graph statistics always describe the original triples.
  • Expand QA paraphrases starts unchecked. Enable it to expand grouped alternative wordings when question-number families are available. QA collections with an explicit Question-Family-ID, including PQ/PQL, are not duplicated.
  • Random-walk dead ends are absorbing.

Definitions and help

Hover over an underlined metric name for a short definition. Descriptions are also available when focusing supported terms with the keyboard. Key quantities include:

  • RW-Ans@p-actions: fixed-horizon probability of reaching any target answer when uniformly sampling labeled traversal actions.
  • RW-Ans@p-neighbors: the same target probability when uniformly sampling unique neighboring entities.
  • RW-RefPath@p-actions: probability of executing an annotated reference path with any required self-loop padding.
  • Weakly connected component (WCC): a connected region after ignoring graph-edge direction.
  • Reachable-N-Count: number of possible destinations after exactly N random-walk steps.

Numerical scope

The Lab computes graph summaries and exact fixed-horizon random-walk distributions locally in your browser. Graph and QA sizes may affect processing time and memory use. Expensive optional graph metrics may be skipped under built-in size safeguards.

PQ/PQL local QA notes

  • Use Custom QA to upload the reconstructed pq_qa_nhop.csv or pql_qa_nhop.csv.
  • Upload the active filtered or source graph as Custom graph. The chosen graph determines random-walk probabilities and whether released reference paths are executable. Graph-Answer-Entity, Graph-Multi-Paths, and Graph-Path-Count are supplied by the uploaded QA CSV. This Space does not recompute them for a different graph.
  • Answer-Entity is the primary released-answer target. Graph-Answer-Entity is evaluated separately when present and must not be treated as independently verified gold.
  • The Multi-Paths field supplies released paths in grouped form. Multi-Paths-Key is a list of complete relation chains corresponding to the groups.
  • Path-Count and Graph-Path-Count are retained as separate metrics, as are Multi-Paths and Graph-Multi-Paths.
  • Family-macro results use the released Question-Family-ID. PQ/PQL already explicitly represent paraphrase variants as individual rows.

Tooltip behavior

Technical terms use a single delayed description bubble on hover or keyboard focus. The browser-native duplicate tooltip has been removed.

v11.1: session download cache

The worker stays alive after a successful analysis. Built-in GCS ZIP archives and Hugging Face graph/QA files are stored in memory and reused when the same source is analyzed again. The UI shows the number and compressed/downloaded byte size of the cached resources and has a Clear cached datasets button. Clearing the cache ends the idle worker, frees its archives, clears metadata responses, and causes the next run to fetch them again. Reloading or closing the page also drops the cache. Canceling an in-progress analysis preserves completed downloads. This is not browser disk storage, so reopening the application does not retain downloads. Each file is fetched once per session unless cleared. Different providers have separate cache entries. The displayed sizes are download sizes, not a guarantee of total browser memory usage. Custom local uploads are not copied into this download cache.

v11.3: cancellation preserves cached datasets

Canceling an analysis terminates the active calculation Worker so computational work stops immediately. Completed built-in downloads are backed up as Blob objects in page memory and restored to the next Worker, avoiding another fetch. This includes Google Cloud Storage archives and individual Hugging Face dataset files. A download still in progress or an archive not yet validated at the instant of cancellation is not retained. Clear cached datasets is the explicit way to discard them, and reloading or closing the page also releases the cache. No persistent disk storage is used.

v11.3: selective cache deletion and optional random walks

  • Downloaded Hugging Face files and GCS archives have different cache entries even when they represent the same dataset. Switching provider downloads the requested provider's resource once, without discarding the first provider's cache.
  • Each cached resource is listed with its provider, filename and size, and may be deleted separately with Delete. Clear all cached datasets removes the whole in-memory cache. Cache operations are unavailable while an analysis is running. Removing an entry restarts the idle Worker so it cannot reuse a deleted item, while retaining all other completed page-owned downloads.
  • Calculate random-walk probabilities is checked by default, preserving earlier behavior. Uncheck it to skip expensive fixed-horizon random-walk distributions, reachability, and reference-path probabilities, while retaining KG-only, QA-only, joint answer and annotated-path structure metrics. Report fields for skipped random-walk metrics are null rather than zero.
  • Existing saved presets without the new checkbox restore with random-walk calculation enabled. New presets remember the checkbox.
  • The cache is in memory for the lifetime of the page only. Reloading or closing the tab clears it.