Spaces:

tungpth
/

visual-doc-rag

Running on Zero

App Files Files Community

visual-doc-rag / docs /DATA.md

tungpth

Deploy slim ZeroGPU bundle with pinned model startup downloads (part 6)

86e797b verified 2 days ago

preview code

Raw History Blame Contribute Delete

6.74 kB

# ViDoRe V3 HR: verified data contract

Inspected locally on 2026-10-19 with `uv run vdr download --subset hr` and
`uv run vdr inspect`. Statistics below are computed from downloaded Parquet tables,
not inferred from the dataset card.

## Source and reproducibility

- Dataset: [vidore/vidore_v3_hr](https://huggingface.co/datasets/vidore/vidore_v3_hr).
- Snapshot: `0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3`.
- Four configurations, each with a single `test` split; no new train/dev split.
- Downloaded Parquet size: 442,735,193 bytes (442.74 MB / 422.23 MiB).
- Original PDFs are available upstream but are not needed or downloaded: the corpus
  already contains page images and extracted markdown.
- Raw files: `data/hr/snapshots/<revision>/<config>/*.parquet`.
- Provenance: `data/hr/manifest.json`; full observed schema/statistics:
  `data/hr/inspection.json`. Both are local, ignored artifacts.

To reproduce this snapshot explicitly:

```bash
uv run vdr download --subset hr --revision 0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3
uv run vdr inspect
```

## Counts and suitability

| Item | Observed |
| --- | ---: |
| Documents | 14 |
| Page images / retrieval candidates | 1,110 |
| Queries, all languages | 1,908 |
| English queries (`language == "english"`) | 318 |
| Qrels, all languages | 10,386 |
| Qrels for English queries | 1,731 |
| Distinct relevant pages for English queries | 613 |
| Pages with nonempty markdown | 1,098 |
| Pages with empty/missing markdown | 12 |
| Total markdown characters | 3,443,946 |

Each of `english`, `french`, `spanish`, `italian`, `german`, `portuguese` has
318 queries. Filter by the actual language field; do not infer language from ID
ranges. The loader accepts normalized `english` or `en`.

All 318 English queries have positive relevance judgments. All page/query IDs are
unique, qrel pairs are unique, all qrel references exist, and all corpus documents
have metadata. All nine implemented integrity checks passed.

Document metadata lists charts in 13 documents, tables in 10, and images in 8.
English qrel content labels include Text (1,654), Chart (234), Table (206),
Infographic (65), Image (13), Other (3), and `N/A (If relevance score=0)` (1).
Labels can overlap, so these counts must not be added as disjoint categories.
All observed scores are nevertheless 1 or 2: that single N/A label is an upstream
annotation inconsistency. Preserve it and the original scores unchanged.

Six evidence-selected pages were decoded and manually reviewed in
`data/hr/samples/contact-sheet.jpg`: they contain dense tables, line charts,
scatter plots and pie charts. All six sampled images are 1654 × 2339 pixels;
dimensions of the remaining pages were not measured during initial inspection. These inspection JPEGs are
previews, not the full-resolution inputs for visual indexing.

The subsequent full PNG export measured every page: 967 are 1654 × 2339,
1 is 1653 × 2339, 54 are landscape 2339 × 1654, 5 are 1386 × 1969,
33 are 1386 × 1970, and 50 are 1387 × 1970. Original-size RGB PNGs total
503,571,673 bytes and live under `data/pages/hr/<revision>/`; the page manifest
records each image's dimensions and source attribution.

**Decision:** keep HR. It has useful visual evidence, supplied BM25 text, and a
manageable initial download. Actual visual-index size and hosting fit still need
measurement in the next milestone. HR here means EU employment/labour reports,
not an internal employee handbook; future demo examples should match this domain.

## Observed schemas

### `corpus` — 1,110 rows

| Field | Type | Meaning |
| --- | --- | --- |
| `corpus_id` | int64 | Stable page ID; join to qrels |
| `image` | Hugging Face Image | Embedded page image; decoded as PIL on access |
| `doc_id` | string | Source document; join to document metadata |
| `markdown` | string | Provided OCR/parsed page text for BM25 |
| `page_number_in_doc` | int64 | Source page position; observed range 0–201 |

Use `corpus_id` as the retrieval/citation identifier, never the row offset or
`page_number_in_doc`. Do not assume printed page numbers equal stored positions.
Keep all 1,110 pages as retrieval candidates, including pages without qrels and
the 12 without text. An empty text page contributes an empty BM25 document.

### `queries` — 1,908 rows

| Field | Type |
| --- | --- |
| `query_id` | int64 |
| `query`, `language` | string |
| `query_types` | list[string] |
| `query_format` | string |
| `content_type`, `raw_answers` | list[string] |
| `query_generator`, `query_generation_pipeline` | string |
| `source_type`, `query_type_for_generation` | string |
| `answer` | string |

`raw_answers` contains reference answers and `answer` the combined reference
answer. Neither is used for retrieval indexing or v1 answer evaluation. Benchmark
query/answer text is not copied into committed docs or demo examples.

### `qrels` — 10,386 rows

| Field | Type |
| --- | --- |
| `query_id`, `corpus_id`, `score` | int64 |
| `content_type` | list[string] |
| `bounding_boxes` | list[struct{annotator, x1, x2, y1, y2: int64}] |

Join `queries.query_id → qrels.query_id → corpus.corpus_id`. Use an ID mapping,
not positional indexing. Scope qrels to the selected English query IDs.

Observed score counts across all languages: 1 → 8,700; 2 → 1,686. The
[dataset card](https://huggingface.co/datasets/vidore/vidore_v3_hr/blob/0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3/README.md)
defines 1 as critically relevant (partial evidence) and 2 as fully relevant.
Keep graded scores for NDCG and treat scores greater than zero as relevant for
recall. Never treat unjudged pages as additional positive labels.
Bounding boxes are preserved but unused in v1; their coordinate conventions
have not been validated for drawing overlays.

### `documents_metadata` — 14 rows
| Field | Type |
| --- | --- |
| `file_name`, `doc_id`, `url`, `doc_type`, `doc_language` | string |
| `doc_year` | int64 |
| `visual_types` | list[string] |
| `page_number` | int64 |
| `license` | string |

## Licensing and attribution

The pinned dataset card licenses annotations, qrels and generated metadata under
**CC BY 4.0**. Source documents and parsed markdown inherit the individual
publishers' licenses, as recorded in `documents_metadata.license`; all 14 observed
entries are `cc-by-4.0`.

For demo redistribution, retain ViDoRe attribution, source document URLs and
publisher attribution, link [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
and state image resizing/compression. Keep per-document provenance with exported
demo artifacts. No raw dataset, benchmark queries/answers, or images belong in Git.

Benchmark reference: [Loison et al., ViDoRe V3 (2026)](https://arxiv.org/abs/2601.08620).

	# ViDoRe V3 HR: verified data contract

	Inspected locally on 2026-10-19 with `uv run vdr download --subset hr` and
	`uv run vdr inspect`. Statistics below are computed from downloaded Parquet tables,
	not inferred from the dataset card.

	## Source and reproducibility

	- Dataset: [vidore/vidore_v3_hr](https://huggingface.co/datasets/vidore/vidore_v3_hr).
	- Snapshot: `0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3`.
	- Four configurations, each with a single `test` split; no new train/dev split.
	- Downloaded Parquet size: 442,735,193 bytes (442.74 MB / 422.23 MiB).
	- Original PDFs are available upstream but are not needed or downloaded: the corpus
	already contains page images and extracted markdown.
	- Raw files: `data/hr/snapshots/<revision>/<config>/*.parquet`.
	- Provenance: `data/hr/manifest.json`; full observed schema/statistics:
	`data/hr/inspection.json`. Both are local, ignored artifacts.

	To reproduce this snapshot explicitly:

	```bash
	uv run vdr download --subset hr --revision 0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3
	uv run vdr inspect
	```

	## Counts and suitability

	\| Item \| Observed \|
	\| --- \| ---: \|
	\| Documents \| 14 \|
	\| Page images / retrieval candidates \| 1,110 \|
	\| Queries, all languages \| 1,908 \|
	\| English queries (`language == "english"`) \| 318 \|
	\| Qrels, all languages \| 10,386 \|
	\| Qrels for English queries \| 1,731 \|
	\| Distinct relevant pages for English queries \| 613 \|
	\| Pages with nonempty markdown \| 1,098 \|
	\| Pages with empty/missing markdown \| 12 \|
	\| Total markdown characters \| 3,443,946 \|

	Each of `english`, `french`, `spanish`, `italian`, `german`, `portuguese` has
	318 queries. Filter by the actual language field; do not infer language from ID
	ranges. The loader accepts normalized `english` or `en`.

	All 318 English queries have positive relevance judgments. All page/query IDs are
	unique, qrel pairs are unique, all qrel references exist, and all corpus documents
	have metadata. All nine implemented integrity checks passed.

	Document metadata lists charts in 13 documents, tables in 10, and images in 8.
	English qrel content labels include Text (1,654), Chart (234), Table (206),
	Infographic (65), Image (13), Other (3), and `N/A (If relevance score=0)` (1).
	Labels can overlap, so these counts must not be added as disjoint categories.
	All observed scores are nevertheless 1 or 2: that single N/A label is an upstream
	annotation inconsistency. Preserve it and the original scores unchanged.

	Six evidence-selected pages were decoded and manually reviewed in
	`data/hr/samples/contact-sheet.jpg`: they contain dense tables, line charts,
	scatter plots and pie charts. All six sampled images are 1654 × 2339 pixels;
	dimensions of the remaining pages were not measured during initial inspection. These inspection JPEGs are
	previews, not the full-resolution inputs for visual indexing.

	The subsequent full PNG export measured every page: 967 are 1654 × 2339,
	1 is 1653 × 2339, 54 are landscape 2339 × 1654, 5 are 1386 × 1969,
	33 are 1386 × 1970, and 50 are 1387 × 1970. Original-size RGB PNGs total
	503,571,673 bytes and live under `data/pages/hr/<revision>/`; the page manifest
	records each image's dimensions and source attribution.

	Decision: keep HR. It has useful visual evidence, supplied BM25 text, and a
	manageable initial download. Actual visual-index size and hosting fit still need
	measurement in the next milestone. HR here means EU employment/labour reports,
	not an internal employee handbook; future demo examples should match this domain.

	## Observed schemas

	### `corpus` — 1,110 rows

	\| Field \| Type \| Meaning \|
	\| --- \| --- \| --- \|
	\| `corpus_id` \| int64 \| Stable page ID; join to qrels \|
	\| `image` \| Hugging Face Image \| Embedded page image; decoded as PIL on access \|
	\| `doc_id` \| string \| Source document; join to document metadata \|
	\| `markdown` \| string \| Provided OCR/parsed page text for BM25 \|
	\| `page_number_in_doc` \| int64 \| Source page position; observed range 0–201 \|

	Use `corpus_id` as the retrieval/citation identifier, never the row offset or
	`page_number_in_doc`. Do not assume printed page numbers equal stored positions.
	Keep all 1,110 pages as retrieval candidates, including pages without qrels and
	the 12 without text. An empty text page contributes an empty BM25 document.

	### `queries` — 1,908 rows

	\| Field \| Type \|
	\| --- \| --- \|
	\| `query_id` \| int64 \|
	\| `query`, `language` \| string \|
	\| `query_types` \| list[string] \|
	\| `query_format` \| string \|
	\| `content_type`, `raw_answers` \| list[string] \|
	\| `query_generator`, `query_generation_pipeline` \| string \|
	\| `source_type`, `query_type_for_generation` \| string \|
	\| `answer` \| string \|

	`raw_answers` contains reference answers and `answer` the combined reference
	answer. Neither is used for retrieval indexing or v1 answer evaluation. Benchmark
	query/answer text is not copied into committed docs or demo examples.

	### `qrels` — 10,386 rows

	\| Field \| Type \|
	\| --- \| --- \|
	\| `query_id`, `corpus_id`, `score` \| int64 \|
	\| `content_type` \| list[string] \|
	\| `bounding_boxes` \| list[struct{annotator, x1, x2, y1, y2: int64}] \|

	Join `queries.query_id → qrels.query_id → corpus.corpus_id`. Use an ID mapping,
	not positional indexing. Scope qrels to the selected English query IDs.

	Observed score counts across all languages: 1 → 8,700; 2 → 1,686. The
	[dataset card](https://huggingface.co/datasets/vidore/vidore_v3_hr/blob/0cdf0979f2c5a0fd3e335e6373b9da48a9fe3bc3/README.md)
	defines 1 as critically relevant (partial evidence) and 2 as fully relevant.
	Keep graded scores for NDCG and treat scores greater than zero as relevant for
	recall. Never treat unjudged pages as additional positive labels.
	Bounding boxes are preserved but unused in v1; their coordinate conventions
	have not been validated for drawing overlays.

	### `documents_metadata` — 14 rows

	\| Field \| Type \|
	\| --- \| --- \|
	\| `file_name`, `doc_id`, `url`, `doc_type`, `doc_language` \| string \|
	\| `doc_year` \| int64 \|
	\| `visual_types` \| list[string] \|
	\| `page_number` \| int64 \|
	\| `license` \| string \|

	## Licensing and attribution

	The pinned dataset card licenses annotations, qrels and generated metadata under
	CC BY 4.0. Source documents and parsed markdown inherit the individual
	publishers' licenses, as recorded in `documents_metadata.license`; all 14 observed
	entries are `cc-by-4.0`.

	For demo redistribution, retain ViDoRe attribution, source document URLs and
	publisher attribution, link [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
	and state image resizing/compression. Keep per-document provenance with exported
	demo artifacts. No raw dataset, benchmark queries/answers, or images belong in Git.

	Benchmark reference: [Loison et al., ViDoRe V3 (2026)](https://arxiv.org/abs/2601.08620).