Add files using upload-large-folder tool

README.md

@@ -10,8 +10,8 @@ tags:
 
 # EGMS-QA · Encoder
 
-Frozen spatio-temporal encoder and its 10,000-tile token cache for **EGMS-QA**,
-a natural-language question-answering system over
+Frozen spatio-temporal encoder checkpoint for **EGMS-QA**, a natural-language
+question-answering system over
 [European Ground Motion Service](https://egms.land.copernicus.eu/) (EGMS)
 persistent-scatterer displacement time series.
 
@@ -25,28 +25,26 @@ Companion repos: [`risenyard/egms-qa-dataset`](https://huggingface.co/datasets/r
 
 ## Files
 
-| file | what it is |
+| path | what it is |
 |---|---|
-| `encoder.pt` | frozen encoder checkpoint (`model` state dict + training args) |
-| `encoder_tokens_10k.pt` | token cache for the 10k tiles: `spatial_tokens [10000,65,256]` (X), `token_mask [10000,65]` (m, empty cells = 0), plus `tile_ids`, `splits`, `point_count_per_bin`, `n_points_per_tile`, `metadata` |
-| `encoder_tokens_10k_metadata.json` | extraction metadata |
-| `manifest/split.parquet` | 10k-tile train/val/test split manifest |
-| `manifest/data_config.json` | data configuration reference |
+| `data/encoder/checkpoint/encoder.pt` | frozen encoder checkpoint (`model` state dict + training args) |
+
+The precomputed **token cache** and the **raw tiles** live in the dataset repo
+[`risenyard/egms-qa-dataset`](https://huggingface.co/datasets/risenyard/egms-qa-dataset)
+(`data/encoder/tokens/`, `data/tiles/`); the split manifest and normalization
+are in the code repo under `data/encoder/`.
 
 ## Use
 
-```python
-from huggingface_hub import hf_hub_download
-import torch
+Load the checkpoint with the model code in the GitHub repo (`egms_encoder`), then
+re-extract tokens from the raw tiles:
 
-tok = torch.load(hf_hub_download("risenyard/egms-qa-encoder", "encoder_tokens_10k.pt"),
-                 map_location="cpu", weights_only=False)
-X, m = tok["spatial_tokens"], tok["token_mask"]   # [10000,65,256], [10000,65]
+```bash
+python -m egms_encoder.extract_tokens \
+    --checkpoint data/encoder/checkpoint/encoder.pt \
+    --manifest   data/encoder/manifest/split.parquet \
+    --output-dir outputs/tokens
 ```
 
-Re-extracting tokens from raw tiles, or loading the checkpoint architecture,
-needs the model code in the GitHub repo (`egms_encoder`). Place these files at
-`data/encoder/…` of a checkout to reproduce end to end (see the repo README).
-
-The raw EGMS Level-3 product is Copernicus data and is not redistributed here.
+Most users can skip this and use the released token cache from the dataset repo.
 Licensed CC-BY-4.0.

data/encoder/checkpoint/encoder.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdb0bd2437c063a79c871eb54445dbee6fb82bd9e20186768fa5b947ffd9d474
+size 79442395