Buckets:
| # t5-stream day shards — daily image requests and pageviews for every T5 target, one folder per year | |
| These folders hold the **day shards** of the T5 stream. They are the per-day inputs that the stream's monthly files add | |
| up. Each shard has a row for **every** target file form with at least one request that day, and **every** target article | |
| with at least one view. The published year folders (`t5-stream/<year>/`) carry daily rows only for groups A and B. These | |
| shards are therefore the only daily record of the two large groups, **Cen** (every file form used in an English | |
| Wikipedia article) and **Cpop** (every single-use file of the 14 other wikis). | |
| | folder | days | files | size | | |
| |---|---|---|---| | |
| | `t5-stream/2019-10_2020-09-cday/` | 2019-10-01 to 2020-09-30 | 731 (365 `c_`, 366 `p_`) + manifest | 45.53 GB | | |
| | `t5-stream/2020-10_2021-09-cday/` | 2020-10-01 to 2021-09-30 | 730 (365 `c_`, 365 `p_`) + manifest | 48.79 GB | | |
| | `t5-stream/2021-10_2022-09-cday/` | 2021-10-01 to 2022-09-30 | 730 (365 `c_`, 365 `p_`) + manifest | 50.57 GB | | |
| Each folder lists its files with their SHA-256 in `MANIFEST.public.sha256`. | |
| ## Verification | |
| After upload, every file of each folder was downloaded back from the bucket and its SHA-256 matched the folder's | |
| manifest. | |
| | folder | files (with manifest) | size | downloaded back and matched (UTC) | SHA-256 of `MANIFEST.public.sha256` | | |
| |---|---|---|---|---| | |
| | `2019-10_2020-09-cday` | 731 + 1 = 732 | 45.53 GB | 2026-11-05 09:47:16 | `78d64880b61e3c2faba6ce6faed7d75978bdeff8a9b0cfaaecb58c8e6f7dfe8a` | | |
| | `2020-10_2021-09-cday` | 730 + 1 = 731 | 48.79 GB | 2026-11-05 10:14:40 | `3230f1f73e70630b52ae5ad00163bd2c789367c66564050c0398468020f0217f` | | |
| | `2021-10_2022-09-cday` | 730 + 1 = 731 | 50.57 GB | 2026-11-05 10:47:36 | `2c3ab035f59be7f3f5898348651db4e36f6fce81df6b31c65ef4d9ab60613feb` | | |
| Files per month, `c_` + `p_` together: | |
| - **2019-10_2020-09:** Oct 62, Nov 60, Dec 62, Jan 62, Feb 58, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 59. | |
| September 2020 has 59 because the 2020-09-10 mediacounts file is missing. | |
| - **2020-10_2021-09:** Oct 62, Nov 60, Dec 62, Jan 62, Feb 56, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 60. | |
| - **2021-10_2022-09:** Oct 62, Nov 60, Dec 62, Jan 62, Feb 56, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 60. | |
| ## Files | |
| | path | what | | |
| |---|---| | |
| | `cday/c_<YYYYMMDD>.tsv.gz` | media requests (Wikimedia `mediacounts`) per target file form, one day; every form with total > 0 | | |
| | `cday/p_<YYYYMMDD>.tsv.gz` | pageviews (Wikimedia `pageview_complete`) per target article, one day; every article with a view | | |
| The columns are those of the year folders. | |
| - **`c_`:** `pidx`, `total`, `original`, `tr_image`, the six width bands `w0_199` … `w1000`, and the referer classes `ref_int`, `ref_ext`, `ref_unk`. | |
| - **`p_`:** `aidx`, `wiki`, `page_id`, `u_desktop`, `u_mweb`, `u_app`, `u_noid`, `a_all`, `a_noid`. | |
| Rows are sorted by key, and the gzip header carries mtime 0. | |
| ## How they relate to the year folders | |
| - **Monthly files:** the `monthly/c_<YYYY-MM>` and `monthly/p_<YYYY-MM>` files of a year folder are the sums of these shards over the month. | |
| - Each `monthly/*.done` lists the SHA-256 of every shard it merged. | |
| - Each `done/<mc|pv>_<YYYYMMDD>.done` names its shard's SHA-256. | |
| - So every shard here can be checked against the year folder's own records. | |
| - **Year gate:** it checks that the shards exist. To re-run it, put a year's shards in `cday/` beside its year folder. | |
| - **Cross-check days:** the year folders keep only the shards of the four cross-check days. All other days are here. | |
| ## Decoding `pidx` and `aidx` | |
| The same lists as the year folders decode them: `t5-stream/targets-v2/` (`t5_pathmap.tsv.gz`, `t5_articles.tsv.gz`, `t5_days.tsv`). | |
| They are shared by every year. ⚠ Those lists and the year folders sit in a **different bucket** from these shards: | |
| `Schroedinger1000/Wikidatavector-derivatives`. | |
| ## Caveats | |
| The year folders' caveats apply unchanged. | |
| - **Missing day:** there is no mediacounts shard for 2020-09-10, because the dumps have no file for that day. It is the only missing day in these three years. | |
| - **No agent split in mediacounts:** crawler requests for images are included. | |
| - **Platform changes:** known platform changes move image counts at known dates. | |
| - **Per-file counts:** a file used on several pages or wikis counts all their requests. | |
| - **Size:** 45–51 GB a year, so start with a month, not a whole year. | |
| ## Provenance & licence | |
| Written on Wikimedia Cloud Services by the `t5_packet` scripts (`t5_series.py`, version t5-series-2.x; each `done/*.done` names its version). They were | |
| built in the same runs as the year folders, from one read of each public dump file. Counts are aggregates per file or | |
| per article per day and contain no personal data. File names and article titles originate in Wikimedia projects. | |
| Released under **CC BY-SA 4.0**. | |
| *Card written by Claude (Anthropic) for Lodewijk Gelauff.* | |
Xet Storage Details
- Size:
- 4.97 kB
- Xet hash:
- c3ae9b0a74f4a275c25b93440bd5f35e25c5b07183655fb848590af7867bc4b8
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.