lgelauff/wdrop-data/t5-stream - Storage Bucket

t5-stream day shards — daily image requests and pageviews for every T5 target, one folder per year

These folders hold the day shards of the T5 stream. They are the per-day inputs that the stream's monthly files add up. Each shard has a row for every target file form with at least one request that day, and every target article with at least one view. The published year folders (t5-stream/<year>/) carry daily rows only for groups A and B. These shards are therefore the only daily record of the two large groups, Cen (every file form used in an English Wikipedia article) and Cpop (every single-use file of the 14 other wikis).

folder	days	files	size
t5-stream/2019-10_2020-09-cday/	2019-10-01 to 2020-09-30	731 (365 c_, 366 p_) + manifest	45.53 GB
t5-stream/2020-10_2021-09-cday/	2020-10-01 to 2021-09-30	730 (365 c_, 365 p_) + manifest	48.79 GB
t5-stream/2021-10_2022-09-cday/	2021-10-01 to 2022-09-30	730 (365 c_, 365 p_) + manifest	50.57 GB

Each folder lists its files with their SHA-256 in MANIFEST.public.sha256.

Verification

After upload, every file of each folder was downloaded back from the bucket and its SHA-256 matched the folder's manifest.

folder	files (with manifest)	size	downloaded back and matched (UTC)	SHA-256 of MANIFEST.public.sha256
2019-10_2020-09-cday	731 + 1 = 732	45.53 GB	2026-11-05 09:47:16	78d64880b61e3c2faba6ce6faed7d75978bdeff8a9b0cfaaecb58c8e6f7dfe8a
2020-10_2021-09-cday	730 + 1 = 731	48.79 GB	2026-11-05 10:14:40	3230f1f73e70630b52ae5ad00163bd2c789367c66564050c0398468020f0217f
2021-10_2022-09-cday	730 + 1 = 731	50.57 GB	2026-11-05 10:47:36	2c3ab035f59be7f3f5898348651db4e36f6fce81df6b31c65ef4d9ab60613feb

Files per month, c_ + p_ together:

• 2019-10_2020-09: Oct 62, Nov 60, Dec 62, Jan 62, Feb 58, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 59. September 2020 has 59 because the 2020-09-10 mediacounts file is missing.
• 2020-10_2021-09: Oct 62, Nov 60, Dec 62, Jan 62, Feb 56, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 60.
• 2021-10_2022-09: Oct 62, Nov 60, Dec 62, Jan 62, Feb 56, Mar 62, Apr 60, May 62, Jun 60, Jul 62, Aug 62, Sep 60.

Files

path	what
cday/c_<YYYYMMDD>.tsv.gz	media requests (Wikimedia mediacounts) per target file form, one day; every form with total > 0
cday/p_<YYYYMMDD>.tsv.gz	pageviews (Wikimedia pageview_complete) per target article, one day; every article with a view

The columns are those of the year folders.

• c_: pidx, total, original, tr_image, the six width bands w0_199 … w1000, and the referer classes ref_int, ref_ext, ref_unk.
• p_: aidx, wiki, page_id, u_desktop, u_mweb, u_app, u_noid, a_all, a_noid.

Rows are sorted by key, and the gzip header carries mtime 0.

How they relate to the year folders

• Monthly files: the monthly/c_<YYYY-MM> and monthly/p_<YYYY-MM> files of a year folder are the sums of these shards over the month.
  • Each monthly/*.done lists the SHA-256 of every shard it merged.
  • Each done/<mc|pv>_<YYYYMMDD>.done names its shard's SHA-256.
  • So every shard here can be checked against the year folder's own records.
• Year gate: it checks that the shards exist. To re-run it, put a year's shards in cday/ beside its year folder.
• Cross-check days: the year folders keep only the shards of the four cross-check days. All other days are here.

Decoding pidx and aidx

The same lists as the year folders decode them: t5-stream/targets-v2/ (t5_pathmap.tsv.gz, t5_articles.tsv.gz, t5_days.tsv). They are shared by every year. ⚠ Those lists and the year folders sit in a different bucket from these shards: Schroedinger1000/Wikidatavector-derivatives.

Caveats

The year folders' caveats apply unchanged.

• Missing day: there is no mediacounts shard for 2020-09-10, because the dumps have no file for that day. It is the only missing day in these three years.
• No agent split in mediacounts: crawler requests for images are included.
• Platform changes: known platform changes move image counts at known dates.
• Per-file counts: a file used on several pages or wikis counts all their requests.
• Size: 45–51 GB a year, so start with a month, not a whole year.

Provenance & licence

Written on Wikimedia Cloud Services by the t5_packet scripts (t5_series.py, version t5-series-2.x; each done/*.done names its version). They were built in the same runs as the year folders, from one read of each public dump file. Counts are aggregates per file or per article per day and contain no personal data. File names and article titles originate in Wikimedia projects. Released under CC BY-SA 4.0.

Card written by Claude (Anthropic) for Lodewijk Gelauff.