codehygiene / TELEMETRY.md
brianbellx's picture
Remove Most improved, score every ranked file, and feed ranked files into stats
419a527 verified
|
Raw History Blame Contribute Delete
6.48 kB

Telemetry

The Space keeps a handful of aggregate scoring counters. Source code, file names, paths, user names and IP addresses are never stored, logged or published. Repository names are stored only for public repos that someone explicitly asks to rank (see Leaderboard).

Every number is public at /api/stats, and the leaderboard at /api/leaderboard. Counting logic lives in server/telemetry.py on the server and web/src/telemetry.js in the browser.

What is counted

Counter Counted by When it goes up
files_scored Browser report, checked by the server A file from a repo, folder or snippet is scored for the first time.
issues_found Derived Sum of squiggles_by_kind across scored files.
avg_score_scored Browser report Mean score out of 100 of scored files.
median_score Browser report Median score out of 100 of scored files, from a 101-bin histogram.
score_ms_median, score_ms_samples Browser report Median time the browser took to score one file, and how many files were timed.
by_ext Browser report Files scored per extension and mean score out of 100. Unknown extensions count as other.
by_loc_bucket Browser report Files scored per size band: 0-50, 50-100, 100-300, 300-1000, 1000+ lines.
squiggles_by_kind Browser report Issues flagged in scored files, per kind.
leaderboard_entries Server only Public repos on the leaderboard.

Nothing is counted for the bundled examples, on the page or on the server. The server also ignores the examples' exact contents under any other source.

files_scored and scoring times come from the browser, so they are best effort. The banner at the top of the page shows each number only once its sample is large enough:

Banner item Shown when
Files scored files_scored ≥ 1,000
Issues flagged files_scored ≥ 1,000 and issues_found > 0
Time per file score_ms_samples ≥ 1,000
Leaderboard link leaderboard_entries ≥ 5
Data Retention link Always

The Stats tab of the leaderboard page shows every number from the first count, with its sample size.

The Data Retention link opens /privacy/ (web/src/privacy.html), a plain-language summary of this file for visitors. Any change to what is collected or kept must be reflected there.

What the browser sends

After a repo, folder or snippet is scored locally in the browser, one aggregate request goes to POST /api/telemetry/scan:

{
  "source_kind": "repo",
  "files": [
    {
      "digest16": "9f84c2a10e77b31d",
      "ext": "py",
      "loc_bucket": "100-300",
      "score": 29,
      "squiggles": { "deep_nesting": 2, "swallowed_error": 1 },
      "score_ms": 14
    }
  ]
}
Field Meaning
source_kind repo, folder or snippet. Example runs send nothing.
digest16 First 16 hex characters (64 bits) of SHA-256(ext + "\0" + content), computed in the browser with Web Crypto. Used only to count a file once in memory.
ext Lower-case file extension.
loc_bucket Size band, not the line count.
score Score out of 100.
squiggles Issue counts per kind.
score_ms Milliseconds the browser took to score the file, capped at 60,000. Optional.

Any other field is rejected. A batch holds at most 200 files. Each address may send 10 batches a minute.

What is never kept

  • Source code is scored inside Web Workers in the visitor's browser (23.84 KiB ternary weights) and is never uploaded by the UI.
  • Logs carry no content, file names, folder or repository names, cookies or addresses. The access log is turned off.
  • Rate limiting keys each bucket by a salted hash of the caller's address. The salt is random per process and never stored, and the address itself is dropped after the check.
  • Digests for deduplicating repeated scans live in an in-memory rolling set of at most 100,000 entries and are never written to disk or the dataset.

Where the counters live

Counters are written to stats.json at STATS_PATH (default /tmp/codehygiene_stats.json). When HF_STATS_DATASET_REPO and HF_TOKEN are set, stats.json is read from that Hugging Face Dataset repo on start and written back every 5 minutes and on shutdown. Nothing is uploaded until the existing copy has been read, so a failed read cannot reset the public numbers.

Leaderboard

The leaderboard page (/leaderboard/, opened in a new tab from the banner) ranks public GitHub repos by codehygiene score. A repo is listed only when someone presses the trophy button after loading it, or when it is one of the seed repos in SEED_REPOS in server/leaderboard.py. Scanning a repo on the page does not list it.

The browser sends only the owner and repository name to POST /api/leaderboard/rank. Any other field, a score included, is rejected. The server then:

  1. Reads the repo's metadata from the GitHub API. Private repos and forks are refused. If the API is rate limited, the repo is ranked from its archive alone.
  2. Downloads the default branch as a tar.gz archive from codeload.github.com (at most 80 MB) and reads it in memory. Nothing is written to disk. The commit SHA comes from the archive header.
  3. Keeps hand-written Python, TypeScript, JavaScript, Go and Rust files. Tests, vendored and build folders, minified, generated and type-declaration files, files over 150 KB and binary files are skipped.
  4. Requires at least 5 such files and 1,000 lines.
  5. Scores every kept file in the archive with the Python codehygiene scorer and feeds each file's aggregate digest, extension, size band, score and issue counts into /api/stats. The repo score is the line-weighted mean across all ranked files, out of 100.
  6. Refuses a commit that is already ranked under another name, which catches mirrors and forks.

A ranked repo is reused for 24 hours. Each address may ask for 2 ranks a minute, the whole Space 20, and at most 3 wait at once.

Each entry records the scorer that produced it: the codehygiene version plus the first 12 hex characters of the SHA-256 of weights.npz. Only entries from the running scorer are listed or counted. The Stats tab's "As a reward" figures come from web/src/reward-facts.json, keyed by version.