Deploy 6b6536c from GitHub

README.md

@@ -26,11 +26,13 @@ participant ──► Space (public URL, no login)
                                              → <RESPONSES_REPO>/responses/listNN/<session>.json
 ```
 
-**Counterbalancing.** Only stimuli with `human_study` true are used. The attention checks are the controls whose image plainly contradicts the text (`role` = `control` with a `control_type`); other controls (such as the "every detail true" condition of F pairs) are study stimuli like the rest. Each list gets `CHECKS_PER_LIST` attention checks (default 2, rotating and balanced across lists) and up to `LIST_SIZE` study images (default 14, for a session of about 9 minutes). Two stimuli never go in the same list when they belong to the same minimal pair (`pair_ids`), reuse the same image (`image_id`) or share the same text, so every comparison is between-subjects and nobody sees a text twice. By default there are just enough lists to show every study stimulus at least once, and at least as many as the largest group of stimuli sharing a text, an image or a pair; `LIST_COUNT` forces a number. The search is deterministic: the same dataset always gives the same lists. They are written to `lists.json` (with how many lists each stimulus is in) and `/api/status` shows the sizes. With the current dataset: 10 lists of 16 images (2 checks + 14). Recruit participants in multiples of the number of lists.
+**Lists.** When every study row of the dataset has a `list` (today: 10 lists of 18 stimuli, 17 in the last, 2 of them attention checks, from the dataset's `tools/assign_lists.py`), the Space uses those lists as they are, and the dataset's `attention_check` flag marks the checks; `LIST_COUNT`, `LIST_SIZE` and `CHECKS_PER_LIST` are then unused. Those lists keep the members of a pair and the uses of an image apart but may give one participant two stimuli with the same text; the Space logs each such case at startup. Without a `list` column, the Space builds the lists itself as follows.
+
+**Counterbalancing.** Only stimuli with `human_study` true are used. The attention checks are the controls whose image plainly contradicts the text (`role` = `control` with a `control_type`); other controls (such as the "every detail true" condition of F pairs) are study stimuli like the rest. Each list gets `CHECKS_PER_LIST` attention checks (default 2, rotating and balanced across lists) and up to `LIST_SIZE` study images (default 14, for a session of about 9 minutes). Two stimuli never go in the same list when they belong to the same minimal pair (`pair_ids`), reuse the same image (`image_id`) or share the same text, so every comparison is between-subjects and nobody sees a text twice. By default there are just enough lists to show every study stimulus at least once, and at least as many as the largest group of stimuli sharing a text, an image or a pair; `LIST_COUNT` forces a number. The search is deterministic: the same dataset always gives the same lists. They are written to `lists.json` (with how many lists each stimulus is in) and `/api/status` shows the sizes. With a dataset without lists, on today's stimuli: 10 lists of 16 images (2 checks + 14). Recruit participants in multiples of the number of lists.
 
 **Session.** Consent, a two-question profile, instructions, one practice image ("A butterfly on a flower", `static/practice/`, not saved; a practice ranking with three images also opens the ranking part), then the list in random order. Each image gets one question, the 1–5 match rating. **Question 2 is turned off for now** (commented out in `static/index.html`, marked "question 2"; the segmentation maps are then not loaded either); its fields stay in the saved trials, with `problem` = null. When on, question 2 asks “Is there anything in the image that doesn’t match the text?”. Answering *Yes* requires selecting at least one part: on images with a segmentation in the dataset (`segmentation/<image_id>.png`, 101 images today), a click selects the whole segment under it (saved as `segments`, with each segment's `category` and `names`, plus one point per segment in `marks`; `marking` = `segments`); on the other images, or when a segmentation is missing, unreadable or empty, it places a point (`marking` = `points`). Category names are never sent to the browser. Question 2 is saved as `problem` (sessions before this change saved the older "should not be there" question as `unwanted`). Leaving or reloading the page asks for confirmation, since it loses the answers. The page shows at once: the images of step 1 load while the participant reads the consent and instructions, those of step 2 on the ranking instructions, and a loading screen appears only if an image is not ready when its turn comes. For each image the session stores its `position` (presentation order) and `rt` (time on the image, from display to *Next*).
 
-**Rankings.** After the ratings, each participant ranks `RANKINGS_PER_LIST` groups (default 5). The groups come from the dataset, `groups/groups.jsonl`: one text and its images (`members`, any number from 2; a group with a single image is skipped), so editing that file and restarting the Space changes the rankings. Images that are only in a group (no `sample_id`) are shown only there. The question is match first, preference second: participants order the images by how well each fits the text and, among images that fit equally well (e.g., where the text leaves a detail open), put first the one they would rather get. They tap the images from best to worst, the last one included, shown in random order. Groups are spread evenly over the lists, never two with the same text in one list (16 groups today, each in about 3 of the 10 lists). They come last so that seeing several images of one text cannot affect the between-subjects ratings. Each ranking saves `set_id` (the `group_id`), `shown` (screen order) and `order` (best first) as `image_id`s, `order_sample_ids` (the matching stimulus, null for group-only images), `rt` and `resets`.
+**Rankings.** After the ratings, each participant ranks `RANKINGS_PER_LIST` groups (default 5). The groups come from the dataset, `groups/groups.jsonl`: one text and its images (`members`, any number from 2; a group with a single image is skipped), so editing that file and restarting the Space changes the rankings. Images that are only in a group (no `sample_id`) are shown only there. The question is match first, preference second: participants order the images by how well each fits the text and, among images that fit equally well (e.g., where the text leaves a detail open), put first the one they would rather get. They tap the images from best to worst, the last one included, shown in random order. Groups are spread evenly over the lists, never two with the same text in one list (16 groups today, each in about 3 lists). They come last so that seeing several images of one text cannot affect the between-subjects ratings. Each ranking saves `set_id` (the `group_id`), `shown` (screen order) and `order` (best first) as `image_id`s, `order_sample_ids` (the matching stimulus, null for group-only images), `rt` and `resets`.
 
 **Progress.** Every page shows the same two-step bar, *1 · Rate images* then *2 · Rank images*, with the image or ranking number under it. The page adapts to phones: images fit the screen height, rankings use 2 columns on a phone and images up to 400 px wide on a large screen (3 per row, 2 × 2 for 4 images).
 
@@ -131,6 +133,14 @@ python analysis/collect_responses.py Enzo-py/partial-prompting-responses --out r
 ```
 Downloads every session, flattens to one row per trial, and adds one column per quality check plus `n_failed_checks` and `excluded`. Rankings go to `rankings.parquet`, one row per ranked image.
 
+To fill the dataset's post-study fields (`family_human` in `pairs/pairs.jsonl`, `ranking_human` in `groups/groups.jsonl`):
+
+```bash
+HF_TOKEN=... python analysis/export_human_results.py Enzo-py/partial-prompting-responses --out human_results
+```
+
+It writes both files in the dataset's format (every line and field kept, ready to replace the dataset's), plus `human/` with the numbers behind them: ratings per stimulus, the test-minus-reference difference with its 95% interval per pair, mean ranks per group, and each session's flags. Same exclusions as the Space. A pair is `sensitivity` when the interval lies below -0.5, `invariance` when it lies within ±0.5, else `open`; null under 10 ratings per member (`--margin`, `--min-ratings`, `--min-rankings`). A local folder of session files works in place of the repo id.
+
 ## Files
 
 ```

app.py

@@ -27,8 +27,9 @@ Environment variables (set them as Space secrets / variables):
   STUDY_CONTACT             optional; researcher contact shown on the consent page
   PID_SALT                  optional secret; saves a salted hash of the Prolific ID (never the ID itself)
   LIST_COUNT, LIST_SIZE     optional; number of lists (default: just enough to show every stimulus) and
-                            study images per list (default 14, at most)
-  CHECKS_PER_LIST           optional; attention checks per list, rotating (default 2)
+                            study images per list (default 14, at most); unused when the dataset has its
+                            own lists (`list` on every row)
+  CHECKS_PER_LIST           optional; attention checks per list, rotating (default 2); same
   RANKINGS_PER_LIST         optional; ranking questions after the ratings (default 5, 0 = none)
 """
 import hashlib
@@ -276,9 +277,11 @@ def in_study(s):
 
 
 def is_attention_check(s):
-    """A control whose text the image plainly contradicts (`control_type`: absent, relation, ...): its
-    answer is a rating of 1. Other controls (e.g. "every detail true") are the control condition of a
-    pair and are counterbalanced like any other stimulus. Older datasets mark the check as case E1."""
+    """The dataset's `attention_check` flag when it has one. Older datasets (and sessions saved with
+    them): a control whose text the image plainly contradicts (`control_type`: absent, relation, ...),
+    or case E1; other controls are the control condition of a pair, counterbalanced like the rest."""
+    if "attention_check" in s:
+        return bool(s["attention_check"])
     return s.get("role") == "control" and bool(
         s.get("control_type") or s.get("subcategory") == "Attention check" or s.get("case_id") == "E1")
 
@@ -343,13 +346,34 @@ def design_lists(ids, neighbours, n_lists, size, fixed=None, seed=0, restarts=10
     return best[1] if best else None
 
 
+def dataset_lists(study):
+    """The lists the dataset defines (`list` on every study row, from its tools/assign_lists.py), or
+    None when it does not. Stimuli that conflict (see `conflict`) in one list are only reported."""
+    if not study or any(not isinstance(s.get("list"), int) for s in study):
+        return None
+    lists = defaultdict(list)
+    for s in study:
+        lists[s["list"]].append(s)
+    for k, l in sorted(lists.items()):
+        for i, a in enumerate(l):
+            for b in l[i + 1:]:
+                if conflict(a, b):
+                    print(f"Note: list {k} has {a['sample_id']} and {b['sample_id']} together (same text, image or pair)")
+    return [sorted(s["sample_id"] for s in lists[k]) for k in sorted(lists)]
+
+
 def build_lists(stimuli):
-    """Counterbalanced lists: CHECKS_PER_LIST attention checks plus up to LIST_SIZE study images each,
-    never two conflicting stimuli together (see `conflict`), so each comparison is between-subjects.
+    """The dataset's own lists when every study row has a `list`. Otherwise counterbalanced lists built
+    here: CHECKS_PER_LIST attention checks plus up to LIST_SIZE study images each, never two
+    conflicting stimuli together (see `conflict`), so each comparison is between-subjects.
     Returns (lists, attention checks, stimuli not in the study).
     """
     study = [s for s in stimuli if in_study(s)]
     checks = sorted(s["sample_id"] for s in study if is_attention_check(s))
+    not_shown = [s["sample_id"] for s in stimuli if not in_study(s)]
+    given = dataset_lists(study)
+    if given:
+        return given, checks, not_shown
     items = sorted(s["sample_id"] for s in study if not is_attention_check(s))
     neighbours = {s["sample_id"]: set() for s in study}
     for a in study:
@@ -375,7 +399,6 @@ def build_lists(stimuli):
             break
     else:
         check_lists, lists = [[]], [[]]
-    not_shown = [s["sample_id"] for s in stimuli if not in_study(s)]
     return [sorted(c + l) for c, l in zip(check_lists, lists)], checks, not_shown
 
 
@@ -402,8 +425,8 @@ LIST_RANKINGS = assign_rankings(RANKING_SETS, len(LISTS))
     {"n_lists": len(LISTS), "attention_checks": CHECKS, "not_in_study": NOT_IN_STUDY,
      "exposure": dict(sorted(EXPOSURE.items())), "lists": LISTS,
      "ranking_sets": RANKING_SETS, "list_rankings": LIST_RANKINGS}, indent=2))
-print(f"{len(LISTS)} lists of {sorted({len(l) for l in LISTS})} images, {min(CHECKS_PER_LIST, len(CHECKS))} attention "
-      f"checks each; {len(NOT_IN_STUDY)} stimuli not in the study (human_study = false); "
+print(f"{len(LISTS)} lists of {sorted({len(l) for l in LISTS})} images ({'from the dataset' if all(isinstance(s.get('list'), int) for s in STUDY) else 'built here'}), "
+      f"{sorted({sum(i in CHECKS for i in l) for l in LISTS})} attention checks each; {len(NOT_IN_STUDY)} stimuli not in the study (human_study = false); "
       f"{len(RANKING_SETS)} ranking groups, {len(LIST_RANKINGS[0])} per list")