Spaces:

Enzo-py
/

partial-prompting-study

Running

App Files Files Community

partial-prompting-study / README.md

Enzo-py

Deploy 89e598f from GitHub

8279c9a verified 5 days ago

preview code

Raw History Blame Contribute Delete

17 kB

metadata

title: Image and Text Study
emoji: 🖼️
colorFrom: blue
colorTo: gray
sdk: docker
app_port: 7860
pinned: false
short_description: Human study on unmentioned content in text-to-image prompts

Partial prompt study: annotation Space

Collection interface for the partial-prompt human study. Participants see a text and an image, give a 1–5 match rating, and say whether anything in the image doesn't match the text (selecting those parts). Built with jsPsych; a small FastAPI backend loads the stimuli from the dataset Enzo-py/partial-prompting-ds, assigns counterbalanced lists and saves responses to a private dataset repo.

How it works

startup: download DATASET_REPO (imagefolder or parquet) → write web copies of the images to stimuli_cache/ (WebP, at most 1024 px), build the lists

participant ──► Space (public URL, no login)
                 ├─ POST /api/start   → assigns the least-filled list, returns its stimuli (no labels)
                 ├─ jsPsych trials     → random order, response times per trial
                 └─ POST /api/submit  → writes data/listNN/<session>.json
                                          └─ CommitScheduler pushes data/ every 2 min
                                             → <RESPONSES_REPO>/responses/listNN/<session>.json

Lists. When every study row of the dataset has a list (today: 10 lists of 18 samples, 17 in the last, 2 of them attention checks, from the dataset's tools/assign_lists.py), the Space uses those lists as they are; LIST_COUNT, LIST_SIZE and CHECKS_PER_LIST are then unused. The attention checks are the samples with category = attention_check (older datasets: an attention_check flag, or role = control with a control_type). The dataset's lists never put the same image, text or series twice in one list; the Space logs any list that does at startup. Without a list column, the Space builds the lists itself as follows.

Counterbalancing. Only stimuli with human_study true are used. Each list gets CHECKS_PER_LIST attention checks (default 2, rotating and balanced across lists) and up to LIST_SIZE study images (default 14, for a session of about 9 minutes). Two stimuli never go in the same list when they belong to the same series (series_id; pair_ids in older datasets), reuse the same image (image_id) or share the same text, so every comparison is between-subjects and nobody sees a text twice. By default there are just enough lists to show every study stimulus at least once, and at least as many as the largest group of stimuli sharing a text, an image or a pair; LIST_COUNT forces a number. The search is deterministic: the same dataset always gives the same lists. They are written to lists.json (with how many lists each stimulus is in) and /api/status shows the sizes. With a dataset without lists, on today's stimuli: 10 lists of 16 images (2 checks + 14). Recruit participants in multiples of the number of lists.

Session. Consent, a two-question profile, instructions, one practice image ("A butterfly on a flower", static/practice/, not saved; a practice ranking with three images also opens the ranking part), then the list in random order. Each image gets one question, the 1–5 match rating. Question 2 is turned off for now (commented out in static/index.html, marked "question 2"; the segmentation maps are then not loaded either); its fields stay in the saved trials, with problem = null. When on, question 2 asks “Is there anything in the image that doesn’t match the text?”. Answering Yes requires selecting at least one part: on images with a segmentation in the dataset (segmentation/<image_id>.png, 101 images today), a click selects the whole segment under it (saved as segments, with each segment's category and names, plus one point per segment in marks; marking = segments); on the other images, or when a segmentation is missing, unreadable or empty, it places a point (marking = points). Category names are never sent to the browser. Question 2 is saved as problem (sessions before this change saved the older "should not be there" question as unwanted). Leaving or reloading the page asks for confirmation, since it loses the answers. The page shows at once: the images of step 1 load while the participant reads the consent and instructions, those of step 2 on the ranking instructions, and a loading screen appears only if an image is not ready when its turn comes. For each image the session stores its position (presentation order) and rt (time on the image, from display to Next).

Rankings. After the ratings, each participant ranks the groups of the dataset (groups/groups.jsonl) that belong to their list: each group names its two lists (lists), so 4 groups per list today (20 groups). Without lists, the Space spreads RANKINGS_PER_LIST groups (default 5) evenly over the lists, never two with the same text in one list. An image group (axis = image) shows one text and its images; a prompt group (axis = prompt) shows one image and its texts. Members that are only in a group (no sample_id) are shown only there. The question is match first, preference second: participants order the members by how well each fits and, among images that fit a text equally well (e.g., where the text leaves a detail open), put first the one they would rather get. They tap the members from best to worst, the last one included, shown in random order. The rankings come last so that seeing several images of one text cannot affect the between-subjects ratings. Each ranking saves set_id (the group_id), axis, shown (screen order) and order (best first) as the members' ids (image_id, or the prompt in a prompt group), order_sample_ids (the matching stimulus, null for members ranked only), rt and resets.

Progress. Every page shows the same two-step bar, 1 · Rate images then 2 · Rank images, with the image or ranking number under it. The page adapts to phones: images fit the screen height, rankings use 2 columns on a phone and images up to 400 px wide on a large screen (3 per row, 2 × 2 for 4 images).

Quality checks. Each attention check rated as matching counts as one failed check: 3 or more when the named object is absent, 4 or more for a wrong colour, count, relation or binding, since "Partly" (3) is an honest answer there (CHECK_FAIL_AT). Checks are judged with the labels saved in the session, so later dataset changes do not affect past sessions. 25% or more of the images answered in under 1 s also counts as a failed check, and so do 99% or more identical ratings. A session is excluded from the statistics and the feedback once it fails 2 checks; a single one is a warning in the dashboard (FAST_TRIAL_MS, FAST_SHARE, SAME_ANSWER_SHARE, EXCLUDE_AT in human_export.py, shared by the Space and the export; analysis/collect_responses.py applies the same rules). Excluded sessions still count towards list progress.

Prolific IDs are not saved. The ID is used only to show the completion code. The session stores source (prolific or direct) and, if the PID_SALT secret is set, pid_hash: a salted hash that lets you find the session of a given Prolific submission (to reject or bonus it) by hashing its ID with the same salt, without the ID being in the data.

Taxonomy labels stay server-side. The browser only receives sample_id, prompt and the image. Labels are attached to the saved file on the server.

Deploy

Create the responses repo (private): hf repos create Enzo-py/partial-prompting-responses --repo-type dataset --private. The Space also creates it on first save if it does not exist.
Create a token with write access to that repo (Settings → Access Tokens; a fine-grained token restricted to the repo is safest).
Create the Space: New Space → SDK Docker → blank → visibility Public (participants must reach it without an account).

In the Space's Settings → Variables and secrets, add:

Name	Type	Value
`HF_TOKEN`	Secret	the write token
`DATASET_REPO`	Variable	optional, default `Enzo-py/partial-prompting-ds`
`DATASET_REVISION`	Variable	optional, a branch, tag or commit to pin (default `main`)
`DATASET_SPLIT`	Variable	optional, default `train`
`RESPONSES_REPO`	Variable	`Enzo-py/partial-prompting-responses`
`PROLIFIC_COMPLETION_CODE`	Secret	optional, the code from your Prolific study
`ADMIN_KEY`	Secret	protects `/api/status` and enables the `/admin` dashboard
`STUDY_VERSION`	Variable	optional, e.g. `pilot-v1`
`STUDY_CONTACT`	Variable	optional, shown on the consent page, e.g. `Name, institution, email`
`PID_SALT`	Secret	optional, any long random string; saves a hash of the Prolific ID instead of nothing
`LIST_COUNT` / `LIST_SIZE`	Variable	optional, number of lists (default: enough to show every stimulus) and study images per list (default 14, at most)
`CHECKS_PER_LIST`	Variable	optional, attention checks per list (default 2)
`RANKINGS_PER_LIST`	Variable	optional, ranking questions after the ratings (default 5, 0 = none)

Deploy. Pushing to main on GitHub deploys this folder automatically (see the repository README). To deploy by hand from the repository root:
```
hf upload Enzo-py/partial-prompting-study space . --repo-type space --delete="*"
```
Check https://enzo-py-partial-prompting-study.hf.space/api/status?key=ADMIN_KEY. It should show the dataset commit and stimulus count, and "saving_to": "Enzo-py/partial-prompting-responses".

Prolific

Study URL:

https://enzo-py-partial-prompting-study.hf.space/?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

With PROLIFIC_COMPLETION_CODE set, Prolific participants see the code at the end and a Back to Prolific button (no automatic redirect, so they have time to read their feedback). Without Prolific parameters (colleagues, lab pilot), the page just thanks them.

Using a new version of the dataset

The stimuli are read once, when the Space starts. After you push new images or prompts to the dataset, restart the Space (Settings → Restart). Every saved session records the exact dataset commit (dataset.revision), so responses from different versions stay distinguishable. For the real study, pin DATASET_REVISION to a tag or commit so the stimuli cannot change during collection.

Dataset columns used. The dataset can be an imagefolder (metadata.jsonl with a file_name column next to the image files, the current layout) or parquet files with the images embedded. Required: sample_id (unique), prompt and the image. Optional: list (the annotation list), category and level (attention checks and their kind of error), series_id and image_id (to build lists when the dataset has none). Every other column is copied as a label into the saved responses, server-side only, except the human results and the long descriptive fields (mean_human_score, details_human_score, objects, description, generation_*, segmentation).

Before launching

A free Space goes to sleep after a period without visits and takes a few seconds to wake up. Open the Space yourself just before you publish the study.
Responses are written locally first and pushed every 2 minutes. A restart can lose at most the last 2 minutes; wait for the commit before restarting or redeploying during data collection.
Sample ids of an older dataset may name other samples now. The dashboard, the feedback and the export count a rating only when its saved text is the sample's current text, and a ranking only when it ranked the group's current members.

Saved data (one file per session)

{
  "session_id": "…", "list_id": 5, "study_version": "pilot-v1",
  "dataset": {"repo": "Enzo-py/partial-prompting-ds", "revision": "<commit sha>", "split": "train"},
  "participant": {"source": "prolific", "pid_hash": "3f9a…", "study_id": "…", "english": "fluent", "t2i": "sometimes"},
  "trials": [{
    "position": 1, "sample_id": "S09", "prompt": "an apple on a table",
    "rating": 4, "problem": "yes", "marks": [{"x": 0.147, "y": 0.180}],
    "rt": 5310, "rt_first_rating": 1820, "rating_changes": 1,
    "category": "interference", "level": "fg_simple", "series_id": "IMG-APPLE", "hypothetical_score": 5, "list": 3, "...": "…"
  }],
  "rankings": [{"set_id": "GR-INT-APPLE", "axis": "image", "prompt": "an apple", "position": 1,
                "shown": ["g09_apple_white", "g31_apple_on_tree", "g35_kitchen_small_apple", "g08_apple_other_apples"],
                "order": ["g31_apple_on_tree", "g09_apple_white", "g35_kitchen_small_apple", "g08_apple_other_apples"],
                "order_sample_ids": ["S198", "S102", "S…", "S101"], "rt": 9400, "resets": 0}],
  "total_time_ms": 240000, "client": {"user_agent": "…", "viewport": "…"}, "received_at": "…"
}

Click marks are normalized to the image (0–1 on each axis).

Feedback to participants

After their answers are saved, participants see a short comparison with the other participants: how far their ratings were on average from the others' (in points), on how many images they were within 1 point of the others' average, and the image where they differed most (shown only for a gap of at least 1 point). It is requested separately (POST /api/feedback), so it can never delay or change the saved answers.

Safeguards: it is not announced beforehand, so it cannot push anyone to guess the "expected" answer; it is worded as a comparison, never as a score; controls are never used, so the attention check stays hidden; excluded sessions are left out of the comparison; a stimulus counts only once at least 5 other participants rated it, and nothing is shown below 3 comparable stimuli (FEEDBACK_MIN_OTHERS, FEEDBACK_MIN_ITEMS in app.py). The first participants therefore see no feedback. The feedback is not saved.

Dashboard

https://enzo-py-partial-prompting-study.hf.space/admin shows the study while it runs: sessions completed and in progress, progress per list (click a list to see its images and rankings), exclusions, mean ratings by category and level next to the expected score, each series side by side, the rankings (mean rank of each image or text), a table of every sample (list, category and level, expected score, rating distribution, response time) and the recent sessions.

The page asks for ADMIN_KEY and stays closed until that secret is set. It reads the sessions from RESPONSES_REPO plus the ones not pushed yet, refreshed at most once a minute (the Refresh button forces it). Excluded sessions count in the progress but not in the statistics; the page shows each session's failed checks, as a warning below 2.

Analysis

python analysis/collect_responses.py Enzo-py/partial-prompting-responses --out responses.parquet

Downloads every session, flattens to one row per trial, and adds one column per quality check plus n_failed_checks and excluded. Rankings go to rankings.parquet, one row per ranked image or text.

To fill the dataset's post-study fields (mean_human_score and details_human_score in data/metadata.jsonl, ranking_human and details_ranking_human in groups/groups.jsonl), click Export results on the admin page: it downloads a zip with every saved session, including the ones not pushed to the responses repo yet. The same files from a terminal:

HF_TOKEN=... python analysis/export_human_results.py Enzo-py/partial-prompting-responses --out human_results

It writes both files in the dataset's format (every line and field kept, ready to replace the dataset's), plus human/sessions.jsonl with each session's flags. Same exclusions as the Space. Each rating gives one entry of details_human_score (annotator = the session id, score, list, duration_s) and mean_human_score is their mean; each ranking gives one entry of details_ranking_human, and ranking_human orders the members by mean rank. A local folder of session files works in place of the repo id. The button and the script share space/human_export.py, so they give the same files.

Files

app.py               backend (dataset loading, lists, assignment, saving)
human_export.py      quality checks and the results export (admin button and analysis/export_human_results.py)
static/index.html    jsPsych experiment
static/admin.html    dashboard (/admin)
static/vendor/       jsPsych 7.3.4 (vendored, no CDN dependency)
scripts/             collect_responses.py

Local test

pip install -r requirements.txt
uvicorn app:app --port 7860      # downloads the dataset; responses go to ./data without RESPONSES_REPO