Upload 18 files

DATASET_COLLECTION_GUIDANCE.md

@@ -1,6 +1,6 @@
 # Dataset Collection and Multi-Store Operations Guide
 
-## Schema v3 privacy lifecycle — authoritative
+## Schema v4 privacy and contribution lifecycle — authoritative
 
 > **Run 6 policy:** where older historical sections below describe feedback as
 > training input, cross-source feedback/contribution joins, unversioned consent,
@@ -9,6 +9,19 @@
 - Ordinary `/v1/feedback` is rating telemetry only. Query, answer, note, model,
   page and conversation identity are discarded server-side; durable feedback is
   opt-in and marked `trainingStatus="telemetry"`.
+- Current schema v4 keeps feedback telemetry and explicit content contribution
+  structurally separate. Contribution has two record families: `recordType="qa"`
+  for selected/rated Q&A and `recordType="conversation"` for one ordered
+  `messages[]` record. Whole-conversation mode does not explode turns into
+  independent training rows.
+- The browser's dedicated **Contribute to dataset** sheet is the only content-bearing
+  contribution surface. **This Q&A**, **Rated answers**, and **Whole conversation**
+  converge on the same exact-JSON inspection, privacy preflight, consent, quarantine,
+  receipt-delete, and post-promotion withdrawal lifecycle. Share/export owns no
+  contribution controls.
+- Schema v4 requires contribution consent `2.0.0`. Legacy schema v2/v3 clients may
+  continue using historical consent `1.0.0` for compatibility; that legacy consent
+  is not accepted for schema v4's broader conversation-record contract.
 - `/v1/contribute` requires `consentFlag=true` and the current
   `consentVersion`. Accepted content enters the mutable contribution receipt
   ledger with `trainingStatus="quarantined"`; it is not written directly to
@@ -175,7 +188,7 @@ are readable; Secret values are write-only in the settings interface.
 | `TRAINING_DATASET_REPO` | **Variable** | Legacy HF mode only | Repo ID is normally non-sensitive |
 | `HF_DATASET_TOKEN` | **Secret** | Legacy HF mode | Actual HF credential |
 | `HF_DATASET_TOKEN_TYPE` | **Variable** | Legacy HF mode | Classification only |
-| `FEEDBACK_PERSIST_ENABLED` | **Variable** | Recommended | Boolean config |
+| `FEEDBACK_PERSIST_ENABLED` | **Variable** | Recommended | Server persistence flag; never substitutes for reader telemetry consent |
 
 > **Attention — your existing `TRAINING_DATASET_REPO` Secret works, but it does
 > not normally need to be a Secret.**  
@@ -770,8 +783,12 @@ Both files should contain identical bytes.
 ### Rating telemetry
 
 The browser does not send rating telemetry until the reader explicitly enables
-**Send rating telemetry**. With that preference enabled and
-`FEEDBACK_PERSIST_ENABLED=true`, a rating may produce:
+**Send rating telemetry**. The permission is versioned and stored as a structured
+browser consent record; missing, malformed, stale-version, or historical boolean
+preferences fail closed to **Off**. Every network telemetry request carries
+`telemetryConsent=true`, `telemetryConsentVersion="1.0.0"`, and the grant timestamp;
+the bundled proxy/Worker reject requests that omit the current consent contract.
+With that user permission active and `FEEDBACK_PERSIST_ENABLED=true`, a rating may produce:
 
 ```text
 feedback/YYYY/MM/DD/fb_<recordId>.jsonl
@@ -779,7 +796,10 @@ feedback/YYYY/MM/DD/fb_<recordId>.jsonl
 
 Inspect the row and verify it contains rating/event mechanics only: no query,
 answer, note, model, page URL, or conversation identifier, and
-`trainingStatus="telemetry"`. Storage failure must not break the rating UI.
+`trainingStatus="telemetry"`. Turning telemetry off must stop future rating and
+retraction network requests. Storage failure must not break the local rating UI.
+The public `ai-assistant-feedback` DOM event is content-free for the same reason:
+page listeners receive bounded rating mechanics, not the Q&A/note/model/page tuple.
 
 ### Contribution receipt lifecycle
 
@@ -790,7 +810,7 @@ delete/withdraw capability. The content is training-ineligible at this stage.
 Only an authorized operator review/promotion request may create durable
 `trainingStatus="eligible"` rows. Promotion uses an atomic lifecycle claim so a
 second concurrent reviewer cannot write the same receipt again. After promotion,
-raw Q&A content is cleared from the receipt ledger; lifecycle metadata, server-owned
+raw contributed content is cleared from the receipt ledger; lifecycle metadata, server-owned
 dedup keys, provider paths, and only the delete-capability hash remain.
 
 The same receipt capability remains valid after promotion. DELETE then means
@@ -904,35 +924,31 @@ empty segments, excessive depth, and unsafe names are rejected.
 
 ---
 
-## 14. Why feedback and contributions must be deduplicated
+## 14. Contribution record identity and deduplication
 
-There are two intentional collection paths:
+There are two intentionally separate collection paths:
 
-| Path | Endpoint | Typical folder | Trigger |
+| Path | Endpoint | Typical folder | Content semantics |
 |---|---|---|---|
-| Immediate feedback | `POST /v1/feedback` | `feedback/` | user rates an answer |
-| Explicit contribution | `POST /v1/contribute` | `contributions/` | user consents and contributes conversation data |
+| Rating telemetry | `POST /v1/feedback` | `feedback/` | bounded rating/event mechanics only; never training-eligible |
+| Explicit contribution | `POST /v1/contribute` | `contributions/` | user-reviewed Q&A or one ordered conversation record |
 
-The same Q&A may therefore appear once as feedback and later again as a
-contribution.
+Feedback telemetry no longer carries query/answer content, so it is **not** joined
+with contributions to form training examples. It remains `trainingStatus="telemetry"`
+and is excluded from ordinary training output.
 
-Training both copies would overweight that example.
-
-### Canonical dedup key
-
-```text
-_dedup_key = <conversationId>:<answerIndex>
-```
-
-### Priority
+Contribution dedup keys are server-owned and receipt-scoped rather than stable
+browser conversation identifiers:
 
 ```text
-contribution > feedback
+Q&A record:           <receipt-id>:<answer-index>
+Conversation record:  <receipt-id>:conversation
 ```
 
-For the same `_dedup_key`, the contribution wins.
-
-Within the same source type, the record with the newest server `_ts` wins.
+The same key is reused by later eligible/withdrawal lifecycle records so
+last-write-wins can suppress a withdrawn contribution without exposing a stable
+client identity. Within a key, the newest server `_ts` wins. A whole conversation
+remains one record throughout quarantine, promotion, and withdrawal.
 
 ---
 
@@ -1624,7 +1640,7 @@ conflict resolution could hide corruption or unauthorized modification.
 - exact legacy duplicate records across mirrors suppressed;
 - same canonical record ID + different bytes fails closed;
 - guarded remote archive extraction;
-- historical schema normalization to current schema v3 when `_utils/_dataset_schema.py` is available;
+- historical schema normalization to current schema v4 when `_utils/_dataset_schema.py` is available;
 - default exclusion of feedback telemetry/quarantined/legacy-unreviewed records;
 - training acceptance only for `trainingStatus="eligible"` contributions;
 - tombstone/retraction handling where applicable;

README.md

@@ -146,31 +146,72 @@ caller-supplied forwarding headers.
 
 ### `POST /v1/contribute` — payload schema
 
+The current browser sends **schema v4** with consent **2.0.0**. The endpoint
+supports two explicit contribution record families. A whole conversation is one
+ordered record:
+
 ```json
 {
-  "schemaVersion":  1,
-  "consentFlag":    true,
-  "consentVersion": "v1.0",
-  "sessionId":      "uuid-string",
-  "page":           "https://your-docs-site/index.html",
-  "model":          { "id": "...", "provider": "...", "model": "..." },
+  "schemaVersion": 4,
+  "consentFlag": true,
+  "consentVersion": "2.0.0",
+  "page": "https://your-docs-site/index.html",
+  "model": null,
   "records": [
     {
-      "answerIndex":  0,
-      "query":        "What is a confusion matrix?",
-      "answer":       "A confusion matrix is...",
-      "ratingValue":  2,
-      "ratingLabel":  "👍",
-      "message":      "optional free-text from user",
-      "ts":           1781002584724
+      "recordType": "conversation",
+      "messages": [
+        {"role": "user", "content": "What is a confusion matrix?", "ts": 1781002584000},
+        {
+          "role": "assistant",
+          "content": "A confusion matrix is...",
+          "ts": 1781002584724,
+          "model": {"id": "...", "provider": "...", "model": "..."},
+          "feedback": {"ratingValue": 2, "ratingLabel": "helpful", "note": "optional"}
+        }
+      ],
+      "message": "optional contribution-level reviewer note",
+      "ts": 1781002585000
     }
   ]
 }
 ```
 
-Each accepted record receives a stable content-derived ID, but raw accepted content is **not** written to the provider repositories immediately. It enters the mutable contribution receipt ledger first with `trainingStatus="quarantined"`. Only an independently authorized promotion writes an `eligible` row beneath `contributions/YYYY/MM/DD/<record-id>.jsonl` in the Primary/Mirrors. Only answers the user has explicitly rated are included — unrated answers are skipped by the browser widget before the request is sent.
-
-The receipt's delete capability remains meaningful after promotion. While pending it removes the content from the active review ledger. After promotion the same capability records a privacy-minimal `withdraw` tombstone keyed by the server-owned contribution dedup key and attempts best-effort removal from each provider's current branch/view. Dataset construction applies the withdrawal through last-write-wins and excludes both the withdrawn eligible row and the tombstone itself from training output. Versioned repository history, database pages/WAL remnants, backups, CDN/provider logs, and infrastructure snapshots are **not** claimed physically erased.
+The dedicated **Contribute to dataset** sheet also supports **This Q&A** and
+**Rated answers** scopes. Those produce `recordType="qa"` records using the
+historical `query` / `answer` shape. The current v4 envelope always requires
+`consentVersion="2.0.0"`; legacy schema v2/v3 clients remain accepted with their
+historical `1.0.0` consent for compatibility, but legacy consent is not accepted
+for the broader v4 contract.
+
+Ordinary `/v1/feedback` remains privacy-minimal rating telemetry and never turns
+into content contribution automatically. The browser keeps ratings local unless
+the reader explicitly enables **Send rating telemetry**. Current telemetry uses a
+versioned permission (`telemetryConsent=true`, `telemetryConsentVersion="1.0.0"`)
+and the proxy rejects feedback requests that do not carry that current consent
+contract. Enabling telemetry does not contribute the question, answer, note,
+page, model, or conversation content. Turning telemetry off stops future sends;
+it does not claim erasure of telemetry already accepted by a remote provider.
+
+Each accepted contribution record receives a server receipt-scoped dedup key,
+but raw accepted content is **not** written to provider repositories immediately.
+It enters the mutable contribution receipt ledger first with
+`trainingStatus="quarantined"`. Only an independently authorized promotion writes
+an `eligible` row beneath `contributions/YYYY/MM/DD/<record-id>.jsonl` in the
+Primary/Mirrors. Whole-conversation mode contributes one ordered conversation
+record; rated-answer mode contributes only explicitly rated Q&A records; the
+single-Q&A shortcut contributes only the selected Q&A after the same inspect,
+privacy-review, and consent flow.
+
+The receipt's delete capability remains meaningful after promotion. While
+pending it removes the content from the active review ledger. After promotion
+the same capability records a privacy-minimal `withdraw` tombstone keyed by the
+server-owned contribution dedup key and attempts best-effort removal from each
+provider's current branch/view. Dataset construction applies the withdrawal
+through last-write-wins and excludes both the withdrawn eligible row and the
+tombstone itself from training output. Versioned repository history, database
+pages/WAL remnants, backups, CDN/provider logs, and infrastructure snapshots are
+**not** claimed physically erased.
 
 ---
 
@@ -280,7 +321,7 @@ Effort / Thinking fields and opens a per-model in-memory fallback circuit.
 |---|---|---:|---|
 | `RECORD_STORAGE_TARGETS` | Variable | New multi-store mode | Provider-neutral JSON topology with exactly one Primary and optional Mirrors. Contains token environment-variable **names**, never token values. |
 | `TRAINING_DATASET_REPO` | Variable | Legacy HF mode only | Backward-compatible HF Dataset repo ID. When `RECORD_STORAGE_TARGETS` is present, the explicit target topology is authoritative; this value may remain for rollback/older discovery consumers. |
-| `FEEDBACK_PERSIST_ENABLED` | Variable | No | Opt-in persistence of privacy-minimal `/v1/feedback` telemetry. Default is `false`; feedback is never training-eligible. |
+| `FEEDBACK_PERSIST_ENABLED` | Variable | No | Server-side persistence permission for privacy-minimal `/v1/feedback` telemetry. Default is `false`; it cannot override the browser user-consent gate, and feedback is never training-eligible. |
 | `CONTRIBUTION_REVIEW_TOKEN` | Secret | For promotion | Operator-only token required to promote quarantined contribution intake to durable `trainingStatus=eligible` storage. |
 | `CONTRIBUTION_LEDGER_BACKEND` | Variable | No | `memory` (default), `sqlite` (local transactional/restart-durable), or `redis` (shared transactional authority across replicas in one Redis consistency domain). Redis persistence durability is deployment-conditional and is not inferred by the proxy. |
 | `CONTRIBUTION_LEDGER_SQLITE_PATH` | Variable | When using SQLite | Local SQLite database path. Keep it on deployment-owned persistent storage if restart durability is required. |
@@ -625,7 +666,7 @@ BASE=https://scikit-plots-ai.hf.space
 
 # 1. Liveness probe
 curl $BASE/health
-# {"status":"ok","version":"6.5.2"}
+# {"status":"ok","version":"6.6.1"}
 
 # Optional deterministic stub rig status
 curl -s $BASE/health | python3 -m json.tool
@@ -664,23 +705,23 @@ curl -N $BASE/v1/chat/completions \
 curl $BASE/v1/contribute \
   -H "Content-Type: application/json" \
   -d '{
-    "schemaVersion":  1,
-    "consentFlag":    true,
-    "consentVersion": "v1.0",
-    "sessionId":      "test-session-001",
-    "page":           "https://example.com",
-    "model":          null,
+    "schemaVersion": 4,
+    "consentFlag": true,
+    "consentVersion": "2.0.0",
+    "page": "https://example.com",
+    "model": {"id":"test-model","provider":"test"},
     "records": [{
-      "answerIndex":  0,
-      "query":        "test query",
-      "answer":       "test answer",
-      "ratingValue":  2,
-      "ratingLabel":  "👍",
-      "message":      "",
-      "ts":           1781002584724
+      "recordType": "qa",
+      "answerIndex": 0,
+      "query": "test query",
+      "answer": "test answer",
+      "ratingValue": 2,
+      "ratingLabel": "helpful",
+      "message": "",
+      "ts": 1781002584724
     }]
   }'
-# {"contributed": true, "rows": 1, "recordId": "...", "primary": "...", "mirrors": {...}}
+# {"contributed": true, "status": "quarantined", "rows": 1, "receiptId": "...", "deleteToken": "..."}
 ```
 
 ---

_utils/_dataset_schema.py

@@ -1,4 +1,4 @@
-# scikitplot/_externals/_sphinx_ext/_sphinx_ai_assistant/_hf_spaces_proxy/dataset_schema.py
+# scikitplot/_externals/_sphinx_ext/_sphinx_ai_assistant/_hf_spaces_proxy/_utils/_dataset_schema.py
 #
 # flake8: noqa: D213
 #
@@ -7,16 +7,17 @@
 
 """Canonical schema and normalization for collection records.
 
-Schema v3 separates two data classes:
+Schema v4 separates telemetry from two explicit contribution record families:
 
 * ``feedback`` is privacy-minimal rating telemetry.  Content, model, page and
   conversation identity are discarded even when legacy/direct callers submit them;
   ``trainingStatus`` is always ``telemetry``.
-* ``contribution`` is explicit-content intake.  It carries versioned consent, is
-  initially ``quarantined``, labels model attribution ``client_reported``, and is
-  training-eligible only after an authorised review promotes it.
+* ``contribution`` is explicit-content intake. Q&A records retain the historical
+  ``query``/``answer`` shape while conversation records carry one ordered ``messages``
+  array. Both carry versioned consent, enter ``quarantined`` state, and are
+  training-eligible only after an authorised review promotes them.
 
-Historical v1/v2 rows remain readable through :func:`normalize_record`, but old
+Historical v1/v2/v3 rows remain readable through :func:`normalize_record`, but old
 contributions become ``legacy_unreviewed`` rather than silently entering training.
 Client IP addresses are never dataset fields.  See ``DATASET_COLLECTION_GUIDANCE.md``
 for lifecycle and retention policy.
@@ -40,8 +41,8 @@ logger = logging.getLogger(__name__)
 #: Increment when a breaking field-name change is introduced; additive
 #: changes (new optional columns, wider population of existing columns) bump
 #: this too so consumers can branch on ``schemaVersion`` to know which fields
-#: to expect.  See "Schema version history" above for what changed in v2.
-SCHEMA_VERSION: int = 3
+#: to expect.  See the module docstring and collection guidance for version semantics.
+SCHEMA_VERSION: int = 4
 
 #: Ordered list of canonical column names.  Every stored JSONL row and every
 #: row in the pandas DataFrame will have these columns in exactly this order.
@@ -56,6 +57,7 @@ CANONICAL_COLUMNS: list[str] = [
     "conversationId",  # legacy field; v3 feedback/contribution normalization writes None
     "feedbackId",  # feedback event id only; contributions write None
     # ── Record descriptor ─────────────────────────────────────────────────────
+    "recordType",  # "qa" | "conversation" (feedback writes None)
     "answerIndex",  # 0-based position of answer in the conversation
     "action",  # "rate" | "retract"
     "prevFeedbackId",  # feedbackId of the record this one supersedes/invalidates.
@@ -76,7 +78,8 @@ CANONICAL_COLUMNS: list[str] = [
     "message",  # contribution text only; feedback telemetry writes empty string
     # ── Conversation content ──────────────────────────────────────────────────
     "query",  # contribution user question; feedback telemetry writes empty string
-    "answer",  # contribution model response; feedback telemetry writes empty string
+    "answer",  # Q&A contribution model response; feedback telemetry/conversations write empty string
+    "messages",  # conversation contribution ordered message list; otherwise None
     # ── Model ────────────────────────────────────────────────────────────────
     "model",  # dict | None: normalised 8-key model object (see MODEL_KEYS)
     "modelEvidence",  # None | "client_reported" | "legacy_unverified"
@@ -88,7 +91,7 @@ CANONICAL_COLUMNS: list[str] = [
 ]
 
 #: Required keys for the normalised model sub-object.
-#: Both feedback (3-key shape) and contribution (8-key shape) are expanded to
+#: Legacy/model-bearing contribution shapes are expanded to
 #: this full set; keys absent in the source are filled with ``None``.
 MODEL_KEYS: list[str] = [
     "id",  # canonical model identifier (e.g. "Qwen2.5-Coder-7B-Instruct-hf")
@@ -109,7 +112,10 @@ MODEL_KEYS: list[str] = [
 #: whenever the displayed contribution terms change materially and update the
 #: browser ``CONSENT_VERSION`` in the same run.
 CONSENT_VERSION_ENABLED: bool = True
-RESERVED_CONSENT_VERSION: str = "1.0.0"
+RESERVED_CONSENT_VERSION: str = "2.0.0"
+FEEDBACK_TELEMETRY_CONSENT_VERSION: str = "1.0.0"
+FEEDBACK_TELEMETRY_SCHEMA_VERSION: int = 4
+LEGACY_CONSENT_VERSIONS: frozenset[str] = frozenset({"1.0.0"})
 
 
 def _resolve_consent_version(raw: Any) -> str | None:
@@ -139,8 +145,8 @@ def _resolve_consent_version(raw: Any) -> str | None:
 
     Examples
     --------
-    >>> _resolve_consent_version("1.0.0")
-    '1.0.0'
+    >>> _resolve_consent_version("2.0.0")
+    '2.0.0'
     >>> _resolve_consent_version(None)
     """
     if not CONSENT_VERSION_ENABLED:
@@ -562,6 +568,7 @@ def normalize_feedback_record(
         "_dedup_key": dedup,
         "conversationId": None,
         "feedbackId": feedback_id,
+        "recordType": None,
         "answerIndex": answer_index,
         "action": "retract" if is_retract else "rate",
         "prevFeedbackId": prev_feedback_id,
@@ -575,6 +582,7 @@ def normalize_feedback_record(
         "message": "",
         "query": "",
         "answer": "",
+        "messages": None,
         "model": None,
         "modelEvidence": None,
         "page": "",
@@ -582,6 +590,65 @@ def normalize_feedback_record(
         "ts": payload.get("ts"),
     })
 
+_MAX_CONVERSATION_MESSAGES: int = 100
+_MAX_CONVERSATION_MESSAGE_CHARS: int = 100_000
+_MAX_CONTRIBUTION_NOTE_CHARS: int = 2_000
+
+
+def _bounded_text(value: Any, *, limit: int) -> str:
+    """Return a bounded string for explicit contribution content."""
+    if not isinstance(value, str):
+        return ""
+    return value[:limit]
+
+
+def normalize_conversation_messages(value: Any) -> list[dict[str, Any]]:
+    """Normalize one explicit whole-conversation message array.
+
+    Only ``user`` and ``assistant`` roles are accepted. Error/tool/system rows are
+    deliberately excluded from this training/evaluation contribution family.
+    Per-assistant model and rating metadata remain client-reported evidence.
+    """
+    if not isinstance(value, list):
+        return []
+    out: list[dict[str, Any]] = []
+    for raw in value[:_MAX_CONVERSATION_MESSAGES]:
+        if not isinstance(raw, dict):
+            continue
+        role = raw.get("role")
+        if role not in {"user", "assistant"}:
+            continue
+        content = _bounded_text(raw.get("content"), limit=_MAX_CONVERSATION_MESSAGE_CHARS)
+        if not content:
+            continue
+        item: dict[str, Any] = {
+            "role": role,
+            "content": content,
+            "ts": raw.get("ts") if isinstance(raw.get("ts"), (int, float)) and not isinstance(raw.get("ts"), bool) else None,
+        }
+        if role == "assistant":
+            raw_model = raw.get("model")
+            item["model"] = normalize_model(raw_model) if isinstance(raw_model, dict) else None
+            raw_feedback = raw.get("feedback")
+            if isinstance(raw_feedback, dict):
+                rating = normalize_rating(
+                    raw_feedback.get("ratingValue"), raw_feedback.get("ratingLabel"),
+                    rating_mode=raw_feedback.get("ratingMode"),
+                    rating_title=raw_feedback.get("ratingTitle"), feedback_id=None,
+                )
+                item["feedback"] = {
+                    "ratingValue": raw_feedback.get("ratingValue"),
+                    "ratingSlug": rating["ratingSlug"],
+                    "ratingTitle": rating["ratingTitle"],
+                    "ratingMode": rating["ratingMode"],
+                    "note": _bounded_text(raw_feedback.get("note"), limit=_MAX_CONTRIBUTION_NOTE_CHARS),
+                }
+            else:
+                item["feedback"] = None
+        out.append(item)
+    return out
+
+
 def normalize_contribution_record(
     rec: dict[str, Any],
     *,
@@ -590,12 +657,46 @@ def normalize_contribution_record(
     training_status: str = "quarantined",
     submission_id: str | None = None,
 ) -> dict[str, Any]:
-    """Normalize one explicitly consented contribution record.
+    """Normalize one explicitly consented Q&A or conversation contribution."""
+    if training_status not in {"quarantined", "eligible", "legacy_unreviewed"}:
+        training_status = "quarantined"
+    dedup_base = _safe_id(submission_id) or "pending"
+    declared_type = rec.get("recordType")
+    record_type = "conversation" if declared_type == "conversation" else "qa"
+
+    if record_type == "conversation":
+        messages = normalize_conversation_messages(rec.get("messages"))
+        return _ordered({
+            "schemaVersion": SCHEMA_VERSION,
+            "_source": "contribution",
+            "_ts": server_ts_ms,
+            "_dedup_key": f"{dedup_base}:conversation",
+            "conversationId": None,
+            "feedbackId": None,
+            "recordType": "conversation",
+            "answerIndex": None,
+            "action": "rate",
+            "prevFeedbackId": None,
+            "editCount": 0,
+            "status": "active",
+            "trainingStatus": training_status,
+            "ratingValue": None,
+            "ratingSlug": None,
+            "ratingTitle": None,
+            "ratingMode": None,
+            "message": _bounded_text(rec.get("message"), limit=_MAX_CONTRIBUTION_NOTE_CHARS),
+            "query": "",
+            "answer": "",
+            "messages": messages,
+            "model": None,
+            "modelEvidence": "client_reported_per_message" if any(
+                isinstance(m.get("model"), dict) for m in messages if m.get("role") == "assistant"
+            ) else None,
+            "page": envelope.get("page") or "",
+            "consentVersion": _resolve_consent_version(envelope.get("consentVersion")),
+            "ts": rec.get("ts"),
+        })
 
-    Model/provider metadata is a client report, not verified provenance.
-    ``training_status`` is server-owned and defaults to ``quarantined``; only an
-    authorised review/promotion path may create ``eligible`` records.
-    """
     answer_index = rec.get("answerIndex")
     try:
         answer_index = int(answer_index) if answer_index is not None else None
@@ -606,9 +707,6 @@ def normalize_contribution_record(
         rating_mode=rec.get("ratingMode"), rating_title=rec.get("ratingTitle"),
         feedback_id=None,
     )
-    if training_status not in {"quarantined", "eligible", "legacy_unreviewed"}:
-        training_status = "quarantined"
-    dedup_base = _safe_id(submission_id) or "pending"
     return _ordered({
         "schemaVersion": SCHEMA_VERSION,
         "_source": "contribution",
@@ -616,6 +714,7 @@ def normalize_contribution_record(
         "_dedup_key": f"{dedup_base}:{answer_index}",
         "conversationId": None,
         "feedbackId": None,
+        "recordType": "qa",
         "answerIndex": answer_index,
         "action": "rate",
         "prevFeedbackId": None,
@@ -626,9 +725,10 @@ def normalize_contribution_record(
         "ratingSlug": rating_fields["ratingSlug"],
         "ratingTitle": rating_fields["ratingTitle"],
         "ratingMode": rating_fields["ratingMode"],
-        "message": rec.get("message") or "",
-        "query": rec.get("query") or "",
-        "answer": rec.get("answer") or "",
+        "message": _bounded_text(rec.get("message"), limit=_MAX_CONTRIBUTION_NOTE_CHARS),
+        "query": _bounded_text(rec.get("query"), limit=_MAX_CONVERSATION_MESSAGE_CHARS),
+        "answer": _bounded_text(rec.get("answer"), limit=_MAX_CONVERSATION_MESSAGE_CHARS),
+        "messages": None,
         "model": normalize_model(envelope.get("model")),
         "modelEvidence": "client_reported" if envelope.get("model") else None,
         "page": envelope.get("page") or "",
@@ -665,6 +765,7 @@ def normalize_contribution_withdrawal_record(
         "_dedup_key": key,
         "conversationId": None,
         "feedbackId": None,
+        "recordType": None,
         "answerIndex": answer_index,
         "action": "withdraw",
         "prevFeedbackId": None,
@@ -770,6 +871,7 @@ def normalize_record(raw: dict[str, Any]) -> dict[str, Any]:  # noqa: PLR0912
     # ── Back-fill missing canonical fields (schemaVersion: 1 → 2) ─────────────
     out["schemaVersion"] = SCHEMA_VERSION
     out.setdefault("feedbackId", None)
+    out.setdefault("recordType", "qa" if source == "contribution" else None)
     out.setdefault("action", "rate")
     out.setdefault("prevFeedbackId", None)
     # editCount: None for retraction tombstones (not applicable), 0 for any
@@ -780,6 +882,7 @@ def normalize_record(raw: dict[str, Any]) -> dict[str, Any]:  # noqa: PLR0912
     out.setdefault("message", "")
     out.setdefault("query", "")
     out.setdefault("answer", "")
+    out.setdefault("messages", None)
     out.setdefault("page", "")
     out.setdefault("modelEvidence", "legacy_unverified" if out.get("model") else None)
 

_utils/_shared_logic.py

@@ -220,7 +220,7 @@ __all__ = [  # noqa: RUF022
 # ─────────────────────────────────────────────────────────────────────────────
 
 #: Proxy release version — bump on every breaking change.
-PROXY_VERSION: str = "6.5.2"
+PROXY_VERSION: str = "6.6.1"
 
 #: HuggingFace Inference Providers router base URL (no trailing slash).
 #: Only used for Path 3 (standard provider models) when ``BACKEND_URL`` is

app.py

@@ -110,7 +110,7 @@
 #   RATE_LIMIT_REQUIRE_SHARED When true, local rate limiting is rejected and
 #                       Redis initialization/runtime failure returns HTTP 503.
 #   RATE_LIMIT_REDIS_TIMEOUT_SECONDS Redis operation timeout; clamped 0.25..10 s.
-#   FEEDBACK_PERSIST_ENABLED Persist privacy-minimal rating telemetry only. Default false.
+#   FEEDBACK_PERSIST_ENABLED Persist privacy-minimal consent-gated rating telemetry only. Default false.
 #   CONTRIBUTION_REVIEW_TOKEN Server/operator token required to promote quarantined
 #                       contributions into durable training-eligible storage.
 #   CONTRIBUTION_QUARANTINE_TTL_SECONDS Pending contribution lifetime. Default 86400.
@@ -180,6 +180,9 @@ try:
     from ._utils._dataset_schema import (  # type: ignore[import]
         CONSENT_VERSION_ENABLED,
         RESERVED_CONSENT_VERSION,
+        LEGACY_CONSENT_VERSIONS,
+        FEEDBACK_TELEMETRY_CONSENT_VERSION,
+        FEEDBACK_TELEMETRY_SCHEMA_VERSION,
         normalize_contribution_record,
         normalize_contribution_withdrawal_record,
         normalize_feedback_record,
@@ -188,6 +191,9 @@ except Exception:  # noqa: BLE001
     from _utils._dataset_schema import (  # type: ignore[import]
         CONSENT_VERSION_ENABLED,
         RESERVED_CONSENT_VERSION,
+        LEGACY_CONSENT_VERSIONS,
+        FEEDBACK_TELEMETRY_CONSENT_VERSION,
+        FEEDBACK_TELEMETRY_SCHEMA_VERSION,
         normalize_contribution_record,
         normalize_contribution_withdrawal_record,
         normalize_feedback_record,
@@ -2069,6 +2075,8 @@ async def root() -> JSONResponse:
                 "dataset_repo": None,
                 "contribute_ready": _contribution_pipeline_ready(),
                 "feedback_persist_enabled": FEEDBACK_PERSIST_ENABLED,
+                "feedback_telemetry_schema_version": FEEDBACK_TELEMETRY_SCHEMA_VERSION,
+                "feedback_telemetry_consent_version": FEEDBACK_TELEMETRY_CONSENT_VERSION,
             },
             "tokens": {
                 "hf_token_type": "unknown",
@@ -2430,13 +2438,19 @@ async def contribute(request: Request) -> JSONResponse:
         raise HTTPException(status_code=422, detail="Contribution body must be a JSON object.")
     if payload.get("consentFlag") is not True:
         raise HTTPException(status_code=422, detail="Explicit contribution consent is required.")
-    if payload.get("consentVersion") != RESERVED_CONSENT_VERSION:
+    schema_version = payload.get("schemaVersion")
+    if schema_version not in {2, 3, 4}:
+        raise HTTPException(status_code=422, detail="Unsupported contribution schemaVersion.")
+    consent_version = payload.get("consentVersion")
+    if schema_version == 4:
+        consent_ok = consent_version == RESERVED_CONSENT_VERSION
+    else:
+        consent_ok = consent_version in (LEGACY_CONSENT_VERSIONS | {RESERVED_CONSENT_VERSION})
+    if not consent_ok:
         raise HTTPException(
             status_code=422,
             detail="Consent text changed or is missing. Reload the page and review consent again.",
         )
-    if payload.get("schemaVersion") not in {2, 3}:
-        raise HTTPException(status_code=422, detail="Unsupported contribution schemaVersion.")
     records = payload.get("records")
     if not isinstance(records, list) or not records:
         raise HTTPException(status_code=422, detail="records must be a non-empty list.")
@@ -2462,6 +2476,10 @@ async def contribute(request: Request) -> JSONResponse:
         )
         for rec in records if isinstance(rec, dict)
     ]
+    normalized = [
+        row for row in normalized
+        if row.get("recordType") != "conversation" or bool(row.get("messages"))
+    ]
     if not normalized:
         raise HTTPException(status_code=422, detail="No valid contribution records were supplied.")
     encoded = ("\n".join(json.dumps(r, ensure_ascii=False) for r in normalized)).encode("utf-8")
@@ -3135,9 +3153,10 @@ async def share_delete(share_id: str, request: Request) -> JSONResponse:
 async def feedback(request: Request) -> JSONResponse:
     """Accept privacy-minimal rating telemetry.
 
-    Legacy/direct callers may submit query, answer, comment, model, page or
+    Every request must carry the current explicit telemetry-consent marker.
+    Direct callers may still submit query, answer, comment, model, page or
     conversation identifiers, but the server normalizer discards them before
-    optional persistence.  Persistence is server opt-in
+    optional persistence. Persistence is server opt-in
     (``FEEDBACK_PERSIST_ENABLED=false`` by default) and persisted rows are
     ``trainingStatus=telemetry`` so the training builder excludes them.
     """
@@ -3154,6 +3173,22 @@ async def feedback(request: Request) -> JSONResponse:
             status_code=422, detail="Feedback body must be a JSON object."
         )
 
+    # Network feedback is user-consented telemetry, not an implicit side effect
+    # of clicking a local rating button. Reject legacy/misbehaving clients that
+    # do not carry the current explicit consent marker. This check occurs before
+    # rate-limit/storage work so denied telemetry consumes minimal server state.
+    if payload.get("telemetryConsent") is not True or payload.get("telemetryConsentVersion") != FEEDBACK_TELEMETRY_CONSENT_VERSION:
+        logger.warning(json.dumps({"event": "feedback.consent_required"}))
+        raise HTTPException(
+            status_code=403,
+            detail="Explicit feedback telemetry permission is required.",
+        )
+    if payload.get("schemaVersion") != FEEDBACK_TELEMETRY_SCHEMA_VERSION:
+        raise HTTPException(status_code=422, detail="Unsupported feedback telemetry schema version.")
+    consent_at = payload.get("telemetryConsentAt")
+    if not isinstance(consent_at, (int, float)) or isinstance(consent_at, bool) or consent_at <= 0:
+        raise HTTPException(status_code=422, detail="Feedback telemetry consent timestamp is required.")
+
     # ── Distinguish retraction tombstones from regular ratings ───────────────
     # Retractions are system-generated housekeeping records that invalidate a
     # previous rating in the training dataset.  They carry action="retract" and