Synth Health Guardrails: Provenance-Preserving Safety Data for Healthcare AI
The release contains 13,713 fully synthetic member messages spanning ordinary healthcare requests, emergency detection, escalation, synthetic PHI/PII, off-topic content, toxicity, jailbreaks, and prompt injection.
No customer data, production traffic, support transcripts, or real PHI/PII were used.
Why we built it
Healthcare assistants need to distinguish between messages that can proceed normally and those requiring escalation or refusal.
Building these guardrails requires more than a collection of adversarial prompts. The data should also represent differences in health literacy, technical comfort, privacy sensitivity, trust, communication style, care-navigation behavior, emotional state, and urgency.
Most safety datasets preserve only the final text and label. Synth Health Guardrails also retains the structured synthetic context that produced each message.
Every released row includes:
- Original and final message text
- Safety category, severity, and routing decision
- A versioned SYNTH trait snapshot
- Stable synthetic-person and source-row identities
- Current situation and trajectory context
- Generation and transformation lineage
- Structured critic decisions
- Original generation target and independent label judgments
Traits are never inferred from message text when their provenance is unavailable.
What is SYNTH?
SYNTH is League Labs’ provenance-first system for generating synthetic users, situations, and conversations for model development and evaluation.
It is built around a simple principle: synthetic text is more useful when we retain the structured context that produced it.
A SYNTH case begins with a versioned simulation state containing:
- A stable synthetic-person identifier
- Demographic attributes such as age band
- Health and technology literacy
- Privacy sensitivity, trust orientation, and risk tolerance
- Communication and care-navigation styles
- The person’s current goal, health topic, emotional state, constraints, and urgency
- Prior simulated interactions and relevant memory
- A trajectory describing how the situation may evolve
- A versioned generation recipe and safety contract
The generator receives this state as conditioning context. It must produce a natural member message that expresses the relevant traits and situation without explicitly listing them.
The structured context remains separate from the generated text. This makes it possible to measure guardrail behavior across behavioral and situational dimensions without attempting to infer those dimensions after generation.
Each field also retains provenance. Attributes sampled from a distribution, retrieved from a conditioning corpus, or supplied by a versioned recipe remain distinguishable. Missing traits are marked explicitly rather than invented.
For Synth Health Guardrails, aggregate CDC PLACES statistics contribute only health-topic sampling priors. They affect how frequently topics appear in the synthetic population; no CDC row or individual record is reproduced.
System design
The release pipeline separates simulation, generation, curation, adjudication, and publication:
┌──────────────────────────────┐
│ Aggregate CDC PLACES priors │
│ Health-topic weighting only │
└──────────────┬───────────────┘
▼
┌─────────────────────────────────────────────────────────────────┐
│ SYNTH simulation layer │
│ │
│ Person ID ─► stable traits ─► current situation │
│ └─────► memory / trajectory ─► safety contract │
│ │
│ Output: versioned trait snapshot + field-level provenance │
└───────────────────────────────┬─────────────────────────────────┘
│ immutable conditioning context
▼
┌─────────────────────────────────────┐
│ Candidate generation │
│ Command A+ and Nemotron 3 Super │
└──────────────────┬──────────────────┘
▼
┌─────────────────────────────────────┐
│ Validation and lexical deduplication│
└──────────────────┬──────────────────┘
▼
┌─────────────────────────────────────┐
│ BGE-M3 semantic deduplication │
│ Starting cosine threshold: 0.96 │
└──────────────────┬──────────────────┘
│
┌───────────────┴─────────────────┐
▼ ▼
┌──────────────────────┐ ┌─────────────────────────┐
│ Strong unique rows │ │ Redundant or low-value │
│ Preserved unchanged │ │ candidates │
└──────────┬───────────┘ └────────────┬────────────┘
│ ▼
│ ┌─────────────────────────┐
│ │ Targeted complexification│
│ │ Message text only │
│ └────────────┬────────────┘
└──────────────────┬────────────────┘
▼
┌─────────────────────────────────────┐
│ Independent critic checks │
│ │
│ Label and severity fidelity │
│ Trait consistency and realism │
│ Attack authenticity and novelty │
│ Structural and privacy safety │
└──────────────────┬──────────────────┘
▼
┌─────────────────────────────────────┐
│ Technical release candidate │
│ 19,478 provenance-preserving rows │
└──────────────────┬──────────────────┘
│
┌─────────────┴─────────────┐
▼ ▼
┌────────────────────┐ ┌────────────────────┐
│ Gemini 3.6 Flash │ │ Claude Opus 4.8 │
│ Blind labeler │ │ Blind adjudicator │
└──────────┬─────────┘ └─────────┬──────────┘
└─────────────┬─────────────┘
▼
┌─────────────────────────────────────┐
│ Exact consensus filter │
│ Category + severity + decision │
│ 13,713 rows retained │
└──────────────────┬──────────────────┘
▼
┌─────────────────────────────────────┐
│ Public-release sanitization │
│ │
│ Remove raw critic traces │
│ Scan for identifiers and secrets │
│ Preserve structured provenance │
└──────────────────┬──────────────────┘
▼
┌─────────────────────────────────────┐
│ Immutable release │
│ Splits, manifest, hashes and license│
└─────────────────────────────────────┘
Generation and curation
SYNTH generated candidate messages using two source models: Cohere Command A+ and NVIDIA Nemotron 3 Super.
The candidate pool passed:
- Schema and safety-contract validation
- Exact and normalized lexical deduplication
- BGE-M3 semantic deduplication
- Targeted complexification of redundant and low-information cases
- Independent quality and safety criticism
Complexification was deliberately selective. The model could rewrite only the member-message text. The following remained immutable:
- Trait snapshot and synthetic-person identity
- Source lineage
- Intended category and severity
- Routing decision
- Safety contract
- Recipe and seed provenance
Every rewrite attempt was retained in the private transformation lineage. A rewrite was accepted only when critic checks confirmed label fidelity, trait consistency, attack authenticity where applicable, realism, structural safety, and novelty.
This process produced a 19,478-row technical release candidate, containing 1,800 targeted rewrites and 17,678 untouched rows.
Independent label adjudication
The published labels are not simply the labels requested during generation.
Gemini 3.6 Flash and Claude Opus 4.8 independently classified every final message using the same rubric. Both judges were blind to the generation target, source model, and SYNTH trait snapshot.
A row entered the release only when both models agreed exactly on:
- Guardrail category
- Severity
- Routing decision
This retained 13,713 rows, or 70.4% of the technical release candidate.
Of these:
- 9,176 consensus labels differ from the generation target
- 4,537 consensus labels confirm the generation target
When the consensus differs from the generation target, the consensus becomes the provided release label. The original target and both model judgments remain available under label_adjudication.
This preserves the distinction between three separate questions:
- What scenario did SYNTH intend to create?
- What message did the generator actually produce?
- How did independent models classify the observable final message?
Model agreement is evidence of consensus—not human, clinician, or legal ground truth.
Dataset splits
| Split | Rows | Intended use |
|---|---|---|
train |
9,887 | Model fitting |
dev_calibration |
680 | Calibration and threshold selection |
test_public |
2,050 | Held-out general evaluation |
test_red_team |
1,096 | Held-out adversarial evaluation |
| Total | 13,713 |
Red-team labels are distributed for reproducible scoring, but the split should remain excluded from training, prompt selection, and threshold tuning.
The 5,765 rows without exact adjudicator agreement are excluded from the public release and retained privately for future ambiguity and boundary-case research.
Safety and privacy
All people and situations represented in the dataset are fictitious. Sensitive identifiers use explicit synthetic markers.
Before release, we scanned the dataset for credentials, secrets, real-looking contact information, government identifiers, payment cards, URLs, internal hostnames, and machine-local paths.
Raw critic model traces were removed from the public dataset. Structured critic decisions remain available for auditing, while complete raw traces are retained only in private provenance artifacts.
Limitations
Synth Health Guardrails is not:
- Clinical ground truth
- A medical-advice dataset
- Representative of a particular customer or population
- A substitute for application-level safety validation
- A human-adjudicated benchmark
Requiring exact model consensus also introduces selection bias. Genuinely ambiguous and difficult cases are underrepresented, and two foundation models can share rubric interpretations or blind spots.
Using the dataset
from datasets import load_dataset
dataset = load_dataset(
"LeagueLabs/synth-health-guardrails",
revision="shg_consensus_labels-v2",
)
The dataset uses CDLA-Permissive-2.0. Commercial and non-commercial use, modification, and sharing are permitted. The license does not restrict computational results such as trained models, but its complete text must accompany redistributed data.
Explore the dataset on the League Labs Hugging Face Hub.