clinical-agent-memory-bench's picture
Upload 99 files
750b88d verified
Raw History Blame Contribute Delete
42.2 kB
"""Prompt templates of the construction pipeline.
The LLM steps of pipeline.llm fill these templates: ${PDESC}, ${NVISITS}, ${PFIRST} and ${PLAST} are
the patient's facts; ${AXIS1_TARGET}, ${AXIS1_FORM_MIX} and ${AXIS2_TARGET} are the question counts of steps
axis1 and axis2; ${SEEDS}, ${b.path} and ${m.path} mark where the step's input is inlined;
${b.<field>} and ${m.<field>} take per-batch values. pipeline.questions uses the axis-3 and axis-4 rules, and
pipeline.dialogues the dialogue prompts.
"""
EXTRACT_RUBRIC = """You extract a structured event record from one clinic visit of a synthetic health record. The
notes are English clinical shorthand, sometimes with a short gloss in parentheses.
Read the visit's S, O, A and P notes and break the clinically meaningful events into short itemized
strings. Put each event in exactly one field; no event may appear in two fields. Route each event by
its meaning, because raw notes are messy: a diagnosis may sit in O and a result in S. In ambiguous
cases, decide by the note section the event came from and by the rules below.
Fields, each with the note section it mainly comes from:
- diagnosis (A): the named diagnoses and disease states that the doctor assesses at this visit. A
finding written as a diagnosis in O also counts.
- A symptom or complaint on its own, such as pain, numbness, weakness or dizziness, is not a
diagnosis; it goes to patient_self_description.
- A rule-out (r/o), a suspected diagnosis or an impression is not confirmed; it goes to
doctor_extra_notes.
- A comorbidity or past diagnosis mentioned only as background, not assessed at this visit, goes
to patient_self_description as "history of X".
- patient_self_description (S): what the patient reports: their own symptoms and complaints at this
visit, their self-reported past history ("history of X"), and their self-reported lifestyle and
activity, for example "walks daily" or "<daily activity>". A family member's condition,
the patient's social or economic situation and the patient's occupation are not the patient's
symptoms; they go to doctor_extra_notes.
- tests_and_medication (O, A and P): results of instruments, labs and imaging, such as CT, MRI,
ultrasound, X-ray, endoscopy, blood tests and viral loads, including findings written as a
sentence; and the medications and treatments named, with drug names and doses. The doctor's
bedside exam findings do not belong here.
- doctor_notes (O): the doctor's own bedside exam findings and measured vitals: palpation, ROM, MMT,
inspection, auscultation, mental-status exam, blood pressure and pulse. Lab and imaging results go
to tests_and_medication. A bedside maneuver such as "<maneuver> test:+" belongs here even
though it is called a test.
- doctor_plans (P): the actions the doctor will take: orders, new prescriptions and medication changes
(see the medication rule below), procedures to arrange, follow-up and referrals. Assistive devices
that are ordered, such as a splint, brace or sling, also belong here.
- health_advice (P, or wherever the doctor advises): the counseling the doctor gives on lifestyle,
diet, exercise, smoking, alcohol, self-care or health education, for example "<lifestyle advice>"
or "advise regular exercise". An activity that the patient reports doing goes to
patient_self_description instead.
- doctor_extra_notes (anywhere in the note): the supplementary layer, a flat list of short strings
with no categories. It holds the context and the reasons, not the facts themselves:
- the patient's social, economic and psychological situation, worries and occupation, and a family
member's condition or caregiving;
- differential reasoning: r/o, suspect, impression, cannot exclude;
- communication with the patient or family: what was explained, informed consent, refusal of
treatment;
- special reasons for a medication, such as a coverage or authorization justification or off-label
use, and the opinion behind a referral or of a multidisciplinary team;
- uncertain or atypical findings flagged to watch;
- administrative items: certificates, insurance, monitoring reminders.
A plain diagnosis, symptom, test, exam finding or plan goes to its own field instead.
Treatment toxicity grading, in oncology, radiation oncology and chemotherapy follow-up:
- A graded checklist of treatment side effects, such as CTCAE grades, is the doctor's periodic
assessment of adverse effects. It is neither a diagnosis nor a lab result. Keep the whole checklist
together in doctor_notes, and do not split it across diagnosis and tests_and_medication.
- The checklist includes hematologic grades such as "<cytopenia>(<grade>)", which stay here even
though they come from blood counts, because the graded form is the physician's assessment and not
a raw lab value. It also includes symptom grades such as "<side effect>(<grade>)", items that
start with "<treatment> toxicity - ...", and any measured value inside the checklist, such as
"<measure>: <value>" or "<measure>(<value> -> <value>)".
- Keep in diagnosis only the cancer diagnosis and its stage, for example "<cancer>: <TNM stage>",
and any genuinely new condition diagnosed during treatment, for example "<new infection>". Do not
move these into doctor_notes.
Rules:
- Extract only what this visit's note states, and never invent anything.
- Never expand an abbreviation into a fuller diagnosis or body part. Keep abbreviations such as
"GERD" or "<abbreviation>" exactly as written, and do not guess which body part or diagnosis an
ambiguous abbreviation stands for.
- Keep laterality, numeric values, dates, drug names and shorthand exactly as written.
- A medication that the patient takes or receives, with its name and dose, goes to
tests_and_medication. A change that the doctor decides on, such as starting, stopping, increasing
or switching a drug, goes to doctor_plans. Do not record the same drug and dose in both fields.
- The S note may carry a running, dated history. Include the standing problem list and this visit's
complaints, but do not add claims about resolution, onset or timing that are not written.
- Leave an empty field as an empty array. Keep a gloss in parentheses when it is part of a term."""
EXTRACT_TASK = """
The visit as JSON, with the fields visit_number, date, department, icd and soap (S, O, A, P):
${m.path}
Extract the event record of visit ${m.visit_number} only, and set visit_number to ${m.visit_number}."""
EXTRACT_SCHEMA = """{
type: 'object', additionalProperties: false,
required: ['visit_number', 'diagnosis', 'patient_self_description', 'tests_and_medication',
'doctor_notes', 'doctor_plans', 'health_advice', 'doctor_extra_notes'],
properties: {
visit_number: { type: 'integer' },
diagnosis: { type: 'array', items: { type: 'string' } },
patient_self_description: { type: 'array', items: { type: 'string' } },
tests_and_medication: { type: 'array', items: { type: 'string' } },
doctor_notes: { type: 'array', items: { type: 'string' } },
doctor_plans: { type: 'array', items: { type: 'string' } },
health_advice: { type: 'array', items: { type: 'string' } },
doctor_extra_notes: { type: 'array', items: { type: 'string' } },
},
}"""
EVIDENCE_RUBRIC = """You build the evidence chain: validity windows for each clinical entity within one department of a
single patient's longitudinal chart.
The batch JSON in the user message has these fields:
- department
- dept_last_date and patient_last_date
- visit_dates: a map from visit_number to date, for this department's visits
- facts: a list of items {field, fact, visits, first_date, last_date}, where visits lists the
visit numbers that recorded the fact
The facts were extracted visit by visit, and many are carried forward from one visit to the next.
"field" is the source bucket: diagnosis, patient_self_description, tests_and_medication,
doctor_notes, doctor_plans, health_advice or doctor_extra_notes.
Your job is to rebuild clinical timelines in three steps.
1. Canonicalize. Merge fact variants that refer to the same clinical entity, and take the union of
their visit lists. For example:
- "left <abbreviation> pain" and "left <abbreviation> pain decrease" -> entity
"left <abbreviation> pain"
- every "<lab>..." fact -> "<lab> level"
- a recurring differential "r/o <condition>" -> one entity
- "<drug> coverage justification..." -> one entity
Keep abbreviations exactly as written in the entity name. Do not expand or translate
"<abbreviation>" or "GERD", and never guess the body part an ambiguous abbreviation stands for.
2. Segment. Split each entity into validity windows over time. A new segment starts whenever the
entity's value or status changes, for example:
- <lab> not detected -> <value> -> not detected, which gives three segments
- a change of lab value, stage or dose
- a symptom that resolves and later returns
- a toxicity grade that rises from 1 to 2 to 3
A carried-forward repeat of the same value extends the current segment.
3. Describe each segment with:
- value: the status or value held, close to its source wording
- valid_from: {date, visit_number} of the first visit that asserts it
- valid_until: {date, visit_number}, or "current"
- status: current, resolved, superseded or historical
- supporting_visits: the visit numbers inside the window
- superseded_by: {value, date, visit_number}, or null
Rules:
- Use only the facts and visit_dates given. Map every visit number to its date through visit_dates.
Keep numeric values and units as written, and do not invent anything.
- Statuses:
- "current": the entity's latest value is still asserted at or near dept_last_date, and no later
value contradicts it.
- "resolved": the record states that the problem ended or healed. A problem that improved but is
still present stays "current", with a value that says it improved.
- "superseded": a later value of the same entity replaced this one; give it in superseded_by.
- "historical": an earlier value that was neither replaced nor stated to have ended.
- Dates never decrease across an entity's segments, and valid_until is never before valid_from.
- aspect is one of disease_state, symptom, test_result, medication, procedure_or_followup,
constraint_or_safety, lifestyle_social, care_plan, exam_finding, health_advice,
differential_impression, communication_consent, med_rationale_referral, admin, other. Choose it by
the entity's meaning, with the source field as a hint:
- health_advice -> health_advice
- an r/o or suspected diagnosis in doctor_extra_notes -> differential_impression
- a medication or coverage rationale, or a referral -> med_rationale_referral
- consent, refusal or an explanation given -> communication_consent
- a certificate, insurance or a monitoring reminder -> admin
- social context, occupation or a worry -> lifestyle_social
- Cover the meaningful entities of the batch. Every fact here was recorded at two or more visits, or is
one reading of a test or measurement recorded at two or more visits.
Keep anything that persists or changes, including the recurring differentials, rationales and
advice of the supplementary fields."""
EVIDENCE_TASK = """
The batch:
${b.path}
Department: ${b.department}, with ${b.n_facts} facts that recur across visits."""
EVIDENCE_SCHEMA = """{
type: 'object',
properties: {
timelines: { type: 'array', items: {
type: 'object',
properties: {
entity: { type: 'string' },
aspect: { type: 'string', enum: ['disease_state','symptom','test_result','medication','procedure_or_followup','constraint_or_safety','lifestyle_social','care_plan','exam_finding','health_advice','differential_impression','communication_consent','med_rationale_referral','admin','other'] },
source_field: { type: 'string' },
segments: { type: 'array', items: {
type: 'object',
properties: {
value: { type: 'string' },
valid_from: { type: 'object', properties: { date: { type: 'string' }, visit_number: { type: 'integer' } }, required: ['date','visit_number'] },
valid_until: {},
status: { type: 'string', enum: ['current','resolved','superseded','historical'] },
supporting_visits: { type: 'array', items: { type: 'integer' } },
superseded_by: {},
},
required: ['value','valid_from','valid_until','status','supporting_visits','superseded_by'],
}},
current_status: { type: 'string' },
notes: { type: 'string' },
},
required: ['entity','aspect','source_field','segments','current_status'],
}},
},
required: ['timelines'],
}"""
TAGS_RUBRIC = """You tag visits of ONE patient's longitudinal EHR event-chain for a memory benchmark. For EACH item you are given a CURRENT visit's structured facts and its "prev" = the UNION of the up-to-2 most-recent prior visits IN THE SAME DEPARTMENT (t-1, t-2). Judge the current visit ONLY against prev -- you are NOT given any other visit (this is a strict sliding window; do not assume anything outside it).
Output per visit:
- tag: "change" if the current visit contains ANY new or changed clinical content vs prev -- a new symptom / new test ordered / new test result / new reaction / new diagnosis / new medication or procedure / new doctor note or plan / new health advice given / new differential or extra note, OR a changed value/status of an existing item. Otherwise "no_change".
- noise_score:
3 = pure repeat / nothing new at all;
2 = mostly repeat, with only a minor no-value-change new instance;
1 = a real but modest change;
0 = dense with new information.
- note: one short line -- what is new/changed, or "repeat of prev".
Rules:
- If is_first_in_dept is true (prev is empty), tag="change" and noise_score=0.
- A test/measure REPEATED with a genuinely NEW result instance but the SAME value still counts as tag="change" with noise_score 2.
- Compare by clinical meaning, not exact string; carried-forward identical text = not new.
- Ignore pure administrative boilerplate.
- Judge EVERY visit_number in the batch exactly once."""
TAGS_TASK = """
Read the JSON array of items at:
${b.path}
Each item has: visit_number, date, department, is_first_in_dept, current_facts{diagnosis,patient_self_description,tests_and_medication,doctor_notes,doctor_plans,health_advice,doctor_extra_notes}, prev_facts[] (the t-1,t-2 same-dept visits). Tag all ${b.n} visits (v${b.first_v}..v${b.last_v})."""
AXIS1_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one
patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}.
Input. Read this first. You are given a seed set as JSON:
${SEEDS}
It has the keys axis and seeds. Each seed is a fact that code has already extracted from the
patient's timeline, together with its measured long horizon. Build questions only from these seeds.
Do not invent dates, values, numbers or facts, and copy clinical values, units and dates exactly
from the seed. Each seed has these fields:
- form: "range", a value that stayed unchanged over a long span, or "never_again", a fact that was
never documented again.
- qtype: the clinical type: test, diagnosis, medication, exam, treatment or symptom.
- entity, value: the canonical entity name and its documented value. For a range seed, value is
the answer.
- gt_date: for a range seed, the onset of the fact (its first mention), which must not be disclosed
for a sustained value; for a never_again seed, the date of the last mention.
- window_from, window_to: range seeds only. An interior window {date, visit} strictly inside the
stable span. This is the range to show in a T2 question; it reveals neither the onset nor the
present.
- range_from, range_to: range seeds only. The full documented span, for reference only; do not
show it.
- horizon_visits, horizon_months: the measured horizon without change or recurrence, at least 100
visits.
- n_mentions, evidence_visits: how many times, and at which visits, the fact is documented.
Task. Turn the good seeds into grounded evidence-question pairs whose answer can be verified only
from the cited evidence. Skip weak, vague or duplicate seeds; fewer, better questions are
preferred. For every question fill in entity, form, template, question, answer, answer_type, qtype,
evidence_visits, gt_date, horizon_visits, horizon_months and provenance. provenance is a single line
saying why the answer is grounded, with the measured horizon."""
AXIS1_RULES = """Axis 1: long-horizon recall. Each seed's value either stayed unchanged (form "range") or was
never mentioned again (form "never_again") over horizon_visits elapsed visits, at least 100. Write a
question that can be answered only by recalling across that long gap. Copy the seed's entity (word
for word), form, qtype, horizon_visits, horizon_months, gt_date and evidence_visits onto the question
unchanged.
Diversity matters. Across the whole set, cover different clinical types: a test, lab or imaging
result; a diagnosis or stage; a medication; an exam finding; a treatment or procedure; a symptom.
Do not cluster on one type, and do not ask two near-identical questions. Aim for about
${AXIS1_TARGET} strong questions in total, spanning at least four types and both forms, roughly
${AXIS1_FORM_MIX}.
Range seeds (form "range"):
- The answer is the seed's value, word for word, and answer_type is "open_exact".
- gt_date is the fact's onset; do not disclose it for a sustained value. The seed's window, from
window_from.date to window_to.date, lies strictly inside the stable span, so it reveals neither
the onset nor the present.
- Choose the template by the nature of the fact:
- A one-time result obtained on a specific date, such as an MRI, CT, ultrasound, X-ray, biopsy
or lab test: use T3 or T1, anchored to the test date, which is gt_date. Naming the test by its
date is part of the question, not a leak.
- A sustained value or state that held constant, such as a diagnosis, a stage, an ongoing
treatment or traction regimen, or a persistent exam finding or symptom: always use T2 with the
interior window. Never anchor a sustained value to gt_date, and never use T1 or T3 for it.
Asking "what was X on {date}?" or "according to the {date} record" wrongly implies that a
separate observation was made that day, when the finding is only carried across many visits.
T1 and T3 are only for a one-time test or result actually performed on that date, so all
sustained values use T2.
- Templates:
- T1: "What was the patient's {measure} on {DATE}?"
DATE is gt_date. Only for a one-time test or result performed that day, never for a sustained
exam finding, diagnosis or symptom.
- T2: "Between {window_from.date} and {window_to.date}, what was the patient's {measure}?"
Use the seed's window_from.date and window_to.date exactly. Do not invent dates, and do not use
gt_date or the onset.
- T3: "According to the {gt_date} {test/record}, what {fact}?"
One-time tests only.
- Set gt_date to seed.gt_date.
Never-again seeds (form "never_again"):
- Use template T4. Ask whether a measure, finding, medication or treatment that last appeared on
{gt_date} was documented again at any later visit.
- Use the T4 wording below as written: a natural yes/no question with no hint that the answer is
no. Do not add words such as "still", "ever" or "or not documented".
- The answer is "No — {X} was last documented on {gt_date}." and answer_type is "open_or_tf".
- T4: "Was {X} documented again at any visit after {gt_date}?"
- Set gt_date to seed.gt_date.
Rules for every question:
- There is no separate "dose of drug" template; treat a dose as an ordinary measure. A standing dose
or regimen is a sustained value and uses T2; a single dose given on one date uses T1 or T3.
- The question must never hint at the answer, or that the question might be unanswerable.
- Keep clinical abbreviations exactly as the chart writes them. Never expand, normalize or translate
them: they are this patient's own shorthand, and the answer must match the record word for word.
- Skip a seed whose value is too vague for a crisp answer."""
AXIS1_TASK = """
Read the seed set above and write the axis-1 question set now. Set the "template" field (T1, T2, T3
or T4) on every question, and keep the set diverse."""
AXIS1_SCHEMA = """{
type: 'object', additionalProperties: false, required: ['questions'],
properties: {
questions: {
type: 'array',
items: {
type: 'object', additionalProperties: false,
required: ['entity', 'form', 'template', 'qtype', 'question', 'answer', 'answer_type', 'evidence_visits',
'gt_date', 'horizon_visits', 'horizon_months', 'provenance'],
properties: {
entity: { type: 'string', description: "copy the seed's entity verbatim (maps the question back to its seed; never shown to the answerer)" },
form: { enum: ['range', 'never_again'] },
template: { enum: ['T1', 'T2', 'T3', 'T4'], description: 'T1 point / T2 range / T3 source-anchored / T4 never-again' },
qtype: { type: 'string' },
question: { type: 'string' },
answer: { type: 'string' },
answer_type: { enum: ['open_exact', 'open_or_tf'] },
evidence_visits: { type: 'array', items: { type: 'integer' } },
gt_date: { type: 'string' },
horizon_visits: { type: 'integer' },
horizon_months: { type: 'integer' },
provenance: { type: 'string' },
},
},
},
},
}"""
AXIS2_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one
patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}.
You are given seeds for one axis; each seed is a fact already extracted from the patient's timeline.
Turn the good seeds into grounded evidence-question pairs whose answer can be verified only from the
cited evidence. Use only the seed's fields: do not invent dates, values or numbers, and copy clinical
values and units exactly. Skip weak, vague or duplicate seeds; fewer, better questions are
preferred."""
AXIS2_RULES = """Axis 2: clinical state updating. Each seed is an entity whose clinical state changed over time:
- change_type "evolved": the state was refined or replaced. For example, a diagnosis went from
suspected to confirmed, a disease from active to remission, a scope from knee to hand and knee,
or a differential became the working diagnosis.
- change_type "resolved": a symptom or treatment that ended.
The change is often implicit in the chart. It rarely says "no longer X"; it says "s/p", "NED" or
"improved", or simply stops mentioning it. Write a question that asks for the patient's current
state of that entity or area. The point of this axis is that a model that did not track the update,
or that anchors on the earlier mention, answers with the old state and fails.
Rules. Breaking any of them ruins the question:
- The question must not contain any earlier value: none of the strings in the seed's
old_values_DO_NOT_MENTION, not even paraphrased.
- The question must not contain any date or year.
- Refer to the entity by a neutral, stable handle, such as the body area, the drug or the condition
category, that does not itself reveal the current value, which is the answer. For a diagnosis
that went from r/o to confirmed, ask about "the workup of the patient's <area> pain", not the
confirmed name. For a chronic named disease, ask about "the current status of the patient's
<disease name>": the disease name is stable and safe to say, and its status, such as remission,
relapse or resolved, is the answer.
Templates. Both forms are open status questions that start with "What is the current …?". Never ask
a yes/no question such as "Does the patient still {have/take/experience} …?": it tells the model
that the answer might be "no", which leaks that the state may have ended. Ask openly and let the
model recall the current state.
- evolved: answer_type "open_current".
Question: "What is the current {status/diagnosis} of the patient's {neutral area/condition}?"
Answer: seed.current_value, close to its wording, with abbreviations kept.
- resolved: answer_type "open_current".
Question: "What is the current status of the patient's {neutral entity}?"
Answer, for a seed with qtype medication or treatment: "No longer in use — it was recorded as stopped
or completed."; for any other seed: "No longer an active problem — it was recorded as resolved."
Do not state the resolution date, and do not hint in the question that the problem ended.
Output. Write about ${AXIS2_TARGET} strong questions, mixing evolved and resolved and spanning at
least three clinical types: diagnosis, symptom and treatment. Keep clinical abbreviations exactly
as the chart writes them; never expand, normalize or translate them. Copy entity (word for word from
the seed), change_type, qtype, evidence_visits and gt_date onto each question; gt_date is
seed.gt_date. provenance is a single line giving the change of state and why the
current answer is grounded. You may name the old value in provenance: it is metadata and is never
shown to the answering model."""
AXIS2_TASK = """
The seed set as JSON:
${SEEDS}
It has the keys axis and seeds. Each seed has change_type, entity, aspect, qtype, department,
current_value, old_values_DO_NOT_MENTION, gt_date, gt_visit, evidence_visits and n_segments.
Write the axis-2 current-status question set now."""
AXIS2_SCHEMA = """{
type: 'object', additionalProperties: false, required: ['questions'],
properties: {
questions: {
type: 'array',
items: {
type: 'object', additionalProperties: false,
required: ['entity', 'change_type', 'qtype', 'question', 'answer', 'answer_type', 'evidence_visits',
'gt_date', 'provenance'],
properties: {
entity: { type: 'string', description: "copy the seed's entity verbatim (maps back to old_values for the leak-lint; never shown to the answerer)" },
change_type: { enum: ['evolved', 'resolved'] },
qtype: { type: 'string' },
question: { type: 'string' },
answer: { type: 'string' },
answer_type: { enum: ['open_current', 'open_or_tf'] },
evidence_visits: { type: 'array', items: { type: 'integer' } },
gt_date: { type: 'string' },
provenance: { type: 'string' },
},
},
},
},
}"""
AXIS3_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one
patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}.
You are given one seed, a fact already extracted from the patient's timeline. Build grounded questions whose answer can be verified only from the patient's recalled history. Do
not invent facts about the patient."""
AXIS3_RULES = """Axis 3: drug safety, multiple choice. Each seed is a documented constraint specific to this
patient: an adverse reaction, an intolerance or an interacting medication. Write a realistic
question in the patient's voice for the seed's scenario_hint, for example:
"Doctor, my knees have been aching. I'd like to take an over-the-counter painkiller — which of these
would be most appropriate for me?"
Offer exactly four drug options, all plausible choices that a clinician might consider for that
scenario, with the same therapeutic use. Whether an option is right must depend entirely on
recalling the patient's constraint.
Rules. They are the point of this axis:
- Do not name, describe or hint at the constraint anywhere in the question or the options; the
model must recall it from memory. Phrases such as "given your <condition>", "since you react to
<drug class>", "as you are on <medication>" or "because of <risk>" are forbidden. Ask the way a
patient who does not know the pharmacology would ask.
- Never use the words "safe", "unsafe", "risk", "dangerous" or "avoid" in the question or the
option text; they signal a safety concern. The question asks which option is the most
appropriate or suitable choice for the patient, and the options are plain drug names. The
safe/unsafe judgement appears only in the hidden is_correct flag and the explanation, which the
model never sees.
- Mark each option with is_correct: true if it is appropriate for this patient, false if it is
not. Base this on the recalled constraint and standard pharmacology.
- The correct option, taken from seed.safe_examples, must be a choice that a reader can pick only by
recalling the constraint. It may be the conservative drug when that is the only reasonable
alternative, as long as the three traps are the first-line choices. The three traps, taken from
seed.unsafe_examples, must be exactly the drugs that a clinician, or a model that does not know
this patient's history, would reach for first for seed.scenario_hint: the guideline, textbook or
most popular choices. The question fails if a reader with no memory could pick the right answer
just by being careful or by default. The default pick must be a trap, and the correct answer
must require recalling the constraint.
- Use concrete drug names from the seed's unsafe_examples and safe_examples: the generic name,
with a familiar brand in parentheses where natural. Exactly one of the four options is
appropriate (is_correct: true) and the other three are inappropriate for this patient, so the
question has a single correct answer. Pick one drug from seed.safe_examples
as the appropriate option and three from seed.unsafe_examples as the traps.
- Do not list the options in the question text; they are shown after the question, in an order the
code sets. Return them only in options, each as
{"label": "A", "content": "<drug>", "is_correct": true|false, "explanation": "<why>"}.
For each question also return trap_design with:
- trap_type: for example adverse_reaction, drug_interaction or contraindication
- trap_mechanism: the specific safety fact that makes the textbook choice wrong for this patient
- why_others_fail: why a model without memory picks a trap
- required_patient_info: the exact memory facts needed, copied from seed.required_patient_info
Return each question as {"question", "options", "trap_design"}.
Write one question for the seed, with its scenario matched to seed.scenario_hint. If no question
can meet these rules, return {"questions": []}."""
AXIS4_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one
patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}.
You are given one seed, a fact already extracted from the patient's timeline. Use only the seed's
fields."""
AXIS4_RULES = """Axis 4: entity attribution, the patient or someone else. The seed is a condition or event,
given as entity, whose owner a reader could get wrong. owner says whose it is: a relative, a
household contact, or the patient. kind is relative_owned, contagion or patient_owned. The fact was
documented in atom_verbatim, and why_memory_required says how a model without memory would attribute
it to the wrong person. patient_also_has, when present, is the patient's own documented condition
that matches the entity. Write one natural question that asks whether the patient had {entity},
phrased the same way whoever the owner is, so that a model that did not track whose condition it
was answers wrong.
Rules:
- The question must not reveal the owner. Do not mention mother, father, family, relative,
caregiving or caring for, or whom the condition belongs to; just ask whether the patient had it.
- Phrase it plainly, for example "Did the patient have {entity}?" or "Is {entity} part of the
patient's history?"
- The answer is the gold naming the correct owner:
- owner is someone else and patient_also_has is absent: it starts with "No — it was {owner}'s ...".
- owner is the patient: it starts with "Yes — the patient's own: ...".
- owner is someone else and patient_also_has is present: it starts with "Yes — the patient's own:
..." and adds that {owner}'s {entity} is a separate fact."""
DIALOGUE_RULES = """CONVERSATION RULES (shared -- both speakers follow these)
You are role-playing ONE speaker in a real, two-person clinic conversation; another agent plays the
other person. Make it sound like two real people actually talking -- not two people reciting notes.
Follow these rules every turn:
1. TALK TO EACH OTHER. Respond to what the other person JUST said. React to their last turn first,
then add your part. Never ignore what they just said.
2. ANSWER QUESTIONS. If the other person just asked you something, ANSWER it directly first, before
anything else. Never leave a question hanging.
3. ONE THING PER TURN. Say one point, briefly, the way people speak in a visit -- then stop and let
the other person respond. Do not deliver a paragraph or a checklist.
4. DON'T REPEAT. Say each thing once. Never restate something already said earlier in THIS
conversation. If it was covered, refer to it briefly instead of repeating it.
5. NO FILLER TURNS. Do not use "I understand" / "thank you" / "okay" as a whole turn. Every turn
carries real content -- an answer, a genuine reaction, or your own question.
6. ASK NATURALLY. It's a real visit -- ask things when natural (a patient asks what a finding means;
a doctor asks about the complaint).
7. ENGLISH ONLY. Always speak in natural, fluent English, even if the record contains non-English terms. NEVER
output non-English text -- use the English meaning.
8. GROUNDED BUT HUMAN. Everything you say must be true to the record -- never invent a symptom, test,
result, dose, date, or diagnosis. But say it like a person, not like you're reading a chart. If
you genuinely would not know a detail, say so naturally ("I'm not sure exactly").
9. END ONLY WHEN TOLD. Wrap up the visit only on the turn the steering marks as the final one.
10. OUTPUT. Reply with ONLY a JSON object:
{"text": "<your one spoken turn>",
"covers": ["<atom_id>", ...], // the atom ids from <turn_task> your text actually conveys ([] if none)
"type": "clinical|recap|noise|advice|adherence|intake|closing",
"end_ok": <true ONLY if <steering> says this is the wrap-up turn, else false>}
Only list an atom id in "covers" if your text genuinely conveys that fact, and only ids that
appear in <turn_task>. Never claim to cover something you did not say.
Your role-specific instructions follow. When anything conflicts, these shared rules and the
faithfulness constraints win.
"""
DOCTOR_SYSTEM = """You role-play a DOCTOR at a clinic visit on a given date. Speak naturally, one short turn at a time,
in character. You produce ONLY the doctor's spoken lines -- never the patient's.
WHAT YOU CAN SEE (in the user message; fields named in <>):
- <current_visit_facts>: today's facts, each with an atom id and a novelty flag:
NEW = first appears today -> discuss as a new finding.
CARRIED = already known from this patient's recent same-department visits -> RECAP it,
do NOT re-discover it as novel ("your X-ray still shows the same mild change,
unchanged").
- <recent_visits_here>: up to two previous SAME-department visits -- your recap source.
- <turn_task>: the phase and the exact atom ids to convey THIS turn (with their text).
- <steering>: per-turn note -- phase, small-talk budget, whether this is the wrap-up turn.
You do NOT see any other department's record and nothing dated after today.
HARD RULES:
1. GROUNDING (STRICT) -- state only what is in <current_visit_facts>/<recent_visits_here>. Never
invent or add severity/quantity to any medication, test, result, dose, date, diagnosis, referral,
symptom, or outcome. You MAY convey a documented intake fact (e.g. "no known drug allergy",
"family history was unremarkable") when it is an atom in <current_visit_facts> -- that is the
record, not small talk. What you must not do is INVENT such facts or raise them when not recorded.
2. CONTROLLED RESTATEMENT -- restate a result in plain, patient-friendly words only. Do NOT add
reasoning, prognosis, or conclusions the record doesn't state. If the record contains no
interpretation, do not add one. Keep abbreviations exact if their meaning is not unambiguous from
this visit (never guess a body part or diagnosis).
3. STATUS FIDELITY -- keep r/o and "suspicious" uncertain; "planned"/"will arrange" is NOT done;
NED / remission is NOT active disease. Keep dose/stage/grade numbers exact.
4. REVEAL DISCIPLINE -- reveal objective results ONE at a time, in the order <steering> gives; never
dump the whole record at once; pause for the patient to react.
5. DECOMPOSE MEASUREMENTS -- turn a measurement/exam into a natural multi-turn exchange:
propose it, let the patient respond, then state the reading. E.g. "let's take your blood pressure
-- arm up for me" ... (patient) ... "okay, it reads a bit high today". Same for a repeated test:
"we'll repeat the X-ray" ... then "it looks the same as before".
6. NO FUTURE / NO OTHER-DEPARTMENT -- never mention anything dated after today, or facts belonging to
another department's visit. You only learn of other-department visits if the patient mentions one.
7. SMALL TALK -- brief non-clinical small talk ONLY from these SEVEN topics and nothing else: traffic;
waiting time; the hospital being busy; feeling mildly nervous; coming from work; the weather;
being worried while waiting for results. Stay within the <steering> budget. NEVER introduce any
other topic as chit-chat -- especially new pain, fever, cough, sleep/diet/weight change, exercise
tolerance, smoking, alcohol, medication adherence as a NEW problem, family history, allergies,
pregnancy, or new treatment behavior. Chit-chat never changes your assessment or plan.
8. RECAP GROUPING -- for CARRIED standing conditions, acknowledge them ONCE, briefly, in
fresh varied words (you MAY cover several in one sentence); note they are steady/unchanged. Do NOT
recite them verbatim, do NOT re-discover them as new, do NOT chant "all unchanged" every turn.
OUTPUT -- the shared-rules JSON object ({text, covers, type, end_ok}); "covers" = atom ids from
<turn_task> your line actually conveys.
"""
PATIENT_SYSTEM = """You role-play a PATIENT at a clinic visit. You are the SAME person across all your visits. Speak in
ordinary lay language (no medical jargon, no test read-outs), one short turn at a time. You produce
ONLY the patient's spoken lines -- never the doctor's.
WHAT YOU CAN SEE (in the user message; fields named in <>):
- <your_reasons_today>: YOUR OWN complaints for today (your patient_self_description atoms, each with
an atom id). This is what YOU know and may bring up. When <turn_task> lists one, try to voice it.
- <your_recent_visits_here>: your up-to-two previous SAME-department visits (full record) -- you lived
through these, so you may recall a specific thing you were told, including results.
- <cross_dept_recap>: SOMETIMES a one-line note that you recently saw a DIFFERENT department (roughly
when + which dept only). You may make light small talk about how that went -- you know NO clinical
detail of it beyond what you were told.
- <turn_task> / <steering>: what to do this turn, small-talk budget, wrap-up flag.
CRITICAL -- WHAT YOU DO NOT KNOW:
- You do NOT know today's exam findings, test results, diagnosis, or plan. You learn these from the
DOCTOR now; react as if hearing them for the first time (relief, worry, a lay question). Never state
today's objective result / diagnosis / plan before the doctor has said it.
- Do NOT invent symptoms, sensations, severity, or quantities beyond <your_reasons_today>. If it isn't
in your reasons for today, don't mention it, and don't deny it either.
DECOMPOSED EXCHANGES: when the doctor proposes a measurement or exam ("let's check your
blood pressure, arm up"), respond naturally in the moment ("sure, go ahead") and let the doctor report
the reading -- do not pre-empt the number.
OTHER RULES:
1. You may recall a SPECIFIC fact from a prior visit (from <your_recent_visits_here>); do not narrate
long history and do not reason medically ("so that must mean..."). Stay declarative and lay.
2. CROSS-DEPARTMENT RECAP -- only if <turn_task> provides <cross_dept_recap>, and only briefly: mention
in plain words that you had that other visit and how it went. Never invent a clinical detail.
3. SMALL TALK -- brief, ONLY from this allowed list and within budget: traffic, waiting time, hospital
being busy, feeling mildly nervous, coming from work, weather, being worried while waiting for
results, getting to the clinic. NEVER INVENT a forbidden topic that is NOT in your record (new pain,
fever, cough, sleep/diet/weight change, exercise tolerance, smoking, alcohol, family history,
allergies, pregnancy, new treatment behavior). IMPORTANT EXCEPTION: if such a thing IS one of your
<your_reasons_today> atoms (e.g. a family member was hospitalized, a documented allergy, a lifestyle
note), it is part of YOUR record -- you SHOULD voice it plainly when <turn_task> lists it. The rule
only forbids MAKING UP topics you have no record of. Don't repeat the same small-talk topic.
4. ADHERENCE: if the doctor asks whether you kept up a plan you were given (e.g. a home
exercise or self-care routine), you MAY confirm you did -- briefly, only what's plausible
from the record; never volunteer adherence unprompted.
5. TONE -- speak like a real patient: natural, varied, human, not scripted.
OUTPUT -- the shared-rules JSON object ({text, covers, type, end_ok}); "covers" = atom ids from
<turn_task> your line actually voices (usually your own reasons-for-today).
"""