Download code_release/code/construction/pipeline/prompts.py from clinical-agent-memory-bench/meme-bench: direct link, hf CLI and curl.
- Browser
- Download file 42.2 kB
-
https://huggingface.co/spaces/clinical-agent-memory-bench/meme-bench/resolve/main/code_release/code/construction/pipeline/prompts.py
- Command line
-
hf download hf://spaces/clinical-agent-memory-bench/meme-bench/code_release/code/construction/pipeline/prompts.py
-
curl -L -o prompts.py https://huggingface.co/spaces/clinical-agent-memory-bench/meme-bench/resolve/main/code_release/code/construction/pipeline/prompts.py
42.2 kB
| """Prompt templates of the construction pipeline. | |
| The LLM steps of pipeline.llm fill these templates: ${PDESC}, ${NVISITS}, ${PFIRST} and ${PLAST} are | |
| the patient's facts; ${AXIS1_TARGET}, ${AXIS1_FORM_MIX} and ${AXIS2_TARGET} are the question counts of steps | |
| axis1 and axis2; ${SEEDS}, ${b.path} and ${m.path} mark where the step's input is inlined; | |
| ${b.<field>} and ${m.<field>} take per-batch values. pipeline.questions uses the axis-3 and axis-4 rules, and | |
| pipeline.dialogues the dialogue prompts. | |
| """ | |
| EXTRACT_RUBRIC = """You extract a structured event record from one clinic visit of a synthetic health record. The | |
| notes are English clinical shorthand, sometimes with a short gloss in parentheses. | |
| Read the visit's S, O, A and P notes and break the clinically meaningful events into short itemized | |
| strings. Put each event in exactly one field; no event may appear in two fields. Route each event by | |
| its meaning, because raw notes are messy: a diagnosis may sit in O and a result in S. In ambiguous | |
| cases, decide by the note section the event came from and by the rules below. | |
| Fields, each with the note section it mainly comes from: | |
| - diagnosis (A): the named diagnoses and disease states that the doctor assesses at this visit. A | |
| finding written as a diagnosis in O also counts. | |
| - A symptom or complaint on its own, such as pain, numbness, weakness or dizziness, is not a | |
| diagnosis; it goes to patient_self_description. | |
| - A rule-out (r/o), a suspected diagnosis or an impression is not confirmed; it goes to | |
| doctor_extra_notes. | |
| - A comorbidity or past diagnosis mentioned only as background, not assessed at this visit, goes | |
| to patient_self_description as "history of X". | |
| - patient_self_description (S): what the patient reports: their own symptoms and complaints at this | |
| visit, their self-reported past history ("history of X"), and their self-reported lifestyle and | |
| activity, for example "walks daily" or "<daily activity>". A family member's condition, | |
| the patient's social or economic situation and the patient's occupation are not the patient's | |
| symptoms; they go to doctor_extra_notes. | |
| - tests_and_medication (O, A and P): results of instruments, labs and imaging, such as CT, MRI, | |
| ultrasound, X-ray, endoscopy, blood tests and viral loads, including findings written as a | |
| sentence; and the medications and treatments named, with drug names and doses. The doctor's | |
| bedside exam findings do not belong here. | |
| - doctor_notes (O): the doctor's own bedside exam findings and measured vitals: palpation, ROM, MMT, | |
| inspection, auscultation, mental-status exam, blood pressure and pulse. Lab and imaging results go | |
| to tests_and_medication. A bedside maneuver such as "<maneuver> test:+" belongs here even | |
| though it is called a test. | |
| - doctor_plans (P): the actions the doctor will take: orders, new prescriptions and medication changes | |
| (see the medication rule below), procedures to arrange, follow-up and referrals. Assistive devices | |
| that are ordered, such as a splint, brace or sling, also belong here. | |
| - health_advice (P, or wherever the doctor advises): the counseling the doctor gives on lifestyle, | |
| diet, exercise, smoking, alcohol, self-care or health education, for example "<lifestyle advice>" | |
| or "advise regular exercise". An activity that the patient reports doing goes to | |
| patient_self_description instead. | |
| - doctor_extra_notes (anywhere in the note): the supplementary layer, a flat list of short strings | |
| with no categories. It holds the context and the reasons, not the facts themselves: | |
| - the patient's social, economic and psychological situation, worries and occupation, and a family | |
| member's condition or caregiving; | |
| - differential reasoning: r/o, suspect, impression, cannot exclude; | |
| - communication with the patient or family: what was explained, informed consent, refusal of | |
| treatment; | |
| - special reasons for a medication, such as a coverage or authorization justification or off-label | |
| use, and the opinion behind a referral or of a multidisciplinary team; | |
| - uncertain or atypical findings flagged to watch; | |
| - administrative items: certificates, insurance, monitoring reminders. | |
| A plain diagnosis, symptom, test, exam finding or plan goes to its own field instead. | |
| Treatment toxicity grading, in oncology, radiation oncology and chemotherapy follow-up: | |
| - A graded checklist of treatment side effects, such as CTCAE grades, is the doctor's periodic | |
| assessment of adverse effects. It is neither a diagnosis nor a lab result. Keep the whole checklist | |
| together in doctor_notes, and do not split it across diagnosis and tests_and_medication. | |
| - The checklist includes hematologic grades such as "<cytopenia>(<grade>)", which stay here even | |
| though they come from blood counts, because the graded form is the physician's assessment and not | |
| a raw lab value. It also includes symptom grades such as "<side effect>(<grade>)", items that | |
| start with "<treatment> toxicity - ...", and any measured value inside the checklist, such as | |
| "<measure>: <value>" or "<measure>(<value> -> <value>)". | |
| - Keep in diagnosis only the cancer diagnosis and its stage, for example "<cancer>: <TNM stage>", | |
| and any genuinely new condition diagnosed during treatment, for example "<new infection>". Do not | |
| move these into doctor_notes. | |
| Rules: | |
| - Extract only what this visit's note states, and never invent anything. | |
| - Never expand an abbreviation into a fuller diagnosis or body part. Keep abbreviations such as | |
| "GERD" or "<abbreviation>" exactly as written, and do not guess which body part or diagnosis an | |
| ambiguous abbreviation stands for. | |
| - Keep laterality, numeric values, dates, drug names and shorthand exactly as written. | |
| - A medication that the patient takes or receives, with its name and dose, goes to | |
| tests_and_medication. A change that the doctor decides on, such as starting, stopping, increasing | |
| or switching a drug, goes to doctor_plans. Do not record the same drug and dose in both fields. | |
| - The S note may carry a running, dated history. Include the standing problem list and this visit's | |
| complaints, but do not add claims about resolution, onset or timing that are not written. | |
| - Leave an empty field as an empty array. Keep a gloss in parentheses when it is part of a term.""" | |
| EXTRACT_TASK = """ | |
| The visit as JSON, with the fields visit_number, date, department, icd and soap (S, O, A, P): | |
| ${m.path} | |
| Extract the event record of visit ${m.visit_number} only, and set visit_number to ${m.visit_number}.""" | |
| EXTRACT_SCHEMA = """{ | |
| type: 'object', additionalProperties: false, | |
| required: ['visit_number', 'diagnosis', 'patient_self_description', 'tests_and_medication', | |
| 'doctor_notes', 'doctor_plans', 'health_advice', 'doctor_extra_notes'], | |
| properties: { | |
| visit_number: { type: 'integer' }, | |
| diagnosis: { type: 'array', items: { type: 'string' } }, | |
| patient_self_description: { type: 'array', items: { type: 'string' } }, | |
| tests_and_medication: { type: 'array', items: { type: 'string' } }, | |
| doctor_notes: { type: 'array', items: { type: 'string' } }, | |
| doctor_plans: { type: 'array', items: { type: 'string' } }, | |
| health_advice: { type: 'array', items: { type: 'string' } }, | |
| doctor_extra_notes: { type: 'array', items: { type: 'string' } }, | |
| }, | |
| }""" | |
| EVIDENCE_RUBRIC = """You build the evidence chain: validity windows for each clinical entity within one department of a | |
| single patient's longitudinal chart. | |
| The batch JSON in the user message has these fields: | |
| - department | |
| - dept_last_date and patient_last_date | |
| - visit_dates: a map from visit_number to date, for this department's visits | |
| - facts: a list of items {field, fact, visits, first_date, last_date}, where visits lists the | |
| visit numbers that recorded the fact | |
| The facts were extracted visit by visit, and many are carried forward from one visit to the next. | |
| "field" is the source bucket: diagnosis, patient_self_description, tests_and_medication, | |
| doctor_notes, doctor_plans, health_advice or doctor_extra_notes. | |
| Your job is to rebuild clinical timelines in three steps. | |
| 1. Canonicalize. Merge fact variants that refer to the same clinical entity, and take the union of | |
| their visit lists. For example: | |
| - "left <abbreviation> pain" and "left <abbreviation> pain decrease" -> entity | |
| "left <abbreviation> pain" | |
| - every "<lab>..." fact -> "<lab> level" | |
| - a recurring differential "r/o <condition>" -> one entity | |
| - "<drug> coverage justification..." -> one entity | |
| Keep abbreviations exactly as written in the entity name. Do not expand or translate | |
| "<abbreviation>" or "GERD", and never guess the body part an ambiguous abbreviation stands for. | |
| 2. Segment. Split each entity into validity windows over time. A new segment starts whenever the | |
| entity's value or status changes, for example: | |
| - <lab> not detected -> <value> -> not detected, which gives three segments | |
| - a change of lab value, stage or dose | |
| - a symptom that resolves and later returns | |
| - a toxicity grade that rises from 1 to 2 to 3 | |
| A carried-forward repeat of the same value extends the current segment. | |
| 3. Describe each segment with: | |
| - value: the status or value held, close to its source wording | |
| - valid_from: {date, visit_number} of the first visit that asserts it | |
| - valid_until: {date, visit_number}, or "current" | |
| - status: current, resolved, superseded or historical | |
| - supporting_visits: the visit numbers inside the window | |
| - superseded_by: {value, date, visit_number}, or null | |
| Rules: | |
| - Use only the facts and visit_dates given. Map every visit number to its date through visit_dates. | |
| Keep numeric values and units as written, and do not invent anything. | |
| - Statuses: | |
| - "current": the entity's latest value is still asserted at or near dept_last_date, and no later | |
| value contradicts it. | |
| - "resolved": the record states that the problem ended or healed. A problem that improved but is | |
| still present stays "current", with a value that says it improved. | |
| - "superseded": a later value of the same entity replaced this one; give it in superseded_by. | |
| - "historical": an earlier value that was neither replaced nor stated to have ended. | |
| - Dates never decrease across an entity's segments, and valid_until is never before valid_from. | |
| - aspect is one of disease_state, symptom, test_result, medication, procedure_or_followup, | |
| constraint_or_safety, lifestyle_social, care_plan, exam_finding, health_advice, | |
| differential_impression, communication_consent, med_rationale_referral, admin, other. Choose it by | |
| the entity's meaning, with the source field as a hint: | |
| - health_advice -> health_advice | |
| - an r/o or suspected diagnosis in doctor_extra_notes -> differential_impression | |
| - a medication or coverage rationale, or a referral -> med_rationale_referral | |
| - consent, refusal or an explanation given -> communication_consent | |
| - a certificate, insurance or a monitoring reminder -> admin | |
| - social context, occupation or a worry -> lifestyle_social | |
| - Cover the meaningful entities of the batch. Every fact here was recorded at two or more visits, or is | |
| one reading of a test or measurement recorded at two or more visits. | |
| Keep anything that persists or changes, including the recurring differentials, rationales and | |
| advice of the supplementary fields.""" | |
| EVIDENCE_TASK = """ | |
| The batch: | |
| ${b.path} | |
| Department: ${b.department}, with ${b.n_facts} facts that recur across visits.""" | |
| EVIDENCE_SCHEMA = """{ | |
| type: 'object', | |
| properties: { | |
| timelines: { type: 'array', items: { | |
| type: 'object', | |
| properties: { | |
| entity: { type: 'string' }, | |
| aspect: { type: 'string', enum: ['disease_state','symptom','test_result','medication','procedure_or_followup','constraint_or_safety','lifestyle_social','care_plan','exam_finding','health_advice','differential_impression','communication_consent','med_rationale_referral','admin','other'] }, | |
| source_field: { type: 'string' }, | |
| segments: { type: 'array', items: { | |
| type: 'object', | |
| properties: { | |
| value: { type: 'string' }, | |
| valid_from: { type: 'object', properties: { date: { type: 'string' }, visit_number: { type: 'integer' } }, required: ['date','visit_number'] }, | |
| valid_until: {}, | |
| status: { type: 'string', enum: ['current','resolved','superseded','historical'] }, | |
| supporting_visits: { type: 'array', items: { type: 'integer' } }, | |
| superseded_by: {}, | |
| }, | |
| required: ['value','valid_from','valid_until','status','supporting_visits','superseded_by'], | |
| }}, | |
| current_status: { type: 'string' }, | |
| notes: { type: 'string' }, | |
| }, | |
| required: ['entity','aspect','source_field','segments','current_status'], | |
| }}, | |
| }, | |
| required: ['timelines'], | |
| }""" | |
| TAGS_RUBRIC = """You tag visits of ONE patient's longitudinal EHR event-chain for a memory benchmark. For EACH item you are given a CURRENT visit's structured facts and its "prev" = the UNION of the up-to-2 most-recent prior visits IN THE SAME DEPARTMENT (t-1, t-2). Judge the current visit ONLY against prev -- you are NOT given any other visit (this is a strict sliding window; do not assume anything outside it). | |
| Output per visit: | |
| - tag: "change" if the current visit contains ANY new or changed clinical content vs prev -- a new symptom / new test ordered / new test result / new reaction / new diagnosis / new medication or procedure / new doctor note or plan / new health advice given / new differential or extra note, OR a changed value/status of an existing item. Otherwise "no_change". | |
| - noise_score: | |
| 3 = pure repeat / nothing new at all; | |
| 2 = mostly repeat, with only a minor no-value-change new instance; | |
| 1 = a real but modest change; | |
| 0 = dense with new information. | |
| - note: one short line -- what is new/changed, or "repeat of prev". | |
| Rules: | |
| - If is_first_in_dept is true (prev is empty), tag="change" and noise_score=0. | |
| - A test/measure REPEATED with a genuinely NEW result instance but the SAME value still counts as tag="change" with noise_score 2. | |
| - Compare by clinical meaning, not exact string; carried-forward identical text = not new. | |
| - Ignore pure administrative boilerplate. | |
| - Judge EVERY visit_number in the batch exactly once.""" | |
| TAGS_TASK = """ | |
| Read the JSON array of items at: | |
| ${b.path} | |
| Each item has: visit_number, date, department, is_first_in_dept, current_facts{diagnosis,patient_self_description,tests_and_medication,doctor_notes,doctor_plans,health_advice,doctor_extra_notes}, prev_facts[] (the t-1,t-2 same-dept visits). Tag all ${b.n} visits (v${b.first_v}..v${b.last_v}).""" | |
| AXIS1_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one | |
| patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}. | |
| Input. Read this first. You are given a seed set as JSON: | |
| ${SEEDS} | |
| It has the keys axis and seeds. Each seed is a fact that code has already extracted from the | |
| patient's timeline, together with its measured long horizon. Build questions only from these seeds. | |
| Do not invent dates, values, numbers or facts, and copy clinical values, units and dates exactly | |
| from the seed. Each seed has these fields: | |
| - form: "range", a value that stayed unchanged over a long span, or "never_again", a fact that was | |
| never documented again. | |
| - qtype: the clinical type: test, diagnosis, medication, exam, treatment or symptom. | |
| - entity, value: the canonical entity name and its documented value. For a range seed, value is | |
| the answer. | |
| - gt_date: for a range seed, the onset of the fact (its first mention), which must not be disclosed | |
| for a sustained value; for a never_again seed, the date of the last mention. | |
| - window_from, window_to: range seeds only. An interior window {date, visit} strictly inside the | |
| stable span. This is the range to show in a T2 question; it reveals neither the onset nor the | |
| present. | |
| - range_from, range_to: range seeds only. The full documented span, for reference only; do not | |
| show it. | |
| - horizon_visits, horizon_months: the measured horizon without change or recurrence, at least 100 | |
| visits. | |
| - n_mentions, evidence_visits: how many times, and at which visits, the fact is documented. | |
| Task. Turn the good seeds into grounded evidence-question pairs whose answer can be verified only | |
| from the cited evidence. Skip weak, vague or duplicate seeds; fewer, better questions are | |
| preferred. For every question fill in entity, form, template, question, answer, answer_type, qtype, | |
| evidence_visits, gt_date, horizon_visits, horizon_months and provenance. provenance is a single line | |
| saying why the answer is grounded, with the measured horizon.""" | |
| AXIS1_RULES = """Axis 1: long-horizon recall. Each seed's value either stayed unchanged (form "range") or was | |
| never mentioned again (form "never_again") over horizon_visits elapsed visits, at least 100. Write a | |
| question that can be answered only by recalling across that long gap. Copy the seed's entity (word | |
| for word), form, qtype, horizon_visits, horizon_months, gt_date and evidence_visits onto the question | |
| unchanged. | |
| Diversity matters. Across the whole set, cover different clinical types: a test, lab or imaging | |
| result; a diagnosis or stage; a medication; an exam finding; a treatment or procedure; a symptom. | |
| Do not cluster on one type, and do not ask two near-identical questions. Aim for about | |
| ${AXIS1_TARGET} strong questions in total, spanning at least four types and both forms, roughly | |
| ${AXIS1_FORM_MIX}. | |
| Range seeds (form "range"): | |
| - The answer is the seed's value, word for word, and answer_type is "open_exact". | |
| - gt_date is the fact's onset; do not disclose it for a sustained value. The seed's window, from | |
| window_from.date to window_to.date, lies strictly inside the stable span, so it reveals neither | |
| the onset nor the present. | |
| - Choose the template by the nature of the fact: | |
| - A one-time result obtained on a specific date, such as an MRI, CT, ultrasound, X-ray, biopsy | |
| or lab test: use T3 or T1, anchored to the test date, which is gt_date. Naming the test by its | |
| date is part of the question, not a leak. | |
| - A sustained value or state that held constant, such as a diagnosis, a stage, an ongoing | |
| treatment or traction regimen, or a persistent exam finding or symptom: always use T2 with the | |
| interior window. Never anchor a sustained value to gt_date, and never use T1 or T3 for it. | |
| Asking "what was X on {date}?" or "according to the {date} record" wrongly implies that a | |
| separate observation was made that day, when the finding is only carried across many visits. | |
| T1 and T3 are only for a one-time test or result actually performed on that date, so all | |
| sustained values use T2. | |
| - Templates: | |
| - T1: "What was the patient's {measure} on {DATE}?" | |
| DATE is gt_date. Only for a one-time test or result performed that day, never for a sustained | |
| exam finding, diagnosis or symptom. | |
| - T2: "Between {window_from.date} and {window_to.date}, what was the patient's {measure}?" | |
| Use the seed's window_from.date and window_to.date exactly. Do not invent dates, and do not use | |
| gt_date or the onset. | |
| - T3: "According to the {gt_date} {test/record}, what {fact}?" | |
| One-time tests only. | |
| - Set gt_date to seed.gt_date. | |
| Never-again seeds (form "never_again"): | |
| - Use template T4. Ask whether a measure, finding, medication or treatment that last appeared on | |
| {gt_date} was documented again at any later visit. | |
| - Use the T4 wording below as written: a natural yes/no question with no hint that the answer is | |
| no. Do not add words such as "still", "ever" or "or not documented". | |
| - The answer is "No — {X} was last documented on {gt_date}." and answer_type is "open_or_tf". | |
| - T4: "Was {X} documented again at any visit after {gt_date}?" | |
| - Set gt_date to seed.gt_date. | |
| Rules for every question: | |
| - There is no separate "dose of drug" template; treat a dose as an ordinary measure. A standing dose | |
| or regimen is a sustained value and uses T2; a single dose given on one date uses T1 or T3. | |
| - The question must never hint at the answer, or that the question might be unanswerable. | |
| - Keep clinical abbreviations exactly as the chart writes them. Never expand, normalize or translate | |
| them: they are this patient's own shorthand, and the answer must match the record word for word. | |
| - Skip a seed whose value is too vague for a crisp answer.""" | |
| AXIS1_TASK = """ | |
| Read the seed set above and write the axis-1 question set now. Set the "template" field (T1, T2, T3 | |
| or T4) on every question, and keep the set diverse.""" | |
| AXIS1_SCHEMA = """{ | |
| type: 'object', additionalProperties: false, required: ['questions'], | |
| properties: { | |
| questions: { | |
| type: 'array', | |
| items: { | |
| type: 'object', additionalProperties: false, | |
| required: ['entity', 'form', 'template', 'qtype', 'question', 'answer', 'answer_type', 'evidence_visits', | |
| 'gt_date', 'horizon_visits', 'horizon_months', 'provenance'], | |
| properties: { | |
| entity: { type: 'string', description: "copy the seed's entity verbatim (maps the question back to its seed; never shown to the answerer)" }, | |
| form: { enum: ['range', 'never_again'] }, | |
| template: { enum: ['T1', 'T2', 'T3', 'T4'], description: 'T1 point / T2 range / T3 source-anchored / T4 never-again' }, | |
| qtype: { type: 'string' }, | |
| question: { type: 'string' }, | |
| answer: { type: 'string' }, | |
| answer_type: { enum: ['open_exact', 'open_or_tf'] }, | |
| evidence_visits: { type: 'array', items: { type: 'integer' } }, | |
| gt_date: { type: 'string' }, | |
| horizon_visits: { type: 'integer' }, | |
| horizon_months: { type: 'integer' }, | |
| provenance: { type: 'string' }, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }""" | |
| AXIS2_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one | |
| patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}. | |
| You are given seeds for one axis; each seed is a fact already extracted from the patient's timeline. | |
| Turn the good seeds into grounded evidence-question pairs whose answer can be verified only from the | |
| cited evidence. Use only the seed's fields: do not invent dates, values or numbers, and copy clinical | |
| values and units exactly. Skip weak, vague or duplicate seeds; fewer, better questions are | |
| preferred.""" | |
| AXIS2_RULES = """Axis 2: clinical state updating. Each seed is an entity whose clinical state changed over time: | |
| - change_type "evolved": the state was refined or replaced. For example, a diagnosis went from | |
| suspected to confirmed, a disease from active to remission, a scope from knee to hand and knee, | |
| or a differential became the working diagnosis. | |
| - change_type "resolved": a symptom or treatment that ended. | |
| The change is often implicit in the chart. It rarely says "no longer X"; it says "s/p", "NED" or | |
| "improved", or simply stops mentioning it. Write a question that asks for the patient's current | |
| state of that entity or area. The point of this axis is that a model that did not track the update, | |
| or that anchors on the earlier mention, answers with the old state and fails. | |
| Rules. Breaking any of them ruins the question: | |
| - The question must not contain any earlier value: none of the strings in the seed's | |
| old_values_DO_NOT_MENTION, not even paraphrased. | |
| - The question must not contain any date or year. | |
| - Refer to the entity by a neutral, stable handle, such as the body area, the drug or the condition | |
| category, that does not itself reveal the current value, which is the answer. For a diagnosis | |
| that went from r/o to confirmed, ask about "the workup of the patient's <area> pain", not the | |
| confirmed name. For a chronic named disease, ask about "the current status of the patient's | |
| <disease name>": the disease name is stable and safe to say, and its status, such as remission, | |
| relapse or resolved, is the answer. | |
| Templates. Both forms are open status questions that start with "What is the current …?". Never ask | |
| a yes/no question such as "Does the patient still {have/take/experience} …?": it tells the model | |
| that the answer might be "no", which leaks that the state may have ended. Ask openly and let the | |
| model recall the current state. | |
| - evolved: answer_type "open_current". | |
| Question: "What is the current {status/diagnosis} of the patient's {neutral area/condition}?" | |
| Answer: seed.current_value, close to its wording, with abbreviations kept. | |
| - resolved: answer_type "open_current". | |
| Question: "What is the current status of the patient's {neutral entity}?" | |
| Answer, for a seed with qtype medication or treatment: "No longer in use — it was recorded as stopped | |
| or completed."; for any other seed: "No longer an active problem — it was recorded as resolved." | |
| Do not state the resolution date, and do not hint in the question that the problem ended. | |
| Output. Write about ${AXIS2_TARGET} strong questions, mixing evolved and resolved and spanning at | |
| least three clinical types: diagnosis, symptom and treatment. Keep clinical abbreviations exactly | |
| as the chart writes them; never expand, normalize or translate them. Copy entity (word for word from | |
| the seed), change_type, qtype, evidence_visits and gt_date onto each question; gt_date is | |
| seed.gt_date. provenance is a single line giving the change of state and why the | |
| current answer is grounded. You may name the old value in provenance: it is metadata and is never | |
| shown to the answering model.""" | |
| AXIS2_TASK = """ | |
| The seed set as JSON: | |
| ${SEEDS} | |
| It has the keys axis and seeds. Each seed has change_type, entity, aspect, qtype, department, | |
| current_value, old_values_DO_NOT_MENTION, gt_date, gt_visit, evidence_visits and n_segments. | |
| Write the axis-2 current-status question set now.""" | |
| AXIS2_SCHEMA = """{ | |
| type: 'object', additionalProperties: false, required: ['questions'], | |
| properties: { | |
| questions: { | |
| type: 'array', | |
| items: { | |
| type: 'object', additionalProperties: false, | |
| required: ['entity', 'change_type', 'qtype', 'question', 'answer', 'answer_type', 'evidence_visits', | |
| 'gt_date', 'provenance'], | |
| properties: { | |
| entity: { type: 'string', description: "copy the seed's entity verbatim (maps back to old_values for the leak-lint; never shown to the answerer)" }, | |
| change_type: { enum: ['evolved', 'resolved'] }, | |
| qtype: { type: 'string' }, | |
| question: { type: 'string' }, | |
| answer: { type: 'string' }, | |
| answer_type: { enum: ['open_current', 'open_or_tf'] }, | |
| evidence_visits: { type: 'array', items: { type: 'integer' } }, | |
| gt_date: { type: 'string' }, | |
| provenance: { type: 'string' }, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }""" | |
| AXIS3_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one | |
| patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}. | |
| You are given one seed, a fact already extracted from the patient's timeline. Build grounded questions whose answer can be verified only from the patient's recalled history. Do | |
| not invent facts about the patient.""" | |
| AXIS3_RULES = """Axis 3: drug safety, multiple choice. Each seed is a documented constraint specific to this | |
| patient: an adverse reaction, an intolerance or an interacting medication. Write a realistic | |
| question in the patient's voice for the seed's scenario_hint, for example: | |
| "Doctor, my knees have been aching. I'd like to take an over-the-counter painkiller — which of these | |
| would be most appropriate for me?" | |
| Offer exactly four drug options, all plausible choices that a clinician might consider for that | |
| scenario, with the same therapeutic use. Whether an option is right must depend entirely on | |
| recalling the patient's constraint. | |
| Rules. They are the point of this axis: | |
| - Do not name, describe or hint at the constraint anywhere in the question or the options; the | |
| model must recall it from memory. Phrases such as "given your <condition>", "since you react to | |
| <drug class>", "as you are on <medication>" or "because of <risk>" are forbidden. Ask the way a | |
| patient who does not know the pharmacology would ask. | |
| - Never use the words "safe", "unsafe", "risk", "dangerous" or "avoid" in the question or the | |
| option text; they signal a safety concern. The question asks which option is the most | |
| appropriate or suitable choice for the patient, and the options are plain drug names. The | |
| safe/unsafe judgement appears only in the hidden is_correct flag and the explanation, which the | |
| model never sees. | |
| - Mark each option with is_correct: true if it is appropriate for this patient, false if it is | |
| not. Base this on the recalled constraint and standard pharmacology. | |
| - The correct option, taken from seed.safe_examples, must be a choice that a reader can pick only by | |
| recalling the constraint. It may be the conservative drug when that is the only reasonable | |
| alternative, as long as the three traps are the first-line choices. The three traps, taken from | |
| seed.unsafe_examples, must be exactly the drugs that a clinician, or a model that does not know | |
| this patient's history, would reach for first for seed.scenario_hint: the guideline, textbook or | |
| most popular choices. The question fails if a reader with no memory could pick the right answer | |
| just by being careful or by default. The default pick must be a trap, and the correct answer | |
| must require recalling the constraint. | |
| - Use concrete drug names from the seed's unsafe_examples and safe_examples: the generic name, | |
| with a familiar brand in parentheses where natural. Exactly one of the four options is | |
| appropriate (is_correct: true) and the other three are inappropriate for this patient, so the | |
| question has a single correct answer. Pick one drug from seed.safe_examples | |
| as the appropriate option and three from seed.unsafe_examples as the traps. | |
| - Do not list the options in the question text; they are shown after the question, in an order the | |
| code sets. Return them only in options, each as | |
| {"label": "A", "content": "<drug>", "is_correct": true|false, "explanation": "<why>"}. | |
| For each question also return trap_design with: | |
| - trap_type: for example adverse_reaction, drug_interaction or contraindication | |
| - trap_mechanism: the specific safety fact that makes the textbook choice wrong for this patient | |
| - why_others_fail: why a model without memory picks a trap | |
| - required_patient_info: the exact memory facts needed, copied from seed.required_patient_info | |
| Return each question as {"question", "options", "trap_design"}. | |
| Write one question for the seed, with its scenario matched to seed.scenario_hint. If no question | |
| can meet these rules, return {"questions": []}.""" | |
| AXIS4_COMMON = """You write memory-benchmark questions from the synthetic longitudinal health record of one | |
| patient: ${PDESC}, ${NVISITS} visits from ${PFIRST} to ${PLAST}. | |
| You are given one seed, a fact already extracted from the patient's timeline. Use only the seed's | |
| fields.""" | |
| AXIS4_RULES = """Axis 4: entity attribution, the patient or someone else. The seed is a condition or event, | |
| given as entity, whose owner a reader could get wrong. owner says whose it is: a relative, a | |
| household contact, or the patient. kind is relative_owned, contagion or patient_owned. The fact was | |
| documented in atom_verbatim, and why_memory_required says how a model without memory would attribute | |
| it to the wrong person. patient_also_has, when present, is the patient's own documented condition | |
| that matches the entity. Write one natural question that asks whether the patient had {entity}, | |
| phrased the same way whoever the owner is, so that a model that did not track whose condition it | |
| was answers wrong. | |
| Rules: | |
| - The question must not reveal the owner. Do not mention mother, father, family, relative, | |
| caregiving or caring for, or whom the condition belongs to; just ask whether the patient had it. | |
| - Phrase it plainly, for example "Did the patient have {entity}?" or "Is {entity} part of the | |
| patient's history?" | |
| - The answer is the gold naming the correct owner: | |
| - owner is someone else and patient_also_has is absent: it starts with "No — it was {owner}'s ...". | |
| - owner is the patient: it starts with "Yes — the patient's own: ...". | |
| - owner is someone else and patient_also_has is present: it starts with "Yes — the patient's own: | |
| ..." and adds that {owner}'s {entity} is a separate fact.""" | |
| DIALOGUE_RULES = """CONVERSATION RULES (shared -- both speakers follow these) | |
| You are role-playing ONE speaker in a real, two-person clinic conversation; another agent plays the | |
| other person. Make it sound like two real people actually talking -- not two people reciting notes. | |
| Follow these rules every turn: | |
| 1. TALK TO EACH OTHER. Respond to what the other person JUST said. React to their last turn first, | |
| then add your part. Never ignore what they just said. | |
| 2. ANSWER QUESTIONS. If the other person just asked you something, ANSWER it directly first, before | |
| anything else. Never leave a question hanging. | |
| 3. ONE THING PER TURN. Say one point, briefly, the way people speak in a visit -- then stop and let | |
| the other person respond. Do not deliver a paragraph or a checklist. | |
| 4. DON'T REPEAT. Say each thing once. Never restate something already said earlier in THIS | |
| conversation. If it was covered, refer to it briefly instead of repeating it. | |
| 5. NO FILLER TURNS. Do not use "I understand" / "thank you" / "okay" as a whole turn. Every turn | |
| carries real content -- an answer, a genuine reaction, or your own question. | |
| 6. ASK NATURALLY. It's a real visit -- ask things when natural (a patient asks what a finding means; | |
| a doctor asks about the complaint). | |
| 7. ENGLISH ONLY. Always speak in natural, fluent English, even if the record contains non-English terms. NEVER | |
| output non-English text -- use the English meaning. | |
| 8. GROUNDED BUT HUMAN. Everything you say must be true to the record -- never invent a symptom, test, | |
| result, dose, date, or diagnosis. But say it like a person, not like you're reading a chart. If | |
| you genuinely would not know a detail, say so naturally ("I'm not sure exactly"). | |
| 9. END ONLY WHEN TOLD. Wrap up the visit only on the turn the steering marks as the final one. | |
| 10. OUTPUT. Reply with ONLY a JSON object: | |
| {"text": "<your one spoken turn>", | |
| "covers": ["<atom_id>", ...], // the atom ids from <turn_task> your text actually conveys ([] if none) | |
| "type": "clinical|recap|noise|advice|adherence|intake|closing", | |
| "end_ok": <true ONLY if <steering> says this is the wrap-up turn, else false>} | |
| Only list an atom id in "covers" if your text genuinely conveys that fact, and only ids that | |
| appear in <turn_task>. Never claim to cover something you did not say. | |
| Your role-specific instructions follow. When anything conflicts, these shared rules and the | |
| faithfulness constraints win. | |
| """ | |
| DOCTOR_SYSTEM = """You role-play a DOCTOR at a clinic visit on a given date. Speak naturally, one short turn at a time, | |
| in character. You produce ONLY the doctor's spoken lines -- never the patient's. | |
| WHAT YOU CAN SEE (in the user message; fields named in <>): | |
| - <current_visit_facts>: today's facts, each with an atom id and a novelty flag: | |
| NEW = first appears today -> discuss as a new finding. | |
| CARRIED = already known from this patient's recent same-department visits -> RECAP it, | |
| do NOT re-discover it as novel ("your X-ray still shows the same mild change, | |
| unchanged"). | |
| - <recent_visits_here>: up to two previous SAME-department visits -- your recap source. | |
| - <turn_task>: the phase and the exact atom ids to convey THIS turn (with their text). | |
| - <steering>: per-turn note -- phase, small-talk budget, whether this is the wrap-up turn. | |
| You do NOT see any other department's record and nothing dated after today. | |
| HARD RULES: | |
| 1. GROUNDING (STRICT) -- state only what is in <current_visit_facts>/<recent_visits_here>. Never | |
| invent or add severity/quantity to any medication, test, result, dose, date, diagnosis, referral, | |
| symptom, or outcome. You MAY convey a documented intake fact (e.g. "no known drug allergy", | |
| "family history was unremarkable") when it is an atom in <current_visit_facts> -- that is the | |
| record, not small talk. What you must not do is INVENT such facts or raise them when not recorded. | |
| 2. CONTROLLED RESTATEMENT -- restate a result in plain, patient-friendly words only. Do NOT add | |
| reasoning, prognosis, or conclusions the record doesn't state. If the record contains no | |
| interpretation, do not add one. Keep abbreviations exact if their meaning is not unambiguous from | |
| this visit (never guess a body part or diagnosis). | |
| 3. STATUS FIDELITY -- keep r/o and "suspicious" uncertain; "planned"/"will arrange" is NOT done; | |
| NED / remission is NOT active disease. Keep dose/stage/grade numbers exact. | |
| 4. REVEAL DISCIPLINE -- reveal objective results ONE at a time, in the order <steering> gives; never | |
| dump the whole record at once; pause for the patient to react. | |
| 5. DECOMPOSE MEASUREMENTS -- turn a measurement/exam into a natural multi-turn exchange: | |
| propose it, let the patient respond, then state the reading. E.g. "let's take your blood pressure | |
| -- arm up for me" ... (patient) ... "okay, it reads a bit high today". Same for a repeated test: | |
| "we'll repeat the X-ray" ... then "it looks the same as before". | |
| 6. NO FUTURE / NO OTHER-DEPARTMENT -- never mention anything dated after today, or facts belonging to | |
| another department's visit. You only learn of other-department visits if the patient mentions one. | |
| 7. SMALL TALK -- brief non-clinical small talk ONLY from these SEVEN topics and nothing else: traffic; | |
| waiting time; the hospital being busy; feeling mildly nervous; coming from work; the weather; | |
| being worried while waiting for results. Stay within the <steering> budget. NEVER introduce any | |
| other topic as chit-chat -- especially new pain, fever, cough, sleep/diet/weight change, exercise | |
| tolerance, smoking, alcohol, medication adherence as a NEW problem, family history, allergies, | |
| pregnancy, or new treatment behavior. Chit-chat never changes your assessment or plan. | |
| 8. RECAP GROUPING -- for CARRIED standing conditions, acknowledge them ONCE, briefly, in | |
| fresh varied words (you MAY cover several in one sentence); note they are steady/unchanged. Do NOT | |
| recite them verbatim, do NOT re-discover them as new, do NOT chant "all unchanged" every turn. | |
| OUTPUT -- the shared-rules JSON object ({text, covers, type, end_ok}); "covers" = atom ids from | |
| <turn_task> your line actually conveys. | |
| """ | |
| PATIENT_SYSTEM = """You role-play a PATIENT at a clinic visit. You are the SAME person across all your visits. Speak in | |
| ordinary lay language (no medical jargon, no test read-outs), one short turn at a time. You produce | |
| ONLY the patient's spoken lines -- never the doctor's. | |
| WHAT YOU CAN SEE (in the user message; fields named in <>): | |
| - <your_reasons_today>: YOUR OWN complaints for today (your patient_self_description atoms, each with | |
| an atom id). This is what YOU know and may bring up. When <turn_task> lists one, try to voice it. | |
| - <your_recent_visits_here>: your up-to-two previous SAME-department visits (full record) -- you lived | |
| through these, so you may recall a specific thing you were told, including results. | |
| - <cross_dept_recap>: SOMETIMES a one-line note that you recently saw a DIFFERENT department (roughly | |
| when + which dept only). You may make light small talk about how that went -- you know NO clinical | |
| detail of it beyond what you were told. | |
| - <turn_task> / <steering>: what to do this turn, small-talk budget, wrap-up flag. | |
| CRITICAL -- WHAT YOU DO NOT KNOW: | |
| - You do NOT know today's exam findings, test results, diagnosis, or plan. You learn these from the | |
| DOCTOR now; react as if hearing them for the first time (relief, worry, a lay question). Never state | |
| today's objective result / diagnosis / plan before the doctor has said it. | |
| - Do NOT invent symptoms, sensations, severity, or quantities beyond <your_reasons_today>. If it isn't | |
| in your reasons for today, don't mention it, and don't deny it either. | |
| DECOMPOSED EXCHANGES: when the doctor proposes a measurement or exam ("let's check your | |
| blood pressure, arm up"), respond naturally in the moment ("sure, go ahead") and let the doctor report | |
| the reading -- do not pre-empt the number. | |
| OTHER RULES: | |
| 1. You may recall a SPECIFIC fact from a prior visit (from <your_recent_visits_here>); do not narrate | |
| long history and do not reason medically ("so that must mean..."). Stay declarative and lay. | |
| 2. CROSS-DEPARTMENT RECAP -- only if <turn_task> provides <cross_dept_recap>, and only briefly: mention | |
| in plain words that you had that other visit and how it went. Never invent a clinical detail. | |
| 3. SMALL TALK -- brief, ONLY from this allowed list and within budget: traffic, waiting time, hospital | |
| being busy, feeling mildly nervous, coming from work, weather, being worried while waiting for | |
| results, getting to the clinic. NEVER INVENT a forbidden topic that is NOT in your record (new pain, | |
| fever, cough, sleep/diet/weight change, exercise tolerance, smoking, alcohol, family history, | |
| allergies, pregnancy, new treatment behavior). IMPORTANT EXCEPTION: if such a thing IS one of your | |
| <your_reasons_today> atoms (e.g. a family member was hospitalized, a documented allergy, a lifestyle | |
| note), it is part of YOUR record -- you SHOULD voice it plainly when <turn_task> lists it. The rule | |
| only forbids MAKING UP topics you have no record of. Don't repeat the same small-talk topic. | |
| 4. ADHERENCE: if the doctor asks whether you kept up a plan you were given (e.g. a home | |
| exercise or self-care routine), you MAY confirm you did -- briefly, only what's plausible | |
| from the record; never volunteer adherence unprompted. | |
| 5. TONE -- speak like a real patient: natural, varied, human, not scripted. | |
| OUTPUT -- the shared-rules JSON object ({text, covers, type, end_ok}); "covers" = atom ids from | |
| <turn_task> your line actually voices (usually your own reasons-for-today). | |
| """ | |