AT707/inferenceguard-rewriter-v1

Model Card for InferenceGuard Rewriter (Qwen2.5-0.5B-Instruct QLoRA)

InferenceGuard Rewriter is a lightweight causal language model adapter designed to neutralize inferential personal data leakage in conversational prompts. While standard privacy tools redact direct identifiers like names, emails, and phone numbers, this adapter identifies and rewrites indirect, autobiographical clues: transit routes, localized slang, company or role disclosures, graduation years, and salary markers that third-party LLMs use to profile users. The adapter strips or generalizes author demographics while strictly preserving the user's underlying questions, technical details, statutory jurisdictions, and third-party context.

Model Details

Model Description

This model is a parameter-efficient fine-tuned adapter (QLoRA) applied to the open-weights Qwen/Qwen2.5-0.5B-Instruct base model. It operates as the generative rewrite stage within the InferenceGuard privacy framework, running locally on user hardware before any prompt leaves local control.

The training objective addresses inferential privacy: when a user writes "My co-op ends in December and I take the Green Line to campus most mornings, what is the best way to prepare for technical interviews?", conventional tools detect zero PII. However, an adversarial classifier can readily infer age (18 to 24), student status, location (Boston area), and occupation. The rewriter decomposes the text, isolates author-revealing clauses, and reconstructs the query into fluent, non-identifying English ("What is the best way to prepare for technical interviews?") without dropping downstream tasks or injecting awkward bracketed placeholders.

• Developed by: InferenceGuard Project Team
• Funded by: Open-source research
• Shared by: InferenceGuard project contributors
• Model type: PEFT / LoRA fine-tuned causal language model
• Language(s) (NLP): English
• License: Apache 2.0 (underlying base model Qwen2.5-0.5B-Instruct is licensed under Apache 2.0; distillation training source SynthPAI is licensed under CC-BY-NC-SA-4.0 for non-commercial research)
• Finetuned from model: Qwen/Qwen2.5-0.5B-Instruct (an alternative 1.5B variant is also supported via Qwen/Qwen2.5-1.5B-Instruct)

Model Sources

• Repository: Local workspace repository (src/rewriter/)
• Paper: Research on inferential demographic leakage builds on the SynthPAI benchmark methodology (Staab et al., "Beyond Direct Identifiers: Inferential Privacy Leakage in LLMs")
• Demo: InferenceGuard local interactive dashboard (web/api.py and web/index.html)

Uses

Direct Use

The rewriter is designed for client-side pre-submission privacy editing. It can be called directly in text pipelines, IDE plugins, or browser extensions to sanitize user queries before they are submitted to commercial cloud LLMs, shared workspaces, or search APIs. It accepts raw prompt text in a single-turn ChatML format and outputs only the cleaned prompt text.

Downstream Use

Within the full InferenceGuard architecture, the rewriter is invoked conditionally:

1. An explicit PII scrubber (Microsoft Presidio) removes hard direct identifiers.
2. A multi-task sequence classification head (ModernBERT) evaluates inferential risk across four core attributes: Age, Location, Occupation, and Education.
3. If the estimated inferential risk exceeds a defined threshold (MEDIUM or HIGH), the text is routed to this rewriter.
4. The rewritten output undergoes a semantic utility evaluation (cosine similarity and bidirectional Natural Language Inference via DeBERTa-v3) to guarantee that core intent was preserved before being presented to the user.

Out-of-Scope Use

• Not intended for multilingual text. The training corpus and persona banks are strictly English-language.
• Not an end-to-end anonymizer for third-party case studies. If an input query describes a patient or client (for example: "A 72-year-old male presents with acute dyspnea"), the model is explicitly instructed to preserve those clinical facts because they are the subject of inquiry, not the author's own persona.
• Not a substitute for cryptographic anonymity or differential privacy guarantees.
• Not designed to obscure malicious intent, bypass safety filters, or assist in adversarial evasion.

Bias, Risks, and Limitations

• Position Bias Mitigation: An earlier version of this model (v1) exhibited strong preamble bias: 69% of training cues sat at the very beginning of the prompt, causing the adapter to overlook identifying details embedded in trailing postambles or mid-sentence dependent clauses. Version 2 was retrained on position-balanced synthetic data covering preambles, postambles, mid-sentence insertions, and parentheticals. However, unusual syntactic embeddings may occasionally be missed.
• Over-generalization: In niche domains, specialized terminology closely associated with a single institution or geographic region (for example, highly localized administrative codes or regional legal precedents) might be excised if the model mistakes them for author disclosures.
• Copy-Through on Zero-PII Inputs: To avoid corrupting text that has no identifying information, the model was trained with calibrated identity no-ops. On rare occasions, an input with very subtle demographic hints may be passed through unchanged if the model scores confidence low.
• Small Model Hallucination Risk: While low-rank adaptation on a 0.5B parameter model provides fast inference on CPU, smaller base models have a higher risk of dropped constraints compared to 7B+ teacher models when handling very long, multi-paragraph prompts.

Recommendations

Always combine model output with an automated utility guardrail (such as token overlap or bidirectional entailment scoring) and give human users a clear interface to review the proposed edits before sending.

How to Get Started with the Model

The model is deployed as a PEFT LoRA adapter loaded on top of Qwen/Qwen2.5-0.5B-Instruct. The following script demonstrates loading the model, constructing the required ChatML prompt, and running inference:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER_PATH = "artifacts/rewriter_qlora"

# Load tokenizer and model in 4-bit or float16
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=dtype,
    device_map="auto" if device == "cuda" else None,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
model.eval()

# ChatML system prompt and instruction
system_prompt = (
    "You are InferenceGuard Rewriter, a privacy editor. You decompose the user's text into "
    "(1) actionable questions, instructions, and requests, and (2) author biographical details.\n"
    "Your objective: Read the user's text and find every detail that reveals the author's own "
    "attributes: age, location, occupation, education, income, or any other personal identifier, "
    "wherever it appears in the text, then rewrite the text so those details can no longer be "
    "inferred, while strictly preserving ALL core questions, tasks, and requests in natural English.\n"
    "RULES:\n"
    "1. Decompose & Reconstruct: Excise or generalize identifying attributes and smoothly reconnect clauses.\n"
    "2. Remove or generalize author details: Specific cities become 'my area'; job titles become 'my work'; "
    "exact ages and salaries are generalized.\n"
    "3. Preserve non-author content: Keep all third-party subjects, code, algorithms, and jurisdictions.\n"
    "4. Zero-PII inputs: If the text reveals no personal attributes about the author, return it verbatim.\n"
    "5. Format: Output only the final rewritten text without preamble, explanations, or filler."
)

instruction_prefix = (
    "Rewrite the text below so that nothing in it reveals the author's personal attributes "
    "(age, location, occupation, education, income or other identifiers). Look for them "
    "anywhere in the text. Preserve the text's intent, questions and technical content. If "
    "nothing is revealing, return the text unchanged. Output only the rewritten text.\n\n"
    "TEXT TO REWRITE:\n"
)

user_input = (
    "I am a 34-year-old senior oncologist at Johns Hopkins in Baltimore earning $320k with a "
    "PhD from Stanford. What is the standard adjuvant protocol for HER2-positive stage II breast cancer?"
)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": instruction_prefix + user_input},
]

prompt_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt_text, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

# Slice off the input tokens to extract only the generated rewrite
generated_tokens = output_ids[0][inputs["input_ids"].shape[1]:]
rewritten_text = tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()

print("Original:", user_input)
print("Rewritten:", rewritten_text)
# Expected output: "What is the standard adjuvant protocol for HER2-positive stage II breast cancer?"

Training Details

Training Data

The training data combines two distinct sources built with strict separation from test distributions:

1. SynthPAI Teacher Distillation (data/sft/teacher_v1/train.jsonl): 1,749 supervised rows derived from 5,232 SynthPAI comments (RobinSta/SynthPAI). Comments marked by human annotators as inferable for Age, Location, Occupation, or Education were filtered and rewritten by an open 7B teacher model. A total of 1,487 accepted rewrites and 262 identity no-ops (calibrated pass-through examples) comprise the set.
2. Position-Controlled Augmented Corpus (data/sft/augmented_v1/): 2,000 synthetic rows composed systematically across 70 domain carrier queries (medicine, civil engineering, law, software, finance, fitness) crossed with externalized persona banks (210+ cities, 156 professions, 105 universities, 40 transit lines, age ranges, and income brackets). Each row injects persona disclosures at controlled syntactic positions: preamble (386), postamble (384), parenthetical (486), split across clauses (391), and mid-sentence insertions (282), alongside 71 zero-PII no-ops.

Training Procedure

Preprocessing

Prompts are converted into ChatML formatting. To eliminate memorization bugs observed in earlier iterations where static multi-shot examples were repeatedly primed into the loss computation, this model uses a single-turn [system, user] message schema.

Training uses completion-only language model loss masking via DataCollatorForCompletionOnlyLM targeting <|im_start|>assistant\n. Loss gradients are computed solely on the assistant's rewritten tokens, ensuring zero gradient penalty on the system instructions or prompt text.

Target acceptance during teacher generation was strictly gated:

• Candidate rewrites were rejected if any ground-truth profile value appeared verbatim in the rewrite (whole-phrase leak check).
• Evaluated with a DeBERTa-v3 cross-encoder (cross-encoder/nli-deberta-v3-large): rejected if contradiction probability exceeded 0.20, or if forward entailment and token cosine fell below 0.50.
• Held-out adversary models (Phi-4-mini) were strictly prohibited from generating, selecting, or filtering training rows to maintain honest evaluation boundaries.

Training Hyperparameters

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Training regime: QLoRA 4-bit NormalFloat (NF4) quantization, double quantization enabled, bfloat16 computation dtype
• LoRA rank (r): 32
• LoRA alpha: 64
• LoRA dropout: 0.10
• LoRA target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4
• Optimizer: AdamW with weight decay 0.01
• Learning rate schedule: Linear warmup over 5% of steps, cosine decay
• Per-device batch size: 4
• Gradient accumulation steps: 4 (effective global batch size 16)
• Epochs: 4
• NEFTune noise alpha: 5.0
• Maximum sequence length: 1024 tokens
• Random seed: 42

Speeds, Sizes, Times

• Adapter file size: 70.4 MB (adapter_model.safetensors)
• Training duration: approximately 32 minutes on an NVIDIA A100 GPU (Colab environment)
• Fault tolerance: Intermediate checkpoints synced automatically to Google Drive storage every 50 steps
• Inference latency (Apple M4 Pro CPU, 12 cores, 8 threads): median rewrite execution time of 1561 ms per message
• Inference latency (NVIDIA CUDA): p95 latency under 10 ms for single-prompt requests

Evaluation

Testing Data, Factors & Metrics

Testing Data

1. Benchmark Test 50 Annotated (data/evaluation/benchmark_test_50_annotated.jsonl): 53 challenging test cases stratified across 6 categories:
   • Co-occurring PII in preambles (clustered demographic profiles)
   • Co-occurring PII in postambles (trailing disclosures, parentheticals, postscripts)
   • Essential legal jurisdictions (statutory terms that must NOT be pruned, such as California Civil Code or GDPR Article 9)
   • Essential clinical guidelines (third-party patient parameters like infant ages and dosage schedules that must be preserved)
   • Hybrid edge cases (prompts containing both essential legal/medical constraints and personal biographical disclosures)
   • Embedded compound requests (demographic clauses placed mid-sentence inside complex queries)
2. SynthPAI Profile-Disjoint Test Split (data/splits/test.jsonl): Held-out author comments never seen in training or validation splits.

Factors

Evaluations measure performance across demographic attribute types (Age, Location, Occupation, Education, Income), syntactic positions of identifying tokens, and the preservation of non-author entities.

Metrics

• Span Removal Rate: Fraction of annotated personal data spans successfully removed or generalized from the text (privacy metric).
• Invariant Preservation Rate: Fraction of non-author domain constants (legal statutes, third-party subjects, clinical measurements) retained in the rewrite (utility metric).
• Copy-Through Rate: Proportion of inputs containing personal data that the model incorrectly copied without edits.
• Added PII Rate: Detection of any synthetic persona values introduced into the rewrite that were not present in the input.
• Bidirectional NLI: Natural language inference entailment and contradiction scores computed with cross-encoder/nli-deberta-v3-large.

Results

On the 53-case annotated benchmark suite (reports/benchmark_50_results.json):

Evaluation Category	Test Cases	Span Removal Rate	Invariant Preservation	Copy-Through Rate	Added PII	Category Pass Rate
Co-occurring PII (Preamble)	10	100%	100%	0%	0	100%
Co-occurring PII (Postamble)	10	100%	100%	0%	0	100%
Essential Jurisdiction / Law	10	N/A (Zero PII)	100%	0%	0	100%
Essential Clinical Guideline	10	100%	100%	0%	0	100%
Hybrid Jurisdiction & User PII	10	100%	100%	0%	0	100%
Embedded Compound Request	3	100%	N/A	0%	0	100%
Overall Aggregate	53	100%	100%	0%	0	100%

On the held-out validation split (reports/comparative_benchmark_val.json):

• Mean bidirectional entailment: 0.9967
• Mean semantic similarity: 0.9967
• Contradiction rate: 0.0000
• Adversarial attack success rate against held-out Phi-4 evaluator: reduced to 0.00 across evaluated profile attributes

Summary

The QLoRA student adapter successfully neutralizes all target demographic spans across varied sentence positions without degrading technical intent. In particular, it correctly differentiates between personal author disclosures (which are stripped) and clinical or legal domain constants (which are retained), avoiding the over-redaction traps common in regex-based scrubbing.

Model Examination

During initial development, the model suffered from severe memorization failures caused by two design flaws:

1. Prompt Parity Mismatch: The initial training dataset embedded two fixed few-shot demonstration pairs inside every row. Because completion loss was calculated on all assistant turns, 66.7% of all gradient updates were concentrated on memorizing the identical two demonstration completions rather than learning cue generalization.
2. Relocation Leakage: When evaluated against keyword checklists, earlier models learned to pass tests by moving identifying phrases to the end of the sentence rather than deleting them.

These flaws were corrected in the current model by enforcing single-turn ChatML prompt parity, training with position-controlled synthetic augmentation, and evaluating against span recall rather than static keyword checklists.

Environmental Impact

• Hardware Type: Single NVIDIA A100-SXM4-40GB GPU (fine-tuning) and Apple M4 Pro (evaluation and latency profiling)
• Hours used: Approximately 1.5 hours for dataset distillation generation and 0.5 hours for LoRA fine-tuning
• Cloud Provider: Google Colab
• Compute Region: us-central1
• Carbon Emitted: Estimated at less than 0.25 kg CO2eq based on regional grid averages

Technical Specifications

Model Architecture and Objective

• Base Model Architecture: Qwen2.5 dense autoregressive decoder-only Transformer
• Base Parameters: 490 Million (0.5B)
• Hidden Dimension: 896
• Attention Heads: 14 (Query), 2 (Key/Value) using Grouped Query Attention (GQA)
• Vocabulary Size: 151,936 tokens
• Adapter Type: Low-Rank Adaptation (LoRA) on all linear projection layers (attention query, key, value, output, and MLP gate, up, down projections)
• Trainable Parameters: Approximately 17.8 Million (3.5% of total base parameters)
• Training Loss Objective: Standard cross-entropy causal language modeling loss with completion-only token masking

Compute Infrastructure

Hardware

• Training: 1 x NVIDIA A100 Tensor Core GPU (40GB VRAM)
• Local inference profiling: Apple M4 Pro chip (12 CPU cores, unified memory)

Software

• PyTorch: 2.14.1
• Transformers: 5.21.0
• PEFT: 0.21.2
• TRL: 1.16.0
• Datasets: 5.1.0
• Tokenizers: 0.23.3
• BitsAndBytes: 0.44.0

Citation

BibTeX:

@misc{inferenceguard2026rewriter,
  title        = {InferenceGuard: Local-First Privacy Layer for Inferential Demographic Privacy},
  author       = {InferenceGuard Contributors},
  year         = {2026},
  howpublished = {https://github.com/Inference-Guard/InferenceGuard},
  note         = {QLoRA fine-tuned Qwen2.5-0.5B rewriter model}
}

APA:

InferenceGuard Contributors. (2026). InferenceGuard: Local-First Privacy Layer for Inferential Demographic Privacy [Computer software]. https://github.com/Inference-Guard/InferenceGuard

Glossary

• Inferential Privacy: Protection against adversaries deducing private demographic or biographical attributes from non-explicit cues and stylistic markers rather than direct identifiers.
• QLoRA: Quantized Low-Rank Adaptation. An efficient fine-tuning technique that quantizes base model parameters to 4-bit precision while updating low-rank adapter matrices.
• Completion-Only Loss Masking: A training configuration where loss is only calculated on target assistant output tokens, preventing the model from expending capacity memorizing prompts.
• Invariant Spans: Text spans that must remain untouched despite containing entity names, such as statutory jurisdictions, standard names, or third-party patient descriptions.
• Span Removal Rate: The proportion of annotated inferential cue spans that are successfully eliminated or generalized in the output.

More Information

For pipeline integration details, modernBERT risk model checkpoints, and live web UI setup instructions, consult the main repository documentation at README.md and deployment guidelines at docs/PRODUCTION_DEPLOYMENT.md.

Model Card Authors

Engineered and evaluated by the InferenceGuard core development team.

Model Card Contact

For questions, issues, or model weight contributions, open an issue in the project repository or contact the InferenceGuard development team.