Verity-14B-v1

Verity-14B-v1 is a privacy analysis model developed for Veilance. It compares browser behavior observed during a particular visit with a supplied website privacy policy and produces a structured JSON report. It is based on Qwen/Qwen3-14B and follows the parameter-efficient fine-tuning approach used in earlier Verity releases.

Verity distinguishes what the policy says, what the browser observed, and what can reasonably be inferred from comparing them. Its findings are leads for review, not findings of legal noncompliance or proof of a website's intent.

Item	Description
Base model	Qwen/Qwen3-14B
Task	Privacy-policy and browser-telemetry comparison
Intended output	Structured JSON with findings and evidence
Inference mode	Qwen3 non-thinking mode (enable_thinking=False)
Intended distribution	PEFT / LoRA adapter for the base model; confirm against the published files
Reported benchmark	98% token accuracy; evaluation protocol not supplied
Release status	v1; task-level privacy-analysis results have not been supplied for this card

Release note: This card is a draft based on the previous Verity model cards, the reported 14B run configuration, and the reported 98% token-accuracy result. Confirm the artifact type, license, final run statistics, prompt, output schema, and evaluation protocol against the actual 14B checkpoint before publication. The base model's Apache-2.0 license does not by itself state the license selected for this adapter.

What the model does

The host application supplies two evidence sources:

1. Browser telemetry: A time-bounded record of observed requests, storage operations, browser API access, and related signals.
2. Policy material: Retrieved and validated privacy-policy text, together with information about whether the policy was found and applies to the observed site.

Verity may analyze cookies, local and session storage, IndexedDB, third-party hosts, tracker classifications, Canvas, WebGL, audio, device characteristics, WebRTC, permissions, service workers, and other browser-observable behavior. Which of those signals are available depends on the collection system and the particular visit.

The model does not fetch a policy, browse the site, or inspect network payloads by itself. Policy retrieval, applicability checks, telemetry collection, and input-size control belong to the host application.

Intended use

Verity is intended for research, product analysis, and human review of a website's privacy disclosures alongside Veilance telemetry or comparably structured observations. It can help identify disclosures that appear to match observed activity, disclosures that are broad relative to what was observed, and items that need closer inspection.

It is not intended to make autonomous legal, regulatory, or reputational determinations about an organization. A short observation cannot establish a site's behavior for every user, account, consent choice, device, region, or time.

Comparison labels

Label	Interpretation
matched	Supplied policy language reasonably describes the observed behavior.
partially_matched	The policy broadly addresses the behavior, but the observed detail is not clearly described at the same level of specificity.
policy_only	The policy describes a practice that was not observed in this visit. This does not show that it never occurs.
observed_only	Behavior was observed, but no sufficiently relevant disclosure was identified in the applicable policy material supplied.
possible_contradiction	Observed evidence may conflict with an explicit statement in the supplied policy. This label requires particularly careful review.
indeterminate	The available telemetry or policy evidence is insufficient for a reliable comparison.

An absent or unusable policy should generally lead to indeterminate analysis, not a series of claims that observed behavior was undisclosed. A missing disclosure in an otherwise applicable policy is not automatically a contradiction.

Inputs and outputs

The earlier Verity adapters use a chat-formatted prompt containing a system instruction and a user message with JSON. Their test-record wrapper has top-level telemetry and policy_document keys. An illustrative record is:

{
  "telemetry": {
    "domain_url": "https://example.com",
    "visit": {
      "snapshot_id": "example-visit",
      "observed_at": "2026-09-28T12:00:00Z",
      "duration_seconds": 30
    },
    "seen_behavior": {
      "observations": [
        {"type": "cookie_read", "count": 3}
      ]
    }
  },
  "policy_document": {
    "url": "https://example.com/privacy",
    "found": true,
    "applicable": true,
    "complete": true,
    "sections": [
      {
        "heading": "Cookies",
        "text": "We use cookies to keep the website working."
      }
    ]
  }
}

This record is synthetic and illustrates the wrapper, not a guarantee that the 14B production input schema uses these exact nested field names. Use records prepared by the actual 14B preprocessing pipeline for meaningful tests.

A Verity report is expected to have fields such as analysis, domain, findings, important_limitations, privacy_policy, and visit. Findings should identify the observed behavior, the relevant policy evidence where available, the comparison label, and the basis for the conclusion. analysis.counts should agree with the findings after deterministic validation by the host application.

Running a PEFT adapter

This example assumes the release contains a LoRA adapter with adapter_config.json and adapter weights. If the published artifact is a merged model, load that model directly instead. Install recent versions of the required packages; Qwen3 requires a Transformers release that supports its architecture.

python3 -m pip install -U torch transformers peft accelerate

Save this example as test_model.py. Run it from the adapter repository so that ADAPTER = "." resolves to its files, or change ADAPTER to the published Hugging Face repository ID.

import json
import sys
from pathlib import Path

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer


BASE_MODEL = "Qwen/Qwen3-14B"
ADAPTER = "."  # Or the published Verity-14B-v1 adapter repository ID.

# Example instruction based on the earlier Verity cards. Replace it with the
# exact system prompt used for 14B fine-tuning if that prompt differs.
SYSTEM_PROMPT = """You are Veilance Privacy Analyst. Compare observed browser telemetry against the supplied website privacy policy.

Keep policy claims, observed behavior, and conclusions separate.

Use only the evidence provided in the input. Do not invent policy language, telemetry, trackers, requests, or browser behavior. A behavior not observed during one visit must not be treated as proof that it never occurs. Missing disclosure is not automatically a contradiction.

Return the Verity structured comparison report as valid JSON only."""


def main(path):
    record = json.loads(Path(path).read_text(encoding="utf-8"))
    if not {"telemetry", "policy_document"}.issubset(record):
        raise ValueError("Record needs telemetry and policy_document keys")

    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
    base = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL,
        torch_dtype="auto",
        device_map="auto",
    )
    model = PeftModel.from_pretrained(base, ADAPTER).eval()

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": (
                "Analyze this Veilance observation and privacy policy:\n"
                + json.dumps(record, ensure_ascii=False, separators=(",", ":"))
            ),
        },
    ]
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
    inputs = tokenizer(
        prompt, return_tensors="pt", add_special_tokens=False
    ).to(model.device)

    # Leave room for the report within the 8,192-token training sequence.
    max_new_tokens = 2600
    if inputs["input_ids"].shape[-1] + max_new_tokens > 8192:
        raise ValueError("Input is too long; select relevant policy sections")

    with torch.inference_mode():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )

    generated = output[0, inputs["input_ids"].shape[-1]:]
    text = tokenizer.decode(generated, skip_special_tokens=True).strip()
    report = json.loads(text)  # Fail visibly if the output is not valid JSON.
    print(json.dumps(report, ensure_ascii=False, indent=2))


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python test_model.py path/to/record.json")
    main(sys.argv[1])

For example, after adding a prepared JSON record:

python test_model.py test_records/test_record_1.json

The test path is illustrative; only use it if that file is actually included in the release. A CUDA-capable GPU with enough memory for the base model, adapter, and generation cache is recommended. Quantized loading is a separate deployment choice and may affect results. Match the final 14B training prompt, chat template, and thinking setting when evaluating the release.

Training and evaluation

The reported 14B-v1 run configuration includes the following. These are configuration details, not evidence that a particular final checkpoint achieved any stated score.

Parameter	Reported configuration
Base model	Qwen/Qwen3-14B
Fine-tuning approach	LoRA / PEFT; verify final adapter configuration
Epochs	3 configured
Distributed world size	4 configured
Maximum training sequence length	8,192 tokens
Evaluation subset	4,096 examples configured
Evaluation interval	250 steps configured

Reported benchmark

Metric	Verity-14B-v1 result	Interpretation
Token accuracy	98%	Reported result; the evaluation split, sample size, scoring method, and checkpoint have not been supplied.

Token accuracy measures agreement with reference output tokens. It is not a 98% success rate for privacy-policy analysis, contradiction detection, legal interpretation, or real-world findings. The benchmark's procedure should accompany this result wherever it is quoted.

Not yet documented for this release: actual training and validation example counts, final checkpoint and step, LoRA rank and target modules, learning rate, data provenance and split method, training and evaluation losses, task-level metrics, and evaluation code or dataset version. Record these from the completed 14B run before publishing further performance claims.

The prior Verity-4B-RC card reported roughly 72% on an internal application-specific evaluation and approximately 0.980–0.984 evaluation token accuracy at representative checkpoints. Those are 4B results, not 14B results. Token accuracy measures prediction of reference tokens and must not be presented as accuracy of privacy conclusions. A larger base model alone does not establish improvement on Verity's task.

Useful 14B task evaluations include JSON parse rate, agreement on comparison labels (including per-label precision and recall), support for each finding in the supplied evidence, false-positive contradiction rate, treatment of missing or inapplicable policies, and consistency between findings and aggregate counts. Report the sample size and procedure alongside any percentages.

Evidence and limitations

• A request to a third-party host establishes that a browser request occurred; it does not establish its payload, whether personal data was transferred, or the recipient's purpose.
• Access to Canvas, WebGL, audio, or device characteristics does not, by itself, prove that a persistent fingerprint was created.
• Cookie or storage operations do not reveal values, ownership, retention, or subsequent use unless those facts appear in the supplied evidence.
• A tracker category is a classification of an observed component, not proof of profiling or targeted advertising in the observed session.
• A single visit cannot prove that policy-described behavior never happens. Consent choices, login state, geography, experiments, and time may change behavior.
• Missing, misidentified, truncated, or inapplicable policy text can produce misleading comparisons. The host should validate policy retrieval and applicability.
• Generated JSON can still contain invented evidence, missed matches, incorrect labels, or inconsistent counts. Validate and review it before presenting consequential findings.

For consequential use, review the original policy passage, telemetry record, and collection conditions. Treat possible_contradiction as a reason to investigate, not a legal or factual conclusion about wrongdoing.

Relationship to Veilance

Veilance collects and structures browser-observable privacy signals. Verity compares supplied observations with supplied privacy-policy language. The project code is available through the Veilance GitHub organization.

Before publication

Confirm the published repository ID and artifact type; choose and record the adapter's license; replace the example prompt with the exact 14B training prompt; document the data and final training run; run the inference example against a real included record; document the protocol behind the reported 98% token accuracy; and add task-level evaluation results when available. Do not copy the 4B score into this release.

References

• Qwen3-14B base model and chat-template guidance
• Hugging Face PEFT adapter loading

Disclaimer

Verity is an experimental technical-analysis model. Its output is not legal advice, a determination of regulatory compliance, or definitive evidence of the contents of network communications or a website's intent. Verify material findings against the original evidence.