Veyra-Next-Preview

Architecture preview, not a completed StableCoherence checkpoint.

This repository packages the experimental Reflect + StableCoherence causal decoder as a Hugging Face custom-code model. The base shared-depth weights come from the verified 440,107,008-token Reflect continuation checkpoint. The new coherence.* tensors are deterministically initialized for inspection and experimentation, but they were not trained before the Modal run was stopped.

A stopped H100 retry reached trace step 2,420 and cumulative token count 519,405,568, but it ended before its first periodic checkpoint. Therefore no trained StableCoherence weight tensor exists to publish. training_trace.json preserves that partial run only as provenance; it is not evidence of final model quality.

Architecture

• Parameters: 213,466,112
• Vocabulary: 50,304
• Context: 2,048
• Hidden size: 1,024
• FFN width: 2,688
• Attention heads: 16
• Unique Transformer blocks: 12 shared blocks + 1 StableCoherence block
• Logical depth: 24 shared-depth applications + 1 coherence refinement
• Positional encoding: RoPE
• Normalization: RMSNorm
• Feed-forward: SwiGLU-style gated MLP
• Embeddings: tied input/output
• Coherence: learned local clean/corrupted-sequence scorer and gated residual refinement

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "north-previews/Veyra-Next-Preview"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The capital of France is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=False))

The coherence scorer is also exposed:

scores = model.coherence_scores(inputs["input_ids"])

Do not interpret these scores as factual verification. In this preview they are untrained random-head outputs, not a trained coherence signal.

Provenance

• Source weights: Reflect phase-2 checkpoint
• Source checkpoint parameters: 201,008,128
• Source training tokens: 440,107,008
• Warm-start architecture parameters: 213,466,112
• New parameters: 12,457,984 under model.coherence.*
• Source data: EleutherAI/pile-standard-pythia-preshuffled, revision bac79b6820adb34e451f9a02cc1dc7cd920febf0
• Partial StableCoherence retry: 2,420 updates, 765.94 seconds, cumulative trace token count 519,405,568, no checkpoint saved
• Tokenizer: Pythia tokenizer files included in this repository

Limitations

• The StableCoherence parameters are fresh in this preview; the model card does not claim a trained gated model.
• The base model was trained for fewer than one billion tokens and remains weak at factual recall, arithmetic, instruction-following, and long-form coherence.
• The standard Pile stream is not deduplicated and is not fact-checked or instruction-filtered.
• Coherence scoring is a learned local corruption detector design; even when trained, it cannot guarantee factual or globally coherent output.
• The custom generation path recomputes the full sequence and does not implement a KV cache, so inference is slower than production models.
• Use trust_remote_code=True; inspect modeling_veyra.py before execution as with any custom-code repository.

Files

• model.safetensors: converted architecture-preview state dict
• configuration_veyra.py: HF config class
• modeling_veyra.py: HF causal-LM wrapper and StableCoherence implementation
• training_trace.json: partial stopped-run metrics, for provenance only
• provenance.json: exact checkpoint conversion and stopped-run metadata