Publish full structured rotary report, final model card, all results and tested serving release

e349b70 verified about 12 hours ago

13.6 kB

---
license: mit
language: [en]
library_name: transformers
pipeline_tag: feature-extraction
tags: [sentence-similarity, feature-extraction, consensus-distillation, rotary-embeddings, custom_code, safetensors]
datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts, AbstractPhil/captionbert-8192-v2-consensus]
base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]
---
# CaptionBERT 8192 rotary

A **54.1M-parameter** standalone caption sentence encoder distilled from the
published consensus of five BERT-family teachers. Inference uses this model
and its tokenizer, with **768-dimensional normalized sentence output**.

The default is **Relative 16 heads (N16), seed 1**, trained on
**4,000,000 paired caption rows** for **7,808 steps**. It is the
development-selected release from the completed structured-rotary campaign.
The two 4M architecture means are a near tie; the default follows the declared
preference for fewer additions. Both seed pairs and every other trained option
remain available.

12 layers · hidden width 512 · 16 heads · FFN2048 · trained input ceiling 256.
**8192 is the checked execution capacity; 8K semantic quality was not measured.**

[Plain-English report](REPORT.md) · [Technical companion](TECHNICAL.md) ·
[Training repository](https://github.com/AbstractEyes/captionbert-trainer) ·
[CPU TensorBoard viewer](https://colab.research.google.com/github/AbstractEyes/captionbert-trainer/blob/campaign-structured-20261106/notebooks/captionbert_tensorboard_viewer.ipynb)

## Load the model

~~~python
import torch
from transformers import AutoModel, AutoTokenizer
repo = "AbstractPhil/captionbert-8192-rotary"
revision = "ad14b77f4cc9017d513173f3b340e735af02ef14"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModel.from_pretrained(
    repo, revision=revision, trust_remote_code=True, use_safetensors=True
).eval()
batch = tokenizer(
    ["A cat watches from a windowsill.", "A feline looks out the window."],
    padding=True, truncation=True, max_length=256, return_tensors="pt"
)
with torch.inference_mode():
    output = model(**batch)
embeddings = output.pooler_output       # [2, 768], L2-normalized
tokens = output.last_hidden_state      # [2, sequence_length, 512]
similarity = (embeddings[0] @ embeddings[1]).item()
~~~

The saved tokenizer uses right padding, matching training. No teacher models
or training package are required. Pin the tested revision in the serving
verification report when reproducing exact results.

Weights are **model.safetensors**, saved with **safe_serialization=True** and
PyTorch header metadata **format: pt**. Self-contained configuration/model source
and AutoConfig/AutoModel mappings accompany them.
## Final sentence benchmarks
The default architecture and seed were selected using STS-B **validation**,
before reading these test outcomes. All six larger models are shown.
Spearman correlation, 64-token ceiling, same eight pinned task test splits;
the mean weights tasks equally.
| Recipe / option | Seed | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 4M ×4 epochs / Hierarchical | 1 | 0.5427 | 0.6536 | 0.4882 | 0.5523 | 0.5275 | 0.7023 | 0.6226 | 0.4753 | 0.5706 |
| 4M ×4 epochs / Hierarchical | 2 | 0.5540 | 0.6535 | 0.4932 | 0.5589 | 0.5285 | 0.6981 | 0.6243 | 0.5197 | 0.5788 |
| 4M ×4 epochs / Relative 16 heads | 1 | 0.5385 | 0.6451 | 0.4823 | 0.5466 | 0.5300 | 0.6997 | 0.6232 | 0.5169 | 0.5728 |
| 4M ×4 epochs / Relative 16 heads | 2 | 0.5448 | 0.6492 | 0.4884 | 0.5500 | 0.5316 | 0.7002 | 0.6249 | 0.4791 | 0.5710 |
| 1M ×16 epochs / Hierarchical | 1 | 0.5451 | 0.6553 | 0.4837 | 0.5260 | 0.5173 | 0.6950 | 0.6107 | 0.4657 | 0.5623 |
| 1M ×16 epochs / Hierarchical | 2 | 0.5410 | 0.6570 | 0.4913 | 0.5358 | 0.5244 | 0.7005 | 0.6046 | 0.4975 | 0.5690 |
Pair test means are hierarchical 4M **0.57467**, relative 4M **0.57191**, and hierarchical matched exposure **0.56567**. Hierarchical-minus-relative directions differ between seeds. The strongest individual test row is hierarchical seed 2; it does not replace the development-selected default.
The release is one checkpoint, with both seeds retained for comparison.
Training gradients use caption/consensus targets; STS-B validation labels
influence architecture and seed selection. The historical CaptionBERT v2
numbers are a separate experiment and are not treated as a matched rotary
comparison here.
### Development and geometry
| Recipe | Option | Seed | Dev STS-B | Dev teacher cosine | Dev participation rank |
| --- | --- | --- | --- | --- | --- |
| 4M ×4 epochs | Hierarchical | 1 | 0.68295 | 0.81017 | 96.60 |
| 4M ×4 epochs | Hierarchical | 2 | 0.68854 | 0.81066 | 96.51 |
| 4M ×4 epochs | Relative 16 heads | 1 | 0.68672 | 0.81769 | 97.20 |
| 4M ×4 epochs | Relative 16 heads | 2 | 0.68633 | 0.81811 | 97.13 |
| 1M ×16 epochs | Hierarchical | 1 | 0.68331 | 0.80801 | 95.63 |
| 1M ×16 epochs | Hierarchical | 2 | 0.67775 | 0.80879 | 95.51 |
| Recipe / option | Seed | Chunk060 sample | n | Teacher cosine | Participation rank | Self cosine |
| --- | --- | --- | --- | --- | --- | --- |
| 4M ×4 epochs / Hierarchical | 1 | All sampled rows | 5000 | 0.809758 | 95.952 | 0.003311 |
| 4M ×4 epochs / Hierarchical | 1 | >128-token rows | 5000 | 0.735772 | 75.211 | 0.008305 |
| 4M ×4 epochs / Hierarchical | 2 | All sampled rows | 5000 | 0.810098 | 95.854 | 0.003095 |
| 4M ×4 epochs / Hierarchical | 2 | >128-token rows | 5000 | 0.735145 | 75.173 | 0.008127 |
| 4M ×4 epochs / Relative 16 heads | 1 | All sampled rows | 5000 | 0.817975 | 96.337 | 0.003996 |
| 4M ×4 epochs / Relative 16 heads | 1 | >128-token rows | 5000 | 0.740089 | 76.073 | 0.010202 |
| 4M ×4 epochs / Relative 16 heads | 2 | All sampled rows | 5000 | 0.818179 | 96.243 | 0.003710 |
| 4M ×4 epochs / Relative 16 heads | 2 | >128-token rows | 5000 | 0.740014 | 76.053 | 0.009820 |
| 1M ×16 epochs / Hierarchical | 1 | All sampled rows | 5000 | 0.807663 | 94.974 | 0.003575 |
| 1M ×16 epochs / Hierarchical | 1 | >128-token rows | 5000 | 0.737254 | 73.677 | 0.011612 |
| 1M ×16 epochs / Hierarchical | 2 | All sampled rows | 5000 | 0.808123 | 94.868 | 0.003176 |
| 1M ×16 epochs / Hierarchical | 2 | >128-token rows | 5000 | 0.737987 | 73.511 | 0.010763 |
Caption geometry uses deterministic 5,000-row samples from the separate chunk 060,
plus a separate 5,000-row >128-token sample. Participation rank uses centered
covariance in FP64; self cosine is the exact off-diagonal mean.
Teacher cosine measures consensus reproduction, separately from capability.
## What the structured experiment found
The screen trained eight options under both weight seeds and repeated its
sixteen-head relative reference, for 18 screen models. The larger stage adds
the two4M architecture pairs and a1M matched-exposure hierarchical pair,
for 24 trained models in this campaign.
The complete whole-key model's designated lookup head found the correct earlier
record on all 10,146 sampled query positions under both seeds. Mean correct-record
probability was0.999510/0.999516, across1,935 target-bearing captions from 4,096
development rows. Its sentence-development result did not establish a benefit
from that accurate read. The full report separates word-record attention,
teacher reproduction, pooled sentence similarity and serving parity.
The adaptation uses a three-digit base-eight segment address, a zero-born value
path and, in complete binding options, whole-WordPiece key phases, record-only
norm 2.18 and lookup supervision. The default's actual rotary mode is
**NTK-scaled relative RoPE; 16 heads, scale 4; no hierarchical value path or lexical binding**. Complete binding models and controls remain under their
explicit experiment paths; they are not silently replaced by the default.
## Construction and continuation
- Screen:1M paired rows,4 epochs,1,952steps,200-step warmup.
- Larger4M: chunks000,001,002,003,004,006,009,010,4 epochs.
- Exposure control: original1M rows,16 epochs.
- Every larger recipe:7,808steps, batch 2,048,15,990,784 example presentations,
  400-step warmup, peak LR6e-4 and cosine decay to1e-6.
- Pure Adam, weight decay0, clipping1.0, symmetric InfoNCE at 0.07 plus MSE.
  Lookup loss weight0.1 only in its declared screen options.
- FP32 parameters/optimizer, BF16 training autocast, TF32 off.
- Caption/target pairing is exact flat row index. Chunk061 is development,
  chunk 060 caption test. No image-ID join, refit alignment or fallback teacher.
The [artifact census](campaigns/structured-20261106/artifact-census.json) pins every
run, final/best/midpoint weights, independent optimizer/RNG state, logs, raw
queries, data manifests and TensorBoard events. Original experiments retain their
paths. Recovery notebooks verify compatible identities, resume partial state and
skip only verified completed models.
## Numerical and shape verification
The selected default was tested from both its local export directory and the actual Hub repository at immutable revision **ad14b77f4cc9017d513173f3b340e735af02ef14**. Separate local and G4 processes compared it with the independently restored final training state and exact training implementation. Each default check contains twelve precision/shape cases plus two padding comparisons. Every tested native/AutoModel sentence and token output had maximum absolute difference zero. The G4 default check used Transformers 5.22.1, PyTorch 2.15.0+cu132 and Safetensors 0.8.0; the separate binding G4 check used Transformers 5.22.2. The local default checks stayed below 4.2 decimal GB total device use under an 18 GB guard. These are same-device, same-precision comparisons within each environment, not a claim that outputs are bitwise identical between different GPUs.
Measured reports: [local export](campaigns/structured-20261106/validation/deployment-local-parity.json), [Hub load on the local GPU](campaigns/structured-20261106/validation/deployment-remote-parity.json), [Hub load on G4](campaigns/structured-20261106/validation/deployment-colab-parity.json), and the [executed model-card example](campaigns/structured-20261106/validation/model-card-example.json).
Checks compare an independently loaded final training state with the separate
safe AutoModel export, including every state tensor and actual outputs.
They exercise multiple batch/length shapes, all-padding, right padding,
optional attention/pointer diagnostics and FP16/BF16 autocast with FP32 weights.
| Model / load | Transformers | PyTorch | Case entries | Max sentence / token delta | Peak observed device GB |
| --- | --- | --- | --- | --- | --- |
| Complete binding / local | not captured | 2.11.0+cu128 | 12 | 0 / 0 | 3.652 |
| Complete binding / Colab | 5.22.2 | 2.15.0+cu132 | 14 | 0 / 0 | 1.635 |
| Default / local directory | 4.57.6 | 2.11.0+cu128 | 14 | 0 / 0 | 4.120 |
| Default / Hub on local GPU | 4.57.6 | 2.11.0+cu128 | 14 | 0 / 0 | 4.384 |
| Default / Hub on Colab | 5.22.1 | 2.15.0+cu132 | 14 | 0 / 0 | 1.633 |
Outputs through 8,192 tokens are engineering shape/finite-value checks.
Long-context semantic quality,512-address retrieval in this encoder and
cross-hardware bitwise equality are not established by those checks.
Per-case timings are diagnostic wall times, not a throughput study.
## Limits
Training and caption evaluation use at most256 tokens; the sentence suite uses 64.
The data consists of caption rows with an English uncased WordPiece tokenizer;
image-level and near-duplicate independence across every chunk was not established.
Two weight seeds and the screen reference repeat do not provide a full-stage
population significance test.
Whole-word identity is exact after tokenization, including its uncasing.
The encoder is bidirectional and receives the complete input. A word-record read
here does not establish causal external-memory writing, coreference resolution,
or general binding-sensitive sentence quality. Frozen binding-path removal also
bypasses its norm operation; it does not isolate phase necessity for every
trainable head. The missing pointer-only trained control is a proposed next stage.
Use right padding for binding variants: shifting lexical word phases relative
to positional special-token phases can change their native output under left
padding. The serving verifier measures native/AutoModel agreement under both.
## Output convention
| Field | Shape | Meaning |
|---|---|---|
| last_hidden_state | [B,L,512] | Token states |
| pooler_output | [B,768] | L2-normalized sentence vector |
| hidden_states, when requested |13 tensors | Input plus 12 layers |
| attentions, when requested |12 tensors | [B,16,L,L] per layer |
| pointer_loss, when requested | scalar | Auxiliary diagnostic loss |
All-padding sentence vectors are zero. Token states are not promised to be zero
at padding positions. This API does not define an encode method or embedding
alias; use pooler_output for sentence similarity.
## Citation
~~~bibtex
@misc{abstractphil2026captionbertrotary,
  title = {Structured rotary CaptionBERT: precise word reads and the sentence-embedding test},
  author = {AbstractPhil},
  year = {2026},
  url = {https://huggingface.co/AbstractPhil/captionbert-8192-rotary}
}
~~~
MIT. Format follows the [CaptionBERT v2 card](https://huggingface.co/AbstractPhil/captionbert-8192-v2/blob/ed468fe5aed64a3d03a56a7f7811131d20f81ddb/README.md).

	---
	license: mit
	language: [en]
	library_name: transformers
	pipeline_tag: feature-extraction
	tags: [sentence-similarity, feature-extraction, consensus-distillation, rotary-embeddings, custom_code, safetensors]
	datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts, AbstractPhil/captionbert-8192-v2-consensus]
	base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]
	---

	# CaptionBERT 8192 rotary

	A 54.1M-parameter standalone caption sentence encoder distilled from the
	published consensus of five BERT-family teachers. Inference uses this model
	and its tokenizer, with 768-dimensional normalized sentence output.

	The default is Relative 16 heads (N16), seed 1, trained on
	4,000,000 paired caption rows for 7,808 steps. It is the
	development-selected release from the completed structured-rotary campaign.
	The two 4M architecture means are a near tie; the default follows the declared
	preference for fewer additions. Both seed pairs and every other trained option
	remain available.

	12 layers · hidden width 512 · 16 heads · FFN2048 · trained input ceiling 256.
	8192 is the checked execution capacity; 8K semantic quality was not measured.

	[Plain-English report](REPORT.md) · [Technical companion](TECHNICAL.md) ·
	[Training repository](https://github.com/AbstractEyes/captionbert-trainer) ·
	[CPU TensorBoard viewer](https://colab.research.google.com/github/AbstractEyes/captionbert-trainer/blob/campaign-structured-20261106/notebooks/captionbert_tensorboard_viewer.ipynb)

	## Load the model

	~~~python
	import torch
	from transformers import AutoModel, AutoTokenizer

	repo = "AbstractPhil/captionbert-8192-rotary"
	revision = "ad14b77f4cc9017d513173f3b340e735af02ef14"
	tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
	model = AutoModel.from_pretrained(
	repo, revision=revision, trust_remote_code=True, use_safetensors=True
	).eval()

	batch = tokenizer(
	["A cat watches from a windowsill.", "A feline looks out the window."],
	padding=True, truncation=True, max_length=256, return_tensors="pt"
	)
	with torch.inference_mode():
	output = model(**batch)

	embeddings = output.pooler_output # [2, 768], L2-normalized
	tokens = output.last_hidden_state # [2, sequence_length, 512]
	similarity = (embeddings[0] @ embeddings[1]).item()
	~~~

	The saved tokenizer uses right padding, matching training. No teacher models
	or training package are required. Pin the tested revision in the serving
	verification report when reproducing exact results.

	Weights are model.safetensors, saved with safe_serialization=True and
	PyTorch header metadata format: pt. Self-contained configuration/model source
	and AutoConfig/AutoModel mappings accompany them.

	## Final sentence benchmarks

	The default architecture and seed were selected using STS-B validation,
	before reading these test outcomes. All six larger models are shown.
	Spearman correlation, 64-token ceiling, same eight pinned task test splits;
	the mean weights tasks equally.

	\| Recipe / option \| Seed \| STS-B \| SICK-R \| STS12 \| STS13 \| STS14 \| STS15 \| STS16 \| BIOSSES \| Mean \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| 4M ×4 epochs / Hierarchical \| 1 \| 0.5427 \| 0.6536 \| 0.4882 \| 0.5523 \| 0.5275 \| 0.7023 \| 0.6226 \| 0.4753 \| 0.5706 \|
	\| 4M ×4 epochs / Hierarchical \| 2 \| 0.5540 \| 0.6535 \| 0.4932 \| 0.5589 \| 0.5285 \| 0.6981 \| 0.6243 \| 0.5197 \| 0.5788 \|
	\| 4M ×4 epochs / Relative 16 heads \| 1 \| 0.5385 \| 0.6451 \| 0.4823 \| 0.5466 \| 0.5300 \| 0.6997 \| 0.6232 \| 0.5169 \| 0.5728 \|
	\| 4M ×4 epochs / Relative 16 heads \| 2 \| 0.5448 \| 0.6492 \| 0.4884 \| 0.5500 \| 0.5316 \| 0.7002 \| 0.6249 \| 0.4791 \| 0.5710 \|
	\| 1M ×16 epochs / Hierarchical \| 1 \| 0.5451 \| 0.6553 \| 0.4837 \| 0.5260 \| 0.5173 \| 0.6950 \| 0.6107 \| 0.4657 \| 0.5623 \|
	\| 1M ×16 epochs / Hierarchical \| 2 \| 0.5410 \| 0.6570 \| 0.4913 \| 0.5358 \| 0.5244 \| 0.7005 \| 0.6046 \| 0.4975 \| 0.5690 \|

	Pair test means are hierarchical 4M 0.57467, relative 4M 0.57191, and hierarchical matched exposure 0.56567. Hierarchical-minus-relative directions differ between seeds. The strongest individual test row is hierarchical seed 2; it does not replace the development-selected default.

	The release is one checkpoint, with both seeds retained for comparison.
	Training gradients use caption/consensus targets; STS-B validation labels
	influence architecture and seed selection. The historical CaptionBERT v2
	numbers are a separate experiment and are not treated as a matched rotary
	comparison here.

	### Development and geometry

	\| Recipe \| Option \| Seed \| Dev STS-B \| Dev teacher cosine \| Dev participation rank \|
	\| --- \| --- \| --- \| --- \| --- \| --- \|
	\| 4M ×4 epochs \| Hierarchical \| 1 \| 0.68295 \| 0.81017 \| 96.60 \|
	\| 4M ×4 epochs \| Hierarchical \| 2 \| 0.68854 \| 0.81066 \| 96.51 \|
	\| 4M ×4 epochs \| Relative 16 heads \| 1 \| 0.68672 \| 0.81769 \| 97.20 \|
	\| 4M ×4 epochs \| Relative 16 heads \| 2 \| 0.68633 \| 0.81811 \| 97.13 \|
	\| 1M ×16 epochs \| Hierarchical \| 1 \| 0.68331 \| 0.80801 \| 95.63 \|
	\| 1M ×16 epochs \| Hierarchical \| 2 \| 0.67775 \| 0.80879 \| 95.51 \|

	\| Recipe / option \| Seed \| Chunk060 sample \| n \| Teacher cosine \| Participation rank \| Self cosine \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| 4M ×4 epochs / Hierarchical \| 1 \| All sampled rows \| 5000 \| 0.809758 \| 95.952 \| 0.003311 \|
	\| 4M ×4 epochs / Hierarchical \| 1 \| >128-token rows \| 5000 \| 0.735772 \| 75.211 \| 0.008305 \|
	\| 4M ×4 epochs / Hierarchical \| 2 \| All sampled rows \| 5000 \| 0.810098 \| 95.854 \| 0.003095 \|
	\| 4M ×4 epochs / Hierarchical \| 2 \| >128-token rows \| 5000 \| 0.735145 \| 75.173 \| 0.008127 \|
	\| 4M ×4 epochs / Relative 16 heads \| 1 \| All sampled rows \| 5000 \| 0.817975 \| 96.337 \| 0.003996 \|
	\| 4M ×4 epochs / Relative 16 heads \| 1 \| >128-token rows \| 5000 \| 0.740089 \| 76.073 \| 0.010202 \|
	\| 4M ×4 epochs / Relative 16 heads \| 2 \| All sampled rows \| 5000 \| 0.818179 \| 96.243 \| 0.003710 \|
	\| 4M ×4 epochs / Relative 16 heads \| 2 \| >128-token rows \| 5000 \| 0.740014 \| 76.053 \| 0.009820 \|
	\| 1M ×16 epochs / Hierarchical \| 1 \| All sampled rows \| 5000 \| 0.807663 \| 94.974 \| 0.003575 \|
	\| 1M ×16 epochs / Hierarchical \| 1 \| >128-token rows \| 5000 \| 0.737254 \| 73.677 \| 0.011612 \|
	\| 1M ×16 epochs / Hierarchical \| 2 \| All sampled rows \| 5000 \| 0.808123 \| 94.868 \| 0.003176 \|
	\| 1M ×16 epochs / Hierarchical \| 2 \| >128-token rows \| 5000 \| 0.737987 \| 73.511 \| 0.010763 \|

	Caption geometry uses deterministic 5,000-row samples from the separate chunk 060,
	plus a separate 5,000-row >128-token sample. Participation rank uses centered
	covariance in FP64; self cosine is the exact off-diagonal mean.
	Teacher cosine measures consensus reproduction, separately from capability.

	## What the structured experiment found

	The screen trained eight options under both weight seeds and repeated its
	sixteen-head relative reference, for 18 screen models. The larger stage adds
	the two4M architecture pairs and a1M matched-exposure hierarchical pair,
	for 24 trained models in this campaign.

	The complete whole-key model's designated lookup head found the correct earlier
	record on all 10,146 sampled query positions under both seeds. Mean correct-record
	probability was0.999510/0.999516, across1,935 target-bearing captions from 4,096
	development rows. Its sentence-development result did not establish a benefit
	from that accurate read. The full report separates word-record attention,
	teacher reproduction, pooled sentence similarity and serving parity.

	The adaptation uses a three-digit base-eight segment address, a zero-born value
	path and, in complete binding options, whole-WordPiece key phases, record-only
	norm 2.18 and lookup supervision. The default's actual rotary mode is
	NTK-scaled relative RoPE; 16 heads, scale 4; no hierarchical value path or lexical binding. Complete binding models and controls remain under their
	explicit experiment paths; they are not silently replaced by the default.

	## Construction and continuation

	- Screen:1M paired rows,4 epochs,1,952steps,200-step warmup.
	- Larger4M: chunks000,001,002,003,004,006,009,010,4 epochs.
	- Exposure control: original1M rows,16 epochs.
	- Every larger recipe:7,808steps, batch 2,048,15,990,784 example presentations,
	400-step warmup, peak LR6e-4 and cosine decay to1e-6.
	- Pure Adam, weight decay0, clipping1.0, symmetric InfoNCE at 0.07 plus MSE.
	Lookup loss weight0.1 only in its declared screen options.
	- FP32 parameters/optimizer, BF16 training autocast, TF32 off.
	- Caption/target pairing is exact flat row index. Chunk061 is development,
	chunk 060 caption test. No image-ID join, refit alignment or fallback teacher.

	The [artifact census](campaigns/structured-20261106/artifact-census.json) pins every
	run, final/best/midpoint weights, independent optimizer/RNG state, logs, raw
	queries, data manifests and TensorBoard events. Original experiments retain their
	paths. Recovery notebooks verify compatible identities, resume partial state and
	skip only verified completed models.

	## Numerical and shape verification

	The selected default was tested from both its local export directory and the actual Hub repository at immutable revision ad14b77f4cc9017d513173f3b340e735af02ef14. Separate local and G4 processes compared it with the independently restored final training state and exact training implementation. Each default check contains twelve precision/shape cases plus two padding comparisons. Every tested native/AutoModel sentence and token output had maximum absolute difference zero. The G4 default check used Transformers 5.22.1, PyTorch 2.15.0+cu132 and Safetensors 0.8.0; the separate binding G4 check used Transformers 5.22.2. The local default checks stayed below 4.2 decimal GB total device use under an 18 GB guard. These are same-device, same-precision comparisons within each environment, not a claim that outputs are bitwise identical between different GPUs.

	Measured reports: [local export](campaigns/structured-20261106/validation/deployment-local-parity.json), [Hub load on the local GPU](campaigns/structured-20261106/validation/deployment-remote-parity.json), [Hub load on G4](campaigns/structured-20261106/validation/deployment-colab-parity.json), and the [executed model-card example](campaigns/structured-20261106/validation/model-card-example.json).

	Checks compare an independently loaded final training state with the separate
	safe AutoModel export, including every state tensor and actual outputs.
	They exercise multiple batch/length shapes, all-padding, right padding,
	optional attention/pointer diagnostics and FP16/BF16 autocast with FP32 weights.

	\| Model / load \| Transformers \| PyTorch \| Case entries \| Max sentence / token delta \| Peak observed device GB \|
	\| --- \| --- \| --- \| --- \| --- \| --- \|
	\| Complete binding / local \| not captured \| 2.11.0+cu128 \| 12 \| 0 / 0 \| 3.652 \|
	\| Complete binding / Colab \| 5.22.2 \| 2.15.0+cu132 \| 14 \| 0 / 0 \| 1.635 \|
	\| Default / local directory \| 4.57.6 \| 2.11.0+cu128 \| 14 \| 0 / 0 \| 4.120 \|
	\| Default / Hub on local GPU \| 4.57.6 \| 2.11.0+cu128 \| 14 \| 0 / 0 \| 4.384 \|
	\| Default / Hub on Colab \| 5.22.1 \| 2.15.0+cu132 \| 14 \| 0 / 0 \| 1.633 \|

	Outputs through 8,192 tokens are engineering shape/finite-value checks.
	Long-context semantic quality,512-address retrieval in this encoder and
	cross-hardware bitwise equality are not established by those checks.
	Per-case timings are diagnostic wall times, not a throughput study.

	## Limits

	Training and caption evaluation use at most256 tokens; the sentence suite uses 64.
	The data consists of caption rows with an English uncased WordPiece tokenizer;
	image-level and near-duplicate independence across every chunk was not established.
	Two weight seeds and the screen reference repeat do not provide a full-stage
	population significance test.

	Whole-word identity is exact after tokenization, including its uncasing.
	The encoder is bidirectional and receives the complete input. A word-record read
	here does not establish causal external-memory writing, coreference resolution,
	or general binding-sensitive sentence quality. Frozen binding-path removal also
	bypasses its norm operation; it does not isolate phase necessity for every
	trainable head. The missing pointer-only trained control is a proposed next stage.

	Use right padding for binding variants: shifting lexical word phases relative
	to positional special-token phases can change their native output under left
	padding. The serving verifier measures native/AutoModel agreement under both.

	## Output convention

	\| Field \| Shape \| Meaning \|
	\|---\|---\|---\|
	\| last_hidden_state \| [B,L,512] \| Token states \|
	\| pooler_output \| [B,768] \| L2-normalized sentence vector \|
	\| hidden_states, when requested \|13 tensors \| Input plus 12 layers \|
	\| attentions, when requested \|12 tensors \| [B,16,L,L] per layer \|
	\| pointer_loss, when requested \| scalar \| Auxiliary diagnostic loss \|

	All-padding sentence vectors are zero. Token states are not promised to be zero
	at padding positions. This API does not define an encode method or embedding
	alias; use pooler_output for sentence similarity.

	## Citation

	~~~bibtex
	@misc{abstractphil2026captionbertrotary,
	title = {Structured rotary CaptionBERT: precise word reads and the sentence-embedding test},
	author = {AbstractPhil},
	year = {2026},
	url = {https://huggingface.co/AbstractPhil/captionbert-8192-rotary}
	}
	~~~

	MIT. Format follows the [CaptionBERT v2 card](https://huggingface.co/AbstractPhil/captionbert-8192-v2/blob/ed468fe5aed64a3d03a56a7f7811131d20f81ddb/README.md).