Support Transformers 4 and 5 in Cosmos-Embed1-336p remote code
Summary
Make Cosmos-Embed1 remote code load the existing checkpoints with both Transformers 4.57.6 and 5.17.0, without changing model weights or configuration.
- Vendor the Q-Former pruning and attention-mask helpers removed from Transformers 5; call
post_init()explicitly for model initialization. - Keep normalization constants outside meta-device model construction and move them to the input device during inference.
- Share bicubic positional-embedding interpolation between the Transformers 4 state-dict pre-hook and a Transformers 5
WeightConverter. The 336p and 448p checkpoints store a 257-token (224p-grid) positional embedding and need interpolation to 577 and 1025 tokens, respectively;ignore_mismatched_sizes=Truewould reinitialize this parameter rather than preserve it. - Add weight-free tests for both versions and update the model-card usage example and compatibility guidance. No model weights, tokenizer files, or config files are changed.
Validation
Fixed inputs: eight deterministic JPEG frames generated with NumPy seed 1234 and four fixed text prompts. The real adapter recorded video/text vectors and ranked scores with the original 4.57.6 code, patched 4.57.6 code, and patched 5.17.0 code using the same checkpoint weights.
| Check | 224p | 336p | 448p |
|---|---|---|---|
| Patched 4.57.6 vs original 4.57.6, CPU FP32: max absolute difference in video/text/scores | 0.0 / 0.0 / 0.0 | 0.0 / 0.0 / 0.0 | 0.0 / 0.0 / 0.0 |
| Patched 5.17.0 vs original 4.57.6, CPU FP32 | bitwise exact | bitwise exact | bitwise exact |
| Patched 5.17.0 vs original 4.57.6, Apple MPS FP32 | bitwise exact | bitwise exact | bitwise exact |
| Patched 5.17.0 vs original 4.57.6, CUDA (FP32 weights, BF16 autocast) | bitwise exact | bitwise exact | bitwise exact |
The weight-free tests in this PR pass under Transformers 4.57.6 (6 passed, 2 v5-only tests skipped) and 5.17.0 (8 passed). These measurements establish parity for the tested inputs and devices, not performance or downstream retrieval accuracy. Apple MPS is community-validation evidence, not a change to NVIDIA's supported-platform statement.
Review notes
The converter is registered only when the Transformers 5 conversion API is available; the 4.x pre-hook remains active. Both paths use the same interpolation arithmetic. The model-card example keeps module weights FP32, uses the upstream nvidia/* model ID, and does not require a fork. Equivalent changes are proposed independently for the other two resolution repositories.