OpenModel-Labs/Comet-1-Preview-mlp-gain-110

Comet-1-Preview — MLP gain ×1.10

⚠️ Experimental weight edit — not validated

This is not an official checkpoint. It is a byte-level modification of OpenModel-Labs/Comet-1-Preview made to test one hypothesis. No quality measurement of any kind has been run on these weights. They may be better than the base model, the same, or materially worse — including degraded coherence, repetition, or broken instruction following. Do not use this for anything that matters, and do not report benchmark numbers for it as if they describe Comet-1-Preview.

If you want the real model, use the base repository.

What was changed

Exactly four mlp_scale vectors — 4,096 float32 values each — in the MLP blocks of layers 1, 2, 6 and 8 were multiplied by 1.10:

Tensor	Old mean	New mean
blocks.1.mlp_scale	0.134716	0.148188
blocks.2.mlp_scale	0.136419	0.150060
blocks.6.mlp_scale	0.151504	0.166654
blocks.8.mlp_scale	0.088125	0.096937

Every element was scaled; the ratio is 1.1000000 to float32 precision. The edit is not behaviour-preserving and not quality-validated. The rationale is that these are per-channel output gains on the early-layer MLP path, so a uniform ×1.1 is the cheapest possible probe of whether a little more early MLP signal helps or hurts.

Nothing else is different. The file is the same 28,005,783,576 bytes with the same tensor layout and header as the base checkpoint.

Provenance

Source	OpenModel-Labs/Comet-1-Preview
Source revision	e7702ddcd418dda8eb9856c18aa94c79f67a377d
Edit	mlp_scale ×1.10 in layers 1, 2, 6, 8
Tool	apply_checked_patch — checksum-gated copy, applied on Molab
Report	edit_report.json (source and edited SHA-256 per tensor)

Verification performed on the output file as published here:

• All four edited tensors match edited_sha256 in edit_report.json. ✅
• The safetensors header is byte-identical to the base checkpoint's header. ✅
• All 28.01 GB outside the four patched 16 KB ranges is byte-identical to the base checkpoint, checked region by region: 5 regions, 28,005,783,576 bytes total compared. ✅
• The file is exactly the expected size and was copied from a download whose header hash matched the audited revision. ✅

The base model's own equivalence tests (custom code vs. train.py, cache/padding behaviour, logits in float32) are inherited claims from the base repository — they were run against the base weights, not re-run here. A ×1.1 scale does not change tensor shapes, so the architecture loads exactly as the base does.

Quick start

Identical to the base model — the architecture is not built into Transformers, so loading needs trust_remote_code=True, which runs modeling_comet.py and configuration_comet.py from this repository. Read them first.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "OpenModel-Labs/Comet-1-Preview-mlp-gain-110"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,   # on older 4.x releases that reject `dtype`, use torch_dtype=
    device_map="auto",      # needs `accelerate`
)

inputs = tokenizer("Photosynthesis is the process by which", return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Memory: about 14 GB in bfloat16, 28 GB in float32. Requires a recent PyTorch 2.x and transformers 4.46 or newer.

How to evaluate this properly

If you want to know whether the edit does anything, compare this checkpoint against the base checkpoint on identical inputs. A sensible protocol:

1. Same prompts, same decoding settings (do_sample=False first, so runs are reproducible).
2. Fixed seeds for any sampling; watch for repetition and never-ending loops.
3. Perplexity on held-out text of the same genres the base was trained on (web text, maths, code).
4. Spot-check greedy completions on the base model's own prompt table — the base README lists the base continuations, so any divergence here is the edit showing through.
5. Report base vs. edited side by side. A result for one without the other says nothing.

If you run this, please open a discussion or PR with the numbers so the card can be updated.

Architecture and training (inherited, unchanged)

Comet-1-Preview is a 7.0B-parameter causal language model from OpenModel-Labs, trained from scratch with the single-file pipeline in train.py (internal name: OpenModel-1). Three out of every four blocks are grouped-query attention; the fourth replaces attention with a gated, dilated causal convolution. It is a base model: it continues text, it does not follow chat turns.

Parameters	7,001,436,928
Weights	model.safetensors, float32, 28 GB
Context	trained on 4,096-token sequences; position table covers 8,192
Tokenizer	GPT-NeoX-20B BPE, 50,277 tokens
Language	English web text, with maths and source code

<|endoftext|> (id 0) is the document separator and EOS; <|padding|> (id 1) is the pad token. No chat template is shipped. See the base model card for the full architecture, training and tokenizer description, including the base model's greedy-continuation examples — which do not describe this checkpoint.

Files

File	Notes
model.safetensors	28 GB, float32, 353 tensors, four mlp_scale vectors edited
config.json, configuration_comet.py, modeling_comet.py	Transformers custom code
tokenizer.json, tokenizer_config.json, special_tokens_map.json, generation_config.json	unchanged from base
train.py	unchanged from base
edit_report.json	not part of the model — provenance record of the edit

Limitations

• Unvalidated. As above: no benchmarks, no quality claims, no safety evaluation specific to this variant. The edit could plausibly degrade text quality.
• It is a base model with no instruction or safety tuning. It can produce incorrect, biased or offensive text, and it will state false things fluently.
• Training data is English web text, maths and code. Expect weak results in other languages.
• Sequences longer than 4,096 tokens were never seen in training; up to 8,192 the model runs, but quality there has not been measured.
• No license has been declared for the base weights, and this derivative inherits that ambiguity.