Upload release files

README.md

@@ -0,0 +1,86 @@
+---
+license: apache-2.0
+base_model: ByteDance-Seed/BAGEL-7B-MoT
+pipeline_tag: any-to-any
+library_name: bagel
+tags:
+- bagel
+- bagel-mot
+- thinkmorph-ssl
+- primer
+- multimodal
+- pretext
+datasets:
+- ThinkMorph/Pretext-270K
+---
+
+# ThinkMorph-SSL-Stage1
+
+This is the **intermediate Stage 1 model** of the PRIMER recipe. It is the EMA weights at
+step 5,500 of the pretext stage, trained from BAGEL-7B-MoT on
+[Pretext-270K](https://huggingface.co/datasets/ThinkMorph/Pretext-270K). The final model,
+[ThinkMorph-SSL](https://huggingface.co/ThinkMorph/ThinkMorph-SSL), starts its Stage 2 training from these weights.
+
+> **This model has only had pretext training, with no question-answer supervision.** Its
+> exercises take a corrupted input image and output a description, a restored image and a
+> second description. It is not meant to answer questions directly. The paper's appendix
+> reports roughly 40% on V\* and MMMU for this checkpoint. For question answering, use
+> ThinkMorph-SSL.
+
+## Files
+
+| File | Bytes | SHA256 |
+| --- | ---: | --- |
+| `ema.safetensors` | 29,351,872,574 | `f1dd6a3fb88c9145ac8c3d69fd60877cb359ec3996f3a8218580f621e47943ae` |
+| `ae.safetensors` | 167,664,678 | `e49d48aa3474df744ad718d11736d87f38d52bf7ca57dee6b8417978bfbfc9ad` |
+| `config.json` | 5,175 | `4bc85a4daaef591f0b539216d09a1ee2294a1f3a3bda9f95885c13ed191ae701` |
+| `llm_config.json` | 2,136 | `cb884fc65d117b8e5ec3856090224cd739c511e01c5c1de488cede5a54c8e994` |
+| `vit_config.json` | 1,830 | `78664013bf6778ad4ecaebe78c153ef7fd46f7638fa78c0df1cb17d5bd7df0ab` |
+| `generation_config.json` | 243 | `3a8f9087e486054c8a4a08dae2e5a3ba62e23da212b5b8c08bc42cb983c3459f` |
+| `vocab.json` | 3,383,407 | `87a257b04b17642a0688c98cd1df89c398bda4fee532d6f88b38a659ecb4ac8d` |
+| `merges.txt` | 1,671,853 | `8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5` |
+| `added_tokens.json` | 630 | `a0582f48ed8e795fda5b7b8eb2d98b492842dc229e78b66793bf545ce7ec52c8` |
+| `special_tokens_map.json` | 613 | `76862e765266b85aa9459767e33cbaf13970f327a0e88d1c65846c2ddd3a1ecd` |
+| `tokenizer_config.json` | 4,864 | `b7af0d4c9b917db05150006b466cfa2e8424b3915ff5fe2023a6f512a5e16dd5` |
+
+All files except `ema.safetensors` are byte-identical to the ThinkMorph-SSL release. The
+checkpoint's own config and tokenizer files were compared with those of ThinkMorph-SSL and
+are the same. `ema.safetensors` uses the same format as ThinkMorph-SSL: 1,451 bfloat16 tensors
+with the same names and shapes. The unused 26th SigLIP layer is left out.
+
+## How the file was made
+
+1. The source is the FSDP2 checkpoint of run `0503_stage0_270k_32gpu` at step 5,500
+   (32 `.distcp` shards, 117,699,388,909 bytes). It holds both the regular weights and the
+   EMA weights, all 1,467 tensors of each, in fp32.
+2. `scripts/merge_checkpoint.py --checkpoint-type ema` from the PRIMER code release merged
+   the EMA state into a single fp32 `model.safetensors`. All 1,467 weights came from the EMA
+   state; none were filled in from a base model.
+3. The 1,451 tensors that ThinkMorph-SSL ships were cast once to bfloat16, with no other
+   change, and saved as `ema.safetensors`. A spot check confirmed that the cast values match
+   exactly. The VAE weights are identical to ThinkMorph-SSL's, as expected, because the VAE
+   is frozen in both stages.
+
+## Training
+
+| Item | Value |
+| --- | --- |
+| Run | `0503_stage0_270k_32gpu` |
+| Initialization | BAGEL-7B-MoT (`lmms-lab/BAGEL-7B-MoT-ver.LE` layout and VAE) |
+| Data | Pretext-270K, 270,896 train rows; the val files are not used |
+| GPUs | 32 (4 nodes × 8) |
+| Learning rate | 2.83e-5, warmup 1,000 |
+| Configured horizon | 10,000 steps; this release is the step-5,500 EMA save |
+| EMA | decay 0.9998 from step 1,000, fp32 shadow |
+| Losses | text cross-entropy and image flow matching, weights 1 and 1; VAE frozen |
+
+## Loading
+
+Loading works the same way as for ThinkMorph-SSL. With the official BAGEL inference code or
+the PRIMER code release, point `model_path` at a local copy. The loader reads
+`ema.safetensors`.
+
+## License
+
+The weights are released under Apache-2.0, following BAGEL-7B-MoT. Pretext-270K has its own
+per-source terms; see its card.

added_tokens.json

@@ -0,0 +1,25 @@
+{
+  "</tool_call>": 151658,
+  "<gen_image>": 151665,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

ae.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e49d48aa3474df744ad718d11736d87f38d52bf7ca57dee6b8417978bfbfc9ad
+size 167664678

config.json

@@ -0,0 +1,202 @@
+{
+  "architectures": [
+    "Bagel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_bagel.BagelConfig"
+  },
+  "ce_loss_reweighting": true,
+  "ce_weight": 1.0,
+  "connector_act": "gelu_pytorch_tanh",
+  "dtype": "float32",
+  "interpolate_pos": false,
+  "latent_patch_size": 2,
+  "llm_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "Qwen2ForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 151643,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 151645,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "freeze_und": false,
+    "hidden_act": "silu",
+    "hidden_size": 3584,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 18944,
+    "is_causal": true,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_module": "Qwen2MoTDecoderLayer",
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 28,
+    "min_length": 0,
+    "model_type": "qwen2",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 28,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 4,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "qk_norm": true,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000.0,
+    "sep_token_id": null,
+    "sliding_window": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 152064
+  },
+  "max_latent_size": 64,
+  "model_type": "bagel",
+  "mse_timestep_weighting": false,
+  "mse_weight": 1.0,
+  "timestep_shift": 1.0,
+  "transformers_version": "4.57.1",
+  "vae_config": {
+    "ch": 128,
+    "ch_mult": [
+      1,
+      2,
+      4,
+      4
+    ],
+    "downsample": 8,
+    "in_channels": 3,
+    "num_res_blocks": 2,
+    "out_ch": 3,
+    "resolution": 256,
+    "scale_factor": 0.3611,
+    "shift_factor": 0.1159,
+    "z_channels": 16
+  },
+  "visual_gen": true,
+  "visual_und": true,
+  "vit_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": null,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 980,
+    "intermediate_size": 4304,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "siglip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 26,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 14,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rope": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false
+  },
+  "vit_max_num_patch_per_side": 70,
+  "vit_rope": false,
+  "vit_select_layer": -2
+}

ema.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1dd6a3fb88c9145ac8c3d69fd60877cb359ec3996f3a8218580f621e47943ae
+size 29351872574

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "pad_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_p": 0.8,
+  "top_k": 20,
+  "transformers_version": "4.37.0"
+}

llm_config.json

@@ -0,0 +1,86 @@
+{
+  "_name_or_path": "",
+  "add_cross_attention": false,
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bad_words_ids": null,
+  "begin_suppress_tokens": null,
+  "bos_token_id": 151643,
+  "chunk_size_feed_forward": 0,
+  "cross_attention_hidden_size": null,
+  "decoder_start_token_id": null,
+  "diversity_penalty": 0.0,
+  "do_sample": false,
+  "dtype": "bfloat16",
+  "early_stopping": false,
+  "encoder_no_repeat_ngram_size": 0,
+  "eos_token_id": 151645,
+  "exponential_decay_length_penalty": null,
+  "finetuning_task": null,
+  "forced_bos_token_id": null,
+  "forced_eos_token_id": null,
+  "freeze_und": false,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "is_causal": true,
+  "is_decoder": false,
+  "is_encoder_decoder": false,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "layer_module": "Qwen2MoTDecoderLayer",
+  "length_penalty": 1.0,
+  "max_length": 20,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "min_length": 0,
+  "model_type": "qwen2",
+  "no_repeat_ngram_size": 0,
+  "num_attention_heads": 28,
+  "num_beam_groups": 1,
+  "num_beams": 1,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "num_return_sequences": 1,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "output_scores": false,
+  "pad_token_id": null,
+  "prefix": null,
+  "problem_type": null,
+  "pruned_heads": {},
+  "qk_norm": true,
+  "remove_invalid_values": false,
+  "repetition_penalty": 1.0,
+  "return_dict": true,
+  "return_dict_in_generate": false,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sep_token_id": null,
+  "sliding_window": null,
+  "suppress_tokens": null,
+  "task_specific_params": null,
+  "temperature": 1.0,
+  "tf_legacy_loss": false,
+  "tie_encoder_decoder": false,
+  "tie_word_embeddings": false,
+  "tokenizer_class": null,
+  "top_k": 50,
+  "top_p": 1.0,
+  "torchscript": false,
+  "typical_p": 1.0,
+  "use_bfloat16": false,
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,215 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<gen_image>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

vit_config.json

@@ -0,0 +1,74 @@
+{
+  "_name_or_path": "",
+  "add_cross_attention": false,
+  "architectures": null,
+  "attention_dropout": 0.0,
+  "bad_words_ids": null,
+  "begin_suppress_tokens": null,
+  "bos_token_id": null,
+  "chunk_size_feed_forward": 0,
+  "cross_attention_hidden_size": null,
+  "decoder_start_token_id": null,
+  "diversity_penalty": 0.0,
+  "do_sample": false,
+  "dtype": null,
+  "early_stopping": false,
+  "encoder_no_repeat_ngram_size": 0,
+  "eos_token_id": null,
+  "exponential_decay_length_penalty": null,
+  "finetuning_task": null,
+  "forced_bos_token_id": null,
+  "forced_eos_token_id": null,
+  "hidden_act": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "image_size": 980,
+  "intermediate_size": 4304,
+  "is_decoder": false,
+  "is_encoder_decoder": false,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "layer_norm_eps": 1e-06,
+  "length_penalty": 1.0,
+  "max_length": 20,
+  "min_length": 0,
+  "model_type": "siglip_vision_model",
+  "no_repeat_ngram_size": 0,
+  "num_attention_heads": 16,
+  "num_beam_groups": 1,
+  "num_beams": 1,
+  "num_channels": 3,
+  "num_hidden_layers": 26,
+  "num_return_sequences": 1,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "output_scores": false,
+  "pad_token_id": null,
+  "patch_size": 14,
+  "prefix": null,
+  "problem_type": null,
+  "pruned_heads": {},
+  "remove_invalid_values": false,
+  "repetition_penalty": 1.0,
+  "return_dict": true,
+  "return_dict_in_generate": false,
+  "rope": false,
+  "sep_token_id": null,
+  "suppress_tokens": null,
+  "task_specific_params": null,
+  "temperature": 1.0,
+  "tf_legacy_loss": false,
+  "tie_encoder_decoder": false,
+  "tie_word_embeddings": true,
+  "tokenizer_class": null,
+  "top_k": 50,
+  "top_p": 1.0,
+  "torchscript": false,
+  "typical_p": 1.0,
+  "use_bfloat16": false
+}