Instructions to use masahiroid/bart-large-cnn-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use masahiroid/bart-large-cnn-mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download masahiroid/bart-large-cnn-mlx --local-dir bart-large-cnn-mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Upload folder using huggingface_hub
Browse files- README.md +95 -0
- bart_mlx.py +219 -0
- model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
tags:
|
| 6 |
+
- mlx
|
| 7 |
+
- summarization
|
| 8 |
+
- bart
|
| 9 |
+
- seq2seq
|
| 10 |
+
base_model: facebook/bart-large-cnn
|
| 11 |
+
pipeline_tag: summarization
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# bart-large-cnn-mlx
|
| 15 |
+
|
| 16 |
+
[facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn)(CNN/DailyMail
|
| 17 |
+
で要約にファインチューニングされた、標準的なBART encoder-decoder)の
|
| 18 |
+
[MLX](https://github.com/ml-explore/mlx)版です。
|
| 19 |
+
|
| 20 |
+
This is an unofficial MLX conversion of
|
| 21 |
+
[facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn).
|
| 22 |
+
All credit for the original model goes to its authors (Meta/Facebook AI).
|
| 23 |
+
|
| 24 |
+
## これはmlx_lmでは読み込めません
|
| 25 |
+
|
| 26 |
+
`mlx_lm`は因果言語モデル(decoder-only)専用で、BARTのようなencoder-decoderモデルには
|
| 27 |
+
対応していないため、**MLXでの実装をゼロから書き起こして**変換しています。同梱の
|
| 28 |
+
`bart_mlx.py`が必要です。KVキャッシュ付きの自己回帰デコードも実装済みです。
|
| 29 |
+
|
| 30 |
+
## 制約
|
| 31 |
+
|
| 32 |
+
- **グリーディデコードのみ**: 元モデルの既定の生成設定はビームサーチ(`num_beams=4`)+
|
| 33 |
+
3-gramの繰り返し禁止ですが、本実装はグリーディ(貪欲法)のみ対応です。
|
| 34 |
+
PyTorch版のビームサーチ結果とは文章として異なる場合があります(teacher forcingでの
|
| 35 |
+
ロジット比較では完全に一致することを確認済み — 下記精度検証を参照)。
|
| 36 |
+
- **パディング無し・単一文書のみ**: バッチで複数文書をまとめて処理する使い方は未対応です。
|
| 37 |
+
- **`forced_bos_token_id`の適用**: 元モデルの生成設定では、デコーダー開始トークンの直後に
|
| 38 |
+
強制的にトークンID 0を出力する設定になっています。これを省略すると出力が著しく劣化する
|
| 39 |
+
(グリーディデコードは最初のトークン選択に非常に敏感)ため、デフォルトで適用しています。
|
| 40 |
+
|
| 41 |
+
## 使い方
|
| 42 |
+
|
| 43 |
+
```python
|
| 44 |
+
import mlx.core as mx
|
| 45 |
+
from mlx.utils import tree_unflatten
|
| 46 |
+
from transformers import AutoTokenizer
|
| 47 |
+
|
| 48 |
+
from bart_mlx import BartMLX
|
| 49 |
+
|
| 50 |
+
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
|
| 51 |
+
model = BartMLX()
|
| 52 |
+
weights = mx.load("model.safetensors")
|
| 53 |
+
model.update(tree_unflatten(list(weights.items())))
|
| 54 |
+
mx.eval(model.parameters())
|
| 55 |
+
|
| 56 |
+
article = "..." # 要約したい文章
|
| 57 |
+
inputs = tokenizer(article, return_tensors="np")
|
| 58 |
+
input_ids = mx.array(inputs["input_ids"])
|
| 59 |
+
|
| 60 |
+
generated_ids = model.generate(input_ids, max_new_tokens=142)
|
| 61 |
+
summary = tokenizer.decode(generated_ids, skip_special_tokens=True)
|
| 62 |
+
print(summary)
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
## 精度検証
|
| 66 |
+
|
| 67 |
+
PyTorch fp32リファレンス(ビームサーチ出力をteacher forcingで本実装に与えてロジットを比較)
|
| 68 |
+
と、エッフェル塔に関する記事1件で比較:
|
| 69 |
+
|
| 70 |
+
| 精度 | teacher-forcingロジットのコサイン類似度 | 次トークンargmax一致率 |
|
| 71 |
+
|---|---|---|
|
| 72 |
+
| MLX fp32 | 1.0000001 | 100% |
|
| 73 |
+
| MLX fp16(本リリース) | 0.99999356 | 100% |
|
| 74 |
+
|
| 75 |
+
グリーディ生成の実文章比較(PyTorch版も`num_beams=1`でグリーディにした場合):
|
| 76 |
+
|
| 77 |
+
- PyTorch greedy: *"The tower is 324 metres (1,063 ft) tall, about the same height as an
|
| 78 |
+
81-storey building. It is the tallest structure in Paris and the second tallest
|
| 79 |
+
free-standing structure in France after the Millau Viaduct."*
|
| 80 |
+
- MLX greedy: 上記と**完全に同一の文章**を生成。
|
| 81 |
+
|
| 82 |
+
## Specs
|
| 83 |
+
|
| 84 |
+
| Item | Value |
|
| 85 |
+
|---|---|
|
| 86 |
+
| Base model | `facebook/bart-large-cnn`(BART-large、406M params) |
|
| 87 |
+
| Precision | float16 |
|
| 88 |
+
| Framework | MLX(`bart_mlx.py`によるゼロからの実装、KVキャッシュ付き自己回帰デコード) |
|
| 89 |
+
|
| 90 |
+
## Notes
|
| 91 |
+
|
| 92 |
+
- This is a community conversion, not an official release from Meta.
|
| 93 |
+
- 本変換は非公式のコミュニティ版です。
|
| 94 |
+
- セキュリティー監査には[model-audit-lite](https://github.com/masahirocom/model-audit-lite)を
|
| 95 |
+
使用しています(詳細は`SECURITY.md`)。
|
bart_mlx.py
ADDED
|
@@ -0,0 +1,219 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""From-scratch MLX reimplementation of facebook/bart-large-cnn (standard BART
|
| 2 |
+
encoder-decoder for abstractive summarization).
|
| 3 |
+
|
| 4 |
+
No mlx_lm support (BART is encoder-decoder, not a causal LM) and no
|
| 5 |
+
mlx-vlm/mlx-embeddings coverage either (this isn't a vision-language or
|
| 6 |
+
embedding/reranker model), so reimplemented directly against the HF
|
| 7 |
+
`transformers` BART source (`modeling_bart.py`).
|
| 8 |
+
|
| 9 |
+
Simplification, matching the DETR/YOLOS patterns in this toolkit: assumes a
|
| 10 |
+
single sequence with no padding (no attention_mask handling on the encoder
|
| 11 |
+
side). Summarize one document at a time; no batched mixed-length inputs.
|
| 12 |
+
|
| 13 |
+
Greedy decoding only -- the reference pipeline's default generation config
|
| 14 |
+
uses beam search (num_beams=4) plus a no-repeat-3-gram constraint, which are
|
| 15 |
+
NOT implemented here. Teacher-forced logits (not generated text) are what's
|
| 16 |
+
validated against the PyTorch reference; greedy MLX generation is expected to
|
| 17 |
+
differ in wording from the official beam-search summaries, not just in exact
|
| 18 |
+
tokens.
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
import math
|
| 22 |
+
|
| 23 |
+
import mlx.core as mx
|
| 24 |
+
import mlx.nn as nn
|
| 25 |
+
|
| 26 |
+
D_MODEL = 1024
|
| 27 |
+
NUM_HEADS = 16
|
| 28 |
+
HEAD_DIM = D_MODEL // NUM_HEADS
|
| 29 |
+
FFN_DIM = 4096
|
| 30 |
+
NUM_ENCODER_LAYERS = 12
|
| 31 |
+
NUM_DECODER_LAYERS = 12
|
| 32 |
+
VOCAB_SIZE = 50264
|
| 33 |
+
MAX_POSITION_EMBEDDINGS = 1024
|
| 34 |
+
POSITION_OFFSET = 2
|
| 35 |
+
DECODER_START_TOKEN_ID = 2
|
| 36 |
+
EOS_TOKEN_ID = 2
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
class BartAttention(nn.Module):
|
| 40 |
+
def __init__(self):
|
| 41 |
+
super().__init__()
|
| 42 |
+
self.q_proj = nn.Linear(D_MODEL, D_MODEL)
|
| 43 |
+
self.k_proj = nn.Linear(D_MODEL, D_MODEL)
|
| 44 |
+
self.v_proj = nn.Linear(D_MODEL, D_MODEL)
|
| 45 |
+
self.out_proj = nn.Linear(D_MODEL, D_MODEL)
|
| 46 |
+
|
| 47 |
+
def _split(self, x, b, n):
|
| 48 |
+
return x.reshape(b, n, NUM_HEADS, HEAD_DIM).transpose(0, 2, 1, 3)
|
| 49 |
+
|
| 50 |
+
def __call__(self, x, kv_source=None):
|
| 51 |
+
"""Self-attention over x (kv_source=None) or cross-attention to a
|
| 52 |
+
fixed kv_source. No causal masking here: callers that need causal
|
| 53 |
+
behavior (the decoder) rely on only ever passing already-causal
|
| 54 |
+
inputs (single-token incremental decoding with a growing cache)."""
|
| 55 |
+
b, n, _ = x.shape
|
| 56 |
+
q = self._split(self.q_proj(x), b, n)
|
| 57 |
+
src = x if kv_source is None else kv_source
|
| 58 |
+
src_n = src.shape[1]
|
| 59 |
+
k = self._split(self.k_proj(src), b, src_n)
|
| 60 |
+
v = self._split(self.v_proj(src), b, src_n)
|
| 61 |
+
out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
|
| 62 |
+
out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
|
| 63 |
+
return self.out_proj(out)
|
| 64 |
+
|
| 65 |
+
def project_kv(self, kv_source):
|
| 66 |
+
"""Precompute K/V projections once (used for decoder cross-attention,
|
| 67 |
+
where the encoder output never changes across decoding steps)."""
|
| 68 |
+
b, n, _ = kv_source.shape
|
| 69 |
+
k = self._split(self.k_proj(kv_source), b, n)
|
| 70 |
+
v = self._split(self.v_proj(kv_source), b, n)
|
| 71 |
+
return k, v
|
| 72 |
+
|
| 73 |
+
def with_precomputed_kv(self, x, k, v):
|
| 74 |
+
b, n, _ = x.shape
|
| 75 |
+
q = self._split(self.q_proj(x), b, n)
|
| 76 |
+
out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
|
| 77 |
+
out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
|
| 78 |
+
return self.out_proj(out)
|
| 79 |
+
|
| 80 |
+
def self_attn_with_cache(self, x, cache):
|
| 81 |
+
"""x: (B, 1, D) a single new token. cache: (k, v) each
|
| 82 |
+
(B, num_heads, cached_len, head_dim) or None. Returns (out, new_cache)."""
|
| 83 |
+
b, n, _ = x.shape
|
| 84 |
+
q = self._split(self.q_proj(x), b, n)
|
| 85 |
+
k_new = self._split(self.k_proj(x), b, n)
|
| 86 |
+
v_new = self._split(self.v_proj(x), b, n)
|
| 87 |
+
if cache is not None:
|
| 88 |
+
k_old, v_old = cache
|
| 89 |
+
k = mx.concatenate([k_old, k_new], axis=2)
|
| 90 |
+
v = mx.concatenate([v_old, v_new], axis=2)
|
| 91 |
+
else:
|
| 92 |
+
k, v = k_new, v_new
|
| 93 |
+
out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
|
| 94 |
+
out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
|
| 95 |
+
return self.out_proj(out), (k, v)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
class BartEncoderLayer(nn.Module):
|
| 99 |
+
def __init__(self):
|
| 100 |
+
super().__init__()
|
| 101 |
+
self.self_attn = BartAttention()
|
| 102 |
+
self.self_attn_layer_norm = nn.LayerNorm(D_MODEL)
|
| 103 |
+
self.fc1 = nn.Linear(D_MODEL, FFN_DIM)
|
| 104 |
+
self.fc2 = nn.Linear(FFN_DIM, D_MODEL)
|
| 105 |
+
self.final_layer_norm = nn.LayerNorm(D_MODEL)
|
| 106 |
+
|
| 107 |
+
def __call__(self, x):
|
| 108 |
+
x = x + self.self_attn(x)
|
| 109 |
+
x = self.self_attn_layer_norm(x)
|
| 110 |
+
h = self.fc2(nn.gelu(self.fc1(x)))
|
| 111 |
+
x = x + h
|
| 112 |
+
x = self.final_layer_norm(x)
|
| 113 |
+
return x
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
class BartDecoderLayer(nn.Module):
|
| 117 |
+
def __init__(self):
|
| 118 |
+
super().__init__()
|
| 119 |
+
self.self_attn = BartAttention()
|
| 120 |
+
self.self_attn_layer_norm = nn.LayerNorm(D_MODEL)
|
| 121 |
+
self.encoder_attn = BartAttention()
|
| 122 |
+
self.encoder_attn_layer_norm = nn.LayerNorm(D_MODEL)
|
| 123 |
+
self.fc1 = nn.Linear(D_MODEL, FFN_DIM)
|
| 124 |
+
self.fc2 = nn.Linear(FFN_DIM, D_MODEL)
|
| 125 |
+
self.final_layer_norm = nn.LayerNorm(D_MODEL)
|
| 126 |
+
|
| 127 |
+
def __call__(self, x, self_cache, cross_kv):
|
| 128 |
+
attn_out, new_self_cache = self.self_attn.self_attn_with_cache(x, self_cache)
|
| 129 |
+
x = x + attn_out
|
| 130 |
+
x = self.self_attn_layer_norm(x)
|
| 131 |
+
|
| 132 |
+
cross_k, cross_v = cross_kv
|
| 133 |
+
cross_out = self.encoder_attn.with_precomputed_kv(x, cross_k, cross_v)
|
| 134 |
+
x = x + cross_out
|
| 135 |
+
x = self.encoder_attn_layer_norm(x)
|
| 136 |
+
|
| 137 |
+
h = self.fc2(nn.gelu(self.fc1(x)))
|
| 138 |
+
x = x + h
|
| 139 |
+
x = self.final_layer_norm(x)
|
| 140 |
+
return x, new_self_cache
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
class BartMLX(nn.Module):
|
| 144 |
+
def __init__(self):
|
| 145 |
+
super().__init__()
|
| 146 |
+
self.encoder_embed_tokens = mx.zeros((VOCAB_SIZE, D_MODEL))
|
| 147 |
+
self.encoder_embed_positions = mx.zeros((MAX_POSITION_EMBEDDINGS + POSITION_OFFSET, D_MODEL))
|
| 148 |
+
self.encoder_layernorm_embedding = nn.LayerNorm(D_MODEL)
|
| 149 |
+
self.encoder_layers = [BartEncoderLayer() for _ in range(NUM_ENCODER_LAYERS)]
|
| 150 |
+
|
| 151 |
+
self.decoder_embed_tokens = mx.zeros((VOCAB_SIZE, D_MODEL))
|
| 152 |
+
self.decoder_embed_positions = mx.zeros((MAX_POSITION_EMBEDDINGS + POSITION_OFFSET, D_MODEL))
|
| 153 |
+
self.decoder_layernorm_embedding = nn.LayerNorm(D_MODEL)
|
| 154 |
+
self.decoder_layers = [BartDecoderLayer() for _ in range(NUM_DECODER_LAYERS)]
|
| 155 |
+
|
| 156 |
+
self.lm_head = nn.Linear(D_MODEL, VOCAB_SIZE, bias=False)
|
| 157 |
+
self.final_logits_bias = mx.zeros((1, VOCAB_SIZE))
|
| 158 |
+
|
| 159 |
+
def encode(self, input_ids):
|
| 160 |
+
b, n = input_ids.shape
|
| 161 |
+
x = self.encoder_embed_tokens[input_ids]
|
| 162 |
+
positions = mx.arange(n) + POSITION_OFFSET
|
| 163 |
+
x = x + self.encoder_embed_positions[positions][None]
|
| 164 |
+
x = self.encoder_layernorm_embedding(x)
|
| 165 |
+
for layer in self.encoder_layers:
|
| 166 |
+
x = layer(x)
|
| 167 |
+
return x
|
| 168 |
+
|
| 169 |
+
def decode_step(self, input_ids_step, position, self_caches, cross_kvs):
|
| 170 |
+
"""input_ids_step: (B, 1). position: python int, the absolute decoder
|
| 171 |
+
position of this token. self_caches: list of (k, v) or None per
|
| 172 |
+
layer. cross_kvs: list of (k, v) per layer, precomputed once."""
|
| 173 |
+
b = input_ids_step.shape[0]
|
| 174 |
+
x = self.decoder_embed_tokens[input_ids_step]
|
| 175 |
+
pos_idx = mx.array([position + POSITION_OFFSET])
|
| 176 |
+
x = x + self.decoder_embed_positions[pos_idx][None]
|
| 177 |
+
x = self.decoder_layernorm_embedding(x)
|
| 178 |
+
|
| 179 |
+
new_caches = []
|
| 180 |
+
for layer, cache, cross_kv in zip(self.decoder_layers, self_caches, cross_kvs):
|
| 181 |
+
x, new_cache = layer(x, cache, cross_kv)
|
| 182 |
+
new_caches.append(new_cache)
|
| 183 |
+
|
| 184 |
+
logits = self.lm_head(x) + self.final_logits_bias
|
| 185 |
+
return logits, new_caches
|
| 186 |
+
|
| 187 |
+
def precompute_cross_kv(self, encoder_hidden_states):
|
| 188 |
+
return [layer.encoder_attn.project_kv(encoder_hidden_states) for layer in self.decoder_layers]
|
| 189 |
+
|
| 190 |
+
def generate(self, input_ids, max_new_tokens=142, eos_token_id=EOS_TOKEN_ID, forced_bos_token_id=0):
|
| 191 |
+
"""Greedy decoding. input_ids: (1, src_len) encoder input.
|
| 192 |
+
Returns a list of generated token ids (not including the decoder
|
| 193 |
+
start token, including the eos token if produced).
|
| 194 |
+
|
| 195 |
+
`forced_bos_token_id`: BART's generation config forces this as the
|
| 196 |
+
first generated token after the decoder-start token (not left to
|
| 197 |
+
argmax) -- `facebook/bart-large-cnn`'s config sets this to 0. Skipping
|
| 198 |
+
it produces noticeably degenerate output (greedy decoding is very
|
| 199 |
+
sensitive to the first token), so it's applied here by default to
|
| 200 |
+
match the reference pipeline's behavior; pass `None` to disable it
|
| 201 |
+
and use pure argmax at every step instead."""
|
| 202 |
+
encoder_out = self.encode(input_ids)
|
| 203 |
+
cross_kvs = self.precompute_cross_kv(encoder_out)
|
| 204 |
+
|
| 205 |
+
self_caches = [None] * NUM_DECODER_LAYERS
|
| 206 |
+
token = mx.array([[DECODER_START_TOKEN_ID]])
|
| 207 |
+
generated = []
|
| 208 |
+
for position in range(max_new_tokens):
|
| 209 |
+
logits, self_caches = self.decode_step(token, position, self_caches, cross_kvs)
|
| 210 |
+
if position == 0 and forced_bos_token_id is not None:
|
| 211 |
+
next_id = forced_bos_token_id
|
| 212 |
+
else:
|
| 213 |
+
mx.eval(logits)
|
| 214 |
+
next_id = int(mx.argmax(logits[0, -1]))
|
| 215 |
+
generated.append(next_id)
|
| 216 |
+
if next_id == eos_token_id:
|
| 217 |
+
break
|
| 218 |
+
token = mx.array([[next_id]])
|
| 219 |
+
return generated
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c7eabee28012c9da72b41308aa9388ee248ee371db51e4ed5a48cdb80352dd95
|
| 3 |
+
size 1018619788
|