masahiroid commited on
Commit
fa8d48e
·
verified ·
1 Parent(s): 45037f7

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +95 -0
  2. bart_mlx.py +219 -0
  3. model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ tags:
6
+ - mlx
7
+ - summarization
8
+ - bart
9
+ - seq2seq
10
+ base_model: facebook/bart-large-cnn
11
+ pipeline_tag: summarization
12
+ ---
13
+
14
+ # bart-large-cnn-mlx
15
+
16
+ [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn)(CNN/DailyMail
17
+ で要約にファインチューニングされた、標準的なBART encoder-decoder)の
18
+ [MLX](https://github.com/ml-explore/mlx)版です。
19
+
20
+ This is an unofficial MLX conversion of
21
+ [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn).
22
+ All credit for the original model goes to its authors (Meta/Facebook AI).
23
+
24
+ ## これはmlx_lmでは読み込めません
25
+
26
+ `mlx_lm`は因果言語モデル(decoder-only)専用で、BARTのようなencoder-decoderモデルには
27
+ 対応していないため、**MLXでの実装をゼロから書き起こして**変換しています。同梱の
28
+ `bart_mlx.py`が必要です。KVキャッシュ付きの自己回帰デコードも実装済みです。
29
+
30
+ ## 制約
31
+
32
+ - **グリーディデコードのみ**: 元モデルの既定の生成設定はビームサーチ(`num_beams=4`)+
33
+ 3-gramの繰り返し禁止ですが、本実装はグリーディ(貪欲法)のみ対応です。
34
+ PyTorch版のビームサーチ結果とは文章として異なる場合があります(teacher forcingでの
35
+ ロジット比較では完全に一致することを確認済み — 下記精度検証を参照)。
36
+ - **パディング無し・単一文書のみ**: バッチで複数文書をまとめて処理する使い方は未対応です。
37
+ - **`forced_bos_token_id`の適用**: 元モデルの生成設定では、デコーダー開始トークンの直後に
38
+ 強制的にトークンID 0を出力する設定になっています。これを省略すると出力が著しく劣化する
39
+ (グリーディデコードは最初のトークン選択に非常に敏感)ため、デフォルトで適用しています。
40
+
41
+ ## 使い方
42
+
43
+ ```python
44
+ import mlx.core as mx
45
+ from mlx.utils import tree_unflatten
46
+ from transformers import AutoTokenizer
47
+
48
+ from bart_mlx import BartMLX
49
+
50
+ tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
51
+ model = BartMLX()
52
+ weights = mx.load("model.safetensors")
53
+ model.update(tree_unflatten(list(weights.items())))
54
+ mx.eval(model.parameters())
55
+
56
+ article = "..." # 要約したい文章
57
+ inputs = tokenizer(article, return_tensors="np")
58
+ input_ids = mx.array(inputs["input_ids"])
59
+
60
+ generated_ids = model.generate(input_ids, max_new_tokens=142)
61
+ summary = tokenizer.decode(generated_ids, skip_special_tokens=True)
62
+ print(summary)
63
+ ```
64
+
65
+ ## 精度検証
66
+
67
+ PyTorch fp32リファレンス(ビームサーチ出力をteacher forcingで本実装に与えてロジットを比較)
68
+ と、エッフェル塔に関する記事1件で比較:
69
+
70
+ | 精度 | teacher-forcingロジットのコサイン類似度 | 次トークンargmax一致率 |
71
+ |---|---|---|
72
+ | MLX fp32 | 1.0000001 | 100% |
73
+ | MLX fp16(本リリース) | 0.99999356 | 100% |
74
+
75
+ グリーディ生成の実文章比較(PyTorch版も`num_beams=1`でグリーディにした場合):
76
+
77
+ - PyTorch greedy: *"The tower is 324 metres (1,063 ft) tall, about the same height as an
78
+ 81-storey building. It is the tallest structure in Paris and the second tallest
79
+ free-standing structure in France after the Millau Viaduct."*
80
+ - MLX greedy: 上記と**完全に同一の文章**を生成。
81
+
82
+ ## Specs
83
+
84
+ | Item | Value |
85
+ |---|---|
86
+ | Base model | `facebook/bart-large-cnn`(BART-large、406M params) |
87
+ | Precision | float16 |
88
+ | Framework | MLX(`bart_mlx.py`によるゼロからの実装、KVキャッシュ付き自己回帰デコード) |
89
+
90
+ ## Notes
91
+
92
+ - This is a community conversion, not an official release from Meta.
93
+ - 本変換は非公式のコミュニティ版です。
94
+ - セキュリティー監査には[model-audit-lite](https://github.com/masahirocom/model-audit-lite)を
95
+ 使用しています(詳細は`SECURITY.md`)。
bart_mlx.py ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """From-scratch MLX reimplementation of facebook/bart-large-cnn (standard BART
2
+ encoder-decoder for abstractive summarization).
3
+
4
+ No mlx_lm support (BART is encoder-decoder, not a causal LM) and no
5
+ mlx-vlm/mlx-embeddings coverage either (this isn't a vision-language or
6
+ embedding/reranker model), so reimplemented directly against the HF
7
+ `transformers` BART source (`modeling_bart.py`).
8
+
9
+ Simplification, matching the DETR/YOLOS patterns in this toolkit: assumes a
10
+ single sequence with no padding (no attention_mask handling on the encoder
11
+ side). Summarize one document at a time; no batched mixed-length inputs.
12
+
13
+ Greedy decoding only -- the reference pipeline's default generation config
14
+ uses beam search (num_beams=4) plus a no-repeat-3-gram constraint, which are
15
+ NOT implemented here. Teacher-forced logits (not generated text) are what's
16
+ validated against the PyTorch reference; greedy MLX generation is expected to
17
+ differ in wording from the official beam-search summaries, not just in exact
18
+ tokens.
19
+ """
20
+
21
+ import math
22
+
23
+ import mlx.core as mx
24
+ import mlx.nn as nn
25
+
26
+ D_MODEL = 1024
27
+ NUM_HEADS = 16
28
+ HEAD_DIM = D_MODEL // NUM_HEADS
29
+ FFN_DIM = 4096
30
+ NUM_ENCODER_LAYERS = 12
31
+ NUM_DECODER_LAYERS = 12
32
+ VOCAB_SIZE = 50264
33
+ MAX_POSITION_EMBEDDINGS = 1024
34
+ POSITION_OFFSET = 2
35
+ DECODER_START_TOKEN_ID = 2
36
+ EOS_TOKEN_ID = 2
37
+
38
+
39
+ class BartAttention(nn.Module):
40
+ def __init__(self):
41
+ super().__init__()
42
+ self.q_proj = nn.Linear(D_MODEL, D_MODEL)
43
+ self.k_proj = nn.Linear(D_MODEL, D_MODEL)
44
+ self.v_proj = nn.Linear(D_MODEL, D_MODEL)
45
+ self.out_proj = nn.Linear(D_MODEL, D_MODEL)
46
+
47
+ def _split(self, x, b, n):
48
+ return x.reshape(b, n, NUM_HEADS, HEAD_DIM).transpose(0, 2, 1, 3)
49
+
50
+ def __call__(self, x, kv_source=None):
51
+ """Self-attention over x (kv_source=None) or cross-attention to a
52
+ fixed kv_source. No causal masking here: callers that need causal
53
+ behavior (the decoder) rely on only ever passing already-causal
54
+ inputs (single-token incremental decoding with a growing cache)."""
55
+ b, n, _ = x.shape
56
+ q = self._split(self.q_proj(x), b, n)
57
+ src = x if kv_source is None else kv_source
58
+ src_n = src.shape[1]
59
+ k = self._split(self.k_proj(src), b, src_n)
60
+ v = self._split(self.v_proj(src), b, src_n)
61
+ out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
62
+ out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
63
+ return self.out_proj(out)
64
+
65
+ def project_kv(self, kv_source):
66
+ """Precompute K/V projections once (used for decoder cross-attention,
67
+ where the encoder output never changes across decoding steps)."""
68
+ b, n, _ = kv_source.shape
69
+ k = self._split(self.k_proj(kv_source), b, n)
70
+ v = self._split(self.v_proj(kv_source), b, n)
71
+ return k, v
72
+
73
+ def with_precomputed_kv(self, x, k, v):
74
+ b, n, _ = x.shape
75
+ q = self._split(self.q_proj(x), b, n)
76
+ out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
77
+ out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
78
+ return self.out_proj(out)
79
+
80
+ def self_attn_with_cache(self, x, cache):
81
+ """x: (B, 1, D) a single new token. cache: (k, v) each
82
+ (B, num_heads, cached_len, head_dim) or None. Returns (out, new_cache)."""
83
+ b, n, _ = x.shape
84
+ q = self._split(self.q_proj(x), b, n)
85
+ k_new = self._split(self.k_proj(x), b, n)
86
+ v_new = self._split(self.v_proj(x), b, n)
87
+ if cache is not None:
88
+ k_old, v_old = cache
89
+ k = mx.concatenate([k_old, k_new], axis=2)
90
+ v = mx.concatenate([v_old, v_new], axis=2)
91
+ else:
92
+ k, v = k_new, v_new
93
+ out = mx.fast.scaled_dot_product_attention(q, k, v, scale=1.0 / math.sqrt(HEAD_DIM))
94
+ out = out.transpose(0, 2, 1, 3).reshape(b, n, D_MODEL)
95
+ return self.out_proj(out), (k, v)
96
+
97
+
98
+ class BartEncoderLayer(nn.Module):
99
+ def __init__(self):
100
+ super().__init__()
101
+ self.self_attn = BartAttention()
102
+ self.self_attn_layer_norm = nn.LayerNorm(D_MODEL)
103
+ self.fc1 = nn.Linear(D_MODEL, FFN_DIM)
104
+ self.fc2 = nn.Linear(FFN_DIM, D_MODEL)
105
+ self.final_layer_norm = nn.LayerNorm(D_MODEL)
106
+
107
+ def __call__(self, x):
108
+ x = x + self.self_attn(x)
109
+ x = self.self_attn_layer_norm(x)
110
+ h = self.fc2(nn.gelu(self.fc1(x)))
111
+ x = x + h
112
+ x = self.final_layer_norm(x)
113
+ return x
114
+
115
+
116
+ class BartDecoderLayer(nn.Module):
117
+ def __init__(self):
118
+ super().__init__()
119
+ self.self_attn = BartAttention()
120
+ self.self_attn_layer_norm = nn.LayerNorm(D_MODEL)
121
+ self.encoder_attn = BartAttention()
122
+ self.encoder_attn_layer_norm = nn.LayerNorm(D_MODEL)
123
+ self.fc1 = nn.Linear(D_MODEL, FFN_DIM)
124
+ self.fc2 = nn.Linear(FFN_DIM, D_MODEL)
125
+ self.final_layer_norm = nn.LayerNorm(D_MODEL)
126
+
127
+ def __call__(self, x, self_cache, cross_kv):
128
+ attn_out, new_self_cache = self.self_attn.self_attn_with_cache(x, self_cache)
129
+ x = x + attn_out
130
+ x = self.self_attn_layer_norm(x)
131
+
132
+ cross_k, cross_v = cross_kv
133
+ cross_out = self.encoder_attn.with_precomputed_kv(x, cross_k, cross_v)
134
+ x = x + cross_out
135
+ x = self.encoder_attn_layer_norm(x)
136
+
137
+ h = self.fc2(nn.gelu(self.fc1(x)))
138
+ x = x + h
139
+ x = self.final_layer_norm(x)
140
+ return x, new_self_cache
141
+
142
+
143
+ class BartMLX(nn.Module):
144
+ def __init__(self):
145
+ super().__init__()
146
+ self.encoder_embed_tokens = mx.zeros((VOCAB_SIZE, D_MODEL))
147
+ self.encoder_embed_positions = mx.zeros((MAX_POSITION_EMBEDDINGS + POSITION_OFFSET, D_MODEL))
148
+ self.encoder_layernorm_embedding = nn.LayerNorm(D_MODEL)
149
+ self.encoder_layers = [BartEncoderLayer() for _ in range(NUM_ENCODER_LAYERS)]
150
+
151
+ self.decoder_embed_tokens = mx.zeros((VOCAB_SIZE, D_MODEL))
152
+ self.decoder_embed_positions = mx.zeros((MAX_POSITION_EMBEDDINGS + POSITION_OFFSET, D_MODEL))
153
+ self.decoder_layernorm_embedding = nn.LayerNorm(D_MODEL)
154
+ self.decoder_layers = [BartDecoderLayer() for _ in range(NUM_DECODER_LAYERS)]
155
+
156
+ self.lm_head = nn.Linear(D_MODEL, VOCAB_SIZE, bias=False)
157
+ self.final_logits_bias = mx.zeros((1, VOCAB_SIZE))
158
+
159
+ def encode(self, input_ids):
160
+ b, n = input_ids.shape
161
+ x = self.encoder_embed_tokens[input_ids]
162
+ positions = mx.arange(n) + POSITION_OFFSET
163
+ x = x + self.encoder_embed_positions[positions][None]
164
+ x = self.encoder_layernorm_embedding(x)
165
+ for layer in self.encoder_layers:
166
+ x = layer(x)
167
+ return x
168
+
169
+ def decode_step(self, input_ids_step, position, self_caches, cross_kvs):
170
+ """input_ids_step: (B, 1). position: python int, the absolute decoder
171
+ position of this token. self_caches: list of (k, v) or None per
172
+ layer. cross_kvs: list of (k, v) per layer, precomputed once."""
173
+ b = input_ids_step.shape[0]
174
+ x = self.decoder_embed_tokens[input_ids_step]
175
+ pos_idx = mx.array([position + POSITION_OFFSET])
176
+ x = x + self.decoder_embed_positions[pos_idx][None]
177
+ x = self.decoder_layernorm_embedding(x)
178
+
179
+ new_caches = []
180
+ for layer, cache, cross_kv in zip(self.decoder_layers, self_caches, cross_kvs):
181
+ x, new_cache = layer(x, cache, cross_kv)
182
+ new_caches.append(new_cache)
183
+
184
+ logits = self.lm_head(x) + self.final_logits_bias
185
+ return logits, new_caches
186
+
187
+ def precompute_cross_kv(self, encoder_hidden_states):
188
+ return [layer.encoder_attn.project_kv(encoder_hidden_states) for layer in self.decoder_layers]
189
+
190
+ def generate(self, input_ids, max_new_tokens=142, eos_token_id=EOS_TOKEN_ID, forced_bos_token_id=0):
191
+ """Greedy decoding. input_ids: (1, src_len) encoder input.
192
+ Returns a list of generated token ids (not including the decoder
193
+ start token, including the eos token if produced).
194
+
195
+ `forced_bos_token_id`: BART's generation config forces this as the
196
+ first generated token after the decoder-start token (not left to
197
+ argmax) -- `facebook/bart-large-cnn`'s config sets this to 0. Skipping
198
+ it produces noticeably degenerate output (greedy decoding is very
199
+ sensitive to the first token), so it's applied here by default to
200
+ match the reference pipeline's behavior; pass `None` to disable it
201
+ and use pure argmax at every step instead."""
202
+ encoder_out = self.encode(input_ids)
203
+ cross_kvs = self.precompute_cross_kv(encoder_out)
204
+
205
+ self_caches = [None] * NUM_DECODER_LAYERS
206
+ token = mx.array([[DECODER_START_TOKEN_ID]])
207
+ generated = []
208
+ for position in range(max_new_tokens):
209
+ logits, self_caches = self.decode_step(token, position, self_caches, cross_kvs)
210
+ if position == 0 and forced_bos_token_id is not None:
211
+ next_id = forced_bos_token_id
212
+ else:
213
+ mx.eval(logits)
214
+ next_id = int(mx.argmax(logits[0, -1]))
215
+ generated.append(next_id)
216
+ if next_id == eos_token_id:
217
+ break
218
+ token = mx.array([[next_id]])
219
+ return generated
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7eabee28012c9da72b41308aa9388ee248ee371db51e4ed5a48cdb80352dd95
3
+ size 1018619788