metadata
license: cc-by-nc-nd-4.0
language:
- en
pipeline_tag: token-classification
tags:
- gpt-neox
- tokenizer
- pythia
- luna
- text-generation
LUNA Tokenizer
The shared tokenizer for the LUNA model family — used by both LUNA-100M and LUNA-300M.
It is the EleutherAI / Pythia-160m (GPT-NeoX) tokenizer, extracted into a standalone repository so both models — and any future LUNA variants — reference a single canonical source.
Details
| Property | Value |
|---|---|
| Tokenizer class | GPTNeoXTokenizer |
| Vocabulary size | 50,304 (50,254 + EOS padding to multiple of 128) |
| BOS token | `< |
| EOS token | `< |
| UNK token | `< |
add_prefix_space |
false |
| Upstream | EleutherAI/Pythia-160m (gpt-neox-20b lineage) |
| License | Apache-2.0 (upstream EleutherAI tokenizer) |
Files
| File | Description |
|---|---|
tokenizer.json |
Tokenizer vocabulary + merges (Hugging Face tokenizers format) |
tokenizer_config.json |
Tokenizer configuration (GPTNeoXTokenizer, special tokens) |
config.json |
Model-side config (gpt_neox, vocab_size = 50304) for reference |
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ASTERIZER/LUNA-Tokenizer")
ids = tok.encode("Hello, LUNA!")
print(tok.decode(ids))