| --- |
| language: |
| - el |
| library_name: transformers |
| pipeline_tag: text-generation |
| license: cc-by-nc-4.0 |
| tags: |
| - greek |
| - text-generation |
| - instruction-tuned |
| - llama |
| --- |
| |
| # LexisLM-500m-Instruct |
|
|
| LexisLM-500m-Instruct is a Greek-focused instruction model fine-tuned from the 502-million-parameter LexisLM-500M base checkpoint. It is a decoder-only model exported in BF16 and compatible with the Transformers Llama architecture. |
|
|
| ## Model details |
|
|
| | Property | Value | |
| |---|---| |
| | Layers | 16 | |
| | Hidden size | 1,536 | |
| | Attention heads | 16 | |
| | Context length | 4,096 tokens | |
| | Vocabulary | 32,004 tokens | |
| | Tokenizer | Greek byte-level BPE with chat tokens | |
| | Fine-tuning checkpoint | `runs/greek_500m_22bn_ctx4k_sft_mixed/best.pt`, step 3,802 | |
| | Fine-tuning data | Greek EuroAlpaca, Aya Greek SFT, and Greek chat SFT mixtures | |
|
|
| The chat template prepends `<eos>`, formats each message as `<|role|>` followed by a newline and content, and ends each turn with `<|endofturn|>`. Generation stops at `<eos>` or `<|endofturn|>`. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "ComputationalLinguisticsPlatform/LexisLM-500m-Instruct" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| torch.set_num_threads(4) |
| model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32) |
| torch.manual_seed(42) |
| messages = [{"role": "user", "content": "Η πρωτεύουσα της Ελλάδας είναι"}] |
| inputs = tokenizer.apply_chat_template( |
| messages, add_generation_prompt=True, tokenize=True, |
| return_tensors="pt", return_dict=True, |
| ) |
| inputs = {key: value.to(model.device) for key, value in inputs.items() |
| if key in ("input_ids", "attention_mask")} |
| output = model.generate( |
| **inputs, max_new_tokens=64, do_sample=True, temperature=0.1, top_p=0.9, |
| eos_token_id=[tokenizer.eos_token_id, |
| tokenizer.convert_tokens_to_ids("<|endofturn|>")], |
| pad_token_id=tokenizer.pad_token_id, |
| ) |
| print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| Output from this example (CPU FP32, seed 42): |
|
|
| ```text |
| Η πρωτεύουσα της Ελλάδας είναι η Αθήνα. |
| ``` |
|
|
| ## Evaluation |
|
|
| The selected SFT checkpoint was evaluated with this project's Greek answer-likelihood protocol. ARC used 25 demonstrations, MMLU used 5, and TruthfulQA used 0. These scores are specific to that protocol and are not instruction-following evaluations. |
|
|
| | Benchmark | Metric | Score | |
| |---|---|---:| |
| | ARC-Easy | Character-normalized accuracy | 41.58% | |
| | ARC-Challenge | Character-normalized accuracy | 25.26% | |
| | MMLU | Accuracy | 26.04% | |
| | TruthfulQA | MC1 accuracy | 23.13% | |
| | TruthfulQA | MC2 score | 41.35% | |
|
|
| ## Limitations |
|
|
| The model may produce incorrect, biased, or unsafe text. Greek is the primary target language. The benchmarks above do not establish reliability for a specific application. |
|
|
| ## Provenance and license |
|
|
| Pretraining used Greek Wikipedia and [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), which is released under ODC-By 1.0. Supervised fine-tuning used the three data mixtures listed above, including [EuroAlpaca](https://huggingface.co/datasets/Telefonica/EuroAlpaca). EuroAlpaca is licensed under CC BY-NC 4.0, so this instruction-tuned model is released for noncommercial use under CC BY-NC 4.0. The export was produced with `scripts/export_hf.py` from the selected training checkpoint. |
|
|
