---
language:
  - el
library_name: transformers
pipeline_tag: text-generation
license: cc-by-nc-4.0
tags:
  - greek
  - text-generation
  - instruction-tuned
  - llama
---

# LexisLM-500m-Instruct

LexisLM-500m-Instruct is a Greek-focused instruction model fine-tuned from the 502-million-parameter LexisLM-500M base checkpoint. It is a decoder-only model exported in BF16 and compatible with the Transformers Llama architecture.

## Model details

| Property | Value |
|---|---|
| Layers | 16 |
| Hidden size | 1,536 |
| Attention heads | 16 |
| Context length | 4,096 tokens |
| Vocabulary | 32,004 tokens |
| Tokenizer | Greek byte-level BPE with chat tokens |
| Fine-tuning checkpoint | `runs/greek_500m_22bn_ctx4k_sft_mixed/best.pt`, step 3,802 |
| Fine-tuning data | Greek EuroAlpaca, Aya Greek SFT, and Greek chat SFT mixtures |

The chat template prepends `<eos>`, formats each message as `<|role|>` followed by a newline and content, and ends each turn with `<|endofturn|>`. Generation stops at `<eos>` or `<|endofturn|>`.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ComputationalLinguisticsPlatform/LexisLM-500m-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
torch.set_num_threads(4)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)
torch.manual_seed(42)
messages = [{"role": "user", "content": "Η πρωτεύουσα της Ελλάδας είναι"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True,
)
inputs = {key: value.to(model.device) for key, value in inputs.items()
          if key in ("input_ids", "attention_mask")}
output = model.generate(
    **inputs, max_new_tokens=64, do_sample=True, temperature=0.1, top_p=0.9,
    eos_token_id=[tokenizer.eos_token_id,
                  tokenizer.convert_tokens_to_ids("<|endofturn|>")],
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

Output from this example (CPU FP32, seed 42):

```text
Η πρωτεύουσα της Ελλάδας είναι η Αθήνα.
```

## Evaluation

The selected SFT checkpoint was evaluated with this project's Greek answer-likelihood protocol. ARC used 25 demonstrations, MMLU used 5, and TruthfulQA used 0. These scores are specific to that protocol and are not instruction-following evaluations.

| Benchmark | Metric | Score |
|---|---|---:|
| ARC-Easy | Character-normalized accuracy | 41.58% |
| ARC-Challenge | Character-normalized accuracy | 25.26% |
| MMLU | Accuracy | 26.04% |
| TruthfulQA | MC1 accuracy | 23.13% |
| TruthfulQA | MC2 score | 41.35% |

## Limitations

The model may produce incorrect, biased, or unsafe text. Greek is the primary target language. The benchmarks above do not establish reliability for a specific application.

## Provenance and license

Pretraining used Greek Wikipedia and [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), which is released under ODC-By 1.0. Supervised fine-tuning used the three data mixtures listed above, including [EuroAlpaca](https://huggingface.co/datasets/Telefonica/EuroAlpaca). EuroAlpaca is licensed under CC BY-NC 4.0, so this instruction-tuned model is released for noncommercial use under CC BY-NC 4.0. The export was produced with `scripts/export_hf.py` from the selected training checkpoint.