Blame for README.md

---
language:
  - el
library_name: transformers
pipeline_tag: text-generation
license: cc-by-nc-4.0
tags:
  - greek
  - text-generation
  - instruction-tuned
  - llama
base_model: Konstantinos/LexisLM-1B
---

# LexisLM-1B-Instruct

LexisLM-1B-Instruct is a Greek-focused instruction model fine-tuned from LexisLM-1B. It is a 1.093-billion-parameter decoder-only model exported in BF16 and compatible with the Transformers Llama architecture.

## Model details

| Property | Value |
|---|---|
| Layers | 20 |
| Hidden size | 2,048 |
| Attention heads | 16 |
| Context length | 4,096 tokens |
| Vocabulary | 32,004 tokens |
| Tokenizer | Greek byte-level BPE with chat tokens |
| Fine-tuning checkpoint | `runs/greek_1b_22bn_sft_mixed/best.pt`, step 3,802 |
| Fine-tuning data | Greek EuroAlpaca, Aya Greek SFT, and Greek chat SFT mixtures |

The chat template prepends `<eos>`, formats each message as `<|role|>` followed by a newline and content, and ends each turn with `<|endofturn|>`. Generation stops at `<eos>` or `<|endofturn|>`.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ComputationalLinguisticsPlatform/LexisLM-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
torch.set_num_threads(4)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)
torch.manual_seed(42)
messages = [{"role": "user", "content": "Η πρωτεύουσα της Ελλάδας είναι"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True,
)
inputs = {key: value.to(model.device) for key, value in inputs.items()
          if key in ("input_ids", "attention_mask")}
output = model.generate(
    **inputs, max_new_tokens=64, do_sample=True, temperature=0.1, top_p=0.9,
    eos_token_id=[tokenizer.eos_token_id,
                  tokenizer.convert_tokens_to_ids("<|endofturn|>")],
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

Output from this example (CPU FP32, seed 42):

```text
Η Αθήνα είναι η πρωτεύουσα της Ελλάδας από το 1834.
```

## Evaluation


| Benchmark | Metric | Score |
|---|---|---:|
| ARC-Easy | Character-normalized accuracy | 44.15% |
| ARC-Challenge | Character-normalized accuracy | 27.91% |
| MMLU | Accuracy | 24.67% |
| TruthfulQA | MC1 accuracy | 22.52% |
| TruthfulQA | MC2 score | 40.90% |

## Limitations

The model may produce incorrect, biased, or unsafe text. Greek is the primary target language. The benchmarks above do not establish reliability for a specific application.

## Provenance

Pretraining used Greek Wikipedia and FineWeb-2. Supervised fine-tuning used the three data mixtures listed above, including [EuroAlpaca](https://huggingface.co/datasets/Telefonica/EuroAlpaca). EuroAlpaca is licensed under CC BY-NC 4.0, so this instruction-tuned model is released for noncommercial use under CC BY-NC 4.0.

## Citation
```
@misc{perifanos2026compactgreeklanguagemodels,
      title={Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers}, 
      author={Konstantinos Perifanos and Anne Laure Poite and Giannis Tsilikas and Ilias Katis and Stavros Deligiannidis and Dionysis Goutsos},
      year={2026},
      eprint={2611.10209},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2611.10209}, 
}
```