---
language: el
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# LexisLM-1B

LexisLM-1B is a Greek language, decoder-only model for text completion. It was pretrained for  on approximately 22 billion tokens from FineWeb2 and Greek Wikipedia. This release is the **base model**; it has not been instruction tuned for chat or question answering.

## Model details

| Property | Value |
| --- | --- |
| Architecture | Llama-compatible causal language model |
| Transformer layers | 20 |
| Hidden size | 2,048 |
| Attention heads | 16 |
| Context length | 4,096 tokens |
| Vocabulary size | 32,004 tokens |

The repository includes the model weights in `safetensors` format, the configuration, and the tokenizer. It can be loaded with Hugging Face Transformers.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ComputationalLinguisticsLab/LexisLM-1B"  # Or a local path to hf_pretrained
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

# Pretraining documents were separated by <eos>. Start a fresh document with it.
prompt = "Η Ελλάδα είναι μια χώρα"
prompt_ids = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids
eos_id = torch.tensor([[tokenizer.eos_token_id]])
input_ids = torch.cat([eos_id, prompt_ids], dim=1).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output_ids[0, input_ids.shape[1] :], skip_special_tokens=True))
```

Install `torch`, `transformers`, and `accelerate` to run this example. Keep the prompt and generated tokens within the 4,096-token context window.

## Evaluation results

The following results were reported with **0 few-shot examples**. Scores are shown as percentages. ARC uses normalized accuracy; the other benchmarks use their reported accuracy or multiple-choice metrics.

| Benchmark | Metric | Score | Questions |
| --- | --- | ---: | ---: |
| ARC-Challenge | Normalized accuracy | 25.34% | 1,168 |
| ARC-Easy | Normalized accuracy | 33.38% | 2,376 |
| MMLU | Accuracy | 23.15% | 14,042 |
| TruthfulQA | MC1 | 24.60% | 817 |
| TruthfulQA | MC2 | 43.49% | 817 |

## Intended use and limitations

LexisLM-1B is intended for experimenting with Greek text generation and as a starting point for further fine-tuning. Because it is a base model, chat-style instructions may not produce reliable answers. Generated text can be inaccurate or reflect biases in the training data; review it before using it in consequential settings.

## License


LexisLM-1B model weights are released under Apache-2.0. Training data included FineWeb2 (HuggingFaceFW), available under ODC-By-1.0. Dataset and underlying content rights remain with their respective holders.


## Citation 

``` 
@misc{perifanos2026compactgreeklanguagemodels,
      title={Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers}, 
      author={Konstantinos Perifanos and Anne Laure Poite and Giannis Tsilikas and Ilias Katis and Stavros Deligiannidis and Dionysis Goutsos},
      year={2026},
      eprint={2611.10209},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2611.10209}, 
}
```