| --- |
| language: el |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # LexisLM-1B |
|
|
| LexisLM-1B is a Greek language, decoder-only model for text completion. It was pretrained for on approximately 22 billion tokens from FineWeb2 and Greek Wikipedia. This release is the **base model**; it has not been instruction tuned for chat or question answering. |
|
|
| ## Model details |
|
|
| | Property | Value | |
| | --- | --- | |
| | Architecture | Llama-compatible causal language model | |
| | Transformer layers | 20 | |
| | Hidden size | 2,048 | |
| | Attention heads | 16 | |
| | Context length | 4,096 tokens | |
| | Vocabulary size | 32,004 tokens | |
|
|
| The repository includes the model weights in `safetensors` format, the configuration, and the tokenizer. It can be loaded with Hugging Face Transformers. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "ComputationalLinguisticsLab/LexisLM-1B" # Or a local path to hf_pretrained |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto") |
| |
| # Pretraining documents were separated by <eos>. Start a fresh document with it. |
| prompt = "Η Ελλάδα είναι μια χώρα" |
| prompt_ids = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids |
| eos_id = torch.tensor([[tokenizer.eos_token_id]]) |
| input_ids = torch.cat([eos_id, prompt_ids], dim=1).to(model.device) |
| |
| with torch.inference_mode(): |
| output_ids = model.generate( |
| input_ids, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.9, |
| pad_token_id=tokenizer.pad_token_id, |
| ) |
| |
| print(tokenizer.decode(output_ids[0, input_ids.shape[1] :], skip_special_tokens=True)) |
| ``` |
|
|
| Install `torch`, `transformers`, and `accelerate` to run this example. Keep the prompt and generated tokens within the 4,096-token context window. |
|
|
| ## Evaluation results |
|
|
| The following results were reported with **0 few-shot examples**. Scores are shown as percentages. ARC uses normalized accuracy; the other benchmarks use their reported accuracy or multiple-choice metrics. |
|
|
| | Benchmark | Metric | Score | Questions | |
| | --- | --- | ---: | ---: | |
| | ARC-Challenge | Normalized accuracy | 25.34% | 1,168 | |
| | ARC-Easy | Normalized accuracy | 33.38% | 2,376 | |
| | MMLU | Accuracy | 23.15% | 14,042 | |
| | TruthfulQA | MC1 | 24.60% | 817 | |
| | TruthfulQA | MC2 | 43.49% | 817 | |
|
|
| ## Intended use and limitations |
|
|
| LexisLM-1B is intended for experimenting with Greek text generation and as a starting point for further fine-tuning. Because it is a base model, chat-style instructions may not produce reliable answers. Generated text can be inaccurate or reflect biases in the training data; review it before using it in consequential settings. |
|
|
| ## License |
|
|
|
|
| LexisLM-1B model weights are released under Apache-2.0. Training data included FineWeb2 (HuggingFaceFW), available under ODC-By-1.0. Dataset and underlying content rights remain with their respective holders. |
|
|
|
|
| ## Citation |
|
|
| ``` |
| @misc{perifanos2026compactgreeklanguagemodels, |
| title={Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers}, |
| author={Konstantinos Perifanos and Anne Laure Poite and Giannis Tsilikas and Ilias Katis and Stavros Deligiannidis and Dionysis Goutsos}, |
| year={2026}, |
| eprint={2611.10209}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2611.10209}, |
| } |
| ``` |
