PLLuM: A Family of Polish Large Language Models

Overview

PLLuM is a family of large language models (LLMs) specialized in Polish with additional English data incorporated for broader generalization. Developed through an extensive collaboration with various data providers, PLLuM models are built on high-quality text corpora and refined through instruction tuning, preference learning, and advanced alignment techniques. These models are intended to generate contextually coherent text, offer assistance in various tasks (e.g., question answering, summarization), and serve as a foundation for specialized applications such as domain-specific intelligent assistants.

In 2024, PLLuM models were developed by the PLLuM consortium. Since 2025, their development has continued under HIVE AI, a broader alliance of research institutions and organizations delivering digital public services, focused on open language technologies for Polish public administration.

Key Highlights

  • Extensive Data Collection
    We gathered large-scale, high-quality text data in Polish and English, focusing on rigorous cleaning and deduplication to ensure the highest training standards.

  • Organic Instruction Dataset
    We curated the largest Polish collection of manually created β€œorganic instructions”. This human-authored instruction set is based on an extensive typology of human-model interactions and it covers a range of subtle aspects of supervised fine-tuning (SFT) that might be overlooked with automated approaches (including large scale distillation of 'strong LLMs'). It was also designed to mitigate negative linguistic transfer from non-Polish textual data used in the pre-training phase.

  • Polish Preference Corpus

We created and expanded the first Polish-language preference corpus, featuring prompts and multiple model responses manually assessed by a demographically diverse team of annotators. This dataset teaches the model not only correctness (factual and linguistic) but also balance and safetyβ€”especially for potentially controversial or adversarial topics.

  • Evaluation Benchmarks

We developed custom benchmarks to evaluate our models on tasks relevant to Polish public administration, where PLLuM achieved top scores among all tested models. In broader Polish-language tasks, PLLuM models also attain state-of-the-art results.

Model Description

Below is a summary of the new PLLuM (denoted as 2512) models, created in December 2025. All model names link to their respective Hugging Face resources, while the base models and licenses link to the relevant source models or license references. The model with the -chat suffix has been aligned to human preferences and is generally safer and more efficient for use in dialogue and general-purpose scenarios.

Model Development

  • Pretraining: Models were continued-pretrained on large-scale Polish corpora combined with a range of English texts. Following rigorous cleaning and deduplication, the dataset comprised ~6.7B whitespace.
  • Instruction Fine-Tuning: We fine-tuned the models using approximately 70k manually curated Polish "organic instructions," 33k programmatically derived instructions from high-quality Polish corpora, 15k RAG-style context-processing instructions, and 45k synthetic, context-aware instructions generated by high-performing, permissively licensed LLMs such as Mixtral8x22 and Deepseek V3.
  • Alignment and Preference Learning: Using ~60k manually annotated preference pairs, we taught the models to produce safer, balanced, and contextually appropriate responses, even in adversarial or sensitive cases.

Intended Use Cases

  • General Language Tasks: Text generation, summarization, extraction, question answering, etc.
  • Domain-Specific Assistants: Especially effective for Polish public administration and legal or bureaucratic topics where domain-aware retrieval is required.
  • Research & Development: Building blocks for downstream AI applications in academic or industrial settings, where a strong command of the Polish language is essential.

How to Use

Each PLLuM model can be loaded via the Hugging Face Transformers library (or compatible frameworks). For RAG-based scenarios, pair the model with a relevant vector store or document retrieval system.

Below are some recommended steps and code snippets:

1. Installation

Make sure you have the latest versions of transformers and torch (or another compatible deep learning framework) installed:


pip install transformers accelerate torch  

2. Using Transformers (AutoModelForCausalLM)

This is the recommended way for fine-grained control over generation parameters.

device = "cuda" 
model_name = "CYFRAGOVPL/PLLuM-8B-instruct-2512"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device) 

prompt = "Napisz wiersz o wioΕ›nie." 
messages = [{"role": "user", "content": prompt}] 

input_ids = tokenizer.apply_chat_template(messages,  return_tensors="pt").to(device) 

generated_ids = model.generate(**input_ids,  
                             max_new_tokens=1000,  
                             temperature=0.3) 

decoded = tokenizer.batch_decode(generated_ids)  
print(decoded[0])

3. Using Pipeline (Simplified)

from transformers import pipeline 

model_name = "CYFRAGOVPL/PLLuM-8B-instruct-2512" 

pipe = pipeline("text-generation", 
                 model=model_name, 
                 tokenizer=model_name, 
                 device=0) 

messages = [{"role": "user", "content": "Napisz wiersz o wioΕ›nie."}] 
outputs = pipe(messages, max_new_tokens=1000, do_sample=True, return_full_text=False) 
print(outputs[0]['generated_text'])

Your results may vary depending on model parameters (e.g., temperature, top_k, top_p), hardware, and other settings.

4. Retrieval Augmented Generation (RAG)

During post-training, the models were additionally trained to perform well in Retrieval-Augmented Generation (RAG) settings. The prompt is in .jinja format, where docs is a list of document texts and question is a query that should be answered based on the provided documents. If there is no answer in the provided documents model generates "Nie udaΕ‚o mi siΔ™ odnaleΕΊΔ‡ odpowiedzi na pytanie".

Prompt:

Numerowana lista dokumentΓ³w jest poniΕΌej: 
--------------------- 
<results>{% for doc in docs %} 
Dokument: {{ loop.index0 }} 

{{ doc }} 

{% endfor %}</results> 
--------------------- 
Odpowiedz na pytanie uΕΌytkownika wykorzystujΔ…c tylko informacje znajdujΔ…ce siΔ™ w dokumentach, a nie wczeΕ›niejszΔ… wiedzΔ™. 
Udziel wysokiej jakości, poprawnej gramatycznie odpowiedzi w języku polskim. Odpowiedź powinna zawierać cytowania do dokumentów, z których pochodzą informacje. Zacytuj dokument za pomocą symbolu [nr_dokumentu] powołując się na fragment np. [0] dla fragmentu z dokumentu 0. Jeżeli w dokumentach nie ma informacji potrzebnych do odpowiedzi na pytanie, zamiast odpowiedzi zwróć tekst: "Nie udało mi się odnaleźć odpowiedzi na pytanie". 

Pytanie: {{ question }}

Training Procedure

  • hyperparameters: Vary based on model size, typically including Adam or AdamW optimizers, a range of batch sizes, and carefully tuned learning rates.
  • Hardware & Duration: Training using Bem2 (up to 300xH100 GPUs) and Helios HPCs.

Evaluation and Benchmarks

  • Public Administration: PLLuM models demonstrated top-tier performance in specialized tasks relevant to government services.

  • Polish Language Tasks: Across a variety of internal benchmarks and standard corpora, PLLuM consistently outperforms other models in accuracy, coherence, and safety metrics.

  • Custom Tests: A unique preference corpus and alignment tests ensure robust, safe, and contextually accurate responses.

Limitations and Bias

  • Potential Hallucinations: Like other LLMs, PLLuM may occasionally produce factually incorrect or fabricated content.
  • Sensitivity & Bias: While extensive preference learning has been done, biases might still emerge, especially in controversial or subjective topics.
  • Context Length: Very long context tasks may challenge certain models, depending on memory constraints.

Ethical Considerations

PLLuM models are designed for constructive and responsible usage. Users should exercise caution when deploying them in production scenarios, especially for sensitive or regulated domains. Despite efforts to minimize harmful outputs, there is always a risk of generating offensive, biased, or inappropriate text. Human oversight and due diligence are advised.

Legal Aspects

AI Act Compliant Public Summary of Training Content for General Purpose GenAI models

Citation

If you use PLLuM models or any part of this repository in your research or deployment, please cite as follows (BibTeX):

@article{kocon2025pllum, 
title = {PLLuM: A Family of Polish Large Language Models}, 
author = {Jan KocoΕ„ and Maciej Piasecki and Arkadiusz Janz and Teddy Ferdinan and Łukasz RadliΕ„ski and BartΕ‚omiej Koptyra and Marcin Oleksy and  StanisΕ‚aw WoΕΊniak and PaweΕ‚ Walkowiak and Konrad Wojtasik and Julia Moska and Tomasz NaskrΔ™t and Bartosz Walkowiak and Mateusz Gniewkowski and Kamil Szyc and Dawid Motyka and Dawid Banach and Jonatan DalasiΕ„ski and Ewa Rudnicka and BartΕ‚omiej Alberski and Tomasz Walkowiak and Aleksander SzczΔ™sny and Maciej Markiewicz and Tomasz BernaΕ› and Hubert Mazur and Kamil Ε»yta and Mateusz Tykierko and Grzegorz Chodak and Tomasz Kajdanowicz and PrzemysΕ‚aw Kazienko and Agnieszka KarliΕ„ska and Karolina Seweryn and Anna KoΕ‚os and Maciej ChrabΔ…szcz and Katarzyna Lorenc and Aleksandra KrasnodΔ™bska and Artur Wilczek and Katarzyna Dziewulska and Paula Betscher and Zofia CieΕ›liΕ„ska and Katarzyna Kowol and Daria MikoΕ› and Maciej TrzciΕ„ski and Dawid Krutul and Marek KozΕ‚owski and SΕ‚awomir Dadas and RafaΕ‚ PoΕ›wiata and MichaΕ‚ PereΕ‚kiewicz and MaΕ‚gorzata GrΔ™bowiec and Maciej KazuΕ‚a and Marcin BiaΕ‚as and Roman Roszko and Danuta Roszko and Jurgita VaičenonienΔ— and Andrius Utka and PaweΕ‚ Levchuk and PaweΕ‚ Kowalski and Irena Prawdzic-Jankowska and Maciej Ogrodniczuk and Monika Borys and Anna BuliΕ„ska and Wiktoria Gumienna and Witold KieraΕ› and Dorota  KomosiΕ„ska and Katarzyna Krasnowska-KieraΕ› and Łukasz  KobyliΕ„ski and Martyna Lewandowska and Marek ŁaziΕ„ski and  MikoŁaj Łątkowski and Dawid Mastalerz and Beata Milewicz and  Agnieszka Anna Mykowiecka and Angelika Peljak-ŁapiΕ„ska and Sandra  Penno and Zuzanna Przybysz and MichaΕ‚ Rudolf and Piotr Rybak and Karolina  Saputa and Aleksandra Tomaszewska and Aleksander Wawer and Marcin  WoliΕ„ski and Joanna WoΕ‚oszyn and Alina WrΓ³blewska and Bartosz  Ε»uk and Filip Ε»arnecki and Konrad KaczyΕ„ski and Anna Cichosz and  Zuzanna Deckert and Monika Garnys and Izabela Grabarczyk and Wojciech  Janowski and Sylwia KarasiΕ„ska and Aleksandra Kujawiak and Piotr Misztela  and Maria SzymaΕ„ska and Karolina Walkusz and Igor Siek and Jakub Kwiatkowski and Piotr PΔ™zik}, 
year         = 2025, 
journal      = {arXiv preprint arXiv:2511.03823}, 
url          = {https://arxiv.org/abs/2511.03823}, 
eprint       = {2511.03823}, 
archiveprefix = {arXiv}, 
primaryclass = {cs.CL}}

As part of the project, datasets described in the following publications were also created:

@inproceedings{seweryn-etal-2025-pllum, 
title = "{PLL}u{M}-Align: {P}olish Preference Dataset for Large Language Model Alignment", 
author = "Seweryn, Karolina  and KoΕ‚os, Anna  and KarliΕ„ska, Agnieszka  and Lorenc, Katarzyna  and Dziewulska, Katarzyna  and ChrabΔ…szcz, Maciej  and KrasnodΔ™bska, Aleksandra  and Betscher, Paula  and CieΕ›liΕ„ska, Zofia  and Kowol, Katarzyna  and Moska, Julia  and Motyka, Dawid  and Walkowiak, PaweΕ‚  and Ε»uk, Bartosz  and Janz, Arkadiusz",
editor = "Christodoulopoulos, Christos  and Chakraborty, Tanmoy  and Rose, Carolyn  and Peng, Violet", 
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", 
month = nov, 
year = "2025", 
address = "Suzhou, China", 
publisher = "Association for Computational Linguistics", 
url = "https://aclanthology.org/2025.emnlp-main.1219/", 
doi = "10.18653/v1/2025.emnlp-main.1219", 
pages = "23879--23908", 
ISBN = "979-8-89176-332-6"}
@article{pkezik2025pllum, 
author = {Pezik, Piotr and KaczyΕ„ski, Konrad and Cichosz, Anna and Deckert, Zuzanna and Garnys, Monika and Grabarczyk, Izabela and Janowski, Wojciech and KarasiΕ„ska, Sylwia and Kujawiak, Aleksandra and Misztela, Piotr and SzymaΕ„ska, Maria and Walkusz, Karolina and Siek, Igor and Chrabaszcz, Maciej and KoΕ‚os, Anna and KarliΕ„ska, Agnieszka and Seweryn, Karolina and KrasnodΔ™bska, Aleksandra and KocoΕ„, Jan}, 
year = {2025}, 
month = {11}, 
pages = {}, 
title = {The PLLuM Instruction Corpus}, 
doi = {10.48550/arXiv.2511.17161} 
}

Creators & Consortium

The PLLuM second-generation models are developed under the banner of HIVE AI – a new initiative uniting scientific institutions and digital service providers working together to develop and implement open language technologies for the Polish public sector. HIVE AI continues the mission of the original PLLuM project, expanding it into a coordinated effort to build, improve, and deploy Polish language models in real-world administrative applications.

Contact and Support

For questions or contributions, please reach out via: hive@nask.pl

Join the PLLuM Community on Discord

Learn more at the official PLLuM website

We welcome feedback, collaboration, and further exploration of PLLuM models!

Acknowledgements

Project financed by the Minister of Digital Affairs under the targeted subsidy No. 1/WII/DBI/2025: β€œHIVE AI: Development and Pilot Implementation of LLMs in the Polish Public Administration”

Funding Amount: approx. 18.5 mln PLN Contract Signing Date: 2025-03-25

Downloads last month
305
Safetensors
Model size
8B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512

Adapters
1 model
Quantizations
3 models

Space using CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512 1

Collections including CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512

Paper for CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512