---
language:
- cs
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- czech
- chat
- steganography
- small-language-model
- encrypt.click
datasets:
- CZLC/CNC_oral_ortofon
- CZLC/CNC_KSK
- BrunoHays/vystadial-telephony-cs
- CZLC/fb_sentiment_balanced
- lang-uk/Reddit-MultiGEC
- open_subtitles
---

# Kecal

A compact language model (~37.5M parameters) that writes natural, casual Czech chat messages: one person sending a few messages in a row to a friend, the way people actually type on Messenger, WhatsApp or Discord.

Kecal was created for the **Chat cipher** on [encrypt.click](https://encrypt.click), where it turns encrypted data into ordinary-looking cover chat messages and back. It is a specialized, deterministic model for that job, not a chatbot or a general-knowledge assistant.

```
to je pravda
tak jsem ti psal
abys vedela ze to vim
jinak bych ti mohl pomoct neco zjistit
treba ti muzu poslat odkaz na nejaky obchod
ale to jeste nevim
tak ti napisu az budu vedet
jinak jak se mas
uz dlouho jsme se nevideli
```

## Model

| Property | Value |
|---|---|
| Parameters | 37.5 M |
| Architecture | Decoder-only transformer, 8 layers, 8 heads, width 512 |
| Tokens | 24,000 words (lowercase, without diacritics) plus `<pad>`, `<eol>`, `<unk>`, `<turn>` |
| Context | 128 words |

It is a standard GPT-2 compatible model with a word-level tokenizer. Words are stored folded (lowercase, no diacritics); `vocab.json` maps each one back to its most common written form. `<eol>` ends a message, `<turn>` marks a change of writer.

## Usage

With the helper script in this repository, which keeps to one writer and restores diacritics:

```python
from kecal import Kecal

kecal = Kecal.load('danielsebesta/kecal')
print(kecal.generate(60))
```

Or with standard `transformers` (output stays folded):

```python
from transformers import pipeline

kecal = pipeline('text-generation', model='danielsebesta/kecal')
print(kecal('hele zitra', max_new_tokens=30, do_sample=True, top_k=40)[0]['generated_text'])
```

## Training

1. **Pretraining** on full Czech subtitles (OpenSubtitles via OPUS, ~149 million lines, ~812 million words).
2. **Fine-tuning on authentic conversational Czech:**
   - **CNC ORTOFON** (informal spoken Czech transcripts).
   - **CNC KSK** (Corpus of Private Correspondence, letters, notes and personal messages).
   - **Vystadial Telephony** (real conversational telephone dialogues between young Czech speakers).
   - Facebook comments and Reddit posts from `r/czech`.
3. **Targeted fine-tuning on synthetic casual bursts:**
   - Hundreds of thousands of one-writer chat bursts generated across diverse everyday topics (errands, meetings, complaints, small talk).

Validation perplexity on held-out one-writer chat: **14.3** (down from 36 before fine-tuning). Steganographic encoding rate: ~**3.56 bits per word** with 100% bit-exact round-trip recovery.

## Limitations

- Output has no punctuation or capital letters; diacritics follow the most common written form of each word.
- Built for natural-sounding casual filler text, not facts. It invents scenarios and can drift between topics.
- Trained on informal language, so it may occasionally produce colloquial slang.

## Credits

Kecal builds on these excellent resources:

- **ORTOFON & KSK** - Corpora of informal spoken Czech and private correspondence, Institute of the Czech National Corpus, Faculty of Arts, Charles University, Prague ([korpus.cz](https://www.korpus.cz)). Via [CZLC/CNC_oral_ortofon](https://huggingface.co/datasets/CZLC/CNC_oral_ortofon) and [CZLC/CNC_KSK](https://huggingface.co/datasets/CZLC/CNC_KSK) (CC BY-NC-SA 4.0).
- **Vystadial Telephony** - Transcripts of real conversational telephone speech, Charles University (LINDAT/CLARIAH-CZ). Via [BrunoHays/vystadial-telephony-cs](https://huggingface.co/datasets/BrunoHays/vystadial-telephony-cs) (CC BY-SA 3.0).
- **Czech Facebook Sentiment Corpus** - I. Habernal, T. Ptáček, J. Steinberger (2013), University of West Bohemia. Via [CZLC/fb_sentiment_balanced](https://huggingface.co/datasets/CZLC/fb_sentiment_balanced) (CC BY-NC-SA 4.0).
- **Reddit-MultiGEC** - Czech posts from `r/czech` in [lang-uk/Reddit-MultiGEC](https://huggingface.co/datasets/lang-uk/Reddit-MultiGEC) (MIT).
- **OpenSubtitles** - P. Lison and J. Tiedemann (2016), *OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles*, via [OPUS](https://opus.nlpl.eu/).
- **Synthetic Chats** - Large share of the fine-tuning mix; generated for this project with LLMs.

## License

[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), following the licenses of the training data: attribution, non-commercial use, share alike.