Kayra-1-exp — a small Turkish language-model experiment

Kayra-1-exp is an approximately 85M-parameter experimental Turkish causal language model trained from scratch. Its main value is as a study of validation loss versus the reliability of generated facts, not as a production assistant.

Reported training observations

The original project notes reported training on approximately 500,000 Turkish text documents:

Metric	Earlier	Later
Validation loss	~6.0	~3.75
Validation perplexity	~397	~42.7

These are author-reported training measurements. The evaluation split, training script, and complete logs are not packaged here, so the numbers are not independently reproducible.

Even as the reported loss decreased, generations for the prompt “Türkiye'nin başkenti” sometimes incorrectly named Bolu or Konya instead of Ankara. This is a useful qualitative example of a limitation, not a measured hallucination rate.

What the experiment can and cannot show

Cross-entropy validation loss measures how well the model predicts held-out tokens from the chosen data distribution. Lower loss does not guarantee that answers to factual questions become consistently correct.

One prompt at a few training checkpoints cannot isolate the cause of errors. Model capacity, data distribution, prompt wording, decoding randomness, and evaluation design may all contribute. The original notes did not establish a measured frequency distribution for individual capital-city names.

Loading the checkpoint

pip install -U transformers torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sixfingerdev/kayra-1-exp"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
model.eval()

prompt = "Türkiye'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt")
inputs.pop("token_type_ids", None)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=False,
        repetition_penalty=1.2,
    )

new_tokens = outputs[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The model uses custom Python model code. Review the repository files before using trust_remote_code=True.

Limitations

• Small research-scale capacity and unreliable factual recall.
• Possible repetition, grammatical errors, or nonsensical completions.
• No standardized reasoning, safety, or knowledge benchmarks.
• Claims about factual accuracy require a new controlled evaluation with a documented test set.

Data, attribution and license

The original metadata lists Wikipedia and mC4 as source datasets. Their underlying content may impose different attribution and reuse conditions from this repository's declared MIT license. Check the source licenses before redistribution.

Research only: the checkpoint is not recommended for consequential decisions or unverified factual answers.