| --- |
| language: |
| - multilingual |
| - am |
| - ar |
| - eu |
| - bn |
| - bg |
| - ca |
| - zh |
| - cs |
| - en |
| - fil |
| - fi |
| - fr |
| - de |
| - el |
| - ha |
| - he |
| - hi |
| - hu |
| - ig |
| - id |
| - ga |
| - it |
| - ja |
| - jv |
| - km |
| - ko |
| - lt |
| - ms |
| - mt |
| - no |
| - fa |
| - pl |
| - pa |
| - ru |
| - sk |
| - sw |
| - ta |
| - te |
| - th |
| - tr |
| - uk |
| - ur |
| - vi |
| - yo |
| - zu |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - aya |
| - multilingual |
| - long-context |
| - tiny-aya |
| extra_gated_heading: "Acknowledge license to accept the repository" |
| extra_gated_button_content: "Accept license" |
| extra_gated_prompt: > |
| By submitting this form, you agree to the License Agreement and acknowledge |
| that the information you provide will be collected, used, and shared in |
| accordance with Cohere's Privacy Policy. You'll receive email updates about |
| Cohere Labs and Cohere research, events, products and services. You can |
| unsubscribe at any time. |
| extra_gated_fields: |
| Name: text |
| Email: text |
| Affiliation: text |
| Country: country |
| I agree to use this model for non-commercial use ONLY: checkbox |
| --- |
| |
| # Model Card for Tiny Aya Base 32K |
|
|
| ## Model Summary |
|
|
| Cohere Labs Tiny Aya Base 32K is an open-weights research release of a pretrained 3.35 billion parameter model optimized for efficient, strong, and balanced multilingual representation across 70+ languages, including many lower-resourced ones. This variant provides a 32K context window and is designed to support downstream adaptation, instruction tuning and long-context research. |
|
|
| This is a base pretrained model and has not been instruction-tuned or preference-aligned. |
|
|
| This checkpoint is used as the base model for [Tiny Aya L2-Thinker](https://huggingface.co/CohereLabs/tiny-aya-l2-thinker) and [Tiny Aya En-Thinker](https://huggingface.co/CohereLabs/tiny-aya-en-thinker). |
|
|
| Developed by: [Cohere](https://cohere.com/) and [Cohere Labs](https://cohere.com/research) |
|
|
| - Point of Contact: [Cohere Labs](https://cohere.com/research) |
| - License: [CC-BY-NC](https://cohere.com/cohere-labs-cc-by-nc-license), requires also adhering to [Cohere Lab's Acceptable Use Policy](https://docs.cohere.com/docs/cohere-labs-acceptable-use-policy) |
| - Model: tiny-aya-base-32K |
| - Model Size: 3.35B |
| - Context Length: 32K (input + output) |
|
|
| For the 8K release, see [tiny-aya-base](https://huggingface.co/CohereLabs/tiny-aya-base). For the broader family, see [tiny-aya-global](https://huggingface.co/CohereLabs/tiny-aya-global) and the [Tiny Aya collection](https://huggingface.co/collections/CohereLabs/tiny-aya). |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "CohereLabs/tiny-aya-base-32K" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| device_map="auto", |
| torch_dtype="auto", |
| ) |
| |
| prompt = "The capital of Spain is" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=128, |
| do_sample=True, |
| temperature=0.1, |
| top_p=0.9, |
| top_k=50, |
| ) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| The model can also be used with the Transformers `pipeline` abstraction: |
|
|
| ```python |
| from transformers import pipeline |
| |
| generator = pipeline( |
| "text-generation", |
| model="CohereLabs/tiny-aya-base-32K", |
| torch_dtype="auto", |
| device_map="auto", |
| ) |
| |
| output = generator( |
| "The capital of Spain is", |
| max_new_tokens=128, |
| do_sample=True, |
| temperature=0.1, |
| top_p=0.9, |
| repetition_penalty=1.1, |
| ) |
| print(output[0]["generated_text"]) |
| ``` |
|
|
| The combined prompt and generated sequence must fit within the 32K context window. |
|
|
| ## Model Details |
|
|
| **Input:** Text only. |
|
|
| **Output:** Model generates text. |
|
|
| **Model Architecture:** Auto-regressive transformer in the Tiny Aya / Cohere family, using interleaved sliding-window and global-attention layers with rotary positional embeddings. |
|
|
| **Languages covered:** The model was trained on 70+ languages, with a focus on English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian, Russian, Greek, German, Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian, Bulgarian, Arabic, Persian, Urdu, Turkish, Maltese, Hebrew, Hindi, Marathi, Bengali, Gujarati, Punjabi, Tamil, Telugu, Nepali, Tagalog, Malay, Indonesian, Vietnamese, Javanese, Khmer, Thai, Lao, Chinese, Burmese, Japanese, Korean, Amharic, Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba, and Zulu. |
|
|
| **Context Length:** Tiny Aya Base 32K supports a context length of 32K tokens across input and output. |
|
|
| ## Usage and Limitations |
|
|
| ### Intended Usage |
|
|
| Tiny Aya Base 32K is intended for continued pretraining, instruction tuning, and research on multilingual and long-context language modeling. Potential downstream applications include multilingual text generation, summarization, translation, cross-lingual tasks, and adaptation for lower-resource languages. |
|
|
| Because this is a base model, prompts should use text completion rather than a chat template. Additional post-training is recommended before deploying it as a conversational assistant. |
|
|
| ### Strengths |
|
|
| Tiny Aya provides strong open-ended generation across broad language coverage, including many lower-resource languages. Shared training signals across language families and scripts support multilingual generation, translation, summarization, and cross-lingual adaptation. This release supports substantially longer sequences than the 8K base release. |
|
|
| ### Limitations |
|
|
| **Base-model behavior.** This checkpoint is not instruction-tuned or preference-aligned and may not reliably follow instructions or behave like a conversational assistant. |
|
|
| **Reasoning tasks.** The model's strongest performance is on open-ended generation. Chain-of-thought reasoning tasks such as multilingual math are comparatively weaker. |
|
|
| **Factual knowledge.** Outputs may contain incorrect or outdated statements, particularly in lower-resource languages with thinner training data coverage. |
|
|
| **Uneven resource distribution.** High-resource languages benefit from richer training signals and tend to show more consistent quality. The lowest-resource languages may show greater variability, and culturally specific nuance, sarcasm, or figurative language may be less reliable. |
|
|
| **Long-context use.** A 32K context window does not guarantee uniform recall or reasoning quality at every position. Long inputs also require substantially more memory and compute. |
|
|
| ## Model Card Contact |
|
|
| For errors or additional questions about details in this model card, contact [labs@cohere.com](mailto:labs@cohere.com). |
|
|
| ## Terms of Use |
|
|
| This model is governed by a [CC-BY-NC License](https://cohere.com/cohere-labs-cc-by-nc-license) (Non-Commercial) and also requires adhering to [Cohere Lab's Acceptable Use Policy](https://docs.cohere.com/docs/cohere-labs-acceptable-use-policy). If you are interested in commercial use, please contact Cohere’s Sales team. |
|
|
