Model Card for ludolara/aya-vision-32b-americas-grpo-captioning

ludolara/aya-vision-32b-americas-grpo-captioning is a LoRA adapter for Aya Vision 32B trained with a three-stage pipeline:

1. SFT (MT) on Spanish-Indigenous translation data.
2. RLVR (MT) using GRPO with chrF++-based rewards.
3. SFT (IC) for direct Indigenous-language image captioning.

This is the GRPO-enhanced captioning branch reported in the AmericasNLP 2026 system description.

Model Details

Model Description

• Developed by: Mila - Quebec AI Institute (Luis Lara, Param Raval)
• Funded by [optional]: Not explicitly reported
• Shared by [optional]: Luis Lara (ludolara)
• Model type: PEFT LoRA adapter for Aya Vision 32B (multimodal causal language model)
• Language(s) (NLP): Bribri (bzd), Guarani (grn/gn), Nahuatl (nah), Wixárika (hch)
• License: cc-by-nc-4.0 (inherits base-model license and usage constraints)
• Finetuned from model [optional]: CohereLabs/aya-vision-32b

Model Sources [optional]

• Repository: https://github.com/ludolara/americasnlp2026
• Paper [optional]: From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas (AmericasNLP 2026 system description)
• Demo [optional]: Not provided

Uses

Direct Use

• Direct image captioning for Bribri, Guarani, Nahuatl, and Wixárika.
• Research exploration of whether reward-based MT refinement improves downstream captioning.

Downstream Use [optional]

• Ablation reference for SFT (MT) + RLVR (MT) + SFT (IC) in low-resource multilingual captioning.
• Initialization for further captioning adaptation experiments.

Out-of-Scope Use

• Unsupervised production deployment for culturally sensitive content.
• Claims of broad performance gains from GRPO beyond the reported settings.
• Any use that requires guaranteed accuracy, safety, or cultural appropriateness.

Bias, Risks, and Limitations

• RLVR gains are modest and inconsistent across languages in reported experiments.
• Reward design depends on automatic chrF++ and does not directly optimize cultural correctness.
• Low-resource data scarcity and limited held-out validation can lead to unstable behavior.
• Model outputs can still be incorrect, unsafe, or culturally inappropriate.

Recommendations

• Use speaker/community review for any applied setting.
• Treat GRPO improvements as task-specific, not universally transferable.
• Validate quality per language and per domain before broader use.

How to Get Started with the Model

Use the code below to run image captioning with this adapter.

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base_model_id = "CohereLabs/aya-vision-32b"
adapter_id = "ludolara/aya-vision-32b-americas-grpo-captioning"

processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    base_model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

image = Image.open("example.jpg").convert("RGB")
language = "guarani"
prompt = (
    f"Escribe un solo pie de foto en {language} para esta imagen. "
    f"Debe ser una descripcion culturalmente adecuada de la imagen. "
    f"Responde solo con el pie de foto en {language}, sin explicaciones."
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": prompt},
    ],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Training Details

Training Data

This adapter uses three stages:

• Stage 1 (SFT (MT)): Spanish-Indigenous translation data for bzd, grn, nah, hch.
• Stage 2 (RLVR (MT)): Same MT data with GRPO optimization and chrF++ reward.
• Stage 3 (SFT (IC)): AmericasNLP 2026 captioning development data for Bribri, Guarani, Nahuatl, Wixárika.

Training Procedure

Preprocessing [optional]

• MT data is instruction-formatted and bidirectional.
• GRPO stage samples multiple translation candidates and applies reward-based optimization.
• Captioning stage uses a concise Spanish prompt requesting one culturally appropriate caption in the target language.

Training Hyperparameters

• Training regime: Multi-stage LoRA with bf16 + DeepSpeed ZeRO-3
• GRPO stage (configs/lora_grpo.yaml) highlights:
  • epochs: 30
  • learning rate: 1e-6
  • warmup ratio: 0.03
  • per-device batch size: 20
  • generations per prompt: 8
  • temperature: 0.9
  • max completion length: 256
  • eval sample size: 400
  • language sampling alpha: 0.3
• Captioning stage (configs/captioning_lora_sft.yaml) highlights:
  • initialized from GRPO-refined MT adapter
  • epochs: 10
  • learning rate: 2e-5
  • per-device batch size: 1
  • max sequence length: 4096

Speeds, Sizes, Times [optional]

• Scripts target 4 x H100 GPUs on Slurm.
• Full wall-clock duration is not explicitly reported.

Evaluation

Testing Data, Factors & Metrics

Testing Data

• AmericasNLP 2026 shared-task test set for Bribri, Guarani, Nahuatl, and Wixárika.
• Internal 20-example dev subset (5 examples/language, seed 42).

Factors

• Language (bribri, guarani, nahuatl, wixarika)
• Pipeline variant with GRPO intermediate stage

Metrics

• chrF++ for automatic ranking.
• Human-evaluation rank/score in official stage-2 evaluation.

Results

Official test chrF++ (paper Table 1, GRPO variant):

Version	Submission	Overall	Bribri	Guarani	Nahuatl	Wixárika
v1	SFT (MT) + RLVR (MT) + SFT (IC)	17.341	10.886	19.772	19.853	18.853

Internal dev subset chrF++ (paper Table 2):

System	Overall	Bribri	Guarani	Nahuatl	Wixárika
SFT (MT) + RLVR (MT) + SFT (IC)	19.805	11.508	23.767	16.982	22.496

Human evaluation for this branch (paper Table 3):

• Guarani v1: rank 5, mean rating 1.764

Summary

In this setup, GRPO provided a small dev-set gain versus plain SFT (MT) + SFT (IC), but did not consistently improve all languages on the official test set.

Model Examination [optional]

No dedicated interpretability analysis was reported.

Environmental Impact

Carbon emissions can be estimated with the Machine Learning Impact calculator.

• Hardware Type: NVIDIA H100 GPUs (script target: 4 GPUs)
• Hours used: Not reported
• Cloud Provider: Not reported (Slurm/HPC setup)
• Compute Region: Not reported
• Carbon Emitted: Not reported

Technical Specifications [optional]

Model Architecture and Objective

• Base architecture: Aya Vision 32B multimodal autoregressive model
• Adaptation method: LoRA on attention/MLP projection modules
• Objectives:
  • SFT for MT and captioning
  • GRPO optimization with chrF++-based reward in the MT stage

Compute Infrastructure

DeepSpeed ZeRO-3 + Accelerate on Slurm.

Hardware

• Multi-GPU H100 configuration in scripts
• 128 GB host memory target in scripts

Software

• torch>=2.2.0
• transformers>=4.46.0
• datasets>=2.21.0
• accelerate>=0.34.0
• deepspeed>=0.16.0
• trl>=0.11.0
• peft 0.18.1
• huggingface_hub>=0.26.0

Citation [optional]

BibTeX:

@inproceedings{lara-americasnlp-2026,
  title = "From Machine Translation to Image Captioning: Training Vision-Language Models for {I}ndigenous Languages of the {A}mericas",
  author = {Lara, Luis and
  Raval, Param},
  booktitle = "Proceedings of the Sixth Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)",
  month = jul,
  year = "2026",
  address = "San Diego, California",
  publisher = "Association for Computational Linguistics",
}

APA:

Lara, L., & Raval, P. (2026). From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas. AmericasNLP 2026.

Glossary [optional]

• SFT (MT): Supervised fine-tuning on translation data.
• RLVR / GRPO: Reward-based optimization stage applied after MT SFT.
• SFT (IC): Supervised fine-tuning on image-captioning data.

More Information [optional]

This is the GRPO-enhanced captioning branch. For the non-GRPO captioning branch, see ludolara/aya-vision-32b-americas-captioning.

Model Card Authors [optional]

Luis Lara (maintainer), with model-card synthesis from project documentation and system paper.

Model Card Contact

luis.lara@mila.quebec

Framework versions

• PEFT 0.18.1