Instructions to use LangTech-MT/LSC-Parlament_ca-es with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use LangTech-MT/LSC-Parlament_ca-es with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="LangTech-MT/LSC-Parlament_ca-es")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("LangTech-MT/LSC-Parlament_ca-es", device_map="auto") - Notebooks
- Google Colab
- Kaggle
LSC-Parlament SLT Baseline (ca-es) Model Card
This model is a sign language translation (SLT) baseline that translates Catalan Sign Language (LSC, Llengua de signes catalana) video into written Catalan or Spanish text. It was trained on the LSC-Parlament dataset with the MultimodalHugs framework. The model is meant to be used solely for non-commercial research purposes.
It is the bilingual (ca-es) experiment: a single model that produces either Catalan or Spanish output, selected with a target-language prompt token (__ca__ or __es__). The released weights are checkpoint-13000, the checkpoint with the best SacreBLEU on the validation set.
For further details, we refer to our paper: [TBD].
Model Details
Architecture
The model is a video-to-text sequence-to-sequence model made of three parts, all trained end-to-end (no layer is frozen):
- Visual encoder: CLIP ViT-B/32 (openai/clip-vit-base-patch32), which extracts a 512-dimensional feature vector per frame.
- Multimodal mapper: a linear projection with a LayerNorm applied before it and dropout 0.1, mapping the visual features into the backbone's embedding space.
- Seq2seq backbone: M2M-100 418M (facebook/m2m100_418M), whose encoder consumes the projected frame features and whose decoder generates the text.
| Task | Video → text (sign language translation) |
| Source language | Catalan Sign Language (__lsc__) |
| Target languages | Catalan (__ca__), Spanish (__es__) |
| Visual encoder | CLIP ViT-B/32 (feature dim. 512) |
| Mapper | Linear, LayerNorm before, dropout 0.1 |
| Backbone | M2M-100 418M |
| Frame sampling | Every 2nd frame (stride 2), max. 700 frames |
| Max. generated tokens | 100 |
| Training precision | fp16 |
| Framework | MultimodalHugs v0.4.2 |
The text tokenizer is the M2M-100 SentencePiece tokenizer, extended with one new token, __lsc__, which marks the signed-language source.
Intended Use
Direct Use
The model is intended for research on sign language translation, for example as a baseline to compare against, or as a starting point for further experiments on LSC.
Given a video segment of LSC signing, it generates a transcription-style text in Catalan or Spanish, depending on the decoder prompt.
Hardware and Software
Training Framework
The model was trained with MultimodalHugs v0.4.2 (multimodalhugs-setup and multimodalhugs-train, task seq2seq), which builds on Hugging Face Transformers.
Compute Infrastructure
The model was trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.
The accelerated partition is composed of 1,120 nodes with the following specifications:
- 4x Nvidia Hopper GPUs with 64GB HBM2 memory
- 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
- 4x NDR200 (BW per node 800Gb/s)
- 512 GB of Main memory (DDR5)
- 460GB on NVMe storage
How to use
This model is not a plain transformers text model: it takes video as input and needs MultimodalHugs to preprocess the videos and run generation. Install MultimodalHugs following its instructions (the experiments used v0.4.2).
1. Download the model
hf download LangTech-MT/LSC-Parlament_ca-es --local-dir ./lsc_parlament_ca-es
2. Prepare your metadata
Inputs are described in a TSV file with one row per video segment:
| Column | Description |
|---|---|
signal |
Path to the video file |
signal_start |
Segment start in milliseconds |
signal_end |
Segment end in milliseconds |
encoder_prompt |
Sign language tag: __lsc__ |
decoder_prompt |
Target language tag: __ca__ or __es__ |
output |
Reference transcription (used only for scoring) |
Set decoder_prompt to __ca__ to get Catalan output and to __es__ to get Spanish output.
3. Build the dataset and generate
Save the training configuration as config_ca-es.yaml and adjust its paths and metadata files to your data. multimodalhugs-setup creates the processor and dataset artefacts in <setup_dir>, and generation then uses them:
multimodalhugs-setup --modality video2text \
--config_path config_ca-es.yaml
multimodalhugs-generate --task translation \
--metric_name sacrebleu \
--generate_output_dir results/ \
--model_name_or_path ./lsc-ca-es \
--processor_name_or_path <setup_dir>/video2text_processor \
--dataset_dir <setup_dir>/datasets/parlament_ca-es_video \
--config_path config_ca-es.yaml
<setup_dir> is the directory created by multimodalhugs-setup (set by setup.output_dir in the YAML config). Predictions are written to results/.
Data
The model was trained on LSC-Parlament (LangTech-MT/LSC-Parlament), a dataset of LSC video segments aligned with Catalan and Spanish text in the parliamentary domain. The dataset is distributed as four Parquet files, one per split (see Splits). Further dataset description in the dataset card.
The dataset was converted to MultimodalHugs metadata with the following filtering:
- Segments whose Character Error Rate against the ASR reference exceeded 0.3 were discarded (
--max_cer_score 0.3). - A fixed set of sessions with known recording or annotation artefacts was excluded (
--filter_sessions).
Splits
The original plan was to split the data by recording session, so that train, validation and test would each contain different sessions. This proved too hard for this baseline, so segments were instead split randomly into train, validation and test sets (seed 42). This is controlled by the --split_by_sessions parameter of the dataset creation script, which was left disabled (false) for this model. As a result, sessions are spread across the splits, and the same session can appear in the training, validation and test sets. The reported scores are therefore likely optimistic compared with a session-held-out evaluation. See the paper for details.
| Split | File | Samples |
|---|---|---|
| Train | train.parquet |
26,327 |
| Validation | validation.parquet |
3,203 |
| Test (Catalan) | test_ca.parquet |
2,000 |
| Test (Spanish) | test_es.parquet |
1,203 |
Training Procedure
The model was fine-tuned end-to-end from the pretrained CLIP ViT-B/32 and M2M-100 418M weights on the bilingual ca-es data. The configuration is the same as for the monolingual baselines apart from the language-specific fields.
Validation was run every 500 steps, and the checkpoint with the best validation SacreBLEU was kept: step 13,000.
Training hyperparameters
| Hyperparameter | Value |
|---|---|
| Per-device batch size | 4 |
| Gradient accumulation steps | 32 |
| Effective batch size | 128 (1 GPU) |
| Learning rate | 5e-5 |
| LR scheduler | inverse square root |
| Warmup steps | 8,000 |
| Optimizer | Adam (β1 = 0.9, β2 = 0.998) |
| Weight decay | 0 |
| Max. gradient norm | 0.0 (clipping disabled) |
| Dropout (mapper) | 0.1 |
| Max. training steps | 200,000 (configured; best checkpoint at 13,000) |
| Precision | fp16 |
| Max. frames per sample | 700 |
| Frame stride | 2 (every second frame) |
| Max. new tokens (generation) | 100 |
| Evaluation / checkpoint steps | every 500 steps |
| Model selection metric | SacreBLEU (validation) |
| Seed | 3435 |
| Hardware / time | 1× NVIDIA H100, ~48 hours |
Full training configuration (config_ca-es.yaml)
model:
type: multimodal_embedder
feature_extractor_type: clip
pretrained_feature_extractor: models/clip-vit-base-patch32
multimodal_mapper_type: linear
multimodal_mapper_layer_norm_before: true
multimodal_mapper_layer_norm: false
multimodal_mapper_activation: false
multimodal_mapper_dropout: 0.1
backbone_type: m2m_100
pretrained_backbone: models/m2m100_418M
feat_dim: 512
num_labels: 2
freeze_feature_extractor: false
freeze_multimodal_mapper: false
freeze_backbone: false
freeze_decoder_embed_tokens: false
freeze_encoder_embed_tokens: false
freeze_lm_head: false
max_new_tokens: 100
setup:
output_dir: catalan_spanish/
training:
run_name: parlament_ca-es_video
output_dir: trained
logging_dir: logs
do_train: true
do_eval: true
early_stopping: false
predict_with_generate: true
eval_strategy: steps
save_strategy: steps
eval_steps: 500
save_steps: 500
logging_steps: 100
per_device_train_batch_size: 4
per_device_eval_batch_size: 4
gradient_accumulation_steps: 32
learning_rate: 5e-05
weight_decay: 0
adam_beta1: 0.9
adam_beta2: 0.998
max_grad_norm: 0.0
num_train_epochs: 1
max_steps: 200000
lr_scheduler_type: inverse_sqrt
warmup_steps: 8000
save_total_limit: 5
seed: 3435
dataloader_drop_last: false
metric_for_best_model: sacrebleu
metric_name: sacrebleu,chrf
greater_is_better: true
load_best_model_at_end: true
remove_unused_columns: false
dataloader_num_workers: 10
dataloader_prefetch_factor: 4
fp16: true
data:
name: parlament_ca-es_video
train_metadata_file:
ca-es/train.tsv
validation_metadata_file:
ca-es/validation.tsv
test_metadata_file:
ca-es/test.tsv
shuffle: true
max_frames: 700
processor:
text_tokenizer_path: models/m2m100_418M
new_vocabulary: "__lsc__"
custom_preprocessor_path: models/clip-vit-base-patch32
join_chw: false
skip_frames_stride: 2
Evaluation
The model was evaluated on the random-split test set. The model trained on ca+es is evaluated separately on its Catalan (2,000 samples) and Spanish (1,203 samples) test portions, and compared with a Catalan-only model trained with the same recipe.
Evaluation runs in two stages: SacreBLEU and chrF with multimodalhugs-generate, then neural metrics computed from the generated predictions against the references.
| Trained on | Test language | SacreBLEU | chrF | BERTScore F1 | COMET-Kiwi | LaBSE (cosine) | SONAR + BLASER 2 |
|---|---|---|---|---|---|---|---|
ca (separate model) |
ca | 4.23 | 29.13 | 0.634 | 0.142 | 0.522 | 2.563 |
ca+es (this model) |
ca | 4.24 | 28.69 | 0.634 | 0.143 | 0.515 | 2.553 |
ca+es (this model) |
es | 3.08 | 27.60 | 0.603 | 0.101 | 0.443 | 2.510 |
Metric details
SacreBLEUandchrF: computed withmultimodalhugs-generate(SacreBLEU implementation).BERTScore:microsoft/deberta-xlarge-mnli.COMET-Kiwi:Unbabel/wmt23-cometkiwi-da-xl(a quality-estimation model).LaBSE: cosine similarity withsentence-transformers/LaBSE.SONAR + BLASER 2: BLASER 2.0 QE (blaser_2_0_qe) on SONAR embeddings; scores are on roughly a 1–5 scale.- The neural metrics were computed between predictions and references in the same language (source and target language set to the test language).
How to read these results: absolute scores are low (SacreBLEU of 3–4 and chrF below 30), so the outputs are often only loosely related to the reference. On Catalan, the bilingual model is on par with the Catalan-only baseline: SacreBLEU (4.24 vs 4.23), BERTScore F1 (0.634 for both) and COMET-Kiwi (0.143 vs 0.142) are effectively tied, while chrF (28.69 vs 29.13), LaBSE (0.515 vs 0.522) and BLASER (2.553 vs 2.563) are marginally lower. Differences of this size are small and should not be over-interpreted. The practical benefit of the bilingual model is that a single model also produces Spanish output without a clear loss on Catalan. Spanish scores are lower than Catalan scores on every metric, and are computed on a smaller test portion (1,203 samples).
Ethical Considerations and Limitations
Because the model serves as a baseline, it often produces output that is fluent but unfaithful to the signed content. Furthermore, the model has been evaluated optimistically: the random train/validation/test split means recording sessions—and likely signers—overlap across splits, so performance on unseen sessions or signers is expected to be lower.
In addition, the training and evaluation data are restricted in scope. The dataset originates exclusively from parliamentary debates, meaning performance on everyday conversation, other signers, camera set-ups, or sign language varieties has not been evaluated. Similarly, no analysis of bias or performance differences across signers, topics, or demographic groups has been carried out. Finally, the Spanish test portion remains small, consisting of only 1,203 samples.
Additional information
Author
Machine Translation Group, AI Institute, the Barcelona Supercomputing Center (ai_institute_mt@bsc.es).
Contact
For further information, please send an email to ai_institute_mt@bsc.es.
Copyright
Copyright(c) 2026 by AI Institute, Barcelona Supercomputing Center.
Funding
This work has been promoted and financed by the Government of Catalonia through the Aina Project.
This work is supported by MLLM4TRA (PID2024-158157OB-C32) funded by MCIN/AEI/10.13039/501100011033/FEDER, UE.
This work has been funded by the Ministerio de Ciencia, Innovación y Universidades and the Agencia Estatal de Investigación through the project LLM4LS (PID2024-157855OA-C33).
Disclaimer
Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.
The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
License
Research only
See LICENSE file for further details.
The pretrained components used to initialise the model are released by their authors under the MIT License:
- CLIP ViT-B/32: released by OpenAI under the MIT License (openai/CLIP).
- M2M-100 418M: released by Meta AI under the MIT License (facebook/m2m100_418M).
Citation
[TBD]
- Downloads last month
- 14
Model tree for LangTech-MT/LSC-Parlament_ca-es
Base model
facebook/m2m100_418M