LSC-Parlament_ca-es / README.md
fdelucaf's picture
Update license upon due diligence by IPR
cf00ac5 verified
|
Raw History Blame Contribute Delete
17 kB
metadata
library_name: transformers
pipeline_tag: translation
language:
  - ca
  - es
base_model:
  - openai/clip-vit-base-patch32
  - facebook/m2m100_418M
datasets:
  - LangTech-MT/LSC-Parlament
license: other
license_name: research-only-license
license_link: LICENSE.md
tags:
  - sign-language
  - sign-language-translation
  - catalan-sign-language
  - lsc
  - video-to-text
  - multimodalhugs
  - seq2seq

LSC-Parlament SLT Baseline (ca-es) Model Card

This model is a sign language translation (SLT) baseline that translates Catalan Sign Language (LSC, Llengua de signes catalana) video into written Catalan or Spanish text. It was trained on the LSC-Parlament dataset with the MultimodalHugs framework. The model is meant to be used solely for non-commercial research purposes.

It is the bilingual (ca-es) experiment: a single model that produces either Catalan or Spanish output, selected with a target-language prompt token (__ca__ or __es__). The released weights are checkpoint-13000, the checkpoint with the best SacreBLEU on the validation set.

For further details, we refer to our paper: [TBD].


Model Details

Architecture

The model is a video-to-text sequence-to-sequence model made of three parts, all trained end-to-end (no layer is frozen):

  1. Visual encoder: CLIP ViT-B/32 (openai/clip-vit-base-patch32), which extracts a 512-dimensional feature vector per frame.
  2. Multimodal mapper: a linear projection with a LayerNorm applied before it and dropout 0.1, mapping the visual features into the backbone's embedding space.
  3. Seq2seq backbone: M2M-100 418M (facebook/m2m100_418M), whose encoder consumes the projected frame features and whose decoder generates the text.
Task Video → text (sign language translation)
Source language Catalan Sign Language (__lsc__)
Target languages Catalan (__ca__), Spanish (__es__)
Visual encoder CLIP ViT-B/32 (feature dim. 512)
Mapper Linear, LayerNorm before, dropout 0.1
Backbone M2M-100 418M
Frame sampling Every 2nd frame (stride 2), max. 700 frames
Max. generated tokens 100
Training precision fp16
Framework MultimodalHugs v0.4.2

The text tokenizer is the M2M-100 SentencePiece tokenizer, extended with one new token, __lsc__, which marks the signed-language source.


Intended Use

Direct Use

The model is intended for research on sign language translation, for example as a baseline to compare against, or as a starting point for further experiments on LSC.

Given a video segment of LSC signing, it generates a transcription-style text in Catalan or Spanish, depending on the decoder prompt.


Hardware and Software

Training Framework

The model was trained with MultimodalHugs v0.4.2 (multimodalhugs-setup and multimodalhugs-train, task seq2seq), which builds on Hugging Face Transformers.

Compute Infrastructure

The model was trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.

The accelerated partition is composed of 1,120 nodes with the following specifications:

  • 4x Nvidia Hopper GPUs with 64GB HBM2 memory
  • 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
  • 4x NDR200 (BW per node 800Gb/s)
  • 512 GB of Main memory (DDR5)
  • 460GB on NVMe storage

How to use

This model is not a plain transformers text model: it takes video as input and needs MultimodalHugs to preprocess the videos and run generation. Install MultimodalHugs following its instructions (the experiments used v0.4.2).

1. Download the model

hf download LangTech-MT/LSC-Parlament_ca-es --local-dir ./lsc_parlament_ca-es

2. Prepare your metadata

Inputs are described in a TSV file with one row per video segment:

Column Description
signal Path to the video file
signal_start Segment start in milliseconds
signal_end Segment end in milliseconds
encoder_prompt Sign language tag: __lsc__
decoder_prompt Target language tag: __ca__ or __es__
output Reference transcription (used only for scoring)

Set decoder_prompt to __ca__ to get Catalan output and to __es__ to get Spanish output.

3. Build the dataset and generate

Save the training configuration as config_ca-es.yaml and adjust its paths and metadata files to your data. multimodalhugs-setup creates the processor and dataset artefacts in <setup_dir>, and generation then uses them:

multimodalhugs-setup --modality video2text \
    --config_path config_ca-es.yaml

multimodalhugs-generate --task translation \
    --metric_name            sacrebleu \
    --generate_output_dir    results/ \
    --model_name_or_path     ./lsc-ca-es \
    --processor_name_or_path <setup_dir>/video2text_processor \
    --dataset_dir            <setup_dir>/datasets/parlament_ca-es_video \
    --config_path            config_ca-es.yaml

<setup_dir> is the directory created by multimodalhugs-setup (set by setup.output_dir in the YAML config). Predictions are written to results/.


Data

The model was trained on LSC-Parlament (LangTech-MT/LSC-Parlament), a dataset of LSC video segments aligned with Catalan and Spanish text in the parliamentary domain. The dataset is distributed as four Parquet files, one per split (see Splits). Further dataset description in the dataset card.

The dataset was converted to MultimodalHugs metadata with the following filtering:

  • Segments whose Character Error Rate against the ASR reference exceeded 0.3 were discarded (--max_cer_score 0.3).
  • A fixed set of sessions with known recording or annotation artefacts was excluded (--filter_sessions).

Splits

The original plan was to split the data by recording session, so that train, validation and test would each contain different sessions. This proved too hard for this baseline, so segments were instead split randomly into train, validation and test sets (seed 42). This is controlled by the --split_by_sessions parameter of the dataset creation script, which was left disabled (false) for this model. As a result, sessions are spread across the splits, and the same session can appear in the training, validation and test sets. The reported scores are therefore likely optimistic compared with a session-held-out evaluation. See the paper for details.

Split File Samples
Train train.parquet 26,327
Validation validation.parquet 3,203
Test (Catalan) test_ca.parquet 2,000
Test (Spanish) test_es.parquet 1,203

Training Procedure

The model was fine-tuned end-to-end from the pretrained CLIP ViT-B/32 and M2M-100 418M weights on the bilingual ca-es data. The configuration is the same as for the monolingual baselines apart from the language-specific fields.

Validation was run every 500 steps, and the checkpoint with the best validation SacreBLEU was kept: step 13,000.

Training hyperparameters
Hyperparameter Value
Per-device batch size 4
Gradient accumulation steps 32
Effective batch size 128 (1 GPU)
Learning rate 5e-5
LR scheduler inverse square root
Warmup steps 8,000
Optimizer Adam (β1 = 0.9, β2 = 0.998)
Weight decay 0
Max. gradient norm 0.0 (clipping disabled)
Dropout (mapper) 0.1
Max. training steps 200,000 (configured; best checkpoint at 13,000)
Precision fp16
Max. frames per sample 700
Frame stride 2 (every second frame)
Max. new tokens (generation) 100
Evaluation / checkpoint steps every 500 steps
Model selection metric SacreBLEU (validation)
Seed 3435
Hardware / time 1× NVIDIA H100, ~48 hours
Full training configuration (config_ca-es.yaml)
model:
  type: multimodal_embedder
  feature_extractor_type: clip
  pretrained_feature_extractor: models/clip-vit-base-patch32
  multimodal_mapper_type: linear
  multimodal_mapper_layer_norm_before: true
  multimodal_mapper_layer_norm: false
  multimodal_mapper_activation: false
  multimodal_mapper_dropout: 0.1
  backbone_type: m2m_100
  pretrained_backbone: models/m2m100_418M
  feat_dim: 512
  num_labels: 2
  freeze_feature_extractor: false
  freeze_multimodal_mapper: false
  freeze_backbone: false
  freeze_decoder_embed_tokens: false
  freeze_encoder_embed_tokens: false
  freeze_lm_head: false
  max_new_tokens: 100

setup:
  output_dir: catalan_spanish/

training:
  run_name: parlament_ca-es_video
  output_dir: trained
  logging_dir: logs
  do_train: true
  do_eval: true
  early_stopping: false
  predict_with_generate: true
  eval_strategy: steps
  save_strategy: steps
  eval_steps: 500
  save_steps: 500
  logging_steps: 100
  per_device_train_batch_size: 4
  per_device_eval_batch_size: 4
  gradient_accumulation_steps: 32
  learning_rate: 5e-05
  weight_decay: 0
  adam_beta1: 0.9
  adam_beta2: 0.998
  max_grad_norm: 0.0
  num_train_epochs: 1
  max_steps: 200000
  lr_scheduler_type: inverse_sqrt
  warmup_steps: 8000
  save_total_limit: 5
  seed: 3435
  dataloader_drop_last: false
  metric_for_best_model: sacrebleu
  metric_name: sacrebleu,chrf
  greater_is_better: true
  load_best_model_at_end: true
  remove_unused_columns: false
  dataloader_num_workers: 10
  dataloader_prefetch_factor: 4
  fp16: true

data:
  name: parlament_ca-es_video
  train_metadata_file:
    ca-es/train.tsv
  validation_metadata_file:
    ca-es/validation.tsv
  test_metadata_file:
    ca-es/test.tsv
  shuffle: true
  max_frames: 700

processor:
  text_tokenizer_path: models/m2m100_418M
  new_vocabulary: "__lsc__"
  custom_preprocessor_path: models/clip-vit-base-patch32
  join_chw: false
  skip_frames_stride: 2

Evaluation

The model was evaluated on the random-split test set. The model trained on ca+es is evaluated separately on its Catalan (2,000 samples) and Spanish (1,203 samples) test portions, and compared with a Catalan-only model trained with the same recipe.

Evaluation runs in two stages: SacreBLEU and chrF with multimodalhugs-generate, then neural metrics computed from the generated predictions against the references.

Trained on Test language SacreBLEU chrF BERTScore F1 COMET-Kiwi LaBSE (cosine) SONAR + BLASER 2
ca (separate model) ca 4.23 29.13 0.634 0.142 0.522 2.563
ca+es (this model) ca 4.24 28.69 0.634 0.143 0.515 2.553
ca+es (this model) es 3.08 27.60 0.603 0.101 0.443 2.510
Metric details
  • SacreBLEU and chrF: computed with multimodalhugs-generate (SacreBLEU implementation).
  • BERTScore: microsoft/deberta-xlarge-mnli.
  • COMET-Kiwi: Unbabel/wmt23-cometkiwi-da-xl (a quality-estimation model).
  • LaBSE: cosine similarity with sentence-transformers/LaBSE.
  • SONAR + BLASER 2: BLASER 2.0 QE (blaser_2_0_qe) on SONAR embeddings; scores are on roughly a 1–5 scale.
  • The neural metrics were computed between predictions and references in the same language (source and target language set to the test language).

How to read these results: absolute scores are low (SacreBLEU of 3–4 and chrF below 30), so the outputs are often only loosely related to the reference. On Catalan, the bilingual model is on par with the Catalan-only baseline: SacreBLEU (4.24 vs 4.23), BERTScore F1 (0.634 for both) and COMET-Kiwi (0.143 vs 0.142) are effectively tied, while chrF (28.69 vs 29.13), LaBSE (0.515 vs 0.522) and BLASER (2.553 vs 2.563) are marginally lower. Differences of this size are small and should not be over-interpreted. The practical benefit of the bilingual model is that a single model also produces Spanish output without a clear loss on Catalan. Spanish scores are lower than Catalan scores on every metric, and are computed on a smaller test portion (1,203 samples).


Ethical Considerations and Limitations

Because the model serves as a baseline, it often produces output that is fluent but unfaithful to the signed content. Furthermore, the model has been evaluated optimistically: the random train/validation/test split means recording sessions—and likely signers—overlap across splits, so performance on unseen sessions or signers is expected to be lower.

In addition, the training and evaluation data are restricted in scope. The dataset originates exclusively from parliamentary debates, meaning performance on everyday conversation, other signers, camera set-ups, or sign language varieties has not been evaluated. Similarly, no analysis of bias or performance differences across signers, topics, or demographic groups has been carried out. Finally, the Spanish test portion remains small, consisting of only 1,203 samples.


Additional information

Author

Machine Translation Group, AI Institute, the Barcelona Supercomputing Center (ai_institute_mt@bsc.es).

Contact

For further information, please send an email to ai_institute_mt@bsc.es.

Copyright

Copyright(c) 2026 by AI Institute, Barcelona Supercomputing Center.

Funding

This work has been promoted and financed by the Government of Catalonia through the Aina Project.

This work is supported by MLLM4TRA (PID2024-158157OB-C32) funded by MCIN/AEI/10.13039/501100011033/FEDER, UE.

This work has been funded by the Ministerio de Ciencia, Innovación y Universidades and the Agencia Estatal de Investigación through the project LLM4LS (PID2024-157855OA-C33).

Disclaimer

Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.

The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.

License

Research only

See LICENSE file for further details.

The pretrained components used to initialise the model are released by their authors under the MIT License:

Citation

[TBD]