Fill-Mask
Transformers
Safetensors
Greek
English
roberta
greek
english
legal
bilingual
masked-lm
multilingual
Instructions to use novelcore/gem-roberta-bilingual with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use novelcore/gem-roberta-bilingual with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="novelcore/gem-roberta-bilingual")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-roberta-bilingual") model = AutoModelForMaskedLM.from_pretrained("novelcore/gem-roberta-bilingual", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -126,7 +126,7 @@ The data was processed into fixed-size chunks of 512 tokens, respecting document
|
|
| 126 |
|
| 127 |
### Pre-training
|
| 128 |
|
| 129 |
-
The model was pre-trained from scratch for **150,000 steps** on 8x NVIDIA
|
| 130 |
|
| 131 |
The key hyperparameters used were:
|
| 132 |
- **Learning Rate**: 2e-4 (0.0002) with a linear warmup of 9,000 steps
|
|
@@ -144,7 +144,7 @@ The key hyperparameters used were:
|
|
| 144 |
The model achieved the following performance metrics:
|
| 145 |
- **Final Training Loss**: 0.7479
|
| 146 |
- **Final Evaluation Loss**: 0.69405
|
| 147 |
-
- **Training Infrastructure**: 8x NVIDIA H100
|
| 148 |
- **Total Training Steps**: 150,000
|
| 149 |
- **Total Training Time**: 25 hours 7 minutes
|
| 150 |
- **Train/Validation Split**: 95%/5%
|
|
@@ -222,7 +222,7 @@ The model shows stable convergence across both languages:
|
|
| 222 |
- **Model Size**: ~125M parameters
|
| 223 |
- **Architecture**: RoBERTa-base (12 layers, 12 attention heads)
|
| 224 |
- **Languages**: Greek (el) + English (en)
|
| 225 |
-
- **Training Time**: 25 hours 7 minutes on 8x
|
| 226 |
- **Dataset Size**: 26GB bilingual legal corpus
|
| 227 |
- **Language Ratio**: 60.3% Greek, 39.7% English
|
| 228 |
- **Memory Requirements**: Efficient base architecture suitable for production deployment
|
|
|
|
| 126 |
|
| 127 |
### Pre-training
|
| 128 |
|
| 129 |
+
The model was pre-trained from scratch for **150,000 steps** on 8x NVIDIA H100 GPUs, using BFloat16 (`bf16`) mixed-precision for stability and speed. The training took approximately **25 hours and 7 minutes** to complete.
|
| 130 |
|
| 131 |
The key hyperparameters used were:
|
| 132 |
- **Learning Rate**: 2e-4 (0.0002) with a linear warmup of 9,000 steps
|
|
|
|
| 144 |
The model achieved the following performance metrics:
|
| 145 |
- **Final Training Loss**: 0.7479
|
| 146 |
- **Final Evaluation Loss**: 0.69405
|
| 147 |
+
- **Training Infrastructure**: 8x NVIDIA H100 GPUs
|
| 148 |
- **Total Training Steps**: 150,000
|
| 149 |
- **Total Training Time**: 25 hours 7 minutes
|
| 150 |
- **Train/Validation Split**: 95%/5%
|
|
|
|
| 222 |
- **Model Size**: ~125M parameters
|
| 223 |
- **Architecture**: RoBERTa-base (12 layers, 12 attention heads)
|
| 224 |
- **Languages**: Greek (el) + English (en)
|
| 225 |
+
- **Training Time**: 25 hours 7 minutes on 8x H100 GPUs
|
| 226 |
- **Dataset Size**: 26GB bilingual legal corpus
|
| 227 |
- **Language Ratio**: 60.3% Greek, 39.7% English
|
| 228 |
- **Memory Requirements**: Efficient base architecture suitable for production deployment
|