File size: 2,407 Bytes
56bebd8 3413b8d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
language:
- en
- he
---
# Bilingual Language Model for Next Token Prediction
## Overview
This project focuses on building a neural network-based language model for next token prediction using two languages: **English** and **Hebrew**. The model is implemented using an LSTM (Long Short-Term Memory) architecture, designed to predict the next word in a sequence based on the training data provided. The project leverages Recurrent Neural Networks (RNNs) and evaluates the model using the **perplexity** metric to measure the quality of the predictions.
The final model and checkpoints are provided, along with training history including perplexity and loss values.
## Model Architecture
- **Embedding Layer**: Converts tokenized words into dense vector representations.
- **LSTM Layer**: Consists of 128 units to capture long-term dependencies in the sequence data.
- **Dense Output Layer**: Outputs a probability distribution over the vocabulary to predict the next word.
- **Total Vocabulary Size**: The model is trained on a corpus of size `[total_words]` (combining both English and Hebrew datasets).
## Dataset
The model is trained using a combination of English and Hebrew text datasets. The input sequences are tokenized and padded to ensure consistent input length for training the model.
## Training
The model was trained with the following parameters:
- **Optimizer**: Adam
- **Loss Function**: Categorical Crossentropy
- **Batch Size**: 64
- **Epochs**: 20
- **Validation Split**: 20%
## Evaluation Metric: Perplexity
Perplexity is used to measure the model's performance, with lower perplexity indicating better generalization to unseen data. The final perplexity scores are:
- **Final Training Perplexity**: `[Final Training Perplexity]`
- **Final Validation Perplexity**: `[Final Validation Perplexity]`
## Checkpoints
A checkpoint mechanism is used to save the model at its best-performing stage based on validation loss. The best model checkpoint (`best_model.keras`) is included, which can be loaded for inference.
## Results
The model demonstrates competitive performance in predicting next tokens for both English and Hebrew, achieving satisfactory perplexity scores on both training and validation datasets.
## How to Use
To use this model, follow these steps:
1. **Clone the repository**:
```bash
git clone https://huggingface.co/username/model-name
cd model-name
|