| --- |
| language: |
| - en |
| - he |
| --- |
| # Bilingual Language Model for Next Token Prediction |
|
|
| ## Overview |
| This project focuses on building a neural network-based language model for next token prediction using two languages: **English** and **Hebrew**. The model is implemented using an LSTM (Long Short-Term Memory) architecture, designed to predict the next word in a sequence based on the training data provided. The project leverages Recurrent Neural Networks (RNNs) and evaluates the model using the **perplexity** metric to measure the quality of the predictions. |
|
|
| The final model and checkpoints are provided, along with training history including perplexity and loss values. |
|
|
| ## Model Architecture |
| - **Embedding Layer**: Converts tokenized words into dense vector representations. |
| - **LSTM Layer**: Consists of 128 units to capture long-term dependencies in the sequence data. |
| - **Dense Output Layer**: Outputs a probability distribution over the vocabulary to predict the next word. |
| - **Total Vocabulary Size**: The model is trained on a corpus of size `[total_words]` (combining both English and Hebrew datasets). |
|
|
| ## Dataset |
| The model is trained using a combination of English and Hebrew text datasets. The input sequences are tokenized and padded to ensure consistent input length for training the model. |
|
|
| ## Training |
| The model was trained with the following parameters: |
| - **Optimizer**: Adam |
| - **Loss Function**: Categorical Crossentropy |
| - **Batch Size**: 64 |
| - **Epochs**: 20 |
| - **Validation Split**: 20% |
|
|
| ## Evaluation Metric: Perplexity |
| Perplexity is used to measure the model's performance, with lower perplexity indicating better generalization to unseen data. The final perplexity scores are: |
| - **Final Training Perplexity**: `[Final Training Perplexity]` |
| - **Final Validation Perplexity**: `[Final Validation Perplexity]` |
|
|
| ## Checkpoints |
| A checkpoint mechanism is used to save the model at its best-performing stage based on validation loss. The best model checkpoint (`best_model.keras`) is included, which can be loaded for inference. |
|
|
| ## Results |
| The model demonstrates competitive performance in predicting next tokens for both English and Hebrew, achieving satisfactory perplexity scores on both training and validation datasets. |
|
|
| ## How to Use |
| To use this model, follow these steps: |
|
|
| 1. **Clone the repository**: |
| ```bash |
| git clone https://huggingface.co/username/model-name |
| cd model-name |
| |
