| # Neural Network-Based Language Model for Next Token Prediction |
| Project Overview: |
| This project involves the development of a Neural Network-based Language Model for next token prediction. The task is to create a model that can predict the next token (word or character) given a sequence of previous tokens. The model will be trained using two languages: English and Finnish. The main goal is to explore how neural networks can be applied to natural language processing tasks without relying on transformer-based architectures. |
|
|
| This repository contains a Jupyter Notebook implementation of a neural network-based language model designed for next token prediction. The model is trained using both English and assigned language(Finnish), leveraging a custom tokenizer and embedding model. |
|
|
| ## Dataset Description |
| The datasets used for training are corpora in English and Finnish. For English https://huggingface.co/datasets/yahma/alpaca-cleaned was used, and for Finnish https://huggingface.co/datasets/saillab/taco-datasets/tree/main/multilingual-instruction-tuning-dataset%20/multilingual-alpaca-52k-gpt-4/Finnish was used. |
| Each dataset was tokenized using a custom tokenizer, and preprocessing included lowercasing, removing punctuation, and handling out-of-vocabulary (OOV) words. The data was then split into training, validation, and test sets. |
|
|
| ## Features |
|
|
| - **Training with Multiple Languages**: The model supports training in English and an additional language(Finnish). |
| - **Custom Tokenizer**: A tokenizer appropriate for the project requirements. |
| - **Model Architecture**: The implementation utilizes Neural Networks (NN), Recurrent Neural Networks (RNN), or Long Short-Term Memory Networks (LSTM). |
|
|
| ## How to Use |
|
|
| 1. **Clone the repository**: |
| ```bash |
| git clone <repository_url> |
| cd <repository_directory> |
| Install the required packages: Ensure you have the necessary libraries installed. Can be done by running: |
| |
| |
| pip install -r requirements.txt |
| Run the Jupyter Notebook: Open the notebook in Jupyter: |
| |
| |
| jupyter notebook SuryaManjri.ipynb |
| Generate Text: Follow the instructions in the notebook to generate text using the trained model in both languages. |
| |
| Checkpoints |
| The model includes checkpoint implementation to save progress during training, allowing for recovery and continued training. |
| |
| Video Demo |
| A video demonstration of the model’s text generation capabilities can be found here |
