Update README.md

README.md

@@ -1,7 +1,13 @@
 # Neural Network-Based Language Model for Next Token Prediction
+Project Overview:
+This project involves the development of a Neural Network-based Language Model for next token prediction. The task is to create a model that can predict the next token (word or character) given a sequence of previous tokens. The model will be trained using two languages: English and Finnish. The main goal is to explore how neural networks can be applied to natural language processing tasks without relying on transformer-based architectures.
 
 This repository contains a Jupyter Notebook implementation of a neural network-based language model designed for next token prediction. The model is trained using both English and assigned language(Finnish), leveraging a custom tokenizer and embedding model.
 
+## Dataset Description
+The datasets used for training are corpora in English and Finnish. For English https://huggingface.co/datasets/yahma/alpaca-cleaned was used, and for Finnish https://huggingface.co/datasets/saillab/taco-datasets/tree/main/multilingual-instruction-tuning-dataset%20/multilingual-alpaca-52k-gpt-4/Finnish was used.
+Each dataset was tokenized using a custom tokenizer, and preprocessing included lowercasing, removing punctuation, and handling out-of-vocabulary (OOV) words. The data was then split into training, validation, and test sets.
+
 ## Features
 
 - **Training with Multiple Languages**: The model supports training in English and an additional language(Finnish).