| ---
|
| language:
|
| - am
|
| license: mit
|
| tags:
|
| - amharic
|
| - word-embeddings
|
| - skip-gram
|
| - natural-language-processing
|
| - from-scratch
|
| - custom-implementation
|
| metrics:
|
| - loss
|
| library_name: custom
|
| ---
|
|
|
| # Amharic Skip-gram Word Embeddings from Scratch
|
|
|
| A lightweight, pure-Python implementation of a Skip-gram word embedding model trained on a small Amharic corpus. Built entirely from first principles without external numerical or machine learning libraries (no NumPy, PyTorch, or TensorFlow).
|
|
|
| ## Overview
|
|
|
| This project implements a Skip-gram neural network to learn vector representations (word embeddings) for Amharic words. The primary goal is educational: to understand and verify the core mathematical foundations of word embeddings, including:
|
| - Forward propagation and softmax distribution calculation
|
| - Cross-entropy loss formulation
|
| - Analytical gradient derivation for input and output weight matrices
|
| - Stochastic Gradient Descent (SGD) optimization from scratch
|
|
|
| The implementation relies strictly on Python standard library modules (`math`, `random`, lists, and loops).
|
|
|
| ---
|
|
|
| ## Model Architecture & Training Parameters
|
|
|
| | Hyperparameter | Value |
|
| | :--- | :--- |
|
| | **Language** | Amharic (`am`) |
|
| | **Architecture** | Skip-gram with Softmax |
|
| | **Vocabulary Size** | 271 unique tokens |
|
| | **Training Pairs** | 1,976 |
|
| | **Embedding Dimension** | 10 |
|
| | **Context Window Size** | 2 |
|
| | **Learning Rate** | 0.05 |
|
| | **Maximum Epochs** | 50 |
|
| | **Random Seed** | 42 |
|
| | **Best Checkpoint** | Epoch 49 |
|
|
|
| The model optimizes two matrices: an input embedding matrix $E$ and an output context weight matrix $U$. For a given center word, it predicts surrounding context words via a softmax probability distribution over the vocabulary.
|
|
|
| ---
|
|
|
| ## Training & Validation Results
|
|
|
| | Metric | Result |
|
| | :--- | :--- |
|
| | **Initial Mean Training Loss** | 5.602120 |
|
| | **Best Mean Training Loss** | 2.798836 |
|
| | **Best Epoch** | 49 |
|
| | **Epochs Executed** | 50 |
|
| | **Max Absolute Gradient-Check Error** | 0.00000000 |
|
|
|
| > **Note:** The best mean training loss achieved a ~50% reduction from the initial loss. Epoch 49 was selected via early checkpointing as epoch 50 yielded a slightly higher loss value. These metrics validate implementation correctness on the training split and do not reflect generalization performance on unseen text.
|
|
|
| ### Training Loss Curve
|
|
|
| The following plot illustrates the trajectory of the mean training loss recorded before training and after each epoch. The best model checkpoint was saved at epoch 49: |
