---
language:
- am
license: mit
tags:
- amharic
- word-embeddings
- skip-gram
- natural-language-processing
- from-scratch
- custom-implementation
metrics:
- loss
library_name: custom
---

# Amharic Skip-gram Word Embeddings from Scratch

A lightweight, pure-Python implementation of a Skip-gram word embedding model trained on a small Amharic corpus. Built entirely from first principles without external numerical or machine learning libraries (no NumPy, PyTorch, or TensorFlow).

## Overview

This project implements a Skip-gram neural network to learn vector representations (word embeddings) for Amharic words. The primary goal is educational: to understand and verify the core mathematical foundations of word embeddings, including:
- Forward propagation and softmax distribution calculation
- Cross-entropy loss formulation
- Analytical gradient derivation for input and output weight matrices
- Stochastic Gradient Descent (SGD) optimization from scratch

The implementation relies strictly on Python standard library modules (`math`, `random`, lists, and loops).

---

## Model Architecture & Training Parameters

| Hyperparameter | Value |
| :--- | :--- |
| **Language** | Amharic (`am`) |
| **Architecture** | Skip-gram with Softmax |
| **Vocabulary Size** | 271 unique tokens |
| **Training Pairs** | 1,976 |
| **Embedding Dimension** | 10 |
| **Context Window Size** | 2 |
| **Learning Rate** | 0.05 |
| **Maximum Epochs** | 50 |
| **Random Seed** | 42 |
| **Best Checkpoint** | Epoch 49 |

The model optimizes two matrices: an input embedding matrix $E$ and an output context weight matrix $U$. For a given center word, it predicts surrounding context words via a softmax probability distribution over the vocabulary.

---

## Training & Validation Results

| Metric | Result |
| :--- | :--- |
| **Initial Mean Training Loss** | 5.602120 |
| **Best Mean Training Loss** | 2.798836 |
| **Best Epoch** | 49 |
| **Epochs Executed** | 50 |
| **Max Absolute Gradient-Check Error** | 0.00000000 |

> **Note:** The best mean training loss achieved a ~50% reduction from the initial loss. Epoch 49 was selected via early checkpointing as epoch 50 yielded a slightly higher loss value. These metrics validate implementation correctness on the training split and do not reflect generalization performance on unseen text.

### Training Loss Curve

The following plot illustrates the trajectory of the mean training loss recorded before training and after each epoch. The best model checkpoint was saved at epoch 49: