| --- |
| license: mit |
| language: |
| - en |
| --- |
| # Character Embedding Model |
|
|
| A character level embedding for ASCII characters trained on Oxford English Dictionary. |
| <!-- The model learns to represent characters such that `emb('a')` is close to `emb('_pple')` where `_` represents the blank position for 'a'. --> |
|
|
| ## Model Description |
|
|
| This model uses a Transformer-based architecture to create embeddings that capture the contextual relationship between characters and their positions in words. It's trained using a contrastive learning approach where: |
|
|
| - **Positive pairs**: A character and its corresponding word with that character blanked out |
| - **Negative pairs**: Different characters that should have dissimilar embeddings |
|
|
| ### Architecture |
|
|
| - **Embedding Dimension**: 8 |
| - **Hidden Dimension**: 64 |
| - **Transformer Layers**: 2 |
| - **Attention Heads**: 8 |
| - **Vocabulary Size**: 257 (256 ASCII + blank token) |
|
|
| ### Training |
|
|
| The model was trained on word-definition pairs from a dictionary corpus using: |
| - Mixed precision training (FP16) |
| - Contrastive loss with margin-based negative sampling |
| - Periodic embedding stabilization |
| - Best model selection based on quality score (positive similarity - negative similarity) |
|
|
| ## Installation |
|
|
| ```bash |
| pip install torch numpy huggingface_hub |
| ``` |
|
|
| ## Usage |
|
|
| ### Loading the Model |
|
|
| ```python |
| import torch |
| import numpy as np |
| import torch.nn.functional as F |
| from huggingface_hub import hf_hub_download |
| |
| # Example for downloading a single file |
| local_path = hf_hub_download(repo_id="npc0/CharEmb", filename="char_embeddings_best.npy") |
| |
| # Load the pre-computed character embeddings |
| char_embeddings = np.load(local_path, allow_pickle=True).item() |
| |
| # Convert to tensor for efficient operations |
| char_embedding_tensor = {} |
| for char, emb in char_embeddings.items(): |
| char_embedding_tensor[char] = torch.tensor(emb, dtype=torch.float32) |
| ``` |
|
|
| ### Inference: Character to Embedding |
|
|
| ```python |
| def get_character_embedding(char): |
| """Get the embedding for a single character.""" |
| if char in char_embedding_tensor: |
| return char_embedding_tensor[char] |
| else: |
| print(f"Warning: Character '{char}' not found in embeddings") |
| return None |
| |
| # Example usage |
| char = 'a' |
| embedding = get_character_embedding(char) |
| print(f"Embedding for '{char}': {embedding}") |
| print(f"Embedding shape: {embedding.shape}") # Should be (8,) |
| ``` |
|
|
| ### Inference: Embedding to Character (Decoding) |
|
|
| ```python |
| def decode_embedding(query_embedding, top_k=5): |
| """ |
| Find the closest character(s) to a given embedding. |
| |
| Args: |
| query_embedding: torch.Tensor of shape (8,) |
| top_k: Number of closest characters to return |
| |
| Returns: |
| List of (character, similarity_score) tuples |
| """ |
| # Normalize query embedding |
| query_embedding = F.normalize(query_embedding.unsqueeze(0), p=2, dim=-1) |
| |
| similarities = [] |
| for char, emb in char_embedding_tensor.items(): |
| # Normalize character embedding |
| emb_norm = F.normalize(emb.unsqueeze(0), p=2, dim=-1) |
| # Compute cosine similarity |
| sim = F.cosine_similarity(query_embedding, emb_norm, dim=-1).item() |
| similarities.append((char, sim)) |
| |
| # Sort by similarity (descending) |
| similarities.sort(key=lambda x: x[1], reverse=True) |
| |
| return similarities[:top_k] |
| |
| # Example usage |
| test_char = 'e' |
| test_embedding = get_character_embedding(test_char) |
| |
| if test_embedding is not None: |
| top_matches = decode_embedding(test_embedding, top_k=5) |
| print(f"\nTop 5 characters similar to '{test_char}':") |
| for char, sim in top_matches: |
| print(f" '{char}': {sim:.4f}") |
| ``` |
|
|
| ## Model File |
|
|
| - `char_embeddings_best.npy`: Pre-computed character embeddings (numpy dictionary) |
|
|
| ## Limitations |
|
|
| - The model only supports ASCII characters (0-255) plus a special blank token |
| - Embeddings are context-averaged, so they may not capture all nuances of character usage |
| - Performance Limited by the diversity and quality of training data of Oxford English Dictionary |
| - The model uses a relatively small embedding dimension (8) for efficiency |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{character_embedding_model, |
| title={Character Embedding Model with Blank-Filling}, |
| author={Yuan Xu}, |
| year={2025}, |
| howpublished={\url{https://huggingface.co/your-username/character-embedding}} |
| } |
| ``` |
