File size: 2,066 Bytes
f14b99a 6ac6781 8930aea b7ac173 6ac6781 b7ac173 6ac6781 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
license: mit
language:
- en
tags:
- embeddings
- word2vec
---
# Embeddings
## Paper Refs. :
- [Mikolov et al 2013 - Distributed Representations of Words and Phrases (SGNS)](https://arxiv.org/pdf/1310.4546.pdf)
- [Rong, Xin 2014 - word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738)
- [Levy & Goldberg 2014 - Neural Word Embedding as Implicit Matrix Factorization](https://papers.nips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html)
## Results
- **Nearest neighbours: strong** (e.g. `france → spain, italy, germany`)
- **Analogies: ~14% top-1** (semantic > morphological; limited by the small 17M-token corpus)
## Limitations
Small corpus → weak on analogies (esp. capital-country, morphology). For better analogy
accuracy, train on a larger corpus (enwik9+). Lowercased English only; drops OOV.
## Usage snippet
```python
from huggingface_hub import hf_hub_download
import torch
import torch.nn.functional as F
path = hf_hub_download(repo_id="ocdbytes/embeddings", filename="embeddings_200.pt")
# weights_only=False because the checkpoint bundles Python dicts (word2idx/idx2word),
# which the default restricted loader (torch>=2.6) may reject.
ck = torch.load(path, map_location="cpu", weights_only=False)
syn0 = ck["syn0"]
word2idx, idx2word = ck["word2idx"], ck["idx2word"]
emb = F.normalize(syn0, dim=1)
def neighbours(word, n=10):
i = word2idx[word]
sims = emb @ emb[i]
top = sims.topk(n + 1).indices.tolist()
return [idx2word[j] for j in top if j != i][:n]
def analogy(a, b, c, n=5):
t = F.normalize(emb[word2idx[b]] - emb[word2idx[a]] + emb[word2idx[c]], dim=0)
sims = emb @ t
ban = {word2idx[a], word2idx[b], word2idx[c]}
top = sims.topk(n + len(ban)).indices.tolist()
return [idx2word[j] for j in top if j not in ban][:n]
print(neighbours("king")) # -> ['viii', 'elizabeth', 'queen', ...]
print(analogy("france", "paris", "germany")) # -> ['berlin', ...]
```
## Code
- [ocdbytes-ai/embeddings](https://github.com/ocdbytes-ai/embeddings) |