File size: 2,679 Bytes
1a1fb3b e9938b0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
tags:
- markov-chain
- text-generation
- discord
- sqlite
---
# Ruby Chain
Pre-trained Markov chain databases for [Ruby](https://github.com/protocol-luna/ruby), the spontaneous message generator in the Luna Protocol ecosystem.
Trained on **16.9M human messages** from [mookiezi/Discord-Dialogues](https://huggingface.co/datasets/mookiezi/Discord-Dialogues) — a diverse collection of Discord conversations.
## Files
| File | Order | Transitions | Starters | Size | Description |
|------|-------|-------------|----------|------|-------------|
| `chain-order2.db` | 2 | 18,955,474 | 615,483 | 1.1 GB | Original chain, good variety but less coherent |
| `chain-order3.db` | 3 | 6,833,837 | 788,166 | 626 MB | Better coherence, smaller footprint |
| `chain-order4.db` | 4 | 6,891,633 | 1,101,029 | 747 MB | Highest coherence, more unique starters |
| `messages.txt.gz` | — | — | — | 274 MB | 16.9M messages used for training |
**Recommended:** `chain-order3.db` or `chain-order4.db` depending on your size vs coherence tradeoff.
## Database Schema
```sql
CREATE TABLE transitions (
prefix TEXT,
suffix TEXT,
count INTEGER,
channel_id TEXT,
PRIMARY KEY (prefix, suffix, channel_id)
);
CREATE TABLE starters (
prefix TEXT,
channel_id TEXT,
PRIMARY KEY (prefix, channel_id)
);
CREATE INDEX idx_trans_prefix ON transitions(prefix);
```
Prefixes are words joined by `\x00`. The `order` determines how many words form a prefix (e.g., order 3 → 3-word prefix).
The `channel_id` column is always `""` (cross-channel aggregate — the source dataset doesn't include channel metadata).
## Usage
```js
const Database = require('better-sqlite3');
const db = new Database('chain-order3.db');
// Get a random starter
const starter = db.prepare(
'SELECT prefix FROM starters ORDER BY RANDOM() LIMIT 1'
).get()?.prefix;
// Sample the chain
function generate(db, prefix, maxLen = 30) {
const words = prefix.split('\x00');
for (let i = 0; i < maxLen; i++) {
const row = db.prepare(
'SELECT suffix, count FROM transitions WHERE prefix = ? ORDER BY RANDOM()'
).all(prefix);
if (!row.length) break;
const total = row.reduce((s, r) => s + r.count, 0);
let roll = Math.random() * total;
let suffix = row[0].suffix;
for (const r of row) {
roll -= r.count;
if (roll <= 0) { suffix = r.suffix; break; }
}
words.push(suffix);
prefix = words.slice(-3).join('\x00');
}
return words.join(' ');
}
```
## Source
Trained with [Ruby's `tools/train-multi.js`](https://github.com/protocol-luna/ruby) — multi-worker Node.js trainer using better-sqlite3. ~300,000 messages/second on 4 CPU cores.
|