| --- |
| language: en |
| license: mit |
| library_name: pytorch |
| tags: |
| - transformer-decoder |
| - seq2seq |
| - embeddings |
| - mpnet |
| - text-reconstruction |
| - crypto |
| pipeline_tag: text-generation |
| --- |
| |
| ### Aparecium Baseline Model Card |
|
|
| #### Summary |
| - **Task**: Reconstruct natural language posts from token‑level MPNet embeddings (reverse embedding). |
| - **Focus**: Crypto domain, with equities as auxiliary domain. |
| - **Checkpoint**: Baseline model trained with a phased schedule and early stopping. |
| - **Data**: 1.0M synthetic posts (500k crypto + 500k equities), programmatically generated via OpenAI API. No real social‑media content used. |
| - **Input contract**: token‑level MPNet matrix of shape `(seq_len, 768)`, not a pooled vector. |
|
|
| --- |
|
|
| ### Intended use |
| - Research and engineering use for studying reversibility of embedding spaces and for building diagnostics/tools around embedding interpretability. |
| - Not intended to reconstruct private or sensitive content; reconstruction accuracy depends on embedding fidelity and domain match. |
|
|
| --- |
|
|
| ### Model architecture |
| - Encoder side: External; we assume MPNet family encoder (default: `sentence-transformers/all-mpnet-base-v2`) to produce token‑level embeddings. |
| - Decoder: Transformer decoder consuming the MPNet memory: |
| - d_model: 768 |
| - Decoder layers: 2 |
| - Attention heads: 8 |
| - FFN dim: 2048 |
| - Token and positional embeddings; GELU activations |
| - Decoding: |
| - Supports greedy, sampling, and beam search. |
| - Optional embedding‑aware rescoring (cosine similarity between the candidate’s re‑embedded sentence and the pooled MPNet target). |
| - Optional lightweight constraints for hashtag/cashtag/URL continuity. |
| |
| Recommended inference defaults: |
| - `num_beams=8` |
| - `length_penalty_alpha=0.6` |
| - `lambda_sim=0.6` |
| - `rescore_every_k=4`, `rescore_top_m=8` |
| - `beta=10.0` |
| - `enable_constraints=True` |
| - `deterministic=True` |
|
|
| --- |
|
|
| ### Training data and provenance |
| - 1,000,000 synthetic posts total: |
| - 500,000 crypto‑domain posts |
| - 500,000 equities‑domain posts |
| - All posts were programmatically generated via the OpenAI API (synthetic). No real social‑media content was used. |
| - Embeddings: |
| - Token‑level MPNet (default: `sentence-transformers/all-mpnet-base-v2`). |
| - Cached to SQLite to avoid recomputation and allow resumable training. |
|
|
| --- |
|
|
| ### Training procedure (baseline regimen) |
| - Domain emphasis: 80% crypto / 20% equities per training phase. |
| - Phased training (10% of available chunks per phase), evaluate after each phase: |
| - In‑sample: small subset from the phase’s chunks |
| - Out‑of‑sample: small hold‑out from both domains (not seen in the phase) |
| - Early‑stop condition: stop if out‑of‑sample cosine degrades relative to prior phase. |
| - Optimizer: AdamW |
| - Learning rate (baseline finetune): 5e‑5 |
| - Batch size: 16 |
| - Input `max_source_length`: 256 |
| - Target `max_target_length`: 128 |
| - Checkpointing: every 2,000 steps and at phase end. |
|
|
| Notes |
| - Training used early stopping based on out‑of‑sample cosine. |
|
|
| --- |
|
|
| ### Evaluation protocol (for the metrics below) |
| - Sample size: 1,000 examples per domain drawn from cached embedding databases. |
| - Decode config: `num_beams=8`, `length_penalty_alpha=0.6`, `lambda_sim=0.6`, `rescore_every_k=4`, `rescore_top_m=8`, `beta=10.0`, `enable_constraints=True`, `deterministic=True`. |
| - Metrics: |
| - `cosine_mean/median/p10/p90`: cosine between pooled MPNet embedding of generated text and the pooled MPNet target vector (higher is better). |
| - `score_norm_mean`: length‑penalized language model score (more positive is better; negative values are common for log‑scores). |
| - `degenerate_pct`: % of clearly degenerate generations (very short/blank/only hashtags). |
| - `domain_drift_pct`: % of equity‑like terms in crypto outputs (or crypto‑like terms in equities outputs). Heuristic text filter; intended as a rough indicator only. |
|
|
| Results (current `models/baseline` checkpoint) |
| - Crypto (n=1000) |
| - cosine_mean: 0.681 |
| - cosine_median: 0.843 |
| - cosine_p10: 0.000 |
| - cosine_p90: 0.984 |
| - score_norm_mean: −1.977 |
| - degenerate_pct: 5.2% |
| - domain_drift_pct: 0.0% |
| - Equities (n=1000) |
| - cosine_mean: 0.778 |
| - cosine_median: 0.901 |
| - cosine_p10: 0.326 |
| - cosine_p90: 0.986 |
| - score_norm_mean: −1.344 |
| - degenerate_pct: 2.2% |
| - domain_drift_pct: 4.4% |
|
|
| Interpretation |
| - The model reconstructs many posts with strong embedding alignment (p90 ≈ 0.98 cosine in both domains). |
| - Equities shows higher average/median cosine and lower degeneracy than crypto, consistent with the auxiliary‑domain role and data characteristics. |
| - A small fraction of degenerate outputs exists in both domains (crypto ~5.2%, equities ~2.2%). |
| - Domain drift is minimal from crypto→equities (0.0%) and present at a modest rate from equities→crypto (~4.4%) under the chosen heuristic. |
|
|
| --- |
|
|
| ### Input contract and usage |
| - **Input**: MPNet token‑level matrix `(seq_len × 768)` for a single post. Do not pass a pooled vector. |
| - **Tokenizer/model alignment** matters: use the same MPNet tokenizer/model version that produced the embeddings. |
|
|
| --- |
|
|
| ### Limitations and responsible use |
| - Reconstruction is not guaranteed to match the original post text; it optimizes alignment within the MPNet embedding space and LM scoring. |
| - The model can produce generic or incomplete outputs (see `degenerate_pct`). |
| - Domain drift can occur depending on decode settings (see `domain_drift_pct`). |
| - Data are synthetic programmatic generations, not real social‑media posts. Domain semantics may differ from real‑world distributions. |
| - Do not use for reconstructing sensitive/private content or for attempting to de‑anonymize embedding corpora. This model is a research/diagnostic tool. |
|
|
| --- |
|
|
| ### Reproducibility (high‑level) |
| - Prepare embedding caches (not included): build local token‑level MPNet embedding caches for your corpora (e.g., via a data prep script) and store them in your own paths. |
| - Baseline training: iterative 10% phases, 80:20 (crypto:equities), LR=5e‑5, BS=16, early‑stop on out‑of‑sample cosine degradation. |
| - Evaluation: 1,000 samples/domain with the decode settings shown above. |
| - The released checkpoint corresponds to the latest non‑degrading phase under early‑stopping. |
|
|
| --- |
|
|
| ### License |
| - Code: MIT (per repository). |
| - Model weights: same as code unless declared otherwise upon release. |
|
|
| --- |
|
|
| ### Citation |
| If you use this model or codebase, please cite the Aparecium project and this baseline report. |
|
|
|
|
|
|
