Blame for README.md

---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- recurrent-depth
- gated-deltanet
- lfm2
- from-scratch
- experimental
license: apache-2.0
base_model:
- StarpowerTechnology/LiquidWVY-Recurrent-Depth
---

# WVY Liquid Recurent Depth v1

## Tiny-Recursive-LM

LiquidWVY v1 is a tiny experimental model.
The architecture combines three ideas:

1. **Liquid/LFM-style local mixing:** double-gated causal short convolutions cheaply process nearby tokens.
2. **Gated DeltaNet memory:** recurrent fast-weight memory learns when to forget, update, and retrieve information across longer sequences.
3. **Recurrent depth:** a shared core is executed several times before output. This is the loop-transformer or recursive-forward-pass idea: the model performs more latent computation while reusing the same core parameters.

Grouped-query attention remains in the architecture for exact global token-to-token access. Every operator is wrapped with RMS normalization, residual connections, and a SwiGLU feed-forward network.

This combination is experimental. The component ideas have research support separately; LiquidWVY v1 has not yet been pretrained or benchmarked as a combined architecture. Training and evaluation are how its value must be established.

## Model flow

```text
tokens → embeddings
       → prelude: [LFM short convolution → grouped-query attention]
       → recurrent adapter + shared core: [Gated DeltaNet ↔ attention] × N loops
       → coda: [LFM short convolution → grouped-query attention]
       → RMSNorm → tied language-model head → next-token logits
```

The prelude grounds the input. The recurrent adapter injects the prelude representation into every loop. The shared core repeatedly updates the hidden state without adding a new copy of its weights for each recurrence. The coda converts the refined state into the representation used for token prediction.

## Repository layout

| Path | Purpose |
| --- | --- |
| `src/wvy_experimental/modeling_liquid_wvy.py` | Complete LiquidWVY v1 architecture |
| `src/wvy_experimental/configuration_liquid_wvy.py` | Hugging Face configuration |
| `src/wvy_experimental/train.py` | Fresh pretraining, validation, checkpoints, and export |
| `src/wvy_experimental/scan_data.py` | Exact token counter and parameter-target report |
| `src/wvy_experimental/build_config.py` | Searches full configurations for a requested parameter count |
| `src/wvy_experimental/prepare_tokenizer.py` | Byte-level BPE tokenizer training |
| `configs/` | Prepared approximately 10M, 25M, and 50M configurations |
| `assets/tokenizer/` | Public LFM2.5 tokenizer assets for compatibility experiments |
| `notebooks/` | Google Colab and Kaggle training kits |
| `reference/transformers-lfm2/` | Unchanged public LFM2 architecture source used as a reference |
| `huggingface/model-card-template.md` | Template for the trained checkpoint repository |

## Prepared scales

All three configurations use an 8,192-token vocabulary and four recurrent passes by default.

| Configuration | Actual parameters | Physical depth | Effective depth |
| --- | ---: | ---: | ---: |
| `configs/wvy_10m.json` | 9,852,888 | 9 | 24 |
| `configs/wvy_25m.json` | 24,272,848 | 6 | 12 |
| `configs/wvy_50m.json` | 50,210,540 | 7 | 16 |

Physical depth counts unique stored layers. Effective depth counts every pass through the shared recurrent layers. Changing `recurrent_steps` changes computation and effective depth without changing the parameter count.

## Supported training data

Pass one file or a directory. Directories are searched recursively. Supported formats are `.txt`, `.md`, `.json`, `.jsonl`, `.csv`, and `.parquet`. Structured files use a `text` field by default; pass `--text-field content` for another field.

## Local installation

Use Python 3.10 or newer. A CUDA GPU is recommended for real training.

```bash
cd WVY-Experimental
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e .
```

On Windows PowerShell, activate with `.venv\Scripts\Activate.ps1`.

## Prepare a run

Train a compact tokenizer from the corpus:

```bash
wvy-tokenizer --data data/ --vocab-size 8192 --output artifacts/tokenizer
```

The bundled Liquid tokenizer has 65,536 tokens. Its embedding table consumes too much of a 1M–50M budget, so a smaller corpus-specific tokenizer is recommended.

Count the tokens and write the dataset report:

```bash
wvy-scan \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --tokens-per-parameter 20 \
  --output artifacts/data_report.json
```

The ratio is a configurable planning input. No validated scaling law exists for this exact combined architecture. The report therefore presents a suggested starting target rather than a guaranteed optimum.

Generate the nearest model configuration:

```bash
wvy-build \
  --report artifacts/data_report.json \
  --tokenizer artifacts/tokenizer \
  --output artifacts/model_config.json
```

Or prepare all three items with one helper:

```bash
scripts/prepare_data_and_model.sh data/ 8192 20
```

## Train from fresh weights

```bash
wvy-train \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --config artifacts/model_config.json \
  --output outputs/liquidwvy-v1 \
  --sequence-length 1024 \
  --batch-size 4 \
  --gradient-accumulation 8 \
  --learning-rate 3e-4 \
  --epochs 1 \
  --bf16 \
  --gradient-checkpointing
```

Use `--fp16` on GPUs without BF16 support. Remove both precision flags for FP32. Lower the batch size if memory runs out. Resume an interrupted run with the same arguments plus:

```bash
--resume-from-checkpoint outputs/liquidwvy-v1/checkpoint-500
```

The final directory is `outputs/liquidwvy-v1/final/`. It contains trained weights, custom architecture files, configuration, and tokenizer assets. Load that checkpoint with:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "outputs/liquidwvy-v1/final",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "outputs/liquidwvy-v1/final",
    trust_remote_code=True,
)
```

The model accepts `recurrent_steps=` during a forward pass. Training defaults to the number in the configuration. Comparing several recurrence counts is part of evaluating whether additional latent computation helps.

## Google Colab

Open `notebooks/WVY_Colab_Training_Kit.ipynb` in Colab, select a GPU runtime, and run the cells in order. It uploads the kit and corpus, installs dependencies, prepares the tokenizer and configuration, trains fresh weights, and downloads the final checkpoint.

For long runs, mount Google Drive and set `OUTPUT_DIR` to a Drive location so checkpoints survive runtime resets.

## Kaggle

Add `WVY-Experimental.zip` and the corpus as Kaggle notebook inputs, enable a GPU, and open `notebooks/WVY_Kaggle_Training_Kit.ipynb`. Set `KIT_ZIP` and `DATA_PATH` in the first code cell, then run all cells. The trained archive appears in `/kaggle/working/`.

## Hugging Face publishing

Copy `huggingface/model-card-template.md` to the trained `final/` directory as `README.md`, fill in the corpus, compute, loss, evaluation, and intended-use sections, then upload the complete `final/` directory to a new model repository. Loading custom architecture code requires `trust_remote_code=True` until the architecture is integrated into Transformers.

## Implementation status

The Gated DeltaNet path implements the recurrent gated-delta update directly in PyTorch. It is mathematically complete and works on CPU or GPU, but the token recurrence currently executes sequentially. Long-sequence high-throughput training will benefit from a compatible chunkwise CUDA/Triton kernel after the architecture is validated. The standard attention path uses PyTorch scaled-dot-product attention.

No Liquid AI pretrained weights, private corpus, private training pipeline, or teacher logits are included. The `reference/` directory contains public source for study; LiquidWVY v1 runtime code is maintained under `src/wvy_experimental/`.

## Research basis and licensing

- LFM2 public architecture: https://github.com/huggingface/transformers/tree/main/src/transformers/models/lfm2
- Gated Delta Networks: https://arxiv.org/abs/2412.06464
- Recurrent-depth latent reasoning: https://arxiv.org/abs/2502.05171

See `THIRD_PARTY_NOTICES.md` and `licenses/`. The project name describes an experimental design influenced by public research; it does not claim that Liquid AI created or endorsed LiquidWVY v1.