|
Download source/README.md from sriharithebest/nayara-v1: direct link, hf CLI and curl.
- Browser
- Download file 8.33 kB
-
https://huggingface.co/sriharithebest/nayara-v1/resolve/main/source/README.md
- Command line
-
hf download hf://sriharithebest/nayara-v1/source/README.md
-
curl -L -o README.md https://huggingface.co/sriharithebest/nayara-v1/resolve/main/source/README.md
8.33 kB
| # NAYARA v1.0 | |
| NAYARA is a from-scratch, decoder-only 351.57M-parameter code model and a complete 7×RTX 5090 training-to-CPU-demo pipeline. It does not download or initialize from pretrained model weights. The tokenizer is trained from the selected corpus. | |
| The repository is an implementation, not a pretrained checkpoint. This checkout has no CUDA runtime or training corpus, so weights must be produced on the stated GPU host. “100% useful,” “flawless,” and a fixed quality level cannot be guaranteed before evaluation; the sprint has explicit quality gates instead. | |
| For a notebook run today, open `notebooks/NAYARA_Colab.ipynb` and follow `COLAB_QUICKSTART.md`. Its deadline-safe one-GPU L4/A100 profile trains the same 351.57M architecture for 50M tokens and exports a working preview plus Gradio demo. It is a pipeline/behavior preview, not the production-quality 3.8B-token checkpoint. | |
| For a Lightning AI Studio with one H100 80 GB, follow `LIGHTNING_H100_QUICKSTART.md`. It includes exact machine settings, a 2B-token overnight profile, compiled single-GPU training, automatic unique-data clamping, INT8 export, API verification, and Snapshot Deploy settings. | |
| `NAYARA_BUILD_PROMPT.md` is the tightened execution prompt for a GPT-5.6 Codex SOL Ultra run. It replaces untestable superlatives with frozen-cohort quality and CPU performance gates. | |
| ## Locked architecture | |
| | Item | Value | | |
| |---|---:| | |
| | Parameters (tied embedding/head) | 351,568,896 | | |
| | Layers / width | 16 / 1024 | | |
| | Query / KV heads | 8 / 2 | | |
| | Head dimension | 128 | | |
| | SwiGLU intermediate | 5632 | | |
| | Vocabulary / context | 32,000 / 1024 | | |
| | Position / normalization | RoPE / RMSNorm ε=1e-5 | | |
| The 5632 intermediate dimension is intentional. An intermediate size of 2816 would make the proposed network roughly 213M parameters, not 350M. | |
| ## Budget truth table | |
| The usual dense-transformer training estimate is `6 × parameters × tokens`. | |
| | Token target | Approx. model FLOPs | Optimizer steps at 7×32×1024×GA2 | | |
| |---|---:|---:| | |
| | 3.8B (launch default) | 8.016e18 | 8,284 | | |
| | 4.0B (corpus capacity) | 8.438e18 | 8,720 | | |
| Four billion tokens leaves less than one percent of an 8.5e18-FLOP ceiling for approximation error and overhead. The 3.8B default preserves about 5.7% headroom. The wall-clock stop always wins over the token target. | |
| At exactly 45% model-FLOP utilization, finishing 3.8B tokens in 3.5 hours requires about 636 TFLOP/s aggregate measured model throughput. Measure the first 100 steps; do not infer this from marketing TOPS. | |
| ## Data, before the timed sprint | |
| The timed 3.5-hour window cannot include acquiring, filtering, training a tokenizer, and packing four billion tokens. Prepare the three `uint16` files on local NVMe first. | |
| The included source manifest uses an immediately streamable baseline: | |
| - 80% code: CodeParrot Clean Train, restricted to rows marked with permissive licenses and filtered/deduplicated again locally. If Software Heritage access has already been arranged, replace this with Stack-Edu/Stack v2 educational code. | |
| - 15% reasoning: NVIDIA OpenCodeReasoning formatted with NAYARA's user/assistant tokens. | |
| - 5% documentation: deduplicated FineWeb-Edu from the SmolLM corpus. | |
| Review every upstream dataset card and license before distribution. Public code corpora can still contain PII, secrets, malicious code, attribution obligations, or incorrect license labels. The fetcher rejects common embedded-secret patterns and exact normalized duplicates, but it is not a compliance guarantee. | |
| ```bash | |
| python -m venv .venv | |
| source .venv/bin/activate | |
| pip install -e '.[train,demo,dev]' | |
| # This can take many hours and substantial disk/network bandwidth. | |
| python -m nayara.prepare_data fetch --sources data/sources.example.json --raw-dir /mnt/nvme/nayara/raw | |
| python -m nayara.prepare_data tokenizer --raw-dir /mnt/nvme/nayara/raw --output /mnt/nvme/nayara/tokenizer.json | |
| python -m nayara.prepare_data pack --raw-dir /mnt/nvme/nayara/raw --tokenizer /mnt/nvme/nayara/tokenizer.json --output /mnt/nvme/nayara/packed | |
| ``` | |
| Copy `tokenizer.json` into every exported checkpoint directory before inference. The resulting `manifest.json` records the actual token count of every stream. Code documents receive a deterministic 50% fill-in-the-middle transform during packing. Do not start if any source has less than `world_size × micro_batch × (sequence_length + 1)` tokens. | |
| ## Preflight | |
| Use Linux, a current PyTorch/CUDA build that supports Blackwell (`sm_120`), one process per GPU, and a local PCIe Gen 5 NVMe path. FlashAttention-3's original package path was Hopper-specific; NAYARA instead forces PyTorch native fused flash SDPA and executes a real BF16 GQA probe at startup. It aborts rather than silently choosing the math kernel. | |
| Before the real launch: | |
| ```bash | |
| pytest -q | |
| nvidia-smi topo -m | |
| nvidia-smi --query-gpu=index,name,memory.total,pstate,temperature.gpu,power.draw --format=csv | |
| torchrun --standalone --nproc-per-node=7 -m nayara.train \ | |
| --data /mnt/nvme/nayara/packed --output /mnt/nvme/nayara/preflight \ | |
| --micro-batch 2 --grad-accum 1 --max-tokens 143360 --max-minutes 2 | |
| ``` | |
| The production launcher sets NCCL async error handling and a PCIe-appropriate P2P level. Confirm the actual topology first; consumer motherboards may split lanes or route peers through the CPU. If batch 32 OOMs, lower `--micro-batch` and increase accumulation so global tokens per optimizer step remain similar. “32/64 tokens per GPU” is not a valid batch unit here: the implementation interprets 32 as sequences, or 32,768 tokens per GPU per micro-step. | |
| ## Timed launch | |
| Edit the two NVMe paths in `scripts/launch_train.sh`, then: | |
| ```bash | |
| bash scripts/launch_train.sh | |
| ``` | |
| The trainer uses FSDP `FULL_SHARD`, block-level wrapping, BF16 parameters/reductions/buffers, fused AdamW, activation checkpointing, cosine decay with 2% warmup, norm clipping at 1.0, and `torch.compile(mode="reduce-overhead")`. It checkpoints at whichever occurs first: 200 optimizer steps or 15 minutes. `--stop-at 15:15` is interpreted in the host's local time and refuses to run if that time has passed. With an explicit deadline, the default 15-minute shutdown reserve starts the final gather at 15:00 so checkpoint export does not consume the conversion window. | |
| Checkpoint export gathers a CPU full state on rank 0. Budget RAM and fast local storage for the temporary gather; the final SafeTensors model is approximately 671 MiB in BF16 before filesystem metadata. The optimizer checkpoint is for trusted local resume only. | |
| ## Conversion and CPU demo | |
| Copy the tokenizer into the selected final checkpoint, quantize during the reserved conversion window, then launch Gradio: | |
| ```bash | |
| cp /mnt/nvme/nayara/tokenizer.json /mnt/nvme/nayara/checkpoints/step-XXXXXX/tokenizer.json | |
| python -m nayara.quantize /mnt/nvme/nayara/checkpoints/step-XXXXXX | |
| python app.py /mnt/nvme/nayara/checkpoints/step-XXXXXX --host 127.0.0.1 --port 7860 | |
| ``` | |
| The demo uses TorchAO dynamic INT8 linear layers and a KV cache. CPU latency depends heavily on generation length, memory bandwidth, PyTorch build, and core count; benchmark on the target i5 before describing it as “ultra fast.” Keep the server bound to localhost unless authentication and sandboxing are added. | |
| Direct CLI completion: | |
| ```bash | |
| python -m nayara.generate /path/to/checkpoint "def stable_topological_sort(graph):" --completion --max-new-tokens 128 | |
| ``` | |
| For a local OpenAI-style API, install `.[serve]` and run `uvicorn api:app --host 127.0.0.1 --port 8080` with `NAYARA_CHECKPOINT` set to a local checkpoint or S3 prefix. The container and ECS Fargate instructions are in `deploy/aws/README.md`. | |
| ## Release gates | |
| Do not label a checkpoint v1.0 until it passes all of these: | |
| 1. Held-out loss is finite and beats a frequency/unigram baseline. | |
| 2. HumanEval+, MBPP+, and a decontaminated internal completion set are run with recorded sampling settings. | |
| 3. Secret-regurgitation, license-attribution, insecure-code, and prompt-abuse probes are reviewed. | |
| 4. Median time-to-first-token and tokens/second are recorded on the actual target i5 with INT8 enabled. | |
| 5. At least 100 manually reviewed prompts establish the intended concise, direct NAYARA tone. Tone is learned from the formatted reasoning stream; a system prompt cannot substitute for training at this model size. | |