Delete .ipynb_checkpoints/README-checkpoint.md

.ipynb_checkpoints/README-checkpoint.md

@@ -1,194 +0,0 @@
----
-library_name: transformers
-pipeline_tag: text-generation
-license: apache-2.0
-language:
-  - en
-tags:
-  - causal-lm
-  - base-model
-  - custom-code
-  - safetensors
-  - frozen-embeddings
----
-
-# AB-EXT 1.7B Binary16
-
-A decoder-only causal language model whose trainable input embedding table is replaced by canonical fixed 16-bit token-ID codes, repeated deterministically to the model width. The output head remains trainable and untied.
-
-This is a **base pretrained causal language model**. It has not been
-instruction-tuned, preference-optimized, or aligned as a conversational
-assistant.
-
-## Model summary
-
-| Property | Value |
-|---|---|
-| Architecture | Decoder-only causal Transformer |
-| Hidden width | 2048 |
-| Layers | 24 |
-| Attention heads | 32 |
-| FFN intermediate width | 8192 |
-| Context used in training | 2048 tokens |
-| Position encoding | RoPE |
-| Activation | SwiGLU |
-| Normalization | RMSNorm |
-| Input interface | Canonical fixed 16-bit input codes |
-| Output head | Trainable, untied |
-| Training objective | Causal next-token prediction |
-| Training tokens | 100,000,071,680 prediction targets |
-| Tokenizer | `HuggingFaceTB/SmolLM2-1.7B` |
-| Tokenizer revision | `effd688a12921b4cc83e3312b6feb579f70f9c71` |
-
-## Input representation
-
-
-For token ID $t$, the model constructs its canonical little-endian binary code:
-
-$$
-c(t) \in \{0,1\}^{16}.
-$$
-
-Because $d=2048$, the code is repeated $128$ times:
-
-$$
-x(t) =
-\underbrace{
-c(t) \Vert c(t) \Vert \cdots \Vert c(t)
-}_{128\text{ copies}}
-\in \mathbb{R}^{2048}.
-$$
-
-There is no trainable input embedding parameter or trainable input
-projection. The implementation stores the procedurally generated codebook
-as a persistent non-trainable buffer. The output vocabulary projection is
-trainable and untied.
-
-
-## Architecture
-
-The contextual backbone uses causal multi-head self-attention, RoPE,
-RMSNorm and SwiGLU residual blocks. The experiment changes the input
-interface while retaining the same contextual backbone and tokenizer.
-
-The model is not initialized from SmolLM2 weights. SmolLM2 supplies only
-the tokenizer artifacts.
-
-## Training
-
-- **Data:** FineWeb-Edu
-- **Validation:** FineWeb-Edu
-- **Objective:** causal next-token prediction
-- **Optimizer:** AdamW
-- **Peak learning rate:** `0.00015`
-- **Minimum learning rate:** `1e-05`
-- **Warmup:** `2000` optimizer steps
-- **Weight decay:** `0.01`
-- **Sequence length:** `2048`
-- **Precision:** FP32 master parameters and AdamW states; BF16 autocast during forward/backward
-- **Hardware:** 2 x NVIDIA H100
-- **Seed:** `42`
-
-The stated token count is the trainer's actual prediction-target counter,
-not merely the requested stopping threshold.
-
-
-## Evaluation
-
-A frozen evaluation table will be added after all released models and
-comparison baselines have been evaluated with the same pinned commit of
-`lm-evaluation-harness`.
-
-| Benchmark | Shots | Score |
-|---|---:|---:|
-| HellaSwag | 0 | Pending |
-| ARC-Easy | 0 | Pending |
-| ARC-Challenge | 0 | Pending |
-| PIQA | 0 | Pending |
-| WinoGrande | 0 | Pending |
-| OpenBookQA | 0 | Pending |
-| CommonsenseQA | 0 | Pending |
-| MMLU | 5 | Pending |
-| TriviaQA | 0 | Pending |
-| GSM8K | 5 | Pending |
-
-The exact harness commit, task versions, prompts, random seeds and raw JSON
-outputs will be published with the scores.
-
-
-## Usage
-
-This repository contains custom Transformers code. Review the files before
-enabling remote code execution.
-
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_id = "E6E831728/AB-EXT-1.7B-Binary16"
-
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    trust_remote_code=True,
-    torch_dtype=torch.bfloat16,
-    device_map="auto",
-).eval()
-
-inputs = tokenizer(
-    "Gravity is",
-    return_tensors="pt",
-    add_special_tokens=False,
-).to(model.device)
-
-with torch.no_grad():
-    output = model.generate(
-        **inputs,
-        max_new_tokens=32,
-        do_sample=False,
-    )
-
-print(tokenizer.decode(output[0], skip_special_tokens=True))
-```
-
-For strict reproducibility, pin a Hub commit through `revision=...`.
-
-## Limitations
-
-- This is a base completion model, not a chat assistant.
-- The training context was 2048 tokens.
-- Factual outputs may be incorrect or outdated.
-- Generated text can contain biases or harmful content inherited from the
-  pretraining data.
-- Benchmark scores are sensitive to task version, prompt construction,
-  few-shot protocol and decoding settings.
-- Comparisons with SmolLM2 are not controlled comparisons of training
-  compute or data: SmolLM2 uses a different architecture, corpus and token
-  budget.
-- The tokenizer is reused from SmolLM2, but the model weights are newly
-  trained and are not derivatives of SmolLM2 weights.
-
-## Reproducibility and integrity
-
-- Weight SHA-256: `c3e78f24f03dff36b6174985dc5189e8ce37a52685d022ade36fa68c3239801b`
-- Stored tensor values: `1,712,162,816`
-- Stored values by dtype: `{"F32": 1712162816}`
-- Custom loading requires `trust_remote_code=True`.
-
-## Attribution
-
-The tokenizer artifacts are distributed from
-`HuggingFaceTB/SmolLM2-1.7B`, which is released under the Apache-2.0
-license. No SmolLM2 model weights are included.
-
-## Citation
-
-```bibtex
-@misc{attnext2026,
-  title  = {AB-EXT 1.7B Binary16},
-  author = {Anonymous Author(s)},
-  year   = {2026},
-  url    = {https://huggingface.co/E6E831728/AB-EXT-1.7B-Binary16}
-}
-```