MultivexAI commited on
Commit
154ce79
·
verified ·
1 Parent(s): 1d4015f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +190 -0
README.md CHANGED
@@ -1,3 +1,193 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - llama
8
+ - causal-lm
9
+ - slm
10
+ - tiny
11
+ - small
12
+ - AtomixLabs
13
+ - Pascal
14
+ - math
15
+ - mathematics
16
+ - arithmetic
17
+ - sequence-induction
18
+ - symbolic-regression
19
+ - algorithmic-reasoning
20
+ - oeis
21
+ - discrete-mathematics
22
+ - dynamical-systems
23
+ - time-series
24
+ - telemetry
25
+ - multi-token-prediction
26
+ - muon
27
+ - synthetic-data
28
+ library_name: transformers
29
  ---
30
+
31
+ # ⚛️ Pascal-15M-v1.0
32
+
33
+ **Pascal-15M-v1.0** is a compact foundation model developed by **AtomixLabs** for mathematical sequence induction and numeric pattern generalization.
34
+
35
+ Instead of generating natural language text or code, Pascal is trained to model the transition dynamics of numerical series, identifying the latent mathematical rules governing a sequence prefix and extrapolating them forward.
36
+
37
+ Pascal-15M-v1.0 contains **17,716,608 parameters** and was pretrained on approximately **364 million tokens** of structured numerical sequences. Built strictly on native LLaMA transformer blocks with a custom 32-token digit-level vocabulary, it is designed primarily as a lightweight, clean base checkpoint for downstream specialization and fine-tuning.
38
+
39
+ ---
40
+
41
+ ## Task Formulation & Inductive Scope
42
+
43
+ Given a comma-separated sequence prefix, Pascal functions as an autoregressive sequence solver. Rather than retrieving memorized series from an index, the model relies on its attention layers to identify recurrence orders, polynomial progressions, modular cycles, and rational approximations directly from raw input strings.
44
+
45
+ Representative input-output behaviors include:
46
+
47
+ * **Elementary Progressions:** `1, 2, 4, 8, 16,` → `32`
48
+ * **Rational Convergents:** `1, 1/2, 2/3, 3/5, 5/8,` → `8/13` *(Fibonacci convergents to the golden ratio φ)*
49
+ * **Interleaved Independent Streams:** `2, 10, 4, 20, 8, 30,` → `16`
50
+ * **Coupled State Dynamics:** `16, 20, 22, 22, 20, 16, 10,` → `2` *(latent velocity oscillator)*
51
+
52
+ The objective of the pretraining regime is to provide an inductive bias for **symbolic regression in latency**, allowing the network to capture latent dynamic laws from raw observations.
53
+
54
+ ---
55
+
56
+ ## Tokenizer & Numerical Representation
57
+
58
+ Standard language model tokenizers rely on Byte-Pair Encoding (BPE), which merges digits into arbitrary chunks based on corpus frequency (for example, tokenizing `1234` as a single unit while splitting `1235` into two). This arbitrary grouping disrupts place-value relationships and degrades arithmetic consistency.
59
+
60
+ Pascal resolves this by using an exact, **lossless 32-token vocabulary**:
61
+
62
+ * **Digits:** `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`
63
+ * **Separators & Operators:** `,` (delimiter), `-` (sign), `.` (decimal), and `/` (exact rational fraction slash)
64
+ * **Control Tokens:** `<bos>`, `<eos>`, `<pad>`, `<unk>`, and reserved alignment slots
65
+
66
+ > ⚠️ **No `<eos>` Supervision During Training:** While `<eos>` exists as an index slot in the tokenizer vocabulary, **Pascal-15M-v1.0 was not trained with an `<eos>` token**. All pretraining target sequences concluded directly on the final numerical digit of the pattern. Consequently, the model will **never spontaneously emit an `<eos>` token** to halt generation. During inference, stopping criteria must be handled explicitly—either by setting `eos_token_id=tokenizer.convert_tokens_to_ids(",")` to stop at the next number, or by specifying `max_new_tokens`.
67
+
68
+ By processing numbers strictly digit-by-digit or as exact rational ratios (e.g., `8/13`), the model avoids floating-point rounding errors and eliminates tokenization artifacts during sequence evaluation.
69
+
70
+ ---
71
+
72
+ ## Architectural Specification
73
+
74
+ Pascal-15M-v1.0 uses a standard decoder-only LLaMA architecture. The model relies entirely on native PyTorch/Hugging Face operations without custom CUDA kernels, non-standard attention mechanisms, or external repository dependencies (`trust_remote_code=False`).
75
+
76
+ ```python
77
+ # Pascal-15M-v1.0 Structural Specification
78
+ vocab_size: int = 32 # Lossless digit-level numerical vocabulary
79
+ hidden_size: int = 384 # Hidden dimension
80
+ intermediate_size: int = 768 # 2.0x SwiGLU expansion ratio
81
+ num_hidden_layers: int = 12 # Sequential layer depth
82
+ num_attention_heads: int = 12 # Head dimension = 32
83
+ num_key_value_heads: int = 12 # Multi-Head Attention (MHA)
84
+ max_position_embeddings: int = 512 # Context window
85
+ rope_theta: float = 10000.0 # Base rotary frequency
86
+ # Total Trainable Parameters: 17,716,608
87
+ ```
88
+
89
+ At under 18 million parameters, the network operates with low compute requirements and low memory footprint, scaling appropriately for its 32-token vocabulary across its ~364M token training run.
90
+
91
+ ---
92
+
93
+ ## Intended Use & Downstream Adaptation
94
+
95
+ Pascal-15M-v1.0 is intended primarily as a **base model for downstream fine-tuning**.
96
+
97
+ While the base checkpoint exhibits stable out-of-the-box performance on standard arithmetic, polynomial, and recurrence patterns, it is designed to be adapted to specific numerical tasks, including:
98
+
99
+ * **Time-Series & Sensor Modeling:** Fine-tuning on telemetry sequences, systems monitoring, or discrete sensor metrics.
100
+ * **Empirical Symbolic Regression:** Adapting the model to experimental observations to predict underlying dynamic behaviors.
101
+ * **Financial Tick Sequences:** Modeling discrete price levels, spread variations, and rate adjustments.
102
+ * **Dynamical Systems Simulation:** Specializing the architecture on kinematics, coupled oscillations, or orbital transitions.
103
+
104
+ ### Basic Usage
105
+
106
+ ```python
107
+ from transformers import AutoModelForCausalLM, AutoTokenizer
108
+
109
+ model_id = "AtomixLabs/Pascal-15M-v1.0"
110
+
111
+ model = AutoModelForCausalLM.from_pretrained(
112
+ model_id,
113
+ torch_dtype="auto",
114
+ device_map="auto"
115
+ )
116
+
117
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
118
+
119
+ prompt = "1, 1/2, 2/3, 3/5, 5/8,"
120
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
121
+ sep_id = tokenizer.convert_tokens_to_ids(",")
122
+
123
+ outputs = model.generate(
124
+ **inputs,
125
+ max_new_tokens=24,
126
+ do_sample=False,
127
+ eos_token_id=sep_id, # Stop immediately upon completing the next number
128
+ pad_token_id=tokenizer.pad_token_id
129
+ )
130
+
131
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
132
+ ```
133
+
134
+ ---
135
+
136
+ ## Constrained Logit Decoding
137
+
138
+ Because the model uses a compact 32-token vocabulary, outputs can be guided during generation using grammar-constrained logit masking.
139
+
140
+ An external logit processor can enforce structural constraints:
141
+ 1. Suppressing consecutive signs or delimiters (`--`, `..`, `//`).
142
+ 2. Disallowing invalid operations such as division by zero (`/0`).
143
+ 3. Preventing open-ended fraction slashes or decimals before sequence separators.
144
+
145
+ This setup ensures that generated outputs conform to well-formed integers, decimals, or reduced fractions.
146
+
147
+ ---
148
+
149
+ <details>
150
+ <summary><b>Safety Advisory, Technical Limitations & License Notice</b></summary>
151
+
152
+ ### Operational Scope & Technical Limitations
153
+
154
+ **Pascal-15M-v1.0** is an experimental, specialized micro-scale foundation model (17.7M parameters). Because it operates exclusively on numerical sequences rather than natural language, its failure modes differ fundamentally from standard text models. Users and downstream developers should account for the following technical and operational considerations:
155
+
156
+ **1. Vocabulary, Formatting & Context Bounds**
157
+ * **Numeric-Only Scope:** Pascal-15M-v1.0 has a strict 32-token numerical vocabulary. It cannot parse English instructions, hold conversations, or generate source code. Any unsupported characters are mapped to `<unk>` during tokenization, which introduces uninformative tokens into the context and degrades prediction quality.
158
+ * **Formatting Sensitivity:** The model expects consistent comma-separated sequences without extraneous whitespace or mixed notation. Malformed delimiters, non-standard scientific notation, or inconsistent decimal precision can disrupt attention states and produce erratic continuations.
159
+ * **Context Truncation (512 Tokens):** Because sequences are represented digit-by-digit, high-precision floating-point numbers or long series rapidly consume the 512-token context window. Once a sequence exceeds this boundary, earlier terms are truncated, and the model may lose track of long-range recurrence rules or initial boundary conditions.
160
+
161
+ **2. Non-Formal Prediction & Spurious Extrapolation**
162
+ * **Not a Symbolic Engine:** Pascal-15M-v1.0 is an autoregressive neural sequence predictor, not a deterministic computer algebra system (CAS) or formal theorem prover. It approximates latent patterns statistically and does not guarantee exact analytical precision.
163
+ * **False Pattern Induction on Stochastic Data:** The model is trained to detect structural continuity. When prompted with purely stochastic processes, cryptographic outputs, unmodeled exogenous shocks, or chaotic systems, it will attempt to fit and extrapolate spurious patterns rather than recognizing the input as non-deterministic noise.
164
+ * **Verification:** Predictions should be independently validated by deterministic checks or domain experts before being treated as analytically reliable.
165
+
166
+ **3. Downstream Fine-Tuning & Adaptation Responsibility**
167
+ * Pascal-15M-v1.0 is intended primarily as a base weight initialization for downstream specialization. Fine-tuning on domain-specific telemetry, sensor feeds, or financial ticks alters the model's inductive bias and error distributions.
168
+ * Downstream developers are solely responsible for conducting out-of-distribution testing, calibrating confidence thresholds, and monitoring for distribution shift in fine-tuned checkpoints.
169
+
170
+ **4. High-Consequence System Advisory**
171
+ * This checkpoint was developed as a foundational research asset and has not been validated, audited, or certified for safety-critical or high-consequence environments.
172
+ * It is not designed for direct integration into unmonitored physical control loops, industrial actuation, medical diagnostics, aviation/navigation systems, or autonomous financial execution pipelines. Any deployment of raw or fine-tuned weights in environments where numerical misprediction carries physical, operational, or financial risk should incorporate strict external bounds, deterministic fallback logic, and independent oversight.
173
+
174
+ **5. Apache 2.0 Disclaimer of Warranty & Limitation of Liability**
175
+
176
+ Pascal-15M-v1.0 is released by AtomixLabs under the **Apache License, Version 2.0**. In accordance with Sections 7 and 8 of the license, the model weights and associated files are provided strictly on an **"AS IS" BASIS**, without warranties or conditions of any kind, either express or implied, including, without limitation, any warranties or conditions of title, non-infringement, merchantability, or fitness for a particular purpose.
177
+
178
+ To the maximum extent permitted by applicable law, in no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, shall AtomixLabs, its developers, or its contributors be liable for any direct, indirect, special, incidental, exemplary, or consequential damages of any character arising as a result of the use, fine-tuning, modification, or deployment of this model, including but not limited to damages for loss of goodwill, work stoppage, computer failure, system malfunction, data corruption, or financial loss, even if advised of the possibility of such damages. Users assume sole responsibility for determining the appropriateness of using or redistributing the model and assume all risks associated with its exercise of permissions under the license.
179
+
180
+ </details>
181
+
182
+ ## Citation
183
+
184
+ ```bibtex
185
+ @model{atomixlabs_pascal_15m_v1_2026,
186
+ author = {AtomixLabs},
187
+ title = {Pascal-15M-v1.0: A Compact Foundation Transformer for Mathematical Sequence Induction},
188
+ year = {2026},
189
+ publisher = {Hugging Face},
190
+ license = {Apache-2.0},
191
+ url = {https://huggingface.co/AtomixLabs/Pascal-15M-v1.0}
192
+ }
193
+ ```