luisbrel commited on
Commit
e97584b
·
verified ·
1 Parent(s): ff2bf0a

docs(audit): add G.132/G.142 sample-size disclosure (4 unique prompts x 2 trials) and remove legacy ungrounded draft files

Browse files
LINKEDIN_DRAFT_4B_CAPACITY_CEILING.md DELETED
@@ -1,45 +0,0 @@
1
- # LinkedIn Launch & Research Post: The 4B Parameter Capacity Ceiling
2
-
3
- **Model Checkpoint**: `luisbrel/jimmy-4b-zero-regression`
4
- **Target Platform**: LinkedIn
5
- **Theme**: Empirical testing of the 4B edge parameter ceiling, zero-regression representation surgery, and community invitation
6
- **IP Protocol**: High-level empirical framing · No internal mathematical recipes or proprietary code disclosed
7
-
8
- ---
9
-
10
- ```text
11
- Project Jimmy: Closing out our 4B parameter research on Apple Silicon.
12
-
13
- Can post-training weight surgery enable a compact edge model to reason like a frontier model, or does physical parameter volume impose a hard limit?
14
-
15
- Here is the empirical head-to-head across three distinct tiers:
16
-
17
- 1. Base Gemma 4B:
18
- - GSM8K Math: 90.0%
19
- - HumanEval Code: 90.0%
20
- - Visual Critic: 35/100 (acute 180px gap between feet and pedals)
21
-
22
- 2. Jimmy 4B (Our Best Checkpoint):
23
- - GSM8K Math: 90.0% (0.0% regression)
24
- - HumanEval Code: 90.0% (0.0% regression)
25
- - Visual Critic: 75/100 (+40 pts leap; exact pedal-foot lock, hollow rims, amber pouch)
26
-
27
- 3. Frontier Reference (Gemini Flash):
28
- - GSM8K Math: 95.2%
29
- - HumanEval Code: 92.4%
30
- - Visual Critic: 100/100 (publication-grade multi-scale artistry)
31
-
32
- What this proves:
33
- Post-training surgery successfully repaired localized relational defects with zero anchor regression. It pushed the 4B model to its physical Pareto frontier.
34
-
35
- Why we are parking Jimmy:
36
- A 4B architecture has ~4.3 billion parameters. Holding dozens of simultaneous Bezier curves and volumetric proportions in active memory requires parameter bandwidth that 4B models physically lack.
37
-
38
- To bridge the gap from 75 to 100, surgery must be applied to larger capacity tiers: 12B, 14B, and MoE.
39
-
40
- The Jimmy weights and complete 2-trial scorecard are live on Hugging Face (link in first comment). If you test the checkpoint, please leave a comment on the model card with your findings.
41
-
42
- Next stop: scaling beyond 4B.
43
-
44
- #LocalLLM #AppleSilicon #MachineLearning
45
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -31,7 +31,7 @@ pipeline_tag: text-generation
31
 
32
  **luisbrel/jimmy-4b-zero-regression** is an experimental compact language model developed locally on Apple Silicon M4 Max using the MLX framework.
33
 
34
- This project investigates whether a compact edge architecture can undergo targeted structural and behavioral improvements without the classic "alignment tax"—ensuring that foundational mathematical reasoning, code synthesis, and instruction following remain 100% intact with **zero regression**.
35
 
36
  ---
37
 
@@ -41,7 +41,7 @@ Direct comparison across model tiers—from dense edge baseline to our calibrate
41
 
42
  ![Pelican on a Bicycle Benchmark: Base Gemma 4B vs Jimmy 4B Gen 14 vs Gemini Flash Frontier Reference](pelican_3way_comparison.png)
43
 
44
- | Base Gemma-4 (4B) | Jimmy-4B (Gen 14 Champion) | Gemini Flash (Frontier Reference) |
45
  | :---: | :---: | :---: |
46
  | ![Base Gemma-4](pelican_base_gemma_4b.png) | ![Jimmy-4B](pelican_jimmy_4b_gen14.png) | ![Gemini Flash](pelican_gemini_flash_frontier.png) |
47
  | **Score: 35 / 100** | **Score: 75 / 100** | **Score: 100 / 100** |
@@ -49,44 +49,42 @@ Direct comparison across model tiers—from dense edge baseline to our calibrate
49
 
50
  ### Quantitative Comparison Matrix
51
 
52
- | Capability / Benchmark Suite | Base Gemma-4 (4B) | Jimmy-4B (Gen 14 Champion) | Frontier Reference (Gemini Flash) | Analysis & Delta |
53
  | :--- | :---: | :---: | :---: | :--- |
54
  | **Model Scale & Type** | 4.3B Edge Dense | 4.3B Edge Dense (Calibrated) | Frontier Cloud Scale | Compact edge vs calibrated edge vs frontier |
55
- | **GSM8K Math (Multi-Step)** | 90.0% | **90.0%** | 95.2% | Exact 0.0% anchor preservation maintained |
56
- | **HumanEval Code (Python)** | 90.0% | **90.0%** | 92.4% | Exact 0.0% anchor preservation maintained |
57
- | **IFEval (Strict Constraints)** | 100.0% | **100.0%** | 98.0% | Edge model matches and exceeds frontier adherence |
58
- | **ARC-Challenge (Reasoning)** | 80.0% | **80.0%** | 91.0% | Zero regression on scientific reasoning |
59
- | **Pelican Spatial Contact (25 pts)** | 0 / 25 (180px floating gap) | **25 / 25** (Exact pedal lock) | **25 / 25** (Exact pedal lock) | Calibration resolved foot-to-pedal alignment |
60
- | **Mechanical Wheel Rims (25 pts)** | 10 / 25 (Solid black discs) | **25 / 25** (Hollow rims) | **25 / 25** (Spoked rims) | Calibration corrected mechanical rim structure |
61
- | **Plumage Naturalism (25 pts)** | 15 / 25 (Mud brown/olive) | **25 / 25** (White + amber pouch) | **25 / 25** (Hyper-realistic) | Authentic plumage and amber pouch bound |
62
- | **Facial Anatomy & Eye (25 pts)** | 10 / 25 (Missing pupil) | **0 / 25** (Pupil omitted in plan) | **25 / 25** (Articulate pupil & pouch) | 4B trade-off: attention budget focused on pedals |
63
- | **4D Visual Rigor Total Score** | **35 / 100** | **75 / 100** (+40 pts) | **100 / 100** (Reference) | 4B Pareto limit achieved; 100 requires 12B+ |
64
 
65
  ---
66
 
67
- ## 🏆 Local Head-to-Head Evaluation (100% Ground-Truth 2-Trial Battery)
68
 
69
- Evaluated natively on Apple Silicon Metal cores across two independent trials per suite (140 full autoregressive generation passes). Zero mock simulations, zero synthetic surrogates:
70
 
71
- | Benchmark Suite | Base Gemma-4 (Trial 1) | Base Gemma-4 (Trial 2) | Base Mean | Jimmy-4B (Trial 1) | Jimmy-4B (Trial 2) | Jimmy-4B Mean | Net Delta (Δ) |
72
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
73
- | **GSM8K** (Multi-Step Math) | 100.0% | 80.0% | **90.0%** | 100.0% | 80.0% | **90.0%** | **±0.0%** |
74
- | **MATH-500** (Olympiad Math) | 40.0% | 40.0% | **40.0%** | 40.0% | 40.0% | **40.0%** | **±0.0%** |
75
- | **HumanEval** (Python Code Generation) | 100.0% | 80.0% | **90.0%** | 100.0% | 80.0% | **90.0%** | **±0.0%** |
76
- | **IFEval** (Strict Constraint Adherence) | 100.0% | 100.0% | **100.0%** | 100.0% | 100.0% | **100.0%** | **±0.0%** |
77
- | **ARC-Challenge** (Grade School Science) | 80.0% | 80.0% | **80.0%** | 80.0% | 80.0% | **80.0%** | **±0.0%** |
78
- | **Artificial Analysis v5** (Multi-Quadrant) | 100.0% | 100.0% | **100.0%** | 100.0% | 100.0% | **100.0%** | **±0.0%** |
79
- | **GUI & DOM Grounding** (Agentic Web Actions)| 0.0% | 25.0% | **12.5%** | 0.0% | 25.0% | **12.5%** | **±0.0%** |
80
- | **MACRO AVERAGE** | — | — | **73.2%** | — | — | **73.2%** | **0.0% Regression** |
81
 
82
  ---
83
 
84
- ## 🔬 Key Empirical Finding: The 4B Parameter Capacity Frontier
85
 
86
- In our visual and spatial geometry evaluations (the canonical *Pelican on a Bicycle* vector benchmark), we uncovered a crucial insight regarding compact edge models:
87
 
88
- * **Structural Calibration**:
89
- Base Gemma-4 exhibited an acute spatial phase-binding disconnect—rendering feet 180 pixels in mid-air above bicycle pedals with mud-brown plumage. Through targeted post-training calibration, Jimmy-4B completely closed the 180px gap, achieving exact pedal-foot coordinate union at $(x=380, y=480)$ with natural white plumage and amber throat pouch, while preserving 100% of core reasoning capabilities.
90
  * **The Physical Capacity Ceiling**:
91
  While localized geometric bugs can be resolved without anchor degradation, compact 4B models hit an undeniable information-theoretic limit: holding 50+ simultaneous multi-scale Bezier curves and organic volumetric proportions in memory requires parameter bandwidth and active attention head count that 4B architectures physically do not possess.
92
  * **The Takeaway**:
 
31
 
32
  **luisbrel/jimmy-4b-zero-regression** is an experimental compact language model developed locally on Apple Silicon M4 Max using the MLX framework.
33
 
34
+ This project investigates whether a compact 4B edge architecture can undergo targeted geometric calibration while preserving core reasoning behavior on small-sample probe suites (`n=56` total autoregressive evaluations: `4` questions × `2` trials across 7 suites, `69.64% -> 78.57%`, `+8.93%` macro in `outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`, `M4 Max 128GB MLX`; standalone spatial check in `reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`). Under `ParetoAnchorGate.strict_rsi_gate(min_samples_per_suite=50)` (`G.142`, `reports/PARETO_POWERED_GATE_AUDIT.json` @ `353dfb8`, binomial 95% upper bound `52.71%` at `n=4` vs `5.82%` at `n=50`), this `n=4`/`n=8` pilot is classified **`REJECTED_UNDERPOWERED`**. `[DISPUTED — RE-RUN REQUIRED]` for full-scale zero-regression generalizations beyond this `n=56` pilot.
35
 
36
  ---
37
 
 
41
 
42
  ![Pelican on a Bicycle Benchmark: Base Gemma 4B vs Jimmy 4B Gen 14 vs Gemini Flash Frontier Reference](pelican_3way_comparison.png)
43
 
44
+ | Base Gemma-4 (4B) | `jimmy-4b` (Post-Trained 4B) | Gemini Flash (Frontier Reference — Unverified Summary) |
45
  | :---: | :---: | :---: |
46
  | ![Base Gemma-4](pelican_base_gemma_4b.png) | ![Jimmy-4B](pelican_jimmy_4b_gen14.png) | ![Gemini Flash](pelican_gemini_flash_frontier.png) |
47
  | **Score: 35 / 100** | **Score: 75 / 100** | **Score: 100 / 100** |
 
49
 
50
  ### Quantitative Comparison Matrix
51
 
52
+ | Capability / Benchmark Suite | Base Gemma-4 (4B) | `jimmy-4b` (Post-Trained 4B) | Frontier Reference `[DISPUTED — RE-RUN REQUIRED]` | Analysis & Delta |
53
  | :--- | :---: | :---: | :---: | :--- |
54
  | **Model Scale & Type** | 4.3B Edge Dense | 4.3B Edge Dense (Calibrated) | Frontier Cloud Scale | Compact edge vs calibrated edge vs frontier |
55
+ | **GSM8K Math** (`n=8`, 4q × 2 trials) | `50.0%` | **`75.0%`** | `[DISPUTED — RE-RUN REQUIRED]` (legacy ungrounded Frontier `95.2%` removed) | `+25.0%` (`outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`, `M4 Max 128GB MLX`) |
56
+ | **HumanEval Code** (`n=8`, 4q × 2 trials) | `75.0%` | `75.0%` | `[DISPUTED — RE-RUN REQUIRED]` (legacy ungrounded Frontier `92.4%` removed) | `0.0%` (`outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
57
+ | **IFEval (Strict Constraints)** (`n=8`) | `100.0%` | `100.0%` | `[DISPUTED — RE-RUN REQUIRED]` (legacy ungrounded `98.0%` removed) | `0.0%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
58
+ | **ARC-Challenge** (`n=8`, 4q × 2 trials) | `75.0%` | **`87.5%`** | `[DISPUTED — RE-RUN REQUIRED]` | `+12.5%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`; pelican SVG rigor `55` / `perceptual 80` in `reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`, `M4 Max 128GB MLX`) |
59
+ | **Pelican SVG Showcase** (`n=1` static gen14) | `[DISPUTED — RE-RUN REQUIRED]` | `structural 100 / perceptual 80 / rigor 55` | — | `reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1` (`visual_critiques: ["Solid Disc Wheels"]`, `M4 Max 128GB MLX`) |
60
+
61
+ > **Honesty Note on Legacy Qualitative 25-pt Subscores:** `[DISPUTED — RE-RUN REQUIRED]` earlier qualitative claims (`75/100`, hollow rims, `100%` retention, `95.2%`/`92.4%`/`98.0%` Frontier columns) are quarantined; the tracked JSON (`reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`) explicitly records `visual_rigor_score: 55` and `"Solid Disc Wheels"`. Concept DOI: `10.5281/zenodo.24122803`.
 
 
62
 
63
  ---
64
 
65
+ ## 📊 Local Head-to-Head Evaluation (`n=56` Pilot, 7 Suites × 4 Questions × 2 Trials)
66
 
67
+ Evaluated locally on Apple Silicon M4 Max (`128GB` Unified Memory, MLX) across 7 suites (`4` questions × `2` trials = `8` evaluations per suite, `56` total in `outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876` and `EVALUATION_SUMMARY.md` @ `1194cc5`). **Underpowered sample warning:** `n=8`/suite (`n=56` total) is a small pilot and must not be extrapolated to large-scale generalization:
68
 
69
+ | Benchmark Suite | Base Gemma-4 Mean | `jimmy-4b` Mean | Net Delta (Δ) |
70
+ | :--- | :---: | :---: | :--- |
71
+ | **GSM8K** (4q × 2 trials) | `50.0%` | **`75.0%`** | `+25.0%` (`outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`, `M4 Max 128GB MLX`) |
72
+ | **MATH-500** (4q × 2 trials) | `25.0%` | **`37.5%`** | `+12.5%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
73
+ | **HumanEval** (4q × 2 trials) | `75.0%` | `75.0%` | `0.0%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
74
+ | **IFEval** (4q × 2 trials) | `100.0%` | `100.0%` | `0.0%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
75
+ | **ARC-Challenge** (4q × 2 trials) | `75.0%` | **`87.5%`** | `+12.5%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
76
+ | **Artificial Analysis v5** (4q × 2 trials) | `100.0%` | `100.0%` | `0.0%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
77
+ | **GUI & DOM Grounding** (4q × 2 trials) | `62.5%` | **`75.0%`** | `+12.5%` (`HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876`) |
78
+ | **MACRO AVERAGE (`n=56` underpowered pilot)** | `69.64%` | **`78.57%`** | **`+8.93%`** (7-suite macro in `outputs/jimmy_4b_zero_regression/02_findings_and_evaluations/HEAD_TO_HEAD_SCORECARD.json` @ `1e1b876` & `EVALUATION_SUMMARY.md` @ `1194cc5`; see `reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`, `M4 Max 128GB MLX`; legacy `73.2%` table is `[DISPUTED — RE-RUN REQUIRED]`) |
79
 
80
  ---
81
 
82
+ ## 🔬 Key Empirical Observations & Capacity Constraints
83
 
84
+ In our vector-geometry case study (`reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`, `M4 Max 128GB MLX`), compact 4B models show clear trade-offs:
85
 
86
+ * **Audited Pelican Showcase (`jimmy_4b_progressive_gen14`)**:
87
+ Static single-prompt generation completed in `16.198 s` at `113.1 tok/s` (`1832` completion tokens, `n=1` static check vs base Gemma-4 qualitative baselines) with `structural_score: 100`, `perceptual_score: 80`, and `visual_rigor_score: 55` (critique recorded: `"Solid Disc Wheels"`, in `reports/pelican_benchmark_showcase/jimmy_4b_progressive_gen14.json` @ `ba206f1`, `M4 Max 128GB MLX`). `[DISPUTED — RE-RUN REQUIRED]` unverified prose claiming exact `(380, 480)` contact lock or `100%` core reasoning preservation across larger suites.
88
  * **The Physical Capacity Ceiling**:
89
  While localized geometric bugs can be resolved without anchor degradation, compact 4B models hit an undeniable information-theoretic limit: holding 50+ simultaneous multi-scale Bezier curves and organic volumetric proportions in memory requires parameter bandwidth and active attention head count that 4B architectures physically do not possess.
90
  * **The Takeaway**:
linkedin_post_clean.txt DELETED
@@ -1,34 +0,0 @@
1
- Project Jimmy: Closing out our 4B parameter research on Apple Silicon.
2
-
3
- Can targeted post-training calibration enable a compact edge model to reason like a frontier model, or does physical parameter volume impose a hard limit?
4
-
5
- Here is the empirical head-to-head across three distinct tiers:
6
-
7
- 1. Base Gemma 4B:
8
- - GSM8K Math: 90.0%
9
- - HumanEval Code: 90.0%
10
- - Visual Critic: 35/100 (acute 180px gap between feet and pedals)
11
-
12
- 2. Jimmy 4B (Our Best Checkpoint):
13
- - GSM8K Math: 90.0% (0.0% regression)
14
- - HumanEval Code: 90.0% (0.0% regression)
15
- - Visual Critic: 75/100 (+40 pts leap; exact pedal-foot lock, hollow rims, amber pouch)
16
-
17
- 3. Frontier Reference (Gemini Flash):
18
- - GSM8K Math: 95.2%
19
- - HumanEval Code: 92.4%
20
- - Visual Critic: 100/100 (publication-grade multi-scale artistry)
21
-
22
- What this proves:
23
- Targeted post-training calibration successfully resolved localized relational defects with zero anchor regression. It pushed the 4B model to its physical Pareto frontier.
24
-
25
- Why we are parking Jimmy:
26
- A 4B architecture has ~4.3 billion parameters. Holding dozens of simultaneous Bezier curves and volumetric proportions in active memory requires parameter bandwidth that 4B models physically lack.
27
-
28
- To bridge the gap from 75 to 100, these methods must be applied to larger capacity tiers: 12B, 14B, and MoE.
29
-
30
- The Jimmy weights and complete 2-trial scorecard are live on Hugging Face (link in first comment). If you test the checkpoint, please leave a comment on the model card with your findings.
31
-
32
- Next stop: scaling beyond 4B.
33
-
34
- #LocalLLM #AppleSilicon #MachineLearning
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
linkedin_post_visual.txt DELETED
@@ -1,25 +0,0 @@
1
- Can a compact 4B model learn complex spatial relationships, or is there an immutable physical ceiling on 4 billion parameters?
2
-
3
- Look at the image below. When prompted to draw a pelican riding a bicycle in SVG, the baseline 4B model suffered an acute relational failure: the bird's feet floated 180 pixels in mid-air above the pedals.
4
-
5
- I spent the past week testing whether targeted post-training calibration on Apple Silicon could repair this disconnect.
6
-
7
- The calibrated checkpoint (Jimmy-4B) fixed the relational binding:
8
- - Feet locked directly onto the pedals.
9
- - Solid black discs became hollow mechanical rims.
10
- - Natural plumage and amber pouch colors aligned.
11
-
12
- Crucially, core math and code capabilities stayed 100% stable with zero regression.
13
-
14
- Then we compared it against a frontier reference (Gemini Flash).
15
-
16
- Here is the honest finding:
17
- Targeted calibration can repair structural defects and push a small model to its physical Pareto frontier. But it cannot manufacture capacity that does not physically exist.
18
-
19
- A 4B model physically runs out of parameter memory bandwidth. Tracking dozens of simultaneous Bezier curves and volumetric proportions in active attention requires parameter volume that only 12B to 14B+ models possess.
20
-
21
- To reach publication-grade artistry, the next step is taking these methods beyond 4B.
22
-
23
- The weights and evaluation scorecard are open on Hugging Face (link in first comment). If you test the model, please leave a comment on the card with your findings.
24
-
25
- #LocalLLM #AppleSilicon #MachineLearning