Fix broken Markdown table in Preliminary Experimental Results
#1
by BananaMindBot - opened
README.md
CHANGED
|
@@ -54,6 +54,7 @@ The latest experiments compare CECP against a conventional FP32 baseline using t
|
|
| 54 |
- Hardware: NVIDIA Tesla T4
|
| 55 |
|
| 56 |
| Model | Mean Validation Perplexity ↓ |
|
|
|
|
| 57 |
| Dense FP32 | 18.146 ± 0.298 |
|
| 58 |
| **CECP-256** | **17.986 ± 0.082** |
|
| 59 |
|
|
@@ -71,20 +72,20 @@ Its compressed projection representation can achieve a nominal one bit per weigh
|
|
| 71 |
|
| 72 |
The effective compression ratio depends on model architecture, layer dimensions, and configuration.
|
| 73 |
|
| 74 |
-
Early experiments have demonstrated substantial projection weight storage reductions.
|
| 75 |
|
| 76 |
---
|
| 77 |
|
| 78 |
## Potential Scaling
|
| 79 |
|
| 80 |
-
One of the central research questions surrounding CECP is how compression aware learning
|
| 81 |
|
| 82 |
Early experiments suggest that CECP can maintain competitive language model performance across the small model sizes evaluated so far.
|
| 83 |
|
| 84 |
Future research will investigate whether these properties extend to:
|
| 85 |
|
| 86 |
- 100M–500M parameter language models
|
| 87 |
-
-
|
| 88 |
- Larger and more diverse training datasets
|
| 89 |
- Longer context applications
|
| 90 |
- Local deployment on memory constrained consumer hardware
|
|
@@ -95,13 +96,13 @@ However, these scaling benefits remain hypotheses until validated through larger
|
|
| 95 |
|
| 96 |
## Research Status
|
| 97 |
|
| 98 |
-
**Status:
|
| 99 |
|
| 100 |
-
CECP has demonstrated functional compression aware language model training and promising early validation
|
| 101 |
|
| 102 |
Further work is required to establish its performance against established low bit training and quantization methods at larger scales.
|
| 103 |
|
| 104 |
-
Current research priorities include improving training efficiency, evaluating larger models,
|
| 105 |
|
| 106 |
The complete implementation and selected technical details are not publicly available at this stage.
|
| 107 |
|
|
@@ -117,6 +118,6 @@ The long term vision is to explore whether language models can achieve a better
|
|
| 117 |
|
| 118 |
*CECP — Co-Evolving Compositional Pre-Training*
|
| 119 |
|
| 120 |
-
*
|
| 121 |
|
| 122 |
-
*Preliminary results are not evidence of
|
|
|
|
| 54 |
- Hardware: NVIDIA Tesla T4
|
| 55 |
|
| 56 |
| Model | Mean Validation Perplexity ↓ |
|
| 57 |
+
|---|---|
|
| 58 |
| Dense FP32 | 18.146 ± 0.298 |
|
| 59 |
| **CECP-256** | **17.986 ± 0.082** |
|
| 60 |
|
|
|
|
| 72 |
|
| 73 |
The effective compression ratio depends on model architecture, layer dimensions, and configuration.
|
| 74 |
|
| 75 |
+
Early experiments have demonstrated substantial projection weight storage reductions. Large models may benefit from reduced relative representation overhead, although this remains an area of ongoing evaluation.
|
| 76 |
|
| 77 |
---
|
| 78 |
|
| 79 |
## Potential Scaling
|
| 80 |
|
| 81 |
+
One of the central research questions surrounding CECP is how compression aware learning behaviors as model size increases.
|
| 82 |
|
| 83 |
Early experiments suggest that CECP can maintain competitive language model performance across the small model sizes evaluated so far.
|
| 84 |
|
| 85 |
Future research will investigate whether these properties extend to:
|
| 86 |
|
| 87 |
- 100M–500M parameter language models
|
| 88 |
+
- Billon parameter architectures
|
| 89 |
- Larger and more diverse training datasets
|
| 90 |
- Longer context applications
|
| 91 |
- Local deployment on memory constrained consumer hardware
|
|
|
|
| 96 |
|
| 97 |
## Research Status
|
| 98 |
|
| 99 |
+
**Status: Experiment — Active Development**
|
| 100 |
|
| 101 |
+
CECP has demonstrated functional compression aware language model training and promising early validation perplecity results.
|
| 102 |
|
| 103 |
Further work is required to establish its performance against established low bit training and quantization methods at larger scales.
|
| 104 |
|
| 105 |
+
Current research priorities include improving training efficiency, evaluating larger models, verifing end to end deployment memory requirements, and strengthening comparative benckmarks.
|
| 106 |
|
| 107 |
The complete implementation and selected technical details are not publicly available at this stage.
|
| 108 |
|
|
|
|
| 118 |
|
| 119 |
*CECP — Co-Evolving Compositional Pre-Training*
|
| 120 |
|
| 121 |
+
*Indepedent experimental AI architecture research.*
|
| 122 |
|
| 123 |
+
*Preliminary results are not evidence of superiorty over production scale language models or established quantization methods.*
|