Update README.md

README.md

@@ -1,13 +1,109 @@
-CECP - Co-Evolving Compositional Pre-Training
+# CECP — Co-Evolving Compositional Pre-Training
 
-**Experimental Compression-Aware Language Model Training Architecture**
+### Experimental Low Memory Language Model Training Architecture
 
-CECP is an experimental approach to training language models that adapt to highly compressed weight representations during learning.
+**CECP (Co-Evolving Compositional Pre-Training)** is an experimental AI training architecture focused on reducing the memory and storage requirements of large language models while preserving as much of their original performance as possible.
 
-The research aims to reduce deployment memory requirements while preserving model performance.
+Unlike conventional approaches that train a model at higher precision and compress it afterward, CECP explores integrating compression directly into the learning process.
 
-Research status: Active development
+The central idea is simple: **rather than forcing an already trained model into a smaller representation, allow the model to adapt to a compact representation as it learns.**
 
-Latest experiment: 52.8M parameters, three training seeds, and validation perplexity comparable to a dense FP32 baseline.
+This approach aims to minimize performance degradation associated with aggressive model compression, potentially enabling capable AI models to operate on hardware with significantly lower memory requirements.
 
-The full implementation is currently private.
+CECP is currently under active research and development.
+
+---
+
+## Research Objectives
+
+CECP is designed around three primary goals:
+
+- **Extreme model compression:** Reduce model storage requirements without proportionally sacrificing language model performance.
+- **Compression aware learning:** Allow neural networks to adapt to compact representations throughout training.
+- **Accessible local AI:** Make increasingly capable language models practical on consumer hardware with limited RAM and VRAM.
+
+Rather than focusing exclusively on the smallest possible model, CECP investigates the relationship between model capacity, compression, and retained performance.
+
+## Preliminary Experimental Results
+
+CECP has been evaluated on several experimental language model configurations, including models with approximately 21 million and 53 million parameters.
+
+The latest experiments compare CECP against a conventional FP32 baseline using the TinyStories dataset.
+
+### 53M Parameter Language Model
+
+**Experimental configuration:**
+- Model size: approximately 52.8 million parameters
+- Dataset: TinyStories
+- Training duration: 2,500 optimizer steps
+- Evaluation: validation perplexity
+- Independent training seeds: 3
+- Hardware: NVIDIA Tesla T4
+
+| Model | Mean Validation Perplexity ↓ |
+| Dense FP32 | 18.146 ± 0.298 |
+| **CECP-256** | **17.986 ± 0.082** |
+
+In these experiments, CECP achieved validation perplexity comparable to the conventional FP32 baseline despite operating with aggressively compressed weight representations.
+
+The small difference between the two models should be interpreted as approximate performance parity rather than evidence of general superiority.
+
+CECP currently requires more training time than conventional dense training. Improving training efficiency remains an active development priority.
+
+### Compression Efficiency
+
+CECP's weight representation is designed to support extremely low bit storage.
+
+Its compressed projection representation can achieve a nominal one bit per weight index payload, with additional storage required for supporting representation data.
+
+The effective compression ratio depends on model architecture, layer dimensions, and configuration.
+
+Early experiments have demonstrated substantial projection weight storage reductions. Larger models may benefit from reduced relative representation overhead, although this remains an area of ongoing evaluation.
+
+---
+
+## Potential Scaling
+
+One of the central research questions surrounding CECP is how compression aware learning behaves as model size increases.
+
+Early experiments suggest that CECP can maintain competitive language model performance across the small model sizes evaluated so far.
+
+Future research will investigate whether these properties extend to:
+
+- 100M–500M parameter language models
+- Billion parameter architectures
+- Larger and more diverse training datasets
+- Longer context applications
+- Local deployment on memory constrained consumer hardware
+
+As model capacity increases, CECP may offer opportunities to improve the balance between model size and retained capabilities.
+
+However, these scaling benefits remain hypotheses until validated through larger experiments.
+
+## Research Status
+
+**Status: Experimental — Active Development**
+
+CECP has demonstrated functional compression aware language model training and promising early validation perplexity results.
+
+Further work is required to establish its performance against established low bit training and quantization methods at larger scales.
+
+Current research priorities include improving training efficiency, evaluating larger models, verifying end to end deployment memory requirements, and strengthening comparative benchmarks.
+
+The complete implementation and selected technical details are not publicly available at this stage.
+
+## Project Vision
+
+CECP is being developed with a broader objective: making capable AI systems more accessible without requiring increasingly expensive hardware.
+
+The long term vision is to explore whether language models can achieve a better balance between intelligence, memory consumption, and deployment efficiency by adapting to compression throughout training.
+
+**The objective isn't simply to make AI models smaller. It's to make them learn effectively while being smaller.**
+
+---
+
+*CECP — Co-Evolving Compositional Pre-Training*
+
+*Independent experimental AI architecture research.*
+
+*Preliminary results are not evidence of superiority over production scale language models or established quantization methods.*