webxos commited on
Commit
1e61ce5
·
verified ·
1 Parent(s): 92680c6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -92
README.md CHANGED
@@ -72,95 +72,4 @@ webxos.netlify.app/MICROD
72
  </div>
73
 
74
  ## Model Description
75
- This is a distilled language model trained using Group Relative Policy Optimization (GRPO) with VAE filtering.
76
- **MICROD v1.0 (micro-distill-grpo-vae)** is a small template model designed to be built upon for custom ground up builds. It is distillated into a
77
- small set of files the user can use to template their own agents. Designed for educational learning and micro scalling.
78
- Use **MICROD V1.0 (micro-distill-grpo-vae)** in your own custom projects and train it from the ground up.
79
-
80
- The model's architecture details further underscore an educational niche: a hidden size of 512, 8 layers, 8 attention heads, a vocabulary of 50,257 tokens,
81
- and a max sequence length of 1024. Licensed under Apache 2.0, it's openly available for modification, and its small footprint allows quantization,
82
- making it runnable on modest hardware like CPUs or even browsers via TensorFlow.js integration.
83
-
84
- ## Model Details
85
- - **Model type**: micro-distill-grpo-vae
86
- - **Model size**: 42M parameters
87
- - **Language**: English
88
- - **License**: Apache 2.0
89
-
90
- ## Training Methodology
91
- - **GRPO (Group Relative Policy Optimization)**: 8 groups
92
- - **VAE Filtering**: 32D latent space
93
- - **KV-Cache Reuse**: 512 cache size
94
-
95
- ## Architecture Details
96
- - Hidden size: 512
97
- - Number of layers: 8
98
- - Attention heads: 8
99
- - Vocabulary size: 50257
100
- - Maximum sequence length: 1024
101
-
102
- ## Usage
103
-
104
- -Model Distillation Training: Simulate GRPO optimization with VAE filtering for small LLMs (42M-345M params).
105
-
106
- -Policy Experimentation: Test group sizes, KL penalties, cache reuse for RLHF-like training.
107
-
108
- -VAE Filtering: Apply latent space compression to improve distillation quality.
109
-
110
- -Sandbox Testing: Execute safe Python code with feedback masking.
111
-
112
- -Export & Deployment: Generate deployable models for inference in various frameworks.
113
-
114
- -Offline Usage: PWA supports offline training simulation and exports.
115
-
116
- ## Citation
117
-
118
- If you use this data in research, please cite:
119
-
120
- @model{microd_v1_2025,
121
- title={MICROD_v1},
122
- author={webXOS]
123
- year={2025},
124
- publisher={webXOS},
125
- url={webxos.netlify.app}
126
- }
127
-
128
-
129
- ### EXAMPLE: Using Transformers
130
- ```python
131
- from transformers import AutoModelForCausalLM, AutoTokenizer
132
-
133
- model = AutoModelForCausalLM.from_pretrained("micro-distill-grpo-vae")
134
- tokenizer = AutoTokenizer.from_pretrained("micro-distill-grpo-vae")
135
-
136
- inputs = tokenizer("Hello, world!", return_tensors="pt")
137
- outputs = model.generate(**inputs, max_length=50)
138
- print(tokenizer.decode(outputs[0]))
139
- ```
140
-
141
- ### EXAMPLE: USE CASES
142
-
143
- **MICROD_v1** may not rival larger models in breadth, but a focus on accessibility and browser-based AI development opens doors for
144
- innovators balancing all perspectives in the Small Language Model space.
145
-
146
- 1. Prototype without Internet
147
- 2. Offline Simulations in Black Box
148
- 3. Simple Story Generators
149
- 4. Custom Agentic Development
150
- 5. Train on Custom Data
151
- 6. Experiment with max_length
152
- 7. AI agents for custom Games
153
- 8. Educational Fine-Tuning
154
- 9. Prepare Datasets
155
- 10. Fine-tune via GRPO Trainer
156
- 11. Evaluate PY in Sandbox
157
- 12. Create task-specific Variants like Code Tutors
158
-
159
- ### OVERVIEW
160
-
161
- In terms of applications, small distilled models like **MICROD_v1** align with broader trends in SLMs, which prioritize efficiency, accessibility,
162
- and specialization over the scale of large language models (LLMs). For example, they can be fine-tuned for targeted tasks such as customer support
163
- chatbots, where quick responses on edge devices are crucial, or educational tools for teaching natural language processing concepts. In healthcare,
164
- distilled models might power privacy-focused symptom checkers on mobile apps, avoiding data transmission to cloud servers. Automation and control
165
- systems benefit from their low latency, as surveyed in research on tiny language models (TLMs), which use techniques like knowledge distillation and
166
- quantization to enable on-device inference for robotics or IoT devices.
 
72
  </div>
73
 
74
  ## Model Description
75
+ UNDER DEVELOPMENT: MAJOR UPDATE UI FIX