Update README.md
Browse files
README.md
CHANGED
|
@@ -72,95 +72,4 @@ webxos.netlify.app/MICROD
|
|
| 72 |
</div>
|
| 73 |
|
| 74 |
## Model Description
|
| 75 |
-
|
| 76 |
-
**MICROD v1.0 (micro-distill-grpo-vae)** is a small template model designed to be built upon for custom ground up builds. It is distillated into a
|
| 77 |
-
small set of files the user can use to template their own agents. Designed for educational learning and micro scalling.
|
| 78 |
-
Use **MICROD V1.0 (micro-distill-grpo-vae)** in your own custom projects and train it from the ground up.
|
| 79 |
-
|
| 80 |
-
The model's architecture details further underscore an educational niche: a hidden size of 512, 8 layers, 8 attention heads, a vocabulary of 50,257 tokens,
|
| 81 |
-
and a max sequence length of 1024. Licensed under Apache 2.0, it's openly available for modification, and its small footprint allows quantization,
|
| 82 |
-
making it runnable on modest hardware like CPUs or even browsers via TensorFlow.js integration.
|
| 83 |
-
|
| 84 |
-
## Model Details
|
| 85 |
-
- **Model type**: micro-distill-grpo-vae
|
| 86 |
-
- **Model size**: 42M parameters
|
| 87 |
-
- **Language**: English
|
| 88 |
-
- **License**: Apache 2.0
|
| 89 |
-
|
| 90 |
-
## Training Methodology
|
| 91 |
-
- **GRPO (Group Relative Policy Optimization)**: 8 groups
|
| 92 |
-
- **VAE Filtering**: 32D latent space
|
| 93 |
-
- **KV-Cache Reuse**: 512 cache size
|
| 94 |
-
|
| 95 |
-
## Architecture Details
|
| 96 |
-
- Hidden size: 512
|
| 97 |
-
- Number of layers: 8
|
| 98 |
-
- Attention heads: 8
|
| 99 |
-
- Vocabulary size: 50257
|
| 100 |
-
- Maximum sequence length: 1024
|
| 101 |
-
|
| 102 |
-
## Usage
|
| 103 |
-
|
| 104 |
-
-Model Distillation Training: Simulate GRPO optimization with VAE filtering for small LLMs (42M-345M params).
|
| 105 |
-
|
| 106 |
-
-Policy Experimentation: Test group sizes, KL penalties, cache reuse for RLHF-like training.
|
| 107 |
-
|
| 108 |
-
-VAE Filtering: Apply latent space compression to improve distillation quality.
|
| 109 |
-
|
| 110 |
-
-Sandbox Testing: Execute safe Python code with feedback masking.
|
| 111 |
-
|
| 112 |
-
-Export & Deployment: Generate deployable models for inference in various frameworks.
|
| 113 |
-
|
| 114 |
-
-Offline Usage: PWA supports offline training simulation and exports.
|
| 115 |
-
|
| 116 |
-
## Citation
|
| 117 |
-
|
| 118 |
-
If you use this data in research, please cite:
|
| 119 |
-
|
| 120 |
-
@model{microd_v1_2025,
|
| 121 |
-
title={MICROD_v1},
|
| 122 |
-
author={webXOS]
|
| 123 |
-
year={2025},
|
| 124 |
-
publisher={webXOS},
|
| 125 |
-
url={webxos.netlify.app}
|
| 126 |
-
}
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
### EXAMPLE: Using Transformers
|
| 130 |
-
```python
|
| 131 |
-
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 132 |
-
|
| 133 |
-
model = AutoModelForCausalLM.from_pretrained("micro-distill-grpo-vae")
|
| 134 |
-
tokenizer = AutoTokenizer.from_pretrained("micro-distill-grpo-vae")
|
| 135 |
-
|
| 136 |
-
inputs = tokenizer("Hello, world!", return_tensors="pt")
|
| 137 |
-
outputs = model.generate(**inputs, max_length=50)
|
| 138 |
-
print(tokenizer.decode(outputs[0]))
|
| 139 |
-
```
|
| 140 |
-
|
| 141 |
-
### EXAMPLE: USE CASES
|
| 142 |
-
|
| 143 |
-
**MICROD_v1** may not rival larger models in breadth, but a focus on accessibility and browser-based AI development opens doors for
|
| 144 |
-
innovators balancing all perspectives in the Small Language Model space.
|
| 145 |
-
|
| 146 |
-
1. Prototype without Internet
|
| 147 |
-
2. Offline Simulations in Black Box
|
| 148 |
-
3. Simple Story Generators
|
| 149 |
-
4. Custom Agentic Development
|
| 150 |
-
5. Train on Custom Data
|
| 151 |
-
6. Experiment with max_length
|
| 152 |
-
7. AI agents for custom Games
|
| 153 |
-
8. Educational Fine-Tuning
|
| 154 |
-
9. Prepare Datasets
|
| 155 |
-
10. Fine-tune via GRPO Trainer
|
| 156 |
-
11. Evaluate PY in Sandbox
|
| 157 |
-
12. Create task-specific Variants like Code Tutors
|
| 158 |
-
|
| 159 |
-
### OVERVIEW
|
| 160 |
-
|
| 161 |
-
In terms of applications, small distilled models like **MICROD_v1** align with broader trends in SLMs, which prioritize efficiency, accessibility,
|
| 162 |
-
and specialization over the scale of large language models (LLMs). For example, they can be fine-tuned for targeted tasks such as customer support
|
| 163 |
-
chatbots, where quick responses on edge devices are crucial, or educational tools for teaching natural language processing concepts. In healthcare,
|
| 164 |
-
distilled models might power privacy-focused symptom checkers on mobile apps, avoiding data transmission to cloud servers. Automation and control
|
| 165 |
-
systems benefit from their low latency, as surveyed in research on tiny language models (TLMs), which use techniques like knowledge distillation and
|
| 166 |
-
quantization to enable on-device inference for robotics or IoT devices.
|
|
|
|
| 72 |
</div>
|
| 73 |
|
| 74 |
## Model Description
|
| 75 |
+
UNDER DEVELOPMENT: MAJOR UPDATE UI FIX
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|