| --- |
| metrics: |
| - accuracy |
| - code_eval |
| --- |
| # Model Card for Evaluate360M |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| Evaluate360M is a lightweight large language model optimized for reasoning tasks. It is designed to run efficiently on low-end commercial hardware, such as mobile phones, while maintaining strong performance in logical reasoning and general-purpose applications. |
|
|
| - **Developed by:** [More Information Needed] |
| - **Funded by [optional]:** [More Information Needed] |
| - **Shared by [optional]:** [More Information Needed] |
| - **Model type:** Transformer-based decoder model |
| - **Language(s) (NLP):** English |
| - **License:** [More Information Needed] |
| - **Finetuned from model [optional]:** `HuggingFaceTB/SmolLM2-360M-Instruct` |
|
|
| ### Model Sources |
|
|
| - **Repository:** [More Information Needed] |
| - **Paper [optional]:** [More Information Needed] |
| - **Demo [optional]:** [More Information Needed] |
|
|
| ## Uses |
|
|
| ### Direct Use |
| Evaluate360M is intended for general-purpose reasoning tasks and can be used in applications that require lightweight LLMs, such as: |
| - Mobile-based AI assistants |
| - Low-power embedded systems |
| - Edge computing applications |
|
|
| ### Downstream Use |
| It can be further fine-tuned for specific domains, including code generation, summarization, or dialogue systems. |
|
|
| ### Out-of-Scope Use |
| - Not optimized for handling very large context windows |
| - Not designed for generating high-fidelity creative text, such as poetry or fiction |
|
|
| ## Bias, Risks, and Limitations |
|
|
| ### Limitations |
| - Struggles with handling large context windows. |
| - Not evaluated for potential biases yet. |
|
|
| ### Recommendations |
| Users should be aware of the model’s limitations in context length and should evaluate its performance for their specific use cases. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_name = "evaluate360m" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained(model_name) |
| |
| inputs = tokenizer("What is the capital of France?", return_tensors="pt") |
| outputs = model.generate(**inputs) |
| print(tokenizer.decode(outputs[0])) |
| ``` |
|
|
| ## Training Details |
|
|
| ### Training Data |
| - **Dataset:** `HuggingFaceH4/Bespoke-Stratos-17k` |
| - **Preprocessing:** Token packing enabled (`--packing`), sequence length up to 2048 tokens |
|
|
| ### Training Procedure |
| - **Optimizer & Precision:** |
| - `bf16` mixed precision |
| - `gradient_accumulation_steps = 8` |
| - Gradient checkpointing enabled |
| - **Hyperparameters:** |
| - Learning rate: `2e-5` |
| - Epochs: `3` |
| - Batch size: `4` (per device, both training and evaluation) |
| - **Evaluation & Saving:** |
| - Evaluation every `500` steps |
| - Model checkpoint saved every `1000` steps, keeping a max of `2` checkpoints |
|
|
| ### Compute Infrastructure |
| - **Hardware Used:** A100 GPU |
| - **Training Time:** 6 hours |
|
|
| ## Evaluation |
|
|
| - **Benchmarks:** No evaluation conducted yet. |
| - **Metrics:** Not available yet. |
|
|
| ## Environmental Impact |
|
|
| - **Hardware Type:** A100 GPU |
| - **Hours Used:** 6 hours |
| - **Cloud Provider:** [More Information Needed] |
| - **Compute Region:** [More Information Needed] |
| - **Carbon Emitted:** [More Information Needed] |
|
|
| ## Technical Specifications |
|
|
| ### Model Architecture |
| - Similar to SmolLM2-360M |
| - Inspired by MobileLLM |
| - Uses **Grouped-Query Attention (GQA)** |
| - Prioritizes depth over width |
|
|
| ## Citation [optional] |
|
|
| **BibTeX:** |
| [More Information Needed] |
|
|
| **APA:** |
| [More Information Needed] |
|
|
| ## More Information |
|
|
| [More Information Needed] |
|
|
| ## Model Card Authors [optional] |
|
|
| [More Information Needed] |
|
|
| ## Model Card Contact |
|
|
| [More Information Needed] |
