AEGIS-Mini-V1-LoRA

Aegis-mini-1.0 is a small language model for electronics, drones, robotics, math, and physics. This repository holds the LoRA adapter of the model. The adapter works on top of the base model Qwen3-4B. A GGUF file for phones and laptops is in a different repository: Xerxes-28/AEGIS-Mini-V1-GGUF.

---

Table of contents

1. Summary
2. Model details
3. Intended use
4. Safety information
5. How to use the adapter
6. Prompt format
7. Generation settings
8. Training data
9. Training procedure
10. Results
11. Limitations
12. Compute used
13. Files in this repository
14. License
15. Citations
16. Acknowledgements
17. Contact

---

1. Summary

Aegis-mini-1.0 is a fine-tuned version of Qwen3-4B. The author trained it on questions and answers about electronics, drones, robotics, math, and physics. Most of the training data is theory. The model is small. It can run on a phone after conversion to GGUF. The author did not run public benchmarks. Read Section 10 and Section 11 before you use the model.

2. Model details

Item	Value
Model name	Aegis-mini-1.0
Repository	Xerxes-28/AEGIS-Mini-V1-LoRA
Developer	Xerxes-28
Model type	LoRA adapter (PEFT) for a causal language model
Base model	unsloth/Qwen3-4B-unsloth-bnb-4bit (4-bit copy of Qwen3-4B)
Language	English
Task	Text generation (chat)
Training method	Supervised fine-tuning (SFT) with QLoRA
Trainable parameters	33,030,144 (0.81% of 4,055,498,240)
Training context length	2,048 tokens
Training date	2026-10-31
License	Apache-2.0 (same as the base model)
GGUF version	Xerxes-28/AEGIS-Mini-V1-GGUF

3. Intended use

3.1 Use cases

You can use the model for these tasks:

• Explain a concept in electronics, for example Ohm's law, filters, or transistor bias.
• Explain a concept in drones and robotics, for example PID control or sensor fusion.
• Do a step-by-step calculation in math or physics.
• Give a first draft of an answer that you then check.
• Study and learn. The model is a good partner for practice.

3.2 Uses that are not in scope

Do not use the model for these tasks:

• Safety-critical decisions of any kind.
• Final design values for a real circuit, battery pack, or aircraft.
• Medical, legal, or financial advice.
• Exact data from a datasheet, for example register addresses or pin numbers.

4. Safety information

WARNING DO NOT USE THE MODEL OUTPUT AS THE ONLY SOURCE FOR A SAFETY CALCULATION. The model can give a wrong number with a confident tone. A wrong number can cause injury or fire.

WARNING DO NOT TOUCH MAINS VOLTAGE OR HIGH-VOLTAGE CIRCUITS BECAUSE THE MODEL TOLD YOU IT IS SAFE. Electrical shock can cause injury or death. Ask a qualified person.

CAUTION Make sure that you check each value in the datasheet of the part. A wrong value can cause damage to the part, the battery, or the drone.

CAUTION Do not fly a drone with a setting that the model gave you before you test it in a safe place. A wrong control setting can cause a crash.

5. How to use the adapter

You need the adapter and the base model. There are three ways to load them.

5.1 Method A: Unsloth (fastest)

Unsloth loads the base model and the adapter together.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Xerxes-28/AEGIS-Mini-V1-LoRA",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Do not wrap this model again with PeftModel. Unsloth reads adapter_config.json and loads the adapter by itself.

5.2 Method B: Transformers and PEFT

Use this method when you do not want Unsloth. You need transformers, peft, accelerate, and bitsandbytes.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "unsloth/Qwen3-4B-unsloth-bnb-4bit"   # same base that the author used for training
adapter_id = "Xerxes-28/AEGIS-Mini-V1-LoRA"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

5.3 Run one answer

SYSTEM_PROMPT = (
    "You are Aegis-mini-1.0, a compact AI assistant for electronics, drones, robotics, "
    "math and physics. You are strongest at theory and calculations. Be concise: state the "
    "key formula, show the steps with units, then give the result. If you are unsure of an "
    "exact value (register address, pin number, part spec), say so and tell the user to "
    "check the datasheet instead of guessing. For topics outside your focus, say so briefly."
)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "A 5 V supply drives a red LED (2.0 V, 10 mA) through a resistor. Find the resistor value."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.3,
        top_p=0.8,
        top_k=20,
        min_p=0.0,
        repetition_penalty=1.05,
    )

new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

5.4 Merge the adapter into the base model (optional)

Merge the adapter when you want one set of weights, for example to make a GGUF file.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Xerxes-28/AEGIS-Mini-V1-LoRA",
    max_seq_length=2048,
    load_in_4bit=True,
)
model.save_pretrained_merged("aegis-mini-merged", tokenizer, save_method="merged_16bit")

The base weights are in 4-bit form during training. The merge step changes them back to 16-bit numbers. This step can add small rounding errors.

6. Prompt format

The model uses the chat template of Qwen3. The template uses the ChatML structure.

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant

Call tokenizer.apply_chat_template to build this text. Do not type the tags by hand.

6.1 Recommended system prompt

You are Aegis-mini-1.0, a compact AI assistant for electronics, drones, robotics, math and physics. You are strongest at theory and calculations. Be concise: state the key formula, show the steps with units, then give the result. If you are unsure of an exact value (register address, pin number, part spec), say so and tell the user to check the datasheet instead of guessing. For topics outside your focus, say so briefly.

Keep the system prompt short. A long system prompt uses context and makes a phone slow.

7. Generation settings

Setting	Recommended value	Remark
Temperature	0.3 to 0.7	Use 0.3 for calculations and code. Use 0.7 for explanations.
Top-k	20
Top-p	0.8
Min-p	0
Repetition penalty	1.0 to 1.05	Do not go above 1.1.
Maximum new tokens	512 to 1024	Use a higher value for long derivations.
Context length	2,048 to 4,096	The author trained with 2,048 tokens.

CAUTION Do not set a no-repeat n-gram size. Code and equations repeat groups of tokens on purpose. This setting breaks them.

8. Training data

8.1 Sources

| Source | Rows available | Rows used | License | Remark | |---|---|---|---| | Theory dataset (author's own) | 2,667 | 2,667 | Not published | Electronics and embedded-systems questions and answers, formula-to-code tasks, firmware and drone questions. About 74% of the rows are general electronics and embedded-systems material. About 25% of the answers contain code. | | HuggingFaceH4/MATH-500 | 500 | 100 (random sample) | MIT | Competition math problems with worked solutions from the MATH benchmark. | | mrohith29/high-school-physics | 400 | 100 (random sample) | MIT | High-school physics exercises. |

The corpus has 2,867 examples (2,667 + 100 + 100). The Theory dataset is not published at this time. Part of the training data was created by A. Dinesh, a member of the AEGIS V2 team (see Acknowledgements).

8.2 Data preparation

The author did these steps in order:

1. Convert each row into chat messages with a user role and an assistant role.
2. Use the system message of each example, or one common system prompt when that setting was used.
3. Apply the chat template of Qwen3 to make one text for each example.
4. Count the tokens of each example.
5. Remove each example that has more than 2,048 tokens.
6. Shuffle the data with seed 42.
7. Hold out 100 examples for validation.

The result is 2,765 training examples and 100 validation examples.

8.3 Loss mask

The training loss counts only the tokens of the answer. The tokens of the question do not add to the loss. The author used train_on_responses_only from Unsloth for this. The markers are <|im_start|>user and <|im_start|>assistant.

9. Training procedure

9.1 Hardware and software

Item	Value
GPU	NVIDIA Tesla T4 (14.6 GB), Google Colab free tier
Number of GPUs	1
Python	3.13
PyTorch	2.11.0 (CUDA 13.0 build)
Transformers	5.17.0
TRL	1.15.0
PEFT	0.21.2
bitsandbytes	0.50.2
Accelerate	1.15.0
Datasets	4.8.5
Unsloth and unsloth_zoo	2026.10.11
Experiment tracking	Weights & Biases 0.28.1

9.2 LoRA settings

Setting	Value
Rank (r)	16
Alpha	32
Dropout	0
Bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient checkpointing	Unsloth method
Random state	3407

9.3 Training settings

Setting	Value
Epochs	3
Total optimizer steps	519
Batch size on each device	2
Gradient accumulation steps	8
Effective batch size	16
Peak learning rate	2e-4
Learning rate schedule	Cosine, to near zero at the end
Warmup	20 steps
Weight decay	0.01
Optimizer	AdamW 8-bit
Precision	Mixed precision, fp16 (the T4 does not support bf16)
Maximum sequence length	2,048 tokens
Seed	3407
Validation	Every 50 steps
Checkpoint	Every 100 steps (the last 2 are kept)

10. Results

10.1 Final numbers

Metric	Value
Final validation loss	1.1404
Perplexity from the final validation loss	about 3.13 (2.88 at the lowest point)
Training loss (average of the whole run)	0.8454
Lowest validation loss	1.0583 (step 350, epoch 2)
Training loss (last logged value, step 510)	0.5838
Final gradient norm	0.848
Final learning rate	1.98e-7
Epochs finished	3
Steps finished	519
Training time	4,435 seconds (about 1 hour 14 minutes)
Training speed	1.87 samples each second, 0.117 steps each second
Total floating-point operations	4.37 x 10^16
Validation speed	6.07 samples each second

10.2 Loss during training

The Trainer measured the validation loss every 50 steps on 100 held-out examples. The training loss in the table is the value that the Trainer logged at the same step.

Step	Epoch	Training loss	Validation loss	Validation perplexity
50	0.29	1.1866	1.2763	3.58
100	0.58	1.0245	1.1578	3.18
150	0.87	0.9912	1.1150	3.05
200	1.16	0.7534	1.0917	2.98
250	1.45	0.8667	1.0781	2.94
300	1.73	0.7324	1.0733	2.92
350	2.02	0.6131	1.0583	2.88
400	2.31	0.4966	1.1381	3.12
450	2.60	0.5201	1.1466	3.15
500	2.89	0.5285	1.1403	3.13
End (519)	3.00	0.5838 (step 510)	1.1404	3.13

10.3 How to read these numbers

The validation loss went down from 1.2763 at step 50 to 1.0583 at step 350. This is a fall of 17.1%. Step 350 is at the end of epoch 2. An epoch has 173 steps. After step 350 the validation loss went up. It was 1.1381 at step 400 and 1.1404 at the end. In the same period the training loss fell from 0.61 to a range of 0.50 to 0.53. The gap between the two losses grew from 0.45 at step 350 to 0.64 at step 400. This is a clear sign of overfitting in epoch 3. The model fits the training examples too much.

The final model is the model at step 519. Its validation loss is 0.0820 (7.7%) higher than the lowest value. The author did not keep the checkpoint at step 350, because the author saved a checkpoint every 100 steps. A new run with 2 epochs can give a better model.

This card lists only the values that the author recorded. The author did not run public benchmarks, such as GSM8K or MMLU. Do not use MATH-500 to test this model. The training data holds 100 of the 500 problems of MATH-500. Loss is not the same as accuracy. Do not use these numbers as proof of correct answers.

11. Limitations

• Size. The model has 4 billion parameters. It makes more errors than a large model.
• Wrong numbers. The model can make mistakes in arithmetic, in units, and in long derivations.
• Made-up facts. The model can invent exact values, such as register addresses, pin numbers, and part limits. Check each value in the datasheet.
• Theory focus. About three quarters of the training data is general electronics and embedded-systems material, and about a quarter of the answers contain code. The model is weaker for long programs and for real hardware settings.
• No code tests. The author did not compile or run the code that the model writes.
• Language. The model is trained on English text. Other languages can give poor results.
• Reasoning. The training data does not teach long reasoning in thinking mode. Expect short and direct answers.
• Knowledge. The knowledge of the model comes from the base model and the training data. It can be old.
• Safety tuning. The author did not add new safety training. The model keeps the behavior of the base model.
• Data bias. The data is a choice of the author. The topics are not complete. The physics data is at high-school level. The MATH-500 data is at competition level.
• Overfitting. The final model comes from epoch 3. The validation loss was lower at step 350 (epoch 2). See Section 10.3.

12. Compute used

Item	Value
Hardware	1 NVIDIA Tesla T4
Training time	About 1 hour 14 minutes
Platform	Google Colab free tier

13. Files in this repository

File	Use
adapter_model.safetensors	Weights of the LoRA adapter
adapter_config.json	Settings of the adapter and the name of the base model
tokenizer.json, tokenizer_config.json, and related files	Tokenizer and chat template
README.md	This model card

14. License

The adapter has the license Apache-2.0. The base model Qwen3-4B has the license Apache-2.0. The datasets MATH-500 and high-school-physics have the license MIT. You must follow the terms of each license. Keep the license notices when you share the model.

15. Citations

15.1 How to cite this model

@misc{aegis_mini_1_0_lora_2026,
  title        = {Aegis-mini-1.0 (AEGIS-Mini-V1-LoRA): A LoRA Adapter of Qwen3-4B for Electronics, Drones, Robotics, Math, and Physics},
  author       = {Xerxes-28},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Xerxes-28/AEGIS-Mini-V1-LoRA}},
  note         = {Hugging Face model repository}
}

15.2 Base model

@article{yang2025qwen3,
  title   = {Qwen3 Technical Report},
  author  = {{Qwen Team}},
  journal = {arXiv preprint arXiv:2505.09388},
  year    = {2025},
  url     = {https://arxiv.org/abs/2505.09388}
}

15.3 Datasets

@inproceedings{hendrycks2021math,
  title     = {Measuring Mathematical Problem Solving With the {MATH} Dataset},
  author    = {Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob},
  booktitle = {Thirty-fifth Conference on Neural Information Processing Systems, Datasets and Benchmarks Track},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.03874}
}

@inproceedings{lightman2024lets,
  title     = {Let's Verify Step by Step},
  author    = {Lightman, Hunter and Kosaraju, Vineet and Burda, Yura and Edwards, Harri and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl},
  booktitle = {The Twelfth International Conference on Learning Representations},
  year      = {2024},
  url       = {https://arxiv.org/abs/2305.20050}
}

@misc{h4_math500,
  title        = {MATH-500},
  author       = {{Hugging Face H4}},
  howpublished = {\url{https://huggingface.co/datasets/HuggingFaceH4/MATH-500}},
  note         = {Hugging Face dataset. A 500-problem subset of the MATH test split, introduced in Lightman et al. (2024).}
}

@misc{mrohith29_highschoolphysics,
  title        = {high-school-physics},
  author       = {mrohith29},
  howpublished = {\url{https://huggingface.co/datasets/mrohith29/high-school-physics}},
  note         = {Hugging Face dataset, MIT license}
}

15.4 Methods

@inproceedings{hu2022lora,
  title     = {{LoRA}: Low-Rank Adaptation of Large Language Models},
  author    = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  booktitle = {International Conference on Learning Representations},
  year      = {2022},
  url       = {https://arxiv.org/abs/2106.09685}
}

@inproceedings{dettmers2023qlora,
  title     = {{QLoRA}: Efficient Finetuning of Quantized {LLMs}},
  author    = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2023},
  url       = {https://arxiv.org/abs/2305.14314}
}

@inproceedings{dettmers2022optimizers,
  title     = {8-bit Optimizers via Block-wise Quantization},
  author    = {Dettmers, Tim and Lewis, Mike and Shleifer, Sam and Zettlemoyer, Luke},
  booktitle = {International Conference on Learning Representations},
  year      = {2022},
  url       = {https://arxiv.org/abs/2110.02861}
}

@inproceedings{loshchilov2019adamw,
  title     = {Decoupled Weight Decay Regularization},
  author    = {Loshchilov, Ilya and Hutter, Frank},
  booktitle = {International Conference on Learning Representations},
  year      = {2019},
  url       = {https://arxiv.org/abs/1711.05101}
}

@inproceedings{loshchilov2017sgdr,
  title     = {{SGDR}: Stochastic Gradient Descent with Warm Restarts},
  author    = {Loshchilov, Ilya and Hutter, Frank},
  booktitle = {International Conference on Learning Representations},
  year      = {2017},
  url       = {https://arxiv.org/abs/1608.03983}
}

15.5 Software

@misc{unsloth,
  title        = {Unsloth},
  author       = {Han, Daniel and Han, Michael and {Unsloth team}},
  year         = {2023},
  howpublished = {\url{https://github.com/unslothai/unsloth}}
}

@misc{vonwerra2020trl,
  title        = {{TRL}: Transformer Reinforcement Learning},
  author       = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallou{\'e}dec, Quentin},
  year         = {2020},
  howpublished = {\url{https://github.com/huggingface/trl}}
}

@inproceedings{wolf2020transformers,
  title     = {Transformers: State-of-the-Art Natural Language Processing},
  author    = {Wolf, Thomas and others},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  pages     = {38--45},
  year      = {2020},
  url       = {https://aclanthology.org/2020.emnlp-demos.6}
}

@misc{wandb,
  title        = {Experiment Tracking with Weights and Biases},
  author       = {Biewald, Lukas},
  year         = {2020},
  howpublished = {\url{https://www.wandb.com/}}
}

16. Acknowledgements

The author thanks A. Dinesh, a member of the AEGIS V2 team, who created part of the training data. The author also thanks M. Rohith (High School Physics data set), Hugging Face H4 and the authors of MATH and Let's Verify Step by Step (MATH-500), the Qwen team (Qwen3), the Unsloth team, the developers of TRL, Transformers, PEFT, and bitsandbytes, the llama.cpp contributors, the PocketPal AI developers, and Google Colab for free compute.

17. Contact

Open a discussion in the Community tab of this repository. Hugging Face name of the author: Xerxes-28.