Orbit-4-Gemma-E4B-it

Model Description

Orbit-4-Gemma-E4B-it is a language model fine-tuned from Gemma 3n E4B-it, developed to enhance comprehension and knowledge capabilities in the Indonesian language.

The fine-tuning focuses on strengthening the model's ability to follow instructions in Indonesian, understand local context, and generate more natural and relevant responses. The training process utilized a large-scale Indonesian instruction dataset and spanned a full epoch without sampling.

This model retains the multimodal architecture of the Gemma 3n base model, meaning it structurally still supports text, image, audio, and video processing. However, the fine-tuning process was performed exclusively on the text layers, so multimodal performance was not specifically reinforced through this training.

Model Specifications

Item Detail
Base model google/gemma-3n-E4B-it
Fine-tuning method LoRA (r=16, alpha=16)
Epoch 1 full epoch (approx. 29,807 steps)
Effective batch size 4 (batch size 1 x gradient accumulation 4)
Learning rate 2e-4 (linear schedule)
Warmup 3% of total steps
Max sequence length 2048 tokens
Optimizer adamw_8bit
Weight decay 0.01
LoRA dropout 0
Target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Merge precision float32 (performed on CPU)
Training hardware Tesla T4 16GB
Framework Unsloth, TRL, Transformers
Prompt format Gemma 3n chat template
Seed 3407

Prompt Format

The model uses the Gemma 3n chat template as follows:

<bos><start_of_turn>user
{instruction}<end_of_turn>
<start_of_turn>model
{output}<end_of_turn>

Model Capabilities

The model possesses the following capabilities:

  • Comprehension of Indonesian-language instructions, including complex and multi-step instructions
  • General knowledge and local Indonesian context, covering history, culture, geography, and everyday topics
  • Generation of natural Indonesian text for various needs, such as explanations, summaries, and narratives
  • Creative writing tasks in Indonesian, such as poetry (puisi and pantun), and short stories
  • Translation between Indonesian and English
  • Sentiment analysis and Indonesian text processing
  • Basic mathematical reasoning and simple problem-solving
  • Inherent multimodal capabilities (text, image, audio, video input; text output), although not reinforced through this fine-tuning

Limitations

  • The model is optimized for Indonesian text; performance on other languages or non-text modalities may decrease compared to the base model
  • The model has not been comprehensively tested for multimodal tasks within the Indonesian context
  • The model may produce hallucinations; verification of critical facts remains necessary
  • The model has not undergone additional safety tuning; deployment in production applications should be accompanied by supervision
  • Maximum context is 2048 tokens in accordance with training settings

Credits

  • Base model: Google DeepMind (Gemma 3n)
  • Fine-tuning framework: Unsloth
  • Dataset: AksaraLLM
  • Fine-tuning and training: OpenPathAI

License

This model follows Google's Gemma license. Commercial use is subject to the Gemma Terms of Use.

Downloads last month
313
Safetensors
Model size
8B params
Tensor type
F32
ยท
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OpenPathAI/Orbit-4-Gemma-E4B-it

Adapter
(18)
this model

Collection including OpenPathAI/Orbit-4-Gemma-E4B-it