File size: 3,514 Bytes
579f222
51f56ff
 
 
 
 
 
2010d69
51f56ff
 
 
 
 
 
a0d89e1
57a81d2
579f222
51f56ff
 
 
5682cbf
51f56ff
5682cbf
51f56ff
5682cbf
51f56ff
5682cbf
51f56ff
5682cbf
51f56ff
 
 
 
5682cbf
 
 
51f56ff
5682cbf
 
51f56ff
 
 
 
5682cbf
 
51f56ff
5682cbf
51f56ff
 
5682cbf
51f56ff
5682cbf
51f56ff
5682cbf
51f56ff
 
 
a0d89e1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
---
language:
  - id
  - en
license: gemma
library_name: transformers
base_model: google/gemma-3n-E4B-it
tags:
  - gemma
  - gemma-3n
  - unsloth
  - lora
  - fine-tuned
  - indonesian
  - english
  - text-generation
---

# Orbit-4-Gemma-E4B-it

## Model Description

Orbit-4-Gemma-E4B-it is a language model fine-tuned from Gemma 3n E4B-it, developed to enhance comprehension and knowledge capabilities in the Indonesian language.

The fine-tuning focuses on strengthening the model's ability to follow instructions in Indonesian, understand local context, and generate more natural and relevant responses. The training process utilized a large-scale Indonesian instruction dataset and spanned a full epoch without sampling.

This model retains the multimodal architecture of the Gemma 3n base model, meaning it structurally still supports text, image, audio, and video processing. However, the fine-tuning process was performed exclusively on the text layers, so multimodal performance was not specifically reinforced through this training.

## Model Specifications

| Item | Detail |
|---|---|
| Base model | google/gemma-3n-E4B-it |
| Fine-tuning method | LoRA (r=16, alpha=16) |
| Epoch | 1 full epoch (approx. 29,807 steps) |
| Effective batch size | 4 (batch size 1 x gradient accumulation 4) |
| Learning rate | 2e-4 (linear schedule) |
| Warmup | 3% of total steps |
| Max sequence length | 2048 tokens |
| Optimizer | adamw_8bit |
| Weight decay | 0.01 |
| LoRA dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Merge precision | float32 (performed on CPU) |
| Training hardware | Tesla T4 16GB |
| Framework | Unsloth, TRL, Transformers |
| Prompt format | Gemma 3n chat template |
| Seed | 3407 |

### Prompt Format

The model uses the Gemma 3n chat template as follows:

```text
<bos><start_of_turn>user
{instruction}<end_of_turn>
<start_of_turn>model
{output}<end_of_turn>

```

## Model Capabilities

The model possesses the following capabilities:

* Comprehension of Indonesian-language instructions, including complex and multi-step instructions
* General knowledge and local Indonesian context, covering history, culture, geography, and everyday topics
* Generation of natural Indonesian text for various needs, such as explanations, summaries, and narratives
* Creative writing tasks in Indonesian, such as poetry (*puisi* and *pantun*), and short stories
* Translation between Indonesian and English
* Sentiment analysis and Indonesian text processing
* Basic mathematical reasoning and simple problem-solving
* Inherent multimodal capabilities (text, image, audio, video input; text output), although not reinforced through this fine-tuning

### Limitations

* The model is optimized for Indonesian text; performance on other languages or non-text modalities may decrease compared to the base model
* The model has not been comprehensively tested for multimodal tasks within the Indonesian context
* The model may produce hallucinations; verification of critical facts remains necessary
* The model has not undergone additional safety tuning; deployment in production applications should be accompanied by supervision
* Maximum context is 2048 tokens in accordance with training settings

## Credits

* Base model: Google DeepMind (Gemma 3n)
* Fine-tuning framework: Unsloth
* Dataset: AksaraLLM
* Fine-tuning and training: OpenPathAI

## License

This model follows Google's Gemma license. Commercial use is subject to the Gemma Terms of Use.