Smit31 commited on
Commit
59ccb0f
·
verified ·
1 Parent(s): 36b6013

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
3
+ library_name: peft
4
+ tags:
5
+ - base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
6
+ - lora
7
+ - sft
8
+ - transformers
9
+ - trl
10
+ ---
11
+
12
+ # Model Card for Model ID
13
+
14
+ <!-- Provide a quick summary of what the model is/does. -->
15
+
16
+
17
+
18
+ ## Model Details
19
+
20
+ ### Model Description
21
+
22
+ <!-- Provide a longer summary of what this model is. -->
23
+
24
+
25
+
26
+ - **Developed by:** [More Information Needed]
27
+ - **Funded by [optional]:** [More Information Needed]
28
+ - **Shared by [optional]:** [More Information Needed]
29
+ - **Model type:** [More Information Needed]
30
+ - **Language(s) (NLP):** [More Information Needed]
31
+ - **License:** [More Information Needed]
32
+ - **Finetuned from model [optional]:** [More Information Needed]
33
+
34
+ ### Model Sources [optional]
35
+
36
+ <!-- Provide the basic links for the model. -->
37
+
38
+ - **Repository:** [More Information Needed]
39
+ - **Paper [optional]:** [More Information Needed]
40
+ - **Demo [optional]:** [More Information Needed]
41
+
42
+ ## Uses
43
+
44
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
45
+
46
+ ### Direct Use
47
+
48
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Downstream Use [optional]
53
+
54
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
55
+
56
+ [More Information Needed]
57
+
58
+ ### Out-of-Scope Use
59
+
60
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ## Bias, Risks, and Limitations
65
+
66
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
67
+
68
+ [More Information Needed]
69
+
70
+ ### Recommendations
71
+
72
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
73
+
74
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
75
+
76
+ ## How to Get Started with the Model
77
+
78
+ Use the code below to get started with the model.
79
+
80
+ [More Information Needed]
81
+
82
+ ## Training Details
83
+
84
+ ### Training Data
85
+
86
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
87
+
88
+ [More Information Needed]
89
+
90
+ ### Training Procedure
91
+
92
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
93
+
94
+ #### Preprocessing [optional]
95
+
96
+ [More Information Needed]
97
+
98
+
99
+ #### Training Hyperparameters
100
+
101
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
102
+
103
+ #### Speeds, Sizes, Times [optional]
104
+
105
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
106
+
107
+ [More Information Needed]
108
+
109
+ ## Evaluation
110
+
111
+ <!-- This section describes the evaluation protocols and provides the results. -->
112
+
113
+ ### Testing Data, Factors & Metrics
114
+
115
+ #### Testing Data
116
+
117
+ <!-- This should link to a Dataset Card if possible. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Factors
122
+
123
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
124
+
125
+ [More Information Needed]
126
+
127
+ #### Metrics
128
+
129
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
130
+
131
+ [More Information Needed]
132
+
133
+ ### Results
134
+
135
+ [More Information Needed]
136
+
137
+ #### Summary
138
+
139
+
140
+
141
+ ## Model Examination [optional]
142
+
143
+ <!-- Relevant interpretability work for the model goes here -->
144
+
145
+ [More Information Needed]
146
+
147
+ ## Environmental Impact
148
+
149
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
150
+
151
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
152
+
153
+ - **Hardware Type:** [More Information Needed]
154
+ - **Hours used:** [More Information Needed]
155
+ - **Cloud Provider:** [More Information Needed]
156
+ - **Compute Region:** [More Information Needed]
157
+ - **Carbon Emitted:** [More Information Needed]
158
+
159
+ ## Technical Specifications [optional]
160
+
161
+ ### Model Architecture and Objective
162
+
163
+ [More Information Needed]
164
+
165
+ ### Compute Infrastructure
166
+
167
+ [More Information Needed]
168
+
169
+ #### Hardware
170
+
171
+ [More Information Needed]
172
+
173
+ #### Software
174
+
175
+ [More Information Needed]
176
+
177
+ ## Citation [optional]
178
+
179
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
180
+
181
+ **BibTeX:**
182
+
183
+ [More Information Needed]
184
+
185
+ **APA:**
186
+
187
+ [More Information Needed]
188
+
189
+ ## Glossary [optional]
190
+
191
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
192
+
193
+ [More Information Needed]
194
+
195
+ ## More Information [optional]
196
+
197
+ [More Information Needed]
198
+
199
+ ## Model Card Authors [optional]
200
+
201
+ [More Information Needed]
202
+
203
+ ## Model Card Contact
204
+
205
+ [More Information Needed]
206
+ ### Framework versions
207
+
208
+ - PEFT 0.17.1
adapter_config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": {
4
+ "base_model_class": "LlamaForCausalLM",
5
+ "parent_library": "transformers.models.llama.modeling_llama"
6
+ },
7
+ "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
8
+ "bias": "none",
9
+ "corda_config": null,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": null,
25
+ "peft_type": "LORA",
26
+ "qalora_group_size": 16,
27
+ "r": 16,
28
+ "rank_pattern": {},
29
+ "revision": null,
30
+ "target_modules": [
31
+ "q_proj",
32
+ "down_proj",
33
+ "gate_proj",
34
+ "v_proj",
35
+ "o_proj",
36
+ "k_proj",
37
+ "up_proj"
38
+ ],
39
+ "target_parameters": null,
40
+ "task_type": null,
41
+ "trainable_token_indices": null,
42
+ "use_dora": false,
43
+ "use_qalora": false,
44
+ "use_rslora": false
45
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:379f08c09605e978c3e51741a1d44cf601b24ef82119583ad917f25f9378c063
3
+ size 50503544
chat_template.jinja ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {% for message in messages %}
2
+ {% if message['role'] == 'user' %}
3
+ {{ '<|user|>
4
+ ' + message['content'] + eos_token }}
5
+ {% elif message['role'] == 'system' %}
6
+ {{ '<|system|>
7
+ ' + message['content'] + eos_token }}
8
+ {% elif message['role'] == 'assistant' %}
9
+ {{ '<|assistant|>
10
+ ' + message['content'] + eos_token }}
11
+ {% endif %}
12
+ {% if loop.last and add_generation_prompt %}
13
+ {{ '<|assistant|>' }}
14
+ {% endif %}
15
+ {% endfor %}
checkpoint-500/README.md ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
3
+ library_name: peft
4
+ tags:
5
+ - base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
6
+ - lora
7
+ - sft
8
+ - transformers
9
+ - trl
10
+ ---
11
+
12
+ # Model Card for Model ID
13
+
14
+ <!-- Provide a quick summary of what the model is/does. -->
15
+
16
+
17
+
18
+ ## Model Details
19
+
20
+ ### Model Description
21
+
22
+ <!-- Provide a longer summary of what this model is. -->
23
+
24
+
25
+
26
+ - **Developed by:** [More Information Needed]
27
+ - **Funded by [optional]:** [More Information Needed]
28
+ - **Shared by [optional]:** [More Information Needed]
29
+ - **Model type:** [More Information Needed]
30
+ - **Language(s) (NLP):** [More Information Needed]
31
+ - **License:** [More Information Needed]
32
+ - **Finetuned from model [optional]:** [More Information Needed]
33
+
34
+ ### Model Sources [optional]
35
+
36
+ <!-- Provide the basic links for the model. -->
37
+
38
+ - **Repository:** [More Information Needed]
39
+ - **Paper [optional]:** [More Information Needed]
40
+ - **Demo [optional]:** [More Information Needed]
41
+
42
+ ## Uses
43
+
44
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
45
+
46
+ ### Direct Use
47
+
48
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Downstream Use [optional]
53
+
54
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
55
+
56
+ [More Information Needed]
57
+
58
+ ### Out-of-Scope Use
59
+
60
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ## Bias, Risks, and Limitations
65
+
66
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
67
+
68
+ [More Information Needed]
69
+
70
+ ### Recommendations
71
+
72
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
73
+
74
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
75
+
76
+ ## How to Get Started with the Model
77
+
78
+ Use the code below to get started with the model.
79
+
80
+ [More Information Needed]
81
+
82
+ ## Training Details
83
+
84
+ ### Training Data
85
+
86
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
87
+
88
+ [More Information Needed]
89
+
90
+ ### Training Procedure
91
+
92
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
93
+
94
+ #### Preprocessing [optional]
95
+
96
+ [More Information Needed]
97
+
98
+
99
+ #### Training Hyperparameters
100
+
101
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
102
+
103
+ #### Speeds, Sizes, Times [optional]
104
+
105
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
106
+
107
+ [More Information Needed]
108
+
109
+ ## Evaluation
110
+
111
+ <!-- This section describes the evaluation protocols and provides the results. -->
112
+
113
+ ### Testing Data, Factors & Metrics
114
+
115
+ #### Testing Data
116
+
117
+ <!-- This should link to a Dataset Card if possible. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Factors
122
+
123
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
124
+
125
+ [More Information Needed]
126
+
127
+ #### Metrics
128
+
129
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
130
+
131
+ [More Information Needed]
132
+
133
+ ### Results
134
+
135
+ [More Information Needed]
136
+
137
+ #### Summary
138
+
139
+
140
+
141
+ ## Model Examination [optional]
142
+
143
+ <!-- Relevant interpretability work for the model goes here -->
144
+
145
+ [More Information Needed]
146
+
147
+ ## Environmental Impact
148
+
149
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
150
+
151
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
152
+
153
+ - **Hardware Type:** [More Information Needed]
154
+ - **Hours used:** [More Information Needed]
155
+ - **Cloud Provider:** [More Information Needed]
156
+ - **Compute Region:** [More Information Needed]
157
+ - **Carbon Emitted:** [More Information Needed]
158
+
159
+ ## Technical Specifications [optional]
160
+
161
+ ### Model Architecture and Objective
162
+
163
+ [More Information Needed]
164
+
165
+ ### Compute Infrastructure
166
+
167
+ [More Information Needed]
168
+
169
+ #### Hardware
170
+
171
+ [More Information Needed]
172
+
173
+ #### Software
174
+
175
+ [More Information Needed]
176
+
177
+ ## Citation [optional]
178
+
179
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
180
+
181
+ **BibTeX:**
182
+
183
+ [More Information Needed]
184
+
185
+ **APA:**
186
+
187
+ [More Information Needed]
188
+
189
+ ## Glossary [optional]
190
+
191
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
192
+
193
+ [More Information Needed]
194
+
195
+ ## More Information [optional]
196
+
197
+ [More Information Needed]
198
+
199
+ ## Model Card Authors [optional]
200
+
201
+ [More Information Needed]
202
+
203
+ ## Model Card Contact
204
+
205
+ [More Information Needed]
206
+ ### Framework versions
207
+
208
+ - PEFT 0.17.1
checkpoint-500/adapter_config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": {
4
+ "base_model_class": "LlamaForCausalLM",
5
+ "parent_library": "transformers.models.llama.modeling_llama"
6
+ },
7
+ "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
8
+ "bias": "none",
9
+ "corda_config": null,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": null,
25
+ "peft_type": "LORA",
26
+ "qalora_group_size": 16,
27
+ "r": 16,
28
+ "rank_pattern": {},
29
+ "revision": null,
30
+ "target_modules": [
31
+ "q_proj",
32
+ "down_proj",
33
+ "gate_proj",
34
+ "v_proj",
35
+ "o_proj",
36
+ "k_proj",
37
+ "up_proj"
38
+ ],
39
+ "target_parameters": null,
40
+ "task_type": null,
41
+ "trainable_token_indices": null,
42
+ "use_dora": false,
43
+ "use_qalora": false,
44
+ "use_rslora": false
45
+ }
checkpoint-500/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:420e1afffbd6568efc4aa506812035158de8652c4ac4a371dfad640484bade77
3
+ size 50503544
checkpoint-500/chat_template.jinja ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {% for message in messages %}
2
+ {% if message['role'] == 'user' %}
3
+ {{ '<|user|>
4
+ ' + message['content'] + eos_token }}
5
+ {% elif message['role'] == 'system' %}
6
+ {{ '<|system|>
7
+ ' + message['content'] + eos_token }}
8
+ {% elif message['role'] == 'assistant' %}
9
+ {{ '<|assistant|>
10
+ ' + message['content'] + eos_token }}
11
+ {% endif %}
12
+ {% if loop.last and add_generation_prompt %}
13
+ {{ '<|assistant|>' }}
14
+ {% endif %}
15
+ {% endfor %}
checkpoint-500/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9e534c1b03dd1078a62fb585066182ca83441f6da8389278ed16bb1f7ed163f
3
+ size 101188875
checkpoint-500/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4d606bd8ffdc74fbfbe4f270ed8a3f3f8c70508b79db9957eda85e1e7b7d16a
3
+ size 14645
checkpoint-500/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:965475fa922aabe62de60ece69f94d852a7236e2ca018179d098f61320a174f3
3
+ size 1465
checkpoint-500/special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
checkpoint-500/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
checkpoint-500/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
3
+ size 499723
checkpoint-500/tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": null,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<unk>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<s>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "2": {
23
+ "content": "</s>",
24
+ "lstrip": false,
25
+ "normalized": false,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": true
29
+ }
30
+ },
31
+ "bos_token": "<s>",
32
+ "clean_up_tokenization_spaces": false,
33
+ "eos_token": "</s>",
34
+ "extra_special_tokens": {},
35
+ "legacy": false,
36
+ "model_max_length": 2048,
37
+ "pad_token": "</s>",
38
+ "padding_side": "right",
39
+ "sp_model_kwargs": {},
40
+ "tokenizer_class": "LlamaTokenizer",
41
+ "unk_token": "<unk>",
42
+ "use_default_system_prompt": false
43
+ }
checkpoint-500/trainer_state.json ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 0.7195538765965102,
6
+ "eval_steps": 500,
7
+ "global_step": 500,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.028782155063860407,
14
+ "grad_norm": 0.25469791889190674,
15
+ "learning_rate": 0.00019453237410071942,
16
+ "loss": 0.6626,
17
+ "step": 20
18
+ },
19
+ {
20
+ "epoch": 0.057564310127720814,
21
+ "grad_norm": 0.26558175683021545,
22
+ "learning_rate": 0.00018877697841726622,
23
+ "loss": 0.53,
24
+ "step": 40
25
+ },
26
+ {
27
+ "epoch": 0.08634646519158122,
28
+ "grad_norm": 0.2776390314102173,
29
+ "learning_rate": 0.00018302158273381295,
30
+ "loss": 0.5091,
31
+ "step": 60
32
+ },
33
+ {
34
+ "epoch": 0.11512862025544163,
35
+ "grad_norm": 0.25934168696403503,
36
+ "learning_rate": 0.00017726618705035972,
37
+ "loss": 0.4938,
38
+ "step": 80
39
+ },
40
+ {
41
+ "epoch": 0.14391077531930202,
42
+ "grad_norm": 0.24628081917762756,
43
+ "learning_rate": 0.00017151079136690648,
44
+ "loss": 0.4691,
45
+ "step": 100
46
+ },
47
+ {
48
+ "epoch": 0.17269293038316244,
49
+ "grad_norm": 0.24218925833702087,
50
+ "learning_rate": 0.00016575539568345325,
51
+ "loss": 0.4648,
52
+ "step": 120
53
+ },
54
+ {
55
+ "epoch": 0.20147508544702283,
56
+ "grad_norm": 0.2658531963825226,
57
+ "learning_rate": 0.00016,
58
+ "loss": 0.466,
59
+ "step": 140
60
+ },
61
+ {
62
+ "epoch": 0.23025724051088325,
63
+ "grad_norm": 0.2526750862598419,
64
+ "learning_rate": 0.00015424460431654678,
65
+ "loss": 0.4569,
66
+ "step": 160
67
+ },
68
+ {
69
+ "epoch": 0.25903939557474365,
70
+ "grad_norm": 0.2415703982114792,
71
+ "learning_rate": 0.00014848920863309352,
72
+ "loss": 0.4557,
73
+ "step": 180
74
+ },
75
+ {
76
+ "epoch": 0.28782155063860404,
77
+ "grad_norm": 0.26834964752197266,
78
+ "learning_rate": 0.0001427338129496403,
79
+ "loss": 0.4517,
80
+ "step": 200
81
+ },
82
+ {
83
+ "epoch": 0.3166037057024645,
84
+ "grad_norm": 0.26203978061676025,
85
+ "learning_rate": 0.00013697841726618705,
86
+ "loss": 0.4455,
87
+ "step": 220
88
+ },
89
+ {
90
+ "epoch": 0.3453858607663249,
91
+ "grad_norm": 0.2680965065956116,
92
+ "learning_rate": 0.00013122302158273384,
93
+ "loss": 0.4492,
94
+ "step": 240
95
+ },
96
+ {
97
+ "epoch": 0.3741680158301853,
98
+ "grad_norm": 0.2564185857772827,
99
+ "learning_rate": 0.00012546762589928058,
100
+ "loss": 0.4438,
101
+ "step": 260
102
+ },
103
+ {
104
+ "epoch": 0.40295017089404567,
105
+ "grad_norm": 0.26110026240348816,
106
+ "learning_rate": 0.00011971223021582734,
107
+ "loss": 0.443,
108
+ "step": 280
109
+ },
110
+ {
111
+ "epoch": 0.4317323259579061,
112
+ "grad_norm": 0.262195348739624,
113
+ "learning_rate": 0.00011395683453237412,
114
+ "loss": 0.4309,
115
+ "step": 300
116
+ },
117
+ {
118
+ "epoch": 0.4605144810217665,
119
+ "grad_norm": 0.25222480297088623,
120
+ "learning_rate": 0.00010820143884892087,
121
+ "loss": 0.4405,
122
+ "step": 320
123
+ },
124
+ {
125
+ "epoch": 0.4892966360856269,
126
+ "grad_norm": 0.27905553579330444,
127
+ "learning_rate": 0.00010244604316546762,
128
+ "loss": 0.4329,
129
+ "step": 340
130
+ },
131
+ {
132
+ "epoch": 0.5180787911494873,
133
+ "grad_norm": 0.2416505217552185,
134
+ "learning_rate": 9.66906474820144e-05,
135
+ "loss": 0.4312,
136
+ "step": 360
137
+ },
138
+ {
139
+ "epoch": 0.5468609462133477,
140
+ "grad_norm": 0.2609241008758545,
141
+ "learning_rate": 9.093525179856116e-05,
142
+ "loss": 0.4363,
143
+ "step": 380
144
+ },
145
+ {
146
+ "epoch": 0.5756431012772081,
147
+ "grad_norm": 0.24666130542755127,
148
+ "learning_rate": 8.517985611510792e-05,
149
+ "loss": 0.4302,
150
+ "step": 400
151
+ },
152
+ {
153
+ "epoch": 0.6044252563410686,
154
+ "grad_norm": 0.24873825907707214,
155
+ "learning_rate": 7.942446043165468e-05,
156
+ "loss": 0.4266,
157
+ "step": 420
158
+ },
159
+ {
160
+ "epoch": 0.633207411404929,
161
+ "grad_norm": 0.2555878460407257,
162
+ "learning_rate": 7.366906474820145e-05,
163
+ "loss": 0.4287,
164
+ "step": 440
165
+ },
166
+ {
167
+ "epoch": 0.6619895664687894,
168
+ "grad_norm": 0.2684342861175537,
169
+ "learning_rate": 6.791366906474821e-05,
170
+ "loss": 0.4243,
171
+ "step": 460
172
+ },
173
+ {
174
+ "epoch": 0.6907717215326498,
175
+ "grad_norm": 0.26070982217788696,
176
+ "learning_rate": 6.215827338129496e-05,
177
+ "loss": 0.4259,
178
+ "step": 480
179
+ },
180
+ {
181
+ "epoch": 0.7195538765965102,
182
+ "grad_norm": 0.27525463700294495,
183
+ "learning_rate": 5.6402877697841726e-05,
184
+ "loss": 0.4281,
185
+ "step": 500
186
+ }
187
+ ],
188
+ "logging_steps": 20,
189
+ "max_steps": 695,
190
+ "num_input_tokens_seen": 0,
191
+ "num_train_epochs": 1,
192
+ "save_steps": 500,
193
+ "stateful_callbacks": {
194
+ "TrainerControl": {
195
+ "args": {
196
+ "should_epoch_stop": false,
197
+ "should_evaluate": false,
198
+ "should_log": false,
199
+ "should_save": true,
200
+ "should_training_stop": false
201
+ },
202
+ "attributes": {}
203
+ }
204
+ },
205
+ "total_flos": 2.5734219300864e+16,
206
+ "train_batch_size": 1,
207
+ "trial_name": null,
208
+ "trial_params": null
209
+ }
checkpoint-500/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f7ac894a7270c0a6e6d99e6a8699a4d25d42e4c25c1cfa862f43218ec8b6327
3
+ size 6097
checkpoint-695/README.md ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
3
+ library_name: peft
4
+ tags:
5
+ - base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
6
+ - lora
7
+ - sft
8
+ - transformers
9
+ - trl
10
+ ---
11
+
12
+ # Model Card for Model ID
13
+
14
+ <!-- Provide a quick summary of what the model is/does. -->
15
+
16
+
17
+
18
+ ## Model Details
19
+
20
+ ### Model Description
21
+
22
+ <!-- Provide a longer summary of what this model is. -->
23
+
24
+
25
+
26
+ - **Developed by:** [More Information Needed]
27
+ - **Funded by [optional]:** [More Information Needed]
28
+ - **Shared by [optional]:** [More Information Needed]
29
+ - **Model type:** [More Information Needed]
30
+ - **Language(s) (NLP):** [More Information Needed]
31
+ - **License:** [More Information Needed]
32
+ - **Finetuned from model [optional]:** [More Information Needed]
33
+
34
+ ### Model Sources [optional]
35
+
36
+ <!-- Provide the basic links for the model. -->
37
+
38
+ - **Repository:** [More Information Needed]
39
+ - **Paper [optional]:** [More Information Needed]
40
+ - **Demo [optional]:** [More Information Needed]
41
+
42
+ ## Uses
43
+
44
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
45
+
46
+ ### Direct Use
47
+
48
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Downstream Use [optional]
53
+
54
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
55
+
56
+ [More Information Needed]
57
+
58
+ ### Out-of-Scope Use
59
+
60
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ## Bias, Risks, and Limitations
65
+
66
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
67
+
68
+ [More Information Needed]
69
+
70
+ ### Recommendations
71
+
72
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
73
+
74
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
75
+
76
+ ## How to Get Started with the Model
77
+
78
+ Use the code below to get started with the model.
79
+
80
+ [More Information Needed]
81
+
82
+ ## Training Details
83
+
84
+ ### Training Data
85
+
86
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
87
+
88
+ [More Information Needed]
89
+
90
+ ### Training Procedure
91
+
92
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
93
+
94
+ #### Preprocessing [optional]
95
+
96
+ [More Information Needed]
97
+
98
+
99
+ #### Training Hyperparameters
100
+
101
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
102
+
103
+ #### Speeds, Sizes, Times [optional]
104
+
105
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
106
+
107
+ [More Information Needed]
108
+
109
+ ## Evaluation
110
+
111
+ <!-- This section describes the evaluation protocols and provides the results. -->
112
+
113
+ ### Testing Data, Factors & Metrics
114
+
115
+ #### Testing Data
116
+
117
+ <!-- This should link to a Dataset Card if possible. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Factors
122
+
123
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
124
+
125
+ [More Information Needed]
126
+
127
+ #### Metrics
128
+
129
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
130
+
131
+ [More Information Needed]
132
+
133
+ ### Results
134
+
135
+ [More Information Needed]
136
+
137
+ #### Summary
138
+
139
+
140
+
141
+ ## Model Examination [optional]
142
+
143
+ <!-- Relevant interpretability work for the model goes here -->
144
+
145
+ [More Information Needed]
146
+
147
+ ## Environmental Impact
148
+
149
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
150
+
151
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
152
+
153
+ - **Hardware Type:** [More Information Needed]
154
+ - **Hours used:** [More Information Needed]
155
+ - **Cloud Provider:** [More Information Needed]
156
+ - **Compute Region:** [More Information Needed]
157
+ - **Carbon Emitted:** [More Information Needed]
158
+
159
+ ## Technical Specifications [optional]
160
+
161
+ ### Model Architecture and Objective
162
+
163
+ [More Information Needed]
164
+
165
+ ### Compute Infrastructure
166
+
167
+ [More Information Needed]
168
+
169
+ #### Hardware
170
+
171
+ [More Information Needed]
172
+
173
+ #### Software
174
+
175
+ [More Information Needed]
176
+
177
+ ## Citation [optional]
178
+
179
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
180
+
181
+ **BibTeX:**
182
+
183
+ [More Information Needed]
184
+
185
+ **APA:**
186
+
187
+ [More Information Needed]
188
+
189
+ ## Glossary [optional]
190
+
191
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
192
+
193
+ [More Information Needed]
194
+
195
+ ## More Information [optional]
196
+
197
+ [More Information Needed]
198
+
199
+ ## Model Card Authors [optional]
200
+
201
+ [More Information Needed]
202
+
203
+ ## Model Card Contact
204
+
205
+ [More Information Needed]
206
+ ### Framework versions
207
+
208
+ - PEFT 0.17.1
checkpoint-695/adapter_config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": {
4
+ "base_model_class": "LlamaForCausalLM",
5
+ "parent_library": "transformers.models.llama.modeling_llama"
6
+ },
7
+ "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
8
+ "bias": "none",
9
+ "corda_config": null,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": null,
25
+ "peft_type": "LORA",
26
+ "qalora_group_size": 16,
27
+ "r": 16,
28
+ "rank_pattern": {},
29
+ "revision": null,
30
+ "target_modules": [
31
+ "q_proj",
32
+ "down_proj",
33
+ "gate_proj",
34
+ "v_proj",
35
+ "o_proj",
36
+ "k_proj",
37
+ "up_proj"
38
+ ],
39
+ "target_parameters": null,
40
+ "task_type": null,
41
+ "trainable_token_indices": null,
42
+ "use_dora": false,
43
+ "use_qalora": false,
44
+ "use_rslora": false
45
+ }
checkpoint-695/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:379f08c09605e978c3e51741a1d44cf601b24ef82119583ad917f25f9378c063
3
+ size 50503544
checkpoint-695/chat_template.jinja ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {% for message in messages %}
2
+ {% if message['role'] == 'user' %}
3
+ {{ '<|user|>
4
+ ' + message['content'] + eos_token }}
5
+ {% elif message['role'] == 'system' %}
6
+ {{ '<|system|>
7
+ ' + message['content'] + eos_token }}
8
+ {% elif message['role'] == 'assistant' %}
9
+ {{ '<|assistant|>
10
+ ' + message['content'] + eos_token }}
11
+ {% endif %}
12
+ {% if loop.last and add_generation_prompt %}
13
+ {{ '<|assistant|>' }}
14
+ {% endif %}
15
+ {% endfor %}
checkpoint-695/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d3e3a593380e50da0fe0a93491507b82b604657e349acdf8ce2b9b3a28ed557
3
+ size 101188875
checkpoint-695/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e79343406a1dbf96ca6660391ae425ac535dd9aeef72727b7719b64653562c68
3
+ size 14645
checkpoint-695/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de831bf1d62214d0c80d3f2b405e39c42a580135eb3dd22f4851ded57be09dc6
3
+ size 1465
checkpoint-695/special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
checkpoint-695/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
checkpoint-695/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
3
+ size 499723
checkpoint-695/tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": null,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<unk>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<s>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "2": {
23
+ "content": "</s>",
24
+ "lstrip": false,
25
+ "normalized": false,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": true
29
+ }
30
+ },
31
+ "bos_token": "<s>",
32
+ "clean_up_tokenization_spaces": false,
33
+ "eos_token": "</s>",
34
+ "extra_special_tokens": {},
35
+ "legacy": false,
36
+ "model_max_length": 2048,
37
+ "pad_token": "</s>",
38
+ "padding_side": "right",
39
+ "sp_model_kwargs": {},
40
+ "tokenizer_class": "LlamaTokenizer",
41
+ "unk_token": "<unk>",
42
+ "use_default_system_prompt": false
43
+ }
checkpoint-695/trainer_state.json ADDED
@@ -0,0 +1,272 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.0,
6
+ "eval_steps": 500,
7
+ "global_step": 695,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.028782155063860407,
14
+ "grad_norm": 0.25469791889190674,
15
+ "learning_rate": 0.00019453237410071942,
16
+ "loss": 0.6626,
17
+ "step": 20
18
+ },
19
+ {
20
+ "epoch": 0.057564310127720814,
21
+ "grad_norm": 0.26558175683021545,
22
+ "learning_rate": 0.00018877697841726622,
23
+ "loss": 0.53,
24
+ "step": 40
25
+ },
26
+ {
27
+ "epoch": 0.08634646519158122,
28
+ "grad_norm": 0.2776390314102173,
29
+ "learning_rate": 0.00018302158273381295,
30
+ "loss": 0.5091,
31
+ "step": 60
32
+ },
33
+ {
34
+ "epoch": 0.11512862025544163,
35
+ "grad_norm": 0.25934168696403503,
36
+ "learning_rate": 0.00017726618705035972,
37
+ "loss": 0.4938,
38
+ "step": 80
39
+ },
40
+ {
41
+ "epoch": 0.14391077531930202,
42
+ "grad_norm": 0.24628081917762756,
43
+ "learning_rate": 0.00017151079136690648,
44
+ "loss": 0.4691,
45
+ "step": 100
46
+ },
47
+ {
48
+ "epoch": 0.17269293038316244,
49
+ "grad_norm": 0.24218925833702087,
50
+ "learning_rate": 0.00016575539568345325,
51
+ "loss": 0.4648,
52
+ "step": 120
53
+ },
54
+ {
55
+ "epoch": 0.20147508544702283,
56
+ "grad_norm": 0.2658531963825226,
57
+ "learning_rate": 0.00016,
58
+ "loss": 0.466,
59
+ "step": 140
60
+ },
61
+ {
62
+ "epoch": 0.23025724051088325,
63
+ "grad_norm": 0.2526750862598419,
64
+ "learning_rate": 0.00015424460431654678,
65
+ "loss": 0.4569,
66
+ "step": 160
67
+ },
68
+ {
69
+ "epoch": 0.25903939557474365,
70
+ "grad_norm": 0.2415703982114792,
71
+ "learning_rate": 0.00014848920863309352,
72
+ "loss": 0.4557,
73
+ "step": 180
74
+ },
75
+ {
76
+ "epoch": 0.28782155063860404,
77
+ "grad_norm": 0.26834964752197266,
78
+ "learning_rate": 0.0001427338129496403,
79
+ "loss": 0.4517,
80
+ "step": 200
81
+ },
82
+ {
83
+ "epoch": 0.3166037057024645,
84
+ "grad_norm": 0.26203978061676025,
85
+ "learning_rate": 0.00013697841726618705,
86
+ "loss": 0.4455,
87
+ "step": 220
88
+ },
89
+ {
90
+ "epoch": 0.3453858607663249,
91
+ "grad_norm": 0.2680965065956116,
92
+ "learning_rate": 0.00013122302158273384,
93
+ "loss": 0.4492,
94
+ "step": 240
95
+ },
96
+ {
97
+ "epoch": 0.3741680158301853,
98
+ "grad_norm": 0.2564185857772827,
99
+ "learning_rate": 0.00012546762589928058,
100
+ "loss": 0.4438,
101
+ "step": 260
102
+ },
103
+ {
104
+ "epoch": 0.40295017089404567,
105
+ "grad_norm": 0.26110026240348816,
106
+ "learning_rate": 0.00011971223021582734,
107
+ "loss": 0.443,
108
+ "step": 280
109
+ },
110
+ {
111
+ "epoch": 0.4317323259579061,
112
+ "grad_norm": 0.262195348739624,
113
+ "learning_rate": 0.00011395683453237412,
114
+ "loss": 0.4309,
115
+ "step": 300
116
+ },
117
+ {
118
+ "epoch": 0.4605144810217665,
119
+ "grad_norm": 0.25222480297088623,
120
+ "learning_rate": 0.00010820143884892087,
121
+ "loss": 0.4405,
122
+ "step": 320
123
+ },
124
+ {
125
+ "epoch": 0.4892966360856269,
126
+ "grad_norm": 0.27905553579330444,
127
+ "learning_rate": 0.00010244604316546762,
128
+ "loss": 0.4329,
129
+ "step": 340
130
+ },
131
+ {
132
+ "epoch": 0.5180787911494873,
133
+ "grad_norm": 0.2416505217552185,
134
+ "learning_rate": 9.66906474820144e-05,
135
+ "loss": 0.4312,
136
+ "step": 360
137
+ },
138
+ {
139
+ "epoch": 0.5468609462133477,
140
+ "grad_norm": 0.2609241008758545,
141
+ "learning_rate": 9.093525179856116e-05,
142
+ "loss": 0.4363,
143
+ "step": 380
144
+ },
145
+ {
146
+ "epoch": 0.5756431012772081,
147
+ "grad_norm": 0.24666130542755127,
148
+ "learning_rate": 8.517985611510792e-05,
149
+ "loss": 0.4302,
150
+ "step": 400
151
+ },
152
+ {
153
+ "epoch": 0.6044252563410686,
154
+ "grad_norm": 0.24873825907707214,
155
+ "learning_rate": 7.942446043165468e-05,
156
+ "loss": 0.4266,
157
+ "step": 420
158
+ },
159
+ {
160
+ "epoch": 0.633207411404929,
161
+ "grad_norm": 0.2555878460407257,
162
+ "learning_rate": 7.366906474820145e-05,
163
+ "loss": 0.4287,
164
+ "step": 440
165
+ },
166
+ {
167
+ "epoch": 0.6619895664687894,
168
+ "grad_norm": 0.2684342861175537,
169
+ "learning_rate": 6.791366906474821e-05,
170
+ "loss": 0.4243,
171
+ "step": 460
172
+ },
173
+ {
174
+ "epoch": 0.6907717215326498,
175
+ "grad_norm": 0.26070982217788696,
176
+ "learning_rate": 6.215827338129496e-05,
177
+ "loss": 0.4259,
178
+ "step": 480
179
+ },
180
+ {
181
+ "epoch": 0.7195538765965102,
182
+ "grad_norm": 0.27525463700294495,
183
+ "learning_rate": 5.6402877697841726e-05,
184
+ "loss": 0.4281,
185
+ "step": 500
186
+ },
187
+ {
188
+ "epoch": 0.7483360316603705,
189
+ "grad_norm": 0.2588309347629547,
190
+ "learning_rate": 5.064748201438849e-05,
191
+ "loss": 0.4153,
192
+ "step": 520
193
+ },
194
+ {
195
+ "epoch": 0.7771181867242309,
196
+ "grad_norm": 0.2782978415489197,
197
+ "learning_rate": 4.489208633093525e-05,
198
+ "loss": 0.4237,
199
+ "step": 540
200
+ },
201
+ {
202
+ "epoch": 0.8059003417880913,
203
+ "grad_norm": 0.2572264075279236,
204
+ "learning_rate": 3.9136690647482014e-05,
205
+ "loss": 0.4224,
206
+ "step": 560
207
+ },
208
+ {
209
+ "epoch": 0.8346824968519518,
210
+ "grad_norm": 0.2893691062927246,
211
+ "learning_rate": 3.338129496402878e-05,
212
+ "loss": 0.4234,
213
+ "step": 580
214
+ },
215
+ {
216
+ "epoch": 0.8634646519158122,
217
+ "grad_norm": 0.2681804895401001,
218
+ "learning_rate": 2.762589928057554e-05,
219
+ "loss": 0.4204,
220
+ "step": 600
221
+ },
222
+ {
223
+ "epoch": 0.8922468069796726,
224
+ "grad_norm": 0.2793126702308655,
225
+ "learning_rate": 2.1870503597122302e-05,
226
+ "loss": 0.423,
227
+ "step": 620
228
+ },
229
+ {
230
+ "epoch": 0.921028962043533,
231
+ "grad_norm": 0.27879542112350464,
232
+ "learning_rate": 1.6115107913669063e-05,
233
+ "loss": 0.4158,
234
+ "step": 640
235
+ },
236
+ {
237
+ "epoch": 0.9498111171073934,
238
+ "grad_norm": 0.2639375329017639,
239
+ "learning_rate": 1.0359712230215828e-05,
240
+ "loss": 0.419,
241
+ "step": 660
242
+ },
243
+ {
244
+ "epoch": 0.9785932721712538,
245
+ "grad_norm": 0.25821879506111145,
246
+ "learning_rate": 4.60431654676259e-06,
247
+ "loss": 0.4172,
248
+ "step": 680
249
+ }
250
+ ],
251
+ "logging_steps": 20,
252
+ "max_steps": 695,
253
+ "num_input_tokens_seen": 0,
254
+ "num_train_epochs": 1,
255
+ "save_steps": 500,
256
+ "stateful_callbacks": {
257
+ "TrainerControl": {
258
+ "args": {
259
+ "should_epoch_stop": false,
260
+ "should_evaluate": false,
261
+ "should_log": false,
262
+ "should_save": true,
263
+ "should_training_stop": true
264
+ },
265
+ "attributes": {}
266
+ }
267
+ },
268
+ "total_flos": 3.5764131273375744e+16,
269
+ "train_batch_size": 1,
270
+ "trial_name": null,
271
+ "trial_params": null
272
+ }
checkpoint-695/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f7ac894a7270c0a6e6d99e6a8699a4d25d42e4c25c1cfa862f43218ec8b6327
3
+ size 6097
runs/Nov04_23-41-25_4c9315f9425d/events.out.tfevents.1762299691.4c9315f9425d.5540.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9a897036a811cdf5f96724d062fae647f370e53ba0bae4f65f8b46d81b9c278
3
+ size 13357
special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
3
+ size 499723
tokenizer_config.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "add_prefix_space": null,
5
+ "added_tokens_decoder": {
6
+ "0": {
7
+ "content": "<unk>",
8
+ "lstrip": false,
9
+ "normalized": false,
10
+ "rstrip": false,
11
+ "single_word": false,
12
+ "special": true
13
+ },
14
+ "1": {
15
+ "content": "<s>",
16
+ "lstrip": false,
17
+ "normalized": false,
18
+ "rstrip": false,
19
+ "single_word": false,
20
+ "special": true
21
+ },
22
+ "2": {
23
+ "content": "</s>",
24
+ "lstrip": false,
25
+ "normalized": false,
26
+ "rstrip": false,
27
+ "single_word": false,
28
+ "special": true
29
+ }
30
+ },
31
+ "bos_token": "<s>",
32
+ "clean_up_tokenization_spaces": false,
33
+ "eos_token": "</s>",
34
+ "extra_special_tokens": {},
35
+ "legacy": false,
36
+ "model_max_length": 2048,
37
+ "pad_token": "</s>",
38
+ "padding_side": "right",
39
+ "sp_model_kwargs": {},
40
+ "tokenizer_class": "LlamaTokenizer",
41
+ "unk_token": "<unk>",
42
+ "use_default_system_prompt": false
43
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f7ac894a7270c0a6e6d99e6a8699a4d25d42e4c25c1cfa862f43218ec8b6327
3
+ size 6097