Update README.md
Browse files
README.md
CHANGED
|
@@ -30,19 +30,61 @@ nor has it been tested on other downstream tasks.
|
|
| 30 |
|
| 31 |
# Architecture description
|
| 32 |
TODO: get model architecture like on the model card of the previous model
|
|
|
|
| 33 |
Don't forget to describe that during pretraining, the --mlm-prob parameter has been set to 0.4, in accordance to [paper](INSERT LINK)
|
|
|
|
| 34 |
Parameters used: epochs=19, eval_steps=10000, batch_size=512, gradient_accumulation_steps=1, mlm_prob=0.4, bf16=True, tf32=True
|
| 35 |
|
| 36 |
add estimated time it has cost to train:
|
| 37 |
-
pretraining:
|
| 38 |
finetuning: 4 hours on 1 NVIDIA H200
|
| 39 |
|
| 40 |
# Output
|
| 41 |
-
The model outputs embeddings that can be compared using cosine similarity
|
| 42 |
|
| 43 |
# Data
|
| 44 |
TODO: how did we get the data?
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
# Preprocessing
|
| 47 |
Several preprocessing steps have been taken, that differ slightly between architectures.
|
| 48 |
TODO: update to same as architecture.md in repo
|
|
@@ -50,6 +92,5 @@ TODO: update to same as architecture.md in repo
|
|
| 50 |
# Performance
|
| 51 |
Performance was measured in two ways: Mean Reciprocal Rank (MRR) and Accuracy@1
|
| 52 |
|
| 53 |
-
-------------------------------------------------------------------------------------------------------------------------
|
| 54 |
|model|ARM64 mrr|ARM64 acc|AMD64 mrr|AMD64 acc|RISC-V mrr|RISC-V acc|I386 mrr|I386 acc|crosslingual mrr|crosslingual acc|
|
| 55 |
-
-------------------------------------------------------------------------------------------------------------
|
|
|
|
| 30 |
|
| 31 |
# Architecture description
|
| 32 |
TODO: get model architecture like on the model card of the previous model
|
| 33 |
+
|
| 34 |
Don't forget to describe that during pretraining, the --mlm-prob parameter has been set to 0.4, in accordance to [paper](INSERT LINK)
|
| 35 |
+
|
| 36 |
Parameters used: epochs=19, eval_steps=10000, batch_size=512, gradient_accumulation_steps=1, mlm_prob=0.4, bf16=True, tf32=True
|
| 37 |
|
| 38 |
add estimated time it has cost to train:
|
| 39 |
+
pretraining: 33 hours on TODO!
|
| 40 |
finetuning: 4 hours on 1 NVIDIA H200
|
| 41 |
|
| 42 |
# Output
|
| 43 |
+
The model outputs embeddings of size 768 that can be compared using cosine similarity
|
| 44 |
|
| 45 |
# Data
|
| 46 |
TODO: how did we get the data?
|
| 47 |
|
| 48 |
+
Total amount of functions per architecture
|
| 49 |
+
|
| 50 |
+
| Architecture | # functions|
|
| 51 |
+
|--------------|------------|
|
| 52 |
+
| amd64 |8 202 164|
|
| 53 |
+
| i386 |4 868 531|
|
| 54 |
+
| arm64 |4 421 768|
|
| 55 |
+
| riscv64 |3 791 434 |
|
| 56 |
+
-------------------------
|
| 57 |
+
|
| 58 |
+
Amount of functions per architecture per optimization
|
| 59 |
+
|
| 60 |
+
|Architecture | Optimization | # functions |
|
| 61 |
+
|-------------|--------------|-------------|
|
| 62 |
+
|amd64 | O0 | 1762442 |
|
| 63 |
+
|amd64 | O1 | 1448046 |
|
| 64 |
+
|amd64 | O2 | 1403180 |
|
| 65 |
+
|amd64 | O3 | 1308199 |
|
| 66 |
+
|amd64 | Oc | 899892 |
|
| 67 |
+
|amd64 | Os | 1380405 |
|
| 68 |
+
|arm64 | O0 | 918402 |
|
| 69 |
+
|arm64 | O1 | 824291 |
|
| 70 |
+
|arm64 | O2 | 790679 |
|
| 71 |
+
|arm64 | O3 | 741057 |
|
| 72 |
+
|arm64 | Oc | 331736 |
|
| 73 |
+
|arm64 | Os | 815603 |
|
| 74 |
+
|i386 | O0 | 1081475 |
|
| 75 |
+
|i386 | O1 | 873128 |
|
| 76 |
+
|i386 | O2 | 845943 |
|
| 77 |
+
|i386 | O3 | 816186 |
|
| 78 |
+
|i386 | Oc | 412503 |
|
| 79 |
+
|i386 | Os | 839296 |
|
| 80 |
+
|riscv64 | O0 | 845154 |
|
| 81 |
+
|riscv64 | O1 | 689670 |
|
| 82 |
+
|riscv64 | O2 | 670651 |
|
| 83 |
+
|riscv64 | O3 | 641428 |
|
| 84 |
+
|riscv64 | Oc | 259691 |
|
| 85 |
+
|riscv64 | Os | 684840 |
|
| 86 |
+
----------------------------
|
| 87 |
+
|
| 88 |
# Preprocessing
|
| 89 |
Several preprocessing steps have been taken, that differ slightly between architectures.
|
| 90 |
TODO: update to same as architecture.md in repo
|
|
|
|
| 92 |
# Performance
|
| 93 |
Performance was measured in two ways: Mean Reciprocal Rank (MRR) and Accuracy@1
|
| 94 |
|
|
|
|
| 95 |
|model|ARM64 mrr|ARM64 acc|AMD64 mrr|AMD64 acc|RISC-V mrr|RISC-V acc|I386 mrr|I386 acc|crosslingual mrr|crosslingual acc|
|
| 96 |
+
|-----|---------|---------|---------|---------|----------|----------|--------|--------|----------------|----------------|
|