Update README.md

README.md

@@ -30,6 +30,26 @@ nor has it been tested on other downstream tasks.
 
 # Architecture description
 
+This model has been trained similarly to [Wang et al.](https://arxiv.org/pdf/2205.12713). The first step is Masked Language Modelling, where we set the 
+--mlm-prob parameter to 0.4 in accordance to [Wettig et al., 2023](https://aclanthology.org/2023.eacl-main.217.pdf). We have done some experiments with different mlm-probs 
+(namely, 0.15, 0.3 and 0.4) and found that 0.4 yielded the best results. 
+
+One remarkable aspect of [Wang et al.](https://arxiv.org/pdf/2205.12713)'s paper, was that they came up with the concept of "Jump target prediction". As part of the 
+Masked Language Modelling procedure, the model also has to predict jump targets: whenever there is a jump in the code, the model shouldn't only predict it to be a 
+jump instruction, but it should also predict to which line of code it's supposed to jump. The idea is that this leads to better understanding of the long-term relations
+in the code, and as Wang et al yielded good results, we have adopted it. 
+
+Finetuning happens based on triplet loss: given three functions, of which two are similar (see also Data) and one is deviant, the model has to pick which two functions 
+are similar to each other. We used [Batched Semi Hard Triplet Loss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#batchsemihardtripletloss) to 
+ensure that the training task is not too easy. 
+
+
+**Estimated training time:**
+
+pretraining: 38 hours on 4 NVIDIA H200s  
+finetuning: 4 hours on 1 NVIDIA H200
+
+
 **Full architecture:**
 
   ```
@@ -49,7 +69,7 @@ nor has it been tested on other downstream tasks.
   ]
 ```
 
-**Parameters used:**
+**Pretraining parameters used:**
 
 ```
   {
@@ -92,13 +112,6 @@ nor has it been tested on other downstream tasks.
   }
 ```
 
-Note that during pretraining, the --mlm-prob parameter has been set to 0.4. We have done some experiments with different mlm-probs (namely, 0.15, 0.3 and 0.4) and found that
-0.4 yielded the best results, in accordance to [Wettig et al., 2023](https://aclanthology.org/2023.eacl-main.217.pdf).
-
-**Estimated time it has cost to train:**
-
-pretraining: 38 hours on 4 NVIDIA H200s  
-finetuning: 4 hours on 1 NVIDIA H200
 
 # Output
 The model outputs embeddings of size 768 that can be compared using cosine similarity