Update README.md

README.md

@@ -104,7 +104,16 @@ finetuning: 4 hours on 1 NVIDIA H200
 The model outputs embeddings of size 768 that can be compared using cosine similarity
 
 # Data
-TODO: how did we get the data?
+The dataset is built on the official [Debian Repository](https://wiki.debian.org/DebianRepository). To obtain multiple families of
+assembly, we used `apt` to cross-build the same source package to multiple architectures. The idea is that this gives us 
+the same functions for all four architectures. For all four architectures, these functions are compiled with different optimisation:
+O0, O1, O2, O3, Os and manually selected set with advanced instructions further referenced here as Oc for Optimised-Custom.
+This results in a maximum of 24 (6 optimisation * 4 architectures) different functions 
+which are semantically similar. (i.e. they represent the same functionality but are written differently)
+In practise, it was much easier to obtain amd64 functions than riscv64 functions. Thus, not all functions have 24 semantically similar functions.
+
+The dataset is split into a train, test and an evaluation set. This in done on source package, so all binaries and functions belonging to one source package are part of
+either the train or the test set, not both.
 
 **Total amount of functions per architecture**
 
@@ -156,8 +165,8 @@ to reduce token explosion caused by raw numeric values. These numerical values a
 4096 would be 2 ** 12, so #0x2^c, so would 6000, but 9000 would be #0x2^d. As a consequence, small numbers are distinguishable
 in the vocabulary list, but bigger numbers get one batched representation.
 
-Finally, jump tokens are normalised to be relative `JUMP_ADDR_*` tokens (* being the line number of the jump adress in the given 
-function, rather than in the full CFG) CHECK KLOPT DIT?
+Finally, jump tokens are normalised to be relative `JUMP_ADDR_*` tokens. (* being the line number of the jump adress in the given 
+function, rather than in the full CFG)
 
 
 
@@ -171,4 +180,4 @@ but it's also very strict, which is why we used it combined with MRR.
 
 |model|ARM64 mrr|ARM64 acc|AMD64 mrr|AMD64 acc|RISC-V mrr|RISC-V acc|I386 mrr|I386 acc|crosslingual mrr|crosslingual acc|
 |-----|---------|---------|---------|---------|----------|----------|--------|--------|----------------|----------------|
-|Multilingual-ASMBERT|0.761845832197267|0.685|0.8481566008780298|0.803|0.6150729328823104|0.514|0.6600633426585386|0.575|0.6314267393170043|0.53|
+|Multilingual-ASMBERT|0.762|0.685|0.848|0.803|0.615|0.514|0.660|0.575|0.631|0.530|