README.md

---
license: eupl-1.2
tags:
- assembly
- arm64
- amd64
- risc-v
- i386
---

# Model Description
The ASMTransformers model is a machine learning model that calculates embeddings of assembly functions to enable comparison and similarity
operations. Instruction  set architectures (ISAs) ARM, AMD64, RISC-V and i386 assembly are supported.  The aim is to aid in the process of reverse engineering.

# Status
Not actively maintained after publishing

# Relevant links
* [ASMtransformers on GitHub](https://github.com/NetherlandsForensicInstitute/asmtransformers)

# Version
2026-08-06

# Usage
To use this model, use [inference.py in our repository](https://github.com/NetherlandsForensicInstitute/asmtransformers/blob/main/asmtransformers/README.md#inference).

# Intended use
The model has been trained and tested to be used for similarity search of assembly code. It has not been trained/tested on any other languages than arm64, amd64, risc-v or i386, 
nor has it been tested on other downstream tasks.

# Architecture description

The model is a Bert-base architecture with 12 layers, 12 attention heads and a hidden size of 768. It uses a vocabulary of size 6161 derived from a custom assembler architecture-aware tokenizer.

The model is trained in two phases - a masked language modeling (MLM) pretraining phase and a fine-tuning phase for function similarity.

The MLM pre-training phase uses a loss function that includes jump target prediction (see [Wang et al.](https://arxiv.org/pdf/2205.12713)). 
This pretraining was perfomed using a masking probability of 0.4 that was determined through ablation experiments. This is in line with the findings of [Wettig et al., 2023](https://aclanthology.org/2023.eacl-main.217.pdf).

We perform the finetuning phase using [Batched Semi Hard Triplet Loss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#batchsemihardtripletloss) on batches of size 1024. 
The training set for this phase includes triplets that are formed using versions of the same function compiled for different instruction set architectures. 

**Estimated training time:**

pretraining: 38 hours on 4 NVIDIA H200s  
finetuning: 4 hours on 1 NVIDIA H200


**Full architecture:**

  ```
  [
    {
      "idx": 0,
      "name": "0",
      "path": "0_ASMTransformerModule",
      "type": "asmtransformers.models.asmsentencebert.ASMTransformerModule"
    },
    {
      "idx": 1,
      "name": "1",
      "path": "1_Pooling",
      "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
    }
  ]
```

```
  {
    "architectures": [
      "BertForMaskedLM"
    ],
    "attention_probs_dropout_prob": 0.1,
    "classifier_dropout": null,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 768,
    "initializer_range": 0.02,
    "intermediate_size": 3072,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "bert",
    "num_attention_heads": 12,
    "num_hidden_layers": 12,
    "pad_token_id": 0,
    "position_embedding_type": "absolute",
    "torch_dtype": "float32",
    "transformers_version": "4.12.5",
    "type_vocab_size": 2,
    "use_cache": true,
    "vocab_size": 6161
  }
```

**Pre-training parameters used:**

```
  {
    "epochs": 19, 
    "eval_steps": 10000, 
    "batch_size": 512, 
    "gradient_accumulation_steps": 1, 
    "mlm_prob": 0.4, 
    "bf16": True, 
    "tf32": True
  }
```

# Output
The model outputs embeddings of size 768.

# Data
The dataset is derived from the official [Debian Repository](https://wiki.debian.org/DebianRepository). We used the distribution's build and packaging system `apt` to
cross-compile source packages for multiple ISAs and optimisation levels. We used the standard optimisation levels supported by the distribution's compiler, O0, O1, O2, O3 and Os.
We also added an optimization level that includes vectorization and cryptographic instructions that we refer to as Optimized-custom, or Oc.
The binary files from each source package are disassembled using Ghidra and the resulting control-flow graphs (CFGs) are exported.

This results in a maximum of 24 (6 optimisation levels and 4 ISAs) potentially CFGs which are semantically identical, i.e. they represent the same functionality.
However, because not all source packages can be built for all ISAs and because some optimisation levels result in identical code, the actual number is often lower than that.

The dataset is split into a train, test and an evaluation set. This in done on source package basis, so all binaries and functions belonging to one source package are part of
either the training set or the test set, not both.

**Total amount of functions per architecture**

| Architecture | # functions|
|--------------|------------|
| amd64        |8 202 164|
| i386	        |4 868 531|
| arm64	       |4 421 768|
| riscv64	     |3 791 434 |
-------------------------

**Amount of functions per architecture per optimization**

|Architecture | Optimization | # functions |
|-------------|--------------|-------------|
|amd64 |	O0 | 	1762442    |
|amd64 |	O1 | 	1448046    |
|amd64 |	O2 | 	1403180    |
|amd64 |	O3 |	1308199 |
|amd64 |	Oc |	899892 |
|amd64 |	Os |	1380405 |
|arm64 |	O0 |	918402 |
|arm64 |	O1 |	824291 |
|arm64 |	O2 |	790679 |
|arm64 |	O3 |	741057 |
|arm64 |	Oc |	331736 |
|arm64 |	Os |	815603 |
|i386 |	O0 |	1081475 |
|i386 |	O1 |	873128 |
|i386 |	O2 |	845943 |
|i386 |	O3 |	816186 |
|i386 |	Oc |	412503 |
|i386 |	Os |	839296 |
|riscv64 |	O0 |	845154 |
|riscv64 |	O1 |	689670 |
|riscv64 |	O2 |	670651 |
|riscv64 |	O3 |	641428 |
|riscv64 |	Oc |	259691 |
|riscv64  |	Os |	684840 |
----------------------------

# Preprocessing
Several preprocessing steps have been taken. CFGs are processed into a flat token list using ISA aware tokenizers.
Immediate operands and addresses have been normalized to the nearest power-of-two reduce token explosion caused by raw numeric values.
Finally, target addresses of jumps, conditional branches or call instructions are normalised to fucntion-relative `JUMP_ADDR_*` tokens. 

# Performance
We report two performance metrics: Mean Reciprocal Rank (MRR) and Accuracy@1. 

These metrics are suitable for evaluating tasks where a single example is retrieved from a large collection. 
Accuracy@1 shows how many times the positive example ranked first. It is a useful metric, but it's also very strict, which is why we used it combined with MRR.

|model|ARM64 mrr|ARM64 acc|AMD64 mrr|AMD64 acc|RISC-V mrr|RISC-V acc|I386 mrr|I386 acc|crosslingual mrr|crosslingual acc|
|-----|---------|---------|---------|---------|----------|----------|--------|--------|----------------|----------------|
|Multilingual-ASMBERT|0.762|0.685|0.848|0.803|0.615|0.514|0.660|0.575|0.631|0.530|