Commit ·
fbe2b25
1
Parent(s): 2c61d75
Update README.md
Browse files
README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
BioTinyBERT is the result of training the [TinyBERT](https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D) model in a continual learning fashion for 200k training steps using a total batch size of 192 on the PubMed dataset.
|
| 3 |
|
| 4 |
# Initialisation
|
| 5 |
-
We initialise our model with the pre-trained checkpoints of the [TinyBERT](https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D) model available on
|
| 6 |
|
| 7 |
# Architecture
|
| 8 |
This model uses 4 hidden layers with a hidden dimension size and an embedding size of 768 resulting in a total of 15M parameters.
|
|
|
|
| 2 |
BioTinyBERT is the result of training the [TinyBERT](https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D) model in a continual learning fashion for 200k training steps using a total batch size of 192 on the PubMed dataset.
|
| 3 |
|
| 4 |
# Initialisation
|
| 5 |
+
We initialise our model with the pre-trained checkpoints of the [TinyBERT](https://huggingface.co/huawei-noah/TinyBERT_General_4L_312D) model available on Huggingface.
|
| 6 |
|
| 7 |
# Architecture
|
| 8 |
This model uses 4 hidden layers with a hidden dimension size and an embedding size of 768 resulting in a total of 15M parameters.
|