Create README.md

README.md

@@ -0,0 +1,26 @@
+---
+datasets:
+- oscar
+language:
+- da
+widget:
+  - text: Der var engang
+---
+
+# What is this?
+
+GPT-2 model (small version, 124 M parameters) for Danish text generation. 
+
+# Model training
+
+The model is trained using the Danish part of the [oscar dataset](https://huggingface.co/datasets/oscar) ('unshuffled_deduplicated_da') and a context length of 1024 tokens.
+
+The model is initilized from the English [GPT-2 small model](https://huggingface.co/gpt2) with new word token embeddings created for Danish using [WECHSEL](https://github.com/CPJKU/wechsel).
+
+Initially, only the word token embeddings are trained using 50.000 samples. Finally, the whole model is trained using 1.000.000 samples.
+
+Model training is carried out on a 8 GB GPU.
+
+# Notes
+
+This is a pre-trained model, for optimal performance it should be finetuned for new tasks.