File size: 1,560 Bytes
bdb1e76 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
license: other
license_name: openai-gpt-2-modified-mit
license_link: LICENSE
language:
- en
tags:
- gpt-2
- tensorflow
pipeline_tag: text-generation
---
# GPT-2 TensorFlow checkpoints
This repository mirrors OpenAI's original GPT-2 checkpoints for the weight-loading example in [Chapter 5 of Build a Large Language Model (From Scratch)](https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/01_main-chapter-code).
The four folders, `124M`, `355M`, `774M`, and `1558M`, follow OpenAI's directory layout. Each contains the TensorFlow checkpoint, its hyperparameters (`hparams.json`), and the GPT-2 tokenizer files (`encoder.json` and `vocab.bpe`).
## Loading a checkpoint
The following example uses the Chapter 5 downloader with the `source` argument to load the 124M checkpoint directly from Hugging Face.
```python
from gpt_download import download_and_load_gpt2
settings, params = download_and_load_gpt2("124M", models_dir="gpt2", source="hf")
```
The function saves the seven files in `gpt2/124M/` and returns the settings and NumPy arrays used in the chapter. Change `"124M"` to load one of the larger models. You can also download the seven files directly from a [model folder on the Hub](https://huggingface.co/rasbt/gpt2-tensorflow/tree/main/124M).
OpenAI released the weights under its [modified MIT license](https://github.com/openai/gpt-2/blob/master/LICENSE). For the same weights converted to PyTorch state dictionaries and safetensors, see [rasbt/gpt2-from-scratch-pytorch](https://huggingface.co/rasbt/gpt2-from-scratch-pytorch).
|