| license: other | |
| license_name: openai-gpt-2-modified-mit | |
| license_link: LICENSE | |
| language: | |
| - en | |
| tags: | |
| - gpt-2 | |
| - tensorflow | |
| pipeline_tag: text-generation | |
| # GPT-2 TensorFlow checkpoints | |
| This repository mirrors OpenAI's original GPT-2 checkpoints for the weight-loading example in [Chapter 5 of Build a Large Language Model (From Scratch)](https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/01_main-chapter-code). | |
| The four folders, `124M`, `355M`, `774M`, and `1558M`, follow OpenAI's directory layout. Each contains the TensorFlow checkpoint, its hyperparameters (`hparams.json`), and the GPT-2 tokenizer files (`encoder.json` and `vocab.bpe`). | |
| ## Loading a checkpoint | |
| The following example uses the Chapter 5 downloader with the `source` argument to load the 124M checkpoint directly from Hugging Face. | |
| ```python | |
| from gpt_download import download_and_load_gpt2 | |
| settings, params = download_and_load_gpt2("124M", models_dir="gpt2", source="hf") | |
| ``` | |
| The function saves the seven files in `gpt2/124M/` and returns the settings and NumPy arrays used in the chapter. Change `"124M"` to load one of the larger models. You can also download the seven files directly from a [model folder on the Hub](https://huggingface.co/rasbt/gpt2-tensorflow/tree/main/124M). | |
| OpenAI released the weights under its [modified MIT license](https://github.com/openai/gpt-2/blob/master/LICENSE). For the same weights converted to PyTorch state dictionaries and safetensors, see [rasbt/gpt2-from-scratch-pytorch](https://huggingface.co/rasbt/gpt2-from-scratch-pytorch). | |