Smanj/Ex_transformer

Fine-Tuning GPT-2 and Using Transformer-Explainer for Analysis

Project Overview This project involves fine-tuning the GPT-2 model with the assigned language and analyzing its behavior using the Transformer-Explainer package. The objective is to observe how the fine-tuned model processes language inputs and generates outputs, gaining insights into its inner workings.

Objectives

1. Fine-tune GPT-2 with a specific language dataset.
2. Use the Transformer-Explainer package to analyze the behavior of the custom-trained GPT-2 model.
3. Generate 10 examples and observe the model’s performance using the explainer.

Steps to Complete the Project

1. Fine-Tuning GPT-2 Used HuggingFace Transformers to fine-tune the GPT-2 model on finnish language. Prepared the dataset ensuring it represents the linguistic characteristics and diversity of the assigned language.
2. Running Transformer-Explainer Clone the Transformer-Explainer GitHub repository. Integrated the custom GPT-2 model with the explainer package.
3. Observed and analyzed the output on: Attention patterns. Token relationships.

Resources

1. Transformer-Explainer GitHub Repository
2. HuggingFace Transformers Documentation
3. Dataset in the assigned language