Instructions to use eltorio/IDEFICS3_ROCO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use eltorio/IDEFICS3_ROCO with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3") model = PeftModel.from_pretrained(base_model, "eltorio/IDEFICS3_ROCO") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from eltorio/IDEFICS3_ROCO: direct link, hf CLI and curl.
- Browser
- Download file 5.1 kB
-
https://huggingface.co/eltorio/IDEFICS3_ROCO/resolve/6ff184ed91affaac7716e4e786ea9c19e4e4f2df/README.md
- Command line
-
hf download hf://eltorio/IDEFICS3_ROCO@6ff184ed91affaac7716e4e786ea9c19e4e4f2df/README.md
-
curl -L -o README.md https://huggingface.co/eltorio/IDEFICS3_ROCO/resolve/6ff184ed91affaac7716e4e786ea9c19e4e4f2df/README.md
5.1 kB
| license: apache-2.0 | |
| datasets: | |
| - eltorio/ROCO-radiology | |
| language: | |
| - en | |
| - fr | |
| base_model: | |
| - HuggingFaceM4/Idefics3-8B-Llama3 | |
| pipeline_tag: image-text-to-text | |
| library_name: peft | |
| # IDEFICS3_ROCO | |
| [](https://colab.research.google.com/#fileId=https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) | |
| ## Star the project | |
| **If you appreciate my work, please consider giving it a star! 🤩** | |
| **I'm also looking for donations of free GPU time to complete the fine-tuning process.** | |
| **Please contact me if you can help! 🙏** | |
| ## A Fine-tuned Radiology-focused Model based on Hugging Face's Idefics3 Model | |
| This repository contains a fine-tuned version of the Hugging Face [Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) model, built on top of the Meta Llama 3.1 8B architecture. Our model, `IDEFICS3_ROCO`, has been fine-tuned on the [Radiology Objects in Context (ROCO)](https://huggingface.co/datasets/eltorio/ROCO-radiology) dataset, a large-scale medical and multimodal imaging collection. | |
| ## TL;DR | |
| For immediate use, you can load the model directly from Hugging Face: | |
| ```python | |
| from transformers import AutoModelForImageTextToText | |
| model =AutoModelForImageTextToText.from_pretrained("eltorio/IDEFICS3_ROCO") | |
| ``` | |
| ### Model Information | |
| * **Base Model:** Idefics3-8B-Llama3 | |
| * **Fine-tuning Dataset:** Radiology Objects in Context (ROCO) | |
| * **License:** Apache-2.0 | |
| * **Current Status:** Fine-tuning process is finished. Contributions to complete the fine-tuning / vallidation / test processes are welcome! | |
| ### Training Progress Status | |
| * Current checkpoint: 12267 (100% completed) | |
| * Estimated remaining GPU time: 0 hours | |
| * Hardware requirements: T4 GPU with >16GB VRAM | |
| * Last update: november, 12th 2024 | |
| ### Fine-tuning Code | |
| The fine-tuning code is available as a Jupyter Notebook in the [ROCO-radiology dataset repository](https://huggingface.co/datasets/eltorio/ROCO-radiology) on Hugging Face: | |
| * [ROCO-idefics3.ipynb](https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) | |
| The [Junyper Notebook](https://colab.research.google.com/#fileId=https%3A//huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) [](https://colab.research.google.com/#fileId=https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) contains the code to fine-tune the Idefics3-8B-Llama3 model on the ROCO dataset. The fine-tuning process is currently halted at checkpoint 640 (out of 24,000) due to limitations with Colab Free T4 GPU unit. Contributions to complete the fine-tuning process are welcome! | |
| ### Contributions Welcome | |
| If you have the resources to complete the fine-tuning process, we would appreciate your contribution. Please fork this repository, finish the fine-tuning process, and submit a pull request with your updates. | |
| ### Citation | |
| If you use this model in your work, please cite the original Idefics3 model and our fine-tuned model: | |
| * [Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) | |
| * [IDEFICS3_ROCO](https://huggingface.co/eltorio/IDEFICS3_ROCO) | |
| ### Contribution Guide | |
| 1. **Technical Requirements** | |
| * Access to powerful GPU (T4, V100, A100 or equivalent) | |
| * Python environment with PyTorch | |
| * Disk space: ~100GB | |
| 2. **Getting Started** | |
| * Fork the repository | |
| * Resume from checkpoint 12267 | |
| * Follow instructions in [ROCO-idefics3.ipynb](https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) [](https://colab.research.google.com/#fileId=https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/ROCO-idefics3.ipynb) | |
| 3. **Contact** | |
| * For questions: [link to issues/discussions](https://huggingface.co/eltorio/IDEFICS3_ROCO/discussions) | |
| ### Docker Image | |
| A AI training docker image is available for this model. The image and includes all necessary dependencies to run the fine-tuning process. | |
| You need to set the `HF_TOKEN` environment variable to your Hugging Face API token. | |
| You also need to have NVidia Docker container runtime installed. | |
| Finnaly, you need to run the container with GPU support with `--gpus all` option. | |
| The image is available on Docker Hub: | |
| ```bash | |
| export HF_TOKEN=hf_some_token | |
| docker run --gpus all --user=42420:42420 -e HF_TOKEN=$HF_TOKEN -it sctg/roco-idefics3:latest bash -i /start.sh $HF_TOKEN | |
| ``` | |
| The Dockerfile is available in the [IDEFICS_ROCO repository](https://huggingface.co/eltorio/IDEFICS3_ROCO/blob/main/Dockerfile). | |
| ### Acknowledgments | |
| This work was made possible by the [Hugging Face Transformers](https://huggingface.co/) library and the [ROCO-radiology dataset](https://huggingface.co/datasets/eltorio/ROCO-radiology). |