Instructions to use hkondle/CapstoneMainModel with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use hkondle/CapstoneMainModel with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="hkondle/CapstoneMainModel")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("hkondle/CapstoneMainModel") model = AutoModelForMultimodalLM.from_pretrained("hkondle/CapstoneMainModel", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -11,20 +11,7 @@ tags:
|
|
| 11 |
|
| 12 |
# VisAble diagram captioner
|
| 13 |
|
| 14 |
-
BLIP fine-tuned on AI2D-Caption
|
| 15 |
-
low-vision students. The vision encoder was frozen and only the text decoder was trained, using a
|
| 16 |
-
cross entropy loss that up-weights AI2D entity labels by 5.0x.
|
| 17 |
|
| 18 |
Trained for 5 epochs, batch size 8, learning rate 5e-05.
|
| 19 |
|
| 20 |
-
```python
|
| 21 |
-
from transformers import BlipForConditionalGeneration, BlipProcessor
|
| 22 |
-
from PIL import Image
|
| 23 |
-
|
| 24 |
-
processor = BlipProcessor.from_pretrained("hkondle/CapstoneMainModel")
|
| 25 |
-
model = BlipForConditionalGeneration.from_pretrained("hkondle/CapstoneMainModel")
|
| 26 |
-
|
| 27 |
-
image = Image.open("diagram.png").convert("RGB")
|
| 28 |
-
inputs = processor(images=image, return_tensors="pt")
|
| 29 |
-
print(processor.decode(model.generate(**inputs, max_length=256, num_beams=4)[0], skip_special_tokens=True))
|
| 30 |
-
```
|
|
|
|
| 11 |
|
| 12 |
# VisAble diagram captioner
|
| 13 |
|
| 14 |
+
BLIP fine-tuned on AI2D-Caption dataset to produce part-to-whole descriptions for blind students.
|
|
|
|
|
|
|
| 15 |
|
| 16 |
Trained for 5 epochs, batch size 8, learning rate 5e-05.
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|