Instructions to use benchang1110/TaiVisionLM-base-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use benchang1110/TaiVisionLM-base-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="benchang1110/TaiVisionLM-base-v2", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("benchang1110/TaiVisionLM-base-v2", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use benchang1110/TaiVisionLM-base-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "benchang1110/TaiVisionLM-base-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benchang1110/TaiVisionLM-base-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/benchang1110/TaiVisionLM-base-v2

SGLang

How to use benchang1110/TaiVisionLM-base-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "benchang1110/TaiVisionLM-base-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benchang1110/TaiVisionLM-base-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "benchang1110/TaiVisionLM-base-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benchang1110/TaiVisionLM-base-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use benchang1110/TaiVisionLM-base-v2 with Docker Model Runner:
```
docker model run hf.co/benchang1110/TaiVisionLM-base-v2
```

benchang1110 commited on Sep 24, 2024

Commit

f2dbe79

verified ·

1 Parent(s): bf249d1

Update README.md

Browse files

Files changed (1) hide show

README.md +9 -9

README.md CHANGED Viewed

@@ -135,9 +135,10 @@ print(outputs)
 * TaiVisionLM-base-v1:
 卡通插圖描繪掛在家門口的標誌，上下方以卡通插圖的方式呈現。
 * TaiVisionLM-base-v2:
-這張圖描繪了一個單詞「SMILE」經典的卡通字體。該字表面是黑白的主要色彩調色板。詞以貫穿其身體的光滑線條字體書寫。該字具有模糊的質感，與單詞形成平滑而簡約的視覺效果。
-字母「「SMILE」」自豪地表示。顯眼的文字是圖片的焦點，吸引觀眾的注意力到其具有簡潔性的方式。該字在白色背景上顯眼地展示，與黑色字體形成鮮明對比。
-圖片中沒有其他物品或文字。字和底部的文字並沒有提供有關詞「「SMILE」具體含義的任何其他背景信息。然而，詞「「SMILE」」的整體設計使其成為這張影像中的焦點，吸引了注意力到其獨特形狀。圖片中沒有其他物品或文字。
 - **Example 2**
@@ -145,10 +146,9 @@ print(outputs)
 * TaiVisionLM-base-v1:
 這是一幅攝影作品，展示了巴黎的鐵塔被水景所環繞
 * TaiVisionLM-base-v2:
-這張照片捕捉了巴黎，法國標誌性的塔樓和人行道景觀的令人驚嘆的景象。塔樓高聳在清澈的藍天沿著舊有大路的背景之上。它是一座高聳入雲的圓頂金屬圖案，高度被分數精確錯量。塔樓由金屬和石頭結構組成，其統一的形狀證明了其歷史意義。
-塔樓東面延伸的人行道向遠處延伸，邀請路人探索它所有的美麗。這條人行道上排列著樹木，它們翠綠的葉片與藍天形成鮮明的對比。它們的存在為場景增添了一抹綠意，為都市景觀增添了一抹自然元素。
-背景中可以看到巴黎城市景觀。各種大小和設計的建築物可以看到，它們矗立在背景中，它們的建築藝術被塔樓和人行道的視野所突顯。天空是一個清澈的藍色，它延伸到遠方，沒有任何雲彩的陰影。
-這張照片是巴黎豐富歷史和現代性的一個見證。塔樓和人行道標誌著這座經典都市的地標，高聳主權人偶的高度及其證據這座城市獨特的信仰。橫跨整張照片的人行道禮貌地介紹了城市的繁忙路線。
 ### Training Procedure
 Since we don't have enough resources to train the model on the whole dataset, we only use 250k image-text pairs for training. The following training hyperparameters are used in feature alignment and task specific training stages respectively:
@@ -158,7 +158,7 @@ Since we don't have enough resources to train the model on the whole dataset, we
 | Data size    | Global Batch Size | Learning Rate | Epochs | Max Length | Weight Decay |
 |--------------|-------------------|---------------|--------|------------|--------------|
-| 250k        | 2               | 5e-5          | 1      | 2048       | 1e-5            |
 We use full-parameter finetuning for the projector and apply LoRA to the language model.
@@ -166,4 +166,4 @@ We will update the training procedure once we have more resources to train the m
 ![metric](metrics.png)
 ### Compute Infrastructure
 - **Feature Alignment**
-  1xV100(32GB), took approximately 12 GPU hours.

 * TaiVisionLM-base-v1:
 卡通插圖描繪掛在家門口的標誌，上下方以卡通插圖的方式呈現。
 * TaiVisionLM-base-v2:
+這張圖片呈現了發人深省的對比。圖片中央，白色文字中的「Smile」以粗體黑色字母書寫。文字略微有些傾斜，為原本靜止的圖片增添了動感。背景是一個鮮明的白色，突顯文字並確立其在圖片中的重要性。
+背景並非僅僅是白色的；它與黑色文字形成鮮明對比，創造出引人注目的視覺效果。文字、背景和形狀和諧合作，每個元素都互相襯托，形成和諧的構圖。
+圖片底部右角有微妙的脊狀邊緣。脊狀的輪廓為圖片增添了一種深度，吸引觀眾的注意力，探索圖片的整體背景。脊狀邊緣與圖片整體的設計相輔相成，增強了節奏和能量氛圍。
+整體而言，這張圖片是一個色彩和形狀的和諧結合，每個元素都經過精心放置，創造出視覺上令人愉悅的構圖。使用黑色、粗體字和微妙的脊狀邊緣增添了神秘感，將其印象擴展到更深層，既引人入勝又引人思考。
 - **Example 2**
 * TaiVisionLM-base-v1:
 這是一幅攝影作品，展示了巴黎的鐵塔被水景所環繞
 * TaiVisionLM-base-v2:
+這幅圖片捕捉到法國著名地標艾菲爾鐵塔的令人驚嘆的景觀。天空呈現明亮的藍色，與周圍的綠意交織，形成令人驚嘆的構圖。這座高聳的拱門塗上淺棕色的艾菲爾鐵塔，自豪地矗立在畫面右側。它旁邊是河流，它的平靜水域反射著上方的藍天。
+在遠處，其他著名地標的蹤影可見，包括一座標誌性的橋樑和一座城堡般的摩天大樓，為場景增添深度和尺度。前景中的樹木增添了一抹綠意，為鐵塔的淺褐色和藍天的色彩提供了清新的對比。
+這張圖片是從水面上觀看艾菲爾鐵塔的角度拍攝的，提供了對整個景觀的鳥瞰視角。這個視角可以全面地觀察到艾菲爾鐵塔及其周圍環境，展現了它的壯麗以及位於其中的生命。這張圖片中沒有任何虛構的內容，所有描述都是基於圖片中可見的元素。</s>
 ### Training Procedure
 Since we don't have enough resources to train the model on the whole dataset, we only use 250k image-text pairs for training. The following training hyperparameters are used in feature alignment and task specific training stages respectively:
 | Data size    | Global Batch Size | Learning Rate | Epochs | Max Length | Weight Decay |
 |--------------|-------------------|---------------|--------|------------|--------------|
+| 1.35M        | 4               | 5e-3          | 1      | 1024       | 0            |
 We use full-parameter finetuning for the projector and apply LoRA to the language model.
 ![metric](metrics.png)
 ### Compute Infrastructure
 - **Feature Alignment**
+  1xV100(32GB), took approximately 45 GPU hours.