Instructions to use vtava/Tiny-LLM-PDelta3-GDN2-InputRoute with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vtava/Tiny-LLM-PDelta3-GDN2-InputRoute with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="vtava/Tiny-LLM-PDelta3-GDN2-InputRoute")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("vtava/Tiny-LLM-PDelta3-GDN2-InputRoute", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vtava/Tiny-LLM-PDelta3-GDN2-InputRoute with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/vtava/Tiny-LLM-PDelta3-GDN2-InputRoute

SGLang

How to use vtava/Tiny-LLM-PDelta3-GDN2-InputRoute with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vtava/Tiny-LLM-PDelta3-GDN2-InputRoute",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use vtava/Tiny-LLM-PDelta3-GDN2-InputRoute with Docker Model Runner:
```
docker model run hf.co/vtava/Tiny-LLM-PDelta3-GDN2-InputRoute
```

Tiny-LLM-PDelta3-GDN2-InputRoute / test_summary.csv

vtava

Upload conv4_gdn2_inputroute_f96 from TinyCeNN Tiny-LLM compatibility benchmark

c21da77 verified about 2 months ago

Raw History Blame Contribute Delete

3.15 kB

candidate,candidate_nll,candidate_ppl,ci95_high,ci95_low,context,delta_nll,documents,ppl_ratio,selected_efficiency,selected_quality,state_bytes,state_vs_transformer_fp16,transformer_kv_bytes_fp16,transformer_nll,transformer_ppl,verdict
conv4_pdelta_f96,4.367208196057214,78.82326513019325,0.27285944753222996,0.2233854134877523,256,0.24691674444410536,36,1.2800725351912405,False,False,19392,0.197265625,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
conv4_pdelta_f96,4.33470802836948,76.30267742000811,0.27299132612016463,0.23108539316389295,512,0.2514344784948561,36,1.2858686452546269,False,False,19392,0.0986328125,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
conv4_pdelta_f96,4.323580231931475,75.45830348172126,0.28276246786117554,0.24677499135335287,1024,0.26384247011608547,36,1.3019230883530288,False,False,19392,0.04931640625,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
conv4_channel_decay_f96,4.3716577887535095,79.17477801964439,0.27827538384331596,0.22689927286571926,256,0.2513663371404012,36,1.2857810274087214,False,False,19392,0.197265625,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
conv4_channel_decay_f96,4.335691842767927,76.37778203113514,0.27496619357003105,0.23200341727998522,512,0.25241829289330375,36,1.2871343238366606,False,False,19392,0.0986328125,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
conv4_channel_decay_f96,4.325859294997321,75.63047383321955,0.2848877575662401,0.2489156590567695,1024,0.2661215331819322,36,1.3048936369262525,False,False,19392,0.04931640625,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
conv4_gdn2_f96,4.340679241551293,76.75965998556238,0.24643041690190634,0.19631489117940268,256,0.22038778993818495,36,1.2465600403109762,True,False,20736,0.2109375,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
conv4_gdn2_f96,4.314126392205556,74.7482942195864,0.25244420766830444,0.21143459611468846,512,0.23085284233093262,36,1.259673855140897,True,False,20736,0.10546875,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
conv4_gdn2_f96,4.3035025464163885,73.95838324448775,0.2602580918206109,0.22875974575678507,1024,0.24376478460099962,36,1.276044149953423,True,False,20736,0.052734375,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
conv4_gdn2_inputroute_f96,4.336302816867828,76.42446113617407,0.24170496066411337,0.19230133295059204,256,0.21601136525472006,36,1.241116484525499,False,True,20736,0.2109375,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
conv4_gdn2_inputroute_f96,4.310102687941657,74.44813347342465,0.24796597162882486,0.20758027500576443,512,0.22682913806703356,36,1.254615483599082,False,True,20736,0.10546875,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
conv4_gdn2_inputroute_f96,4.299878226386176,73.69081955579698,0.25628456804487443,0.22531060377756754,1024,0.24014046457078722,36,1.2714277283293163,False,True,20736,0.052734375,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin

	candidate,candidate_nll,candidate_ppl,ci95_high,ci95_low,context,delta_nll,documents,ppl_ratio,selected_efficiency,selected_quality,state_bytes,state_vs_transformer_fp16,transformer_kv_bytes_fp16,transformer_nll,transformer_ppl,verdict
	conv4_pdelta_f96,4.367208196057214,78.82326513019325,0.27285944753222996,0.2233854134877523,256,0.24691674444410536,36,1.2800725351912405,False,False,19392,0.197265625,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
	conv4_pdelta_f96,4.33470802836948,76.30267742000811,0.27299132612016463,0.23108539316389295,512,0.2514344784948561,36,1.2858686452546269,False,False,19392,0.0986328125,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
	conv4_pdelta_f96,4.323580231931475,75.45830348172126,0.28276246786117554,0.24677499135335287,1024,0.26384247011608547,36,1.3019230883530288,False,False,19392,0.04931640625,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
	conv4_channel_decay_f96,4.3716577887535095,79.17477801964439,0.27827538384331596,0.22689927286571926,256,0.2513663371404012,36,1.2857810274087214,False,False,19392,0.197265625,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
	conv4_channel_decay_f96,4.335691842767927,76.37778203113514,0.27496619357003105,0.23200341727998522,512,0.25241829289330375,36,1.2871343238366606,False,False,19392,0.0986328125,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
	conv4_channel_decay_f96,4.325859294997321,75.63047383321955,0.2848877575662401,0.2489156590567695,1024,0.2661215331819322,36,1.3048936369262525,False,False,19392,0.04931640625,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
	conv4_gdn2_f96,4.340679241551293,76.75965998556238,0.24643041690190634,0.19631489117940268,256,0.22038778993818495,36,1.2465600403109762,True,False,20736,0.2109375,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
	conv4_gdn2_f96,4.314126392205556,74.7482942195864,0.25244420766830444,0.21143459611468846,512,0.23085284233093262,36,1.259673855140897,True,False,20736,0.10546875,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
	conv4_gdn2_f96,4.3035025464163885,73.95838324448775,0.2602580918206109,0.22875974575678507,1024,0.24376478460099962,36,1.276044149953423,True,False,20736,0.052734375,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin
	conv4_gdn2_inputroute_f96,4.336302816867828,76.42446113617407,0.24170496066411337,0.19230133295059204,256,0.21601136525472006,36,1.241116484525499,False,True,20736,0.2109375,98304,4.120291451613109,61.577186419687706,worse_than_declared_margin
	conv4_gdn2_inputroute_f96,4.310102687941657,74.44813347342465,0.24796597162882486,0.20758027500576443,512,0.22682913806703356,36,1.254615483599082,False,True,20736,0.10546875,196608,4.083273549874623,59.339402746614645,worse_than_declared_margin
	conv4_gdn2_inputroute_f96,4.299878226386176,73.69081955579698,0.25628456804487443,0.22531060377756754,1024,0.24014046457078722,36,1.2714277283293163,False,True,20736,0.052734375,393216,4.059737761815389,57.959109994107415,worse_than_declared_margin