Instructions to use Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2")
model = AutoModelForCausalLM.from_pretrained("Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2

SGLang

How to use Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2 with Docker Model Runner:
```
docker model run hf.co/Zoyd/DavidAU_Psyonic-Cetacean-V1-20B-Ultra-Quality-Float32-2_5bpw_exl2
```

Configuration Parsing Warning:In config.json: "quantization_config.bits" must be an integer

Exllamav2 quant (exl2 / 2.5 bpw) made with ExLlamaV2 v0.1.3

Other EXL2 quants:

Quant	Model Size	lm_head
2.2	5594 MB	6
2.5	6297 MB	6
3.0	7470 MB	6
3.5	8640 MB	6
3.75	9228 MB	6
4.0	9813 MB	6
4.25	10398 MB	6
5.0	12155 MB	6
6.0	14506 MB	8
6.5	15688 MB	8
8.0	16737 MB	8

Master Files for Ultra High Quality Remasters of "Psyonic-Cetacean" 20B

May "Space Whale" swim in the oceans of the universe forever!

This repo contains the full precision (32 bit) master files for 32 bit upscales created by "DavidAU" of:

https://huggingface.co/DavidAU/Psyonic-Cetacean-Ultra-Quality-20b-GGUF-imatrix

And

https://huggingface.co/DavidAU/Psyonic-Cetacean-Ultra-Quality-20b-GGUF

Please view either repo for details on the remaster's results, and other important infomation.

IMPORTANT NOTES For Maximum Results:

These are "final" result files of the full precision rebuild (including end result merge(s)) minus GGUF and Imatrix level upscaling / adjustments which occuring during "GGUFing" processes.

If you use these to create your own GGUFs, please use "outfile" at F32 for best results. If you use F16 this will reduce the quality by a factor of 2 or higher.

Imatrix processes should use a stable dataset(s) of at least 500 "chunks" or more. If smaller dataset(s) are used this may corrupt or reduce the quality of the Imatrix builds.

Due to the precision remaster there will be "greater" distance between each quant - both non imatrix and imatrix.

IE: The jump in quality, instruction following, "ai brainpower", nuance and output between Q4 and Q5 and likewise Q5 and Q6 will be larger than normal.

Same applies to "Imatrix" quants.

In addition there will also be differences between exact Imatrix and non-imatrix quants especially in terms of "creative uses" and/or uses where there is no "right answer".

Finally, in terms of prompts:

You may find longer prompts are no longer required and/or you may need to reduce the size of prompts in usage. This is a factor due to the precision upscale.

Doing this will ensure the quality of the upscale is maximized in the GGUFs.

/* GPTQers:

Suggest 4bit-Act32 TRUE for best results.

/* EXL2ers:

Suggest Min 4.5 BPW or higher ; 6 BPW and up is especially potent. Strongly suggest you do not reduce layer bit count, as this will affect depth and nuance. The more BPW the better.

Happy GGUFing, EXL2ing, GPTQing, AWQing, HQQing and of course "Merging".

LONG LIVE OPEN SOURCE!

DavidAU

/* Drop me a note when up, and I will link the masters to your repos.

Downloads last month: 1