Experimental Apertus v1.5 support for llama.cpp
Hi,
I’ve been working on an experimental Apertus v1.5 integration for llama.cpp, currently available on my fork: https://github.com/MichelRosselli/llama.cpp/tree/model/apertus-v1.5
The branch covers text inference and chat support, together with vision and audio input through llama.cpp’s multimodal infrastructure. I have tested the 8B model locally with GGUF weights; 70B support is implemented structurally but has not been validated by running the full 70B weights.
I’m currently coordinating the overlapping output-vocabulary work with the existing upstream llama.cpp effort before opening a draft PR: https://github.com/ggml-org/llama.cpp/pull/26226
If anyone with more experience in llama.cpp would like to take a look, suggest improvements, or help with any part of the integration, feedback and contributions would be very welcome.
Hi, many thanks for working on this, as well as your work on earlier versions. If you have a working Apertus 1.5 8B GGUF with 4-bit quantization, would you mind sharing it? So far I have only found 8-bit quantized versions, and haven't been able to use the HF GGUF my repo tool with this Apertus release due to transformers incompatibility. Thanks again!
Thanks for your interest! I’m currently waiting for the official Apertus 1.5 integration in Transformers to be merged before opening the upstream PR for llama.cpp.
If you’d like to experiment in the meantime, you can use my model/apertus-v1.5 branch. I’ve done some local testing with the 8B model and it seems to work well so far.
For example, to convert and quantize the text model:
git clone -b model/apertus-v1.5 https://github.com/MichelRosselli/llama.cpp.git
cd llama.cpp
python3 -m pip install -r requirements.txt
cmake -B build
cmake --build build --config Release
python3 convert_hf_to_gguf.py \
--remote swiss-ai/Apertus-v1.5-8B \
--outfile Apertus-v1.5-8B-BF16.gguf \
--outtype bf16
./build/bin/llama-quantize \
Apertus-v1.5-8B-BF16.gguf \
Apertus-v1.5-8B-Q4_K_M.gguf \
Q4_K_M
Any testing or feedback would be very welcome!
Hi, many thanks for working on this, as well as your work on earlier versions. If you have a working Apertus 1.5 8B GGUF with 4-bit quantization, would you mind sharing it? So far I have only found 8-bit quantized versions, and haven't been able to use the HF GGUF my repo tool with this Apertus release due to transformers incompatibility. Thanks again!
Thanks for your work! I'm also waiting for a gguf-File. I'm using llama.cpp as Docker with an Intel Arc GPU, so i dont have many working options :-)
(and sry for my english)
@zodiac715 you can already download weights in formats of Q4_K_M of 8B by @Colby or a larger Q8_0 of 8B by @andreasmartin - please let us know how it goes, and stay tuned for official releases.
@MrMeOrYou thanks for the script, that works well!