Instructions to use anilsuthar2004/qwen-bot with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use anilsuthar2004/qwen-bot with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf anilsuthar2004/qwen-bot # Run inference directly in the terminal: llama cli -hf anilsuthar2004/qwen-bot
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf anilsuthar2004/qwen-bot # Run inference directly in the terminal: llama cli -hf anilsuthar2004/qwen-bot
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf anilsuthar2004/qwen-bot # Run inference directly in the terminal: ./build/bin/llama cli -hf anilsuthar2004/qwen-bot
Use Docker
docker model run hf.co/anilsuthar2004/qwen-bot
- Llama new
- LM Studio
- Jan
- Ollama
How to use anilsuthar2004/qwen-bot with Ollama:
ollama run hf.co/anilsuthar2004/qwen-bot
- Unsloth Desktop
- Docker Model Runner
How to use anilsuthar2004/qwen-bot with Docker Model Runner:
docker model run hf.co/anilsuthar2004/qwen-bot
- Lemonade
How to use anilsuthar2004/qwen-bot with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull anilsuthar2004/qwen-bot
Run and chat with the model
lemonade run user.qwen-bot-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
|
Download llama.cpp/docs/backend/snapdragon/developer.md from anilsuthar2004/qwen-bot: direct link, hf CLI and curl.
- Browser
- Download file 18.5 kB
-
https://huggingface.co/anilsuthar2004/qwen-bot/resolve/main/llama.cpp/docs/backend/snapdragon/developer.md
- Command line
-
hf download hf://anilsuthar2004/qwen-bot/llama.cpp/docs/backend/snapdragon/developer.md
-
curl -L -o developer.md https://huggingface.co/anilsuthar2004/qwen-bot/resolve/main/llama.cpp/docs/backend/snapdragon/developer.md
18.5 kB
| # Hexagon backend developer details | |
| ## Backend libraries | |
| The Hexagon backend consists of two parts: | |
| - `libggml-hexagon` | |
| This is the regular CPU-side GGML backend library, either shared or statically linked. | |
| - `libggml-htp-vNN` | |
| This is the NPU-side (HTP stands for Hexagon Tensor Processor) shared library that contains the Op dispatcher and kernels. | |
| The correct library is selected automatically at runtime based on the HW version. | |
| Here is an example of the build artifacts: | |
| ``` | |
| ~/src/llama.cpp$ ls -l pkg-adb/llama.cpp/lib/libggml* | |
| pkg-adb/llama.cpp/lib/libggml-base.so | |
| pkg-adb/llama.cpp/lib/libggml-cpu.so | |
| pkg-adb/llama.cpp/lib/libggml-hexagon.so <<< CPU library | |
| pkg-adb/llama.cpp/lib/libggml-htp-v73.so <<< HTP op/kernels for Hexagon v73 | |
| pkg-adb/llama.cpp/lib/libggml-htp-v75.so | |
| pkg-adb/llama.cpp/lib/libggml-htp-v79.so | |
| pkg-adb/llama.cpp/lib/libggml-htp-v81.so | |
| ``` | |
| ## Memory buffers | |
| The Hexagon NPU backend takes advantage of Snapdragon unified memory where all DDR buffers are accessible by CPU, GPU, and NPU. | |
| The NPU has dedicated tightly-coupled memory called VTCM (Vector Tightly-Coupled Memory). VTCM is used for intermediate data (such as | |
| dynamically quantized activations) and streaming buffers (chunks of weight and activation tensors fetched via DMA). | |
| ## Large model handling | |
| Hexagon NPU sessions have a 32-bit virtual address space window of around 3.5GB. | |
| In llama.cpp/GGML, each Hexagon session is mapped to a single GGML backend device (e.g., `HTP0:0`, `HTP0:1`, etc. when using | |
| `GGML_HEXAGON_DEVICES`, or `HTP0`, `HTP1` in legacy mode). | |
| To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps buffers: | |
| - Buffers are allocated in shared DDR (RPCMEM) via file descriptors (`fastrpc_mmap` using `FASTRPC_MAP_FD_DELAYED`). | |
| - Pinned buffers (such as KV cache and active compute buffers) remain mapped throughout execution. | |
| - Inactive weight buffers are dynamically mapped into the NPU session via `HAP_mmap()` during batch buffer preparation | |
| (`prep_op_bufs()` in `htp/main.c`) and unmapped via `htp_iface_munmap()` when no longer needed by the active batch. | |
| - This dynamic sliding window allows a single NPU session to execute models that exceed the 3.5GB window. | |
| Alternatively, users can partition and split the model across multiple virtual sessions or physical NPUs using layer-splitting, | |
| tensor-splitting, or row-splitting modes. For user-facing execution modes and examples, see the | |
| [Snapdragon user guide](README.md#multi-device-execution-modes). | |
| ## Op and Kernel Development Guidelines | |
| Writing high-performance operators for Hexagon requires following specific guidelines. | |
| ### DDR -> DMA -> VTCM Execution Pipeline | |
| - Strongly prefer the `DDR -> DMA -> VTCM -> compute (HVX/HMX) -> VTCM -> DMA -> DDR` data flow. | |
| - Direct HVX reads/writes from/to DDR are less efficient and should only be used as a fallback. | |
| - The DMA queue is a strict FIFO where operations must be pushed and popped in strict order. | |
| - Follow the pipelined multi-buffering sequence properly (typically 2x to 16x buffering) so every push has a corresponding pop: | |
| 1. In the prologue, push initial DDR -> VTCM transfers to prime the pipeline. | |
| 2. In the loop body, wait for buffer N via DMA pop, launch HVX/HMX compute on buffer N, push VTCM -> DDR writeback of result N, | |
| and push DDR -> VTCM prefetch of buffer N+2. | |
| 3. In the epilogue, pop all remaining in-flight transfers to drain the pipeline. | |
| - Because every push must be matched by a pop, `dma_queue_flush()` is not required when the pipeline sequence is followed | |
| properly. Flushing is only used in rare exceptions where a batch of operations is pushed without individual pops. | |
| - Use the DMA queue interface from [`dma-queue.h`](../../../ggml/src/ggml-hexagon/htp/dma-queue.h) | |
| (`dma_queue_push_ddr_to_vtcm`, `dma_queue_pop`, `dma_queue_push_vtcm_to_ddr`). | |
| See [`cumsum-ops.c`](../../../ggml/src/ggml-hexagon/htp/cumsum-ops.c) and | |
| [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c) for reference implementations. | |
| ### Avoid Scalar Reads and Writes to VTCM | |
| - Access VTCM data using DMA transfers or HVX/HMX vector instructions rather than scalar reads and writes. | |
| ### Avoid Scalar Division in Inner Loops | |
| - Hexagon cores do not have hardware division instructions. | |
| - For recurring divisions across iterations or threads, use `fastdiv` from | |
| [`hex-fastdiv.h`](../../../ggml/src/ggml-hexagon/htp/hex-fastdiv.h) with precomputed divisors (such as | |
| `octx->ctx->mdev.count_div` or `octx->n_threads_div`). | |
| - Do not call `init_fastdiv_values()` for single-use divisions; use standard compiler division (`/`) instead. | |
| ### Host-Side Precomputation via `kernel_params` | |
| - Precompute tensor shapes, strides, scale conversions, tiling layouts, and validation checks on the host CPU during graph | |
| preparation in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp). | |
| - Pack precomputed parameters into the operator's fixed `kernel_params` structure in `htp_op_node` (such as | |
| `htp_mm_kernel_params`, `htp_unary_kernel_params`, `htp_fa_kernel_params`, `htp_get_rows_kernel_params`). | |
| - The NPU executes directly using `octx->kernel_params` without redundant runtime metadata extraction or validation. | |
| - **Strict Host-Kernel Alignment**: | |
| - Verify that parameters calculated by the host CPU are strictly honored by the NPU kernel. | |
| - Ensure the kernel does not ignore host-computed fields (for example, falling back to `octx->n_threads` instead of | |
| using `kparams->n_threads`, or ignoring precomputed `tasks_per_thread` and chunk counts). | |
| - Both human developers and coding agents must audit both sides of the interface: ensure fields populated in `kernel_params` | |
| in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp) are actively and consistently utilized by the | |
| corresponding operator entry point and worker threads in `htp/*-ops.c`. | |
| ### Tracing Instrumentation | |
| - All kernels must include trace events for performance profiling and timeline visualization in Perfetto | |
| ([`hex-profile.h`](../../../ggml/src/ggml-hexagon/htp/hex-profile.h)). | |
| - Surround compute sections with `htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)` and | |
| `htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)`. | |
| - Use specific event types for major phases: | |
| - `HTP_TRACE_EVT_HVX_COMP`: Vector compute execution. | |
| - `HTP_TRACE_EVT_DMA`: DMA transfer wait or poll cycles. | |
| - `HTP_TRACE_EVT_FENCE`: Multi-device fence barrier synchronization. | |
| - `HTP_TRACE_EVT_L2FLUSH`: L2 cache cleaning operations. | |
| - Pass meaningful progress metrics (such as row index, chunk index, or token index) in the 16-bit `info` parameter. | |
| ### Work Queue and Threading | |
| - Distribute parallel work across NPU worker threads using the thread pool work queue: | |
| ```c | |
| work_queue_run(ctx->work_queue, worker_func, &op_ctx, n_threads); | |
| ``` | |
| - Keep worker functions independent and re-entrant. Worker threads should only operate on their designated chunk of rows or elements. | |
| ### Avoid Redundant Defensive NULL Checks | |
| - Do not add defensive NULL checks or assertions for internal framework pointers or required graph operands and outputs. | |
| Internal pointers include `ctx`, `octx`, local context structs like `*ctx`, `kparams`, and worker callback `data`. | |
| - These pointers are architectural invariants during kernel execution and host-side graph preparation. | |
| Graph compute receives allocated nodes with valid required `node->src[N]` and `node->data` pointers. | |
| - Do not turn an invariant violation into an unsupported operation or missed fusion. | |
| Checks such as `if (!octx || !octx->ctx)` clutter the code, obscure intent, and hide upstream errors. | |
| - **Distinction**: `octx->src[N]` pointers *can* be NULL by design and must be checked when optional. | |
| Examples include attention masks, optional bias or weights in fused kernels, and frequency factors. | |
| ### Multiline Macro Formatting | |
| - Keep trailing backslashes in multiline `#define` macros cleanly aligned to a consistent column. | |
| - Avoid trailing whitespace after macro backslashes. | |
| - Use [`scripts/snapdragon/ggml-hexagon-align-macros.py`](../../../scripts/snapdragon/ggml-hexagon-align-macros.py) to inspect, diff, | |
| or automatically align macro definitions across Hexagon kernel sources: | |
| ```bash | |
| # Check for misaligned macros | |
| python3 scripts/snapdragon/ggml-hexagon-align-macros.py ggml/src/ggml-hexagon/htp/ | |
| # Fix misaligned macros in-place | |
| python3 scripts/snapdragon/ggml-hexagon-align-macros.py --fix ggml/src/ggml-hexagon/htp/ | |
| ``` | |
| ### Binary Inspection and Spill Analysis | |
| Use [`scripts/snapdragon/ggml-hexagon-inspect.py`](../../../scripts/snapdragon/ggml-hexagon-inspect.py) to audit Hexagon binaries for register | |
| spills, unexpected float promotions, or disassembly: | |
| - Always verify that compute kernels have zero in-loop vector spills (`--spills --strict`) and no float promotions (`--promotions`). | |
| - Avoid excessive loop unrolling (`#pragma unroll`), which increases register pressure and causes spills. | |
| ```bash | |
| # Check for vector and scalar register spills | |
| python3 scripts/snapdragon/ggml-hexagon-inspect.py --spills --strict --func "^compute_" | |
| # Check for float promotions | |
| python3 scripts/snapdragon/ggml-hexagon-inspect.py --promotions --func "^compute_" | |
| # Disassemble with annotated loops and spill markers | |
| python3 scripts/snapdragon/ggml-hexagon-inspect.py --disasm compute_same_shape_div_f32 | |
| # Resolve crash addresses to function symbols and lines | |
| python3 scripts/snapdragon/ggml-hexagon-inspect.py --addr2line 0x51a30 0x5ba54 | |
| ``` | |
| ## Multi-Device Partitioning (mdev) | |
| Multi-device (mdev) mode enables row-level tensor parallel execution across multiple physical NPU cores or virtual NPU | |
| sessions. | |
| ### 128-Byte Cache Line Alignment | |
| - Shared tensor buffers reside in DDR (RPCMEM) with a 128-byte cache line granularity | |
| (`HEX_L2_LINE_SIZE` = 128 bytes, `HTP_TENSOR_MDEV_LINE_SIZE`). | |
| - **Rule**: Multi-device work partitions must align destination write regions to 128-byte cache line boundaries so distinct | |
| devices never share or overwrite the same cache line. | |
| ### Partitioning Helpers in `htp-tensor.h` | |
| Common partitioning logic is factored into reusable inline helpers in | |
| [`htp-tensor.h`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h): | |
| 1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L67): | |
| Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes: | |
| ``` | |
| rows_per_chunk = 128 / hex_gcd_u32(row_size, 128) | |
| ``` | |
| If row stride `nb[1]` is already a multiple of 128 bytes, `rows_per_chunk = 1`. | |
| Returns `false` if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout, | |
| or non-128-byte aligned outer strides). | |
| 2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94): | |
| Calculates the per-device work range `struct htp_tensor_mdev_range { uint32_t start; uint32_t count; }` given | |
| `total_units`, `units_per_chunk`, `mdev_idx`, `mdev_count`, and the precomputed `mdev_count_div`. | |
| Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers | |
| single-device fallback when partitioning is unsafe. | |
| ### Row-Partitioned Operators | |
| For row-wise operators | |
| (such as activations in [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c), | |
| binary ops in [`binary-ops.c`](../../../ggml/src/ggml-hexagon/htp/binary-ops.c), | |
| unary ops in [`unary-ops.c`](../../../ggml/src/ggml-hexagon/htp/unary-ops.c), and | |
| sameshape copies in [`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)): | |
| ```c | |
| const uint32_t total_rows = ne01 * ne02 * ne03; | |
| const size_t dst_row_size = dst->ne[0] * elem_size; | |
| uint32_t row_start = 0; | |
| uint32_t nrows = total_rows; | |
| if (octx->ctx->mdev.count > 1) { | |
| uint32_t rows_per_chunk = 0; | |
| htp_tensor_mdev_rows_per_chunk(dst, elem_size, (uint32_t) dst_row_size, &rows_per_chunk); | |
| const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition( | |
| total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div); | |
| row_start = range.start; | |
| nrows = range.count; | |
| } | |
| if (nrows == 0) { | |
| return HTP_STATUS_OK; | |
| } | |
| ``` | |
| ### Element-Partitioned Operators | |
| For flat element-wise operations (such as reshape copies in | |
| [`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)): | |
| - Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (`elems_per_line = (elem_size == 4) ? 32 : 64`). | |
| - Requires strict 1D contiguity: | |
| [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L28) | |
| and 128-byte aligned destination pointer | |
| [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L47). | |
| - If contiguous and aligned, pass `elems_per_line` to | |
| [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94); | |
| otherwise pass 0 to trigger Device 0 fallback. | |
| ### Single-Device Fallback (Device 0) | |
| - Fallback to Device 0 (`mdev.idx == 0`) when partitioning would cause cache line tearing or when work cannot be evenly distributed. | |
| - Triggers: | |
| 1. Destination tensor cannot be safely partitioned (`rows_per_chunk == 0` or non-contiguous/unaligned buffer). | |
| 2. Total aligned chunks < `mdev_count`. | |
| - Device 0 processes the entire tensor `[0, total_units)`. | |
| - Devices 1 ... N-1 receive `count = 0` and return `HTP_STATUS_OK` immediately. | |
| ### Flatten Outer Dimensions Globally | |
| - **Never partition solely on `ne01` (dimension 1).** | |
| - Partitioning only on `ne01` repeats the device boundary across every 2D slice (`ne02`, `ne03`). If each 2D slice is small, | |
| false sharing occurs repeatedly throughout the tensor. | |
| - Always flatten outer dimensions globally: `total_rows = ne01 * ne02 * ne03` and partition once across the combined row space. | |
| ### Stateless Starting Coordinates | |
| - Do not use incremental state variables across slices that assume the thread or device starts at index 0. | |
| - Precompute starting multidimensional coordinates at `r = row_start` (or `e = elem_start`) once using `fastdiv`. | |
| - In inner loops, step base pointers directly (`ptr += stride`) or reset/wrap coordinates explicitly (`if (++i01 == ne01) { ... }`). | |
| ### Clean Range Encapsulation | |
| - Initialize single-device default ranges at declaration: | |
| ```c | |
| uint32_t row_start = 0; | |
| uint32_t nrows = total_rows; | |
| ``` | |
| - Encapsulate all multi-device logic inside `if (octx->ctx->mdev.count > 1)`. If the block is omitted or compiled out, | |
| the operator runs standard single-device execution untouched. | |
| - Do not propagate `mdev_` prefixes to worker functions or context structs. Worker threads are device-agnostic and | |
| should only receive standard range parameters (`ctx.row_start`, `ctx.nrows`). | |
| - In worker threads, calculate row intervals using standard arithmetic: | |
| ```c | |
| const uint32_t ir0 = ctx->row_start + dr * ith; | |
| const uint32_t ir1 = MIN(ir0 + dr, ctx->row_start + ctx->nrows); | |
| ``` | |
| In single-device mode (`row_start == 0`), this naturally simplifies to `dr * ith` and `MIN(ir0 + dr, ctx->nrows)` with zero overhead. | |
| ## Multi-Device Synchronization | |
| Multi-device execution synchronizes worker sessions across devices using explicit barriers and tensor cache flushing. | |
| ### Synchronization Fence Protocol | |
| Multi-device execution synchronizes worker sessions through atomic fence slots and barriers defined in | |
| [`htp-fence.h`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h): | |
| ``` | |
| [NPU Session 0] [NPU Session 1] | |
| | | | |
| (Input Prep) (Input Prep) | |
| | | | |
| Pre-Op Barrier ----------------------------- Pre-Op Barrier | |
| (mdev_sync_fence) (mdev_sync_fence) | |
| | | | |
| Kernel Execution Kernel Execution | |
| (Output Slice 0) (Output Slice 1) | |
| | | | |
| Tensor Cache Flush Tensor Cache Flush | |
| (htp_tensor_flush_all) (htp_tensor_flush_all) | |
| | | | |
| Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier | |
| (htp_mdev_group_barrier) (htp_mdev_group_barrier) | |
| | | | |
| Return Response to Host Return Response to Host | |
| ``` | |
| ### Atomic Fence Slots and Cache Invalidation | |
| - Fence synchronization operates on dedicated RPCMEM shared memory mapped across all participating sessions (`ctx->mdev.fence_base`). | |
| - Each device owns a dedicated 128-byte cache-line aligned fence slot: | |
| ```c | |
| atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx); | |
| ``` | |
| - **Writing to fence ([`htp_fence_write`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L18))**: | |
| Stores `seq` and `status`, issues a `syncht` thread synchronization barrier, and flushes/invalidates the line | |
| using `Q6_dccleaninva_A(fence)`. | |
| - **Reading from peer fence ([`htp_fence_read`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L26))**: | |
| Executes `Q6_dccleaninva_A(fence)` and `syncht` before reading atomic values to ensure fresh data from DDR. | |
| ### Deterministic Monotonic Sequence Numbers | |
| - Barrier fences use monotonically increasing sequence numbers: | |
| ```c | |
| const uint32_t seq = ++ctx->mdev.fence_seq; | |
| ``` | |
| - Comparing sequence numbers with signed arithmetic `(int32_t)(peer_seq - seq) >= 0` prevents race conditions or | |
| misaligned barrier arrivals across iterations. | |
| - If any peer reports an error status (`peer_status > HTP_STATUS_OK`), the barrier propagates the error and unblocks immediately. | |
| ### Tensor Cache Flush and Pipeline Completion | |
| - In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue. | |
| - Use [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) to flush specific dirty tensors back to DDR: | |
| - [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) flushes only modified tensor address ranges, | |
| ensuring peer devices and the host CPU observe consistent data in DDR. | |
| - Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed. | |