Commit History

  1. Optimize model loading and disable speculative decoding for CPU stability

    Sagar
    3ec513c
  2. Disable flash_attn by default for CPU stability

    Sagar
    1d7babc
  3. Fix GZipMiddleware import/usage spelling

    Sagar
    64ef5ac
  4. Optimize LLM CPU infrastructure: standardize configuration, integrate GzipMiddleware, and optimize KV cache

    Sagar
    560cbb7
  5. Fix generator StopIteration future exception by implementing sentinel wrapper

    Sagar
    828f20f
  6. Optimize LLM CPU inference (eager load, 8-bit cache, flash attention, lock, health endpoint)

    Sagar
    017fb0d

  1. fix: set context to 16k for stability

    e1fcb36
  2. fix: restore 32k context and fix library conflict in requirements.txt

    3c25a03
  3. fix: set context to 5120 and add file verification logic

    82efa78
  4. debug: reduce n_ctx to 2048 and enable verbose logging

    4d6b43d
  5. fix: optimize n_ctx and threads for low-RAM CPU space

    5cf1d7b
  6. fix: correct model repo and filename for Qwen 2.5 1.5B

    f0c4e58
  7. fix: switch to docker sdk to use pre-built wheels and avoid OOM

    7cb03ae
  8. fix: downgrade to python 3.11 to avoid OOM build errors

    eceedb3
  9. docs: add mandatory README.md with space metadata

    1085ad0
  10. fix: add missing llama-cpp-python dependency and disable flash_attn for CPU

    f8e6095
  11. Initial commit for SBIG Dispatch Brain v1

    132db06