Commit History

5 days ago

  1. fix gitignore

    Dmitry Beresnev
    94d7e22
  2. exclude config toml

    Dmitry Beresnev
    7a18d61
  3. fix security volnurabilities

    Dmitry Beresnev
    70dee29

6 days ago

  1. fix other security volnurabilities

    Dmitry Beresnev
    6dc320d
  2. fix critical secutity issues

    Dmitry Beresnev
    6471a06

  1. Add inference metrics + bench script, tune threads/batch for 2-vCPU HF Space, backoff on readiness polling

    Dmitry Beresnev
    27c1c3c

  1. Enforce max_tokens upstream, prune request registry, reap dead workers,⁠

    Dmitry Beresnev
    30c6ca2

  1. fix exceptions

    Dmitry Beresnev
    dc32982

  1. fix error format wrapping now applies to /v1/chat/completions and generation stats

    Dmitry Beresnev
    470e737
  2. change timeouts

    Dmitry Beresnev
    2c31416
  3. add token generation speed to ui

    Dmitry Beresnev
    e8080f5
  4. fix request parsing

    Dmitry Beresnev
    3634ca6

  1. Log detailed error bodies for UI failures

    Dmitry Beresnev
    7caa6ba
  2. Fix 400 for llama.cpp web UI completion requests

    Dmitry Beresnev
    677456b
  3. Fix web UI chat by adding buffered SSE fallback

    Dmitry Beresnev
    6379bd0
  4. Fix Docker build for modular llm-manager

    Dmitry Beresnev
    58d70b1
  5. fix description

    Dmitry Beresnev
    952d357
  6. fix build bugs

    Dmitry Beresnev
    acdc6c1
  7. fix readme file

    Dmitry Beresnev
    fe156b2
  8. add new architectural diagram to readme file

    Dmitry Beresnev
    a51a89f
  9. Refactor the C++ LLM manager into modular components, moves Python modules under python/, and keeps the current control-plane behavior intact. The C++ server now has clearer separation for config, model lifecycle, runtime services, request parsing, HTTP helpers, and server routing, while Docker build/runtime paths were updated to compile multiple C++ files and load Python code from the new package folder

    Dmitry Beresnev
    332826f

  1. add auth, token policy, queue scheduler, and cancel flow, etc

    Dmitry Beresnev
    d9ce859

  1. add new endpoint to cancel all processing prompts

    Dmitry Beresnev
    8ef326a

  1. add new build profile

    Dmitry Beresnev
    a97386f
  2. fix encoding

    Dmitry Beresnev
    d211568
  3. fix model config

    Dmitry Beresnev
    057edf0

  1. fix proxied response in llm manager

    Dmitry Beresnev
    53e9f39
  2. fix routing in llm manager

    Dmitry Beresnev
    a4ee76d
  3. add cpp server

    Dmitry Beresnev
    fc0860f

  1. change llm model

    Dmitry Beresnev
    f41621b
  2. change llm model

    Dmitry Beresnev
    4f2dffc
  3. change model to Qwen2.5-Math-7B-Instruct-GGUF

    Dmitry Beresnev
    cca3c7b
  4. change llm model to qwen2 math

    Dmitry Beresnev
    fe7089d
  5. change llm model to mistral

    Dmitry Beresnev
    97d9520
  6. fix dockerfile

    Dmitry Beresnev
    c33410f

  1. change compilation flags

    Dmitry Beresnev
    0e913e4
  2. change compilation flags

    Dmitry Beresnev
    1a4efad
  3. reduce context and batch

    Dmitry Beresnev
    34775a7
  4. fix repo name of model

    Dmitry Beresnev
    dc883f9
  5. fix repo of model

    Dmitry Beresnev
    c7c8563
  6. fix cmd in dockerfile

    Dmitry Beresnev
    0fbce92
  7. fix dockerfile

    Dmitry Beresnev
    c261631
  8. switch to qwen model via cpp server

    Dmitry Beresnev
    9a590ac
  9. Log elapsed time and token rate when the response arrives.⁠

    Dmitry Beresnev
    a8f6b6b
  10. a “slow request” logging, log when a request exceeds the budgeted prompt and gets compacted

    Dmitry Beresnev
    130d9e3
  11. add simple compacting

    Dmitry Beresnev
    6381e7f
  12. fix context window size

    Dmitry Beresnev
    62a5a49
  13. fix payload processing

    Dmitry Beresnev
    e1e4b82
  14. fix logger

    Dmitry Beresnev
    e9b8569
  15. fix app to handle exceptions

    Dmitry Beresnev
    7d65cc9