AlessandroAmodioNGI Claude Sonnet 5 commited on
Commit
9d12a0a
·
1 Parent(s): aae4fa7

Disable scale-to-zero on Fly backend to fix mid-request autostop kills

Browse files

Fly's "excess capacity" autostop was killing the backend machine while a
structured-dialogue reply was still generating (a request mostly waiting
on the LLM API looks idle to its scorer), causing real dropped replies on
the paid tier. kill_timeout=60 reduced but didn't eliminate the race;
min_machines_running=1 removes it entirely by never letting a machine
become autostop-eligible while serving traffic.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. fly.toml +8 -10
fly.toml CHANGED
@@ -24,16 +24,14 @@ kill_timeout = 60
24
  [http_service]
25
  internal_port = 7860
26
  force_https = true
27
- # Scale-to-zero: cost-conscious default, ~$0.50-1/month idle instead of
28
- # ~$11/month always-on. Fly's proxy auto-starts a stopped machine on the
29
- # first incoming request (cold start ~90s-2min, due to torch/transformers/
30
- # faiss imports). The hourly proactive-messaging trigger (a separate Fly
31
- # scheduled machine, see project memory) relies on this same auto-start.
32
- # Set min_machines_running = 1 instead (and drop the two auto_* lines) if you'd
33
- # rather pay for an always-on machine and remove the wake-latency entirely.
34
- auto_stop_machines = "stop"
35
- auto_start_machines = true
36
- min_machines_running = 0
37
 
38
  # torch + transformers + sentence-transformers + faiss-cpu are all real weight —
39
  # a 512MB shared machine WILL OOM loading the embedding model. Start here; bump
 
24
  [http_service]
25
  internal_port = 7860
26
  force_https = true
27
+ # Always-on (~$11/month): switched off scale-to-zero on 2026-09-29 after a
28
+ # real correctness bug, not just cold-start latency — Fly's "excess
29
+ # capacity" autostop was killing machines mid-request during longer
30
+ # structured-dialogue turns (the paid tier's core feature), since a request
31
+ # mostly waiting on the LLM API looks idle to Fly's scorer. kill_timeout=60
32
+ # (above) reduced but did not eliminate this; the only full fix is to never
33
+ # let a machine become autostop-eligible while serving traffic.
34
+ min_machines_running = 1
 
 
35
 
36
  # torch + transformers + sentence-transformers + faiss-cpu are all real weight —
37
  # a 512MB shared machine WILL OOM loading the embedding model. Start here; bump