Spaces:
Sleeping
Sleeping
Commit ·
9d12a0a
1
Parent(s): aae4fa7
Disable scale-to-zero on Fly backend to fix mid-request autostop kills
Browse filesFly's "excess capacity" autostop was killing the backend machine while a
structured-dialogue reply was still generating (a request mostly waiting
on the LLM API looks idle to its scorer), causing real dropped replies on
the paid tier. kill_timeout=60 reduced but didn't eliminate the race;
min_machines_running=1 removes it entirely by never letting a machine
become autostop-eligible while serving traffic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fly.toml
CHANGED
|
@@ -24,16 +24,14 @@ kill_timeout = 60
|
|
| 24 |
[http_service]
|
| 25 |
internal_port = 7860
|
| 26 |
force_https = true
|
| 27 |
-
#
|
| 28 |
-
#
|
| 29 |
-
#
|
| 30 |
-
#
|
| 31 |
-
#
|
| 32 |
-
#
|
| 33 |
-
#
|
| 34 |
-
|
| 35 |
-
auto_start_machines = true
|
| 36 |
-
min_machines_running = 0
|
| 37 |
|
| 38 |
# torch + transformers + sentence-transformers + faiss-cpu are all real weight —
|
| 39 |
# a 512MB shared machine WILL OOM loading the embedding model. Start here; bump
|
|
|
|
| 24 |
[http_service]
|
| 25 |
internal_port = 7860
|
| 26 |
force_https = true
|
| 27 |
+
# Always-on (~$11/month): switched off scale-to-zero on 2026-09-29 after a
|
| 28 |
+
# real correctness bug, not just cold-start latency — Fly's "excess
|
| 29 |
+
# capacity" autostop was killing machines mid-request during longer
|
| 30 |
+
# structured-dialogue turns (the paid tier's core feature), since a request
|
| 31 |
+
# mostly waiting on the LLM API looks idle to Fly's scorer. kill_timeout=60
|
| 32 |
+
# (above) reduced but did not eliminate this; the only full fix is to never
|
| 33 |
+
# let a machine become autostop-eligible while serving traffic.
|
| 34 |
+
min_machines_running = 1
|
|
|
|
|
|
|
| 35 |
|
| 36 |
# torch + transformers + sentence-transformers + faiss-cpu are all real weight —
|
| 37 |
# a 512MB shared machine WILL OOM loading the embedding model. Start here; bump
|