File size: 4,642 Bytes
dddb6a8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
# Rollout Playbook

Three approximations of real rollout mechanisms, all buildable on free-tier
infrastructure. Each is a genuine approximation of a specific real
mechanism β€” worth knowing what the paid version adds so you're not
surprised later.

## 1. In-process canary (default recommendation for a single Space)

**Mechanism:** both champion and challenger checkpoints loaded in the same
FastAPI process; a random percentage of requests routes to the challenger.

**How to use it:**
1. Push the new checkpoint to the Hub under a new revision/tag.
2. Set in `.env` (or Space secrets):
   ```
   CHALLENGER_MODEL_ID=<repo>          # usually same as MODEL_ID
   CHALLENGER_REVISION=<new tag>
   CANARY_PCT=10
   ```
3. Restart the Space. `app/model/registry.py` loads both; `app/routing/canary.py`
   routes `CANARY_PCT`% of traffic to the challenger.
4. Every `/predict` response includes `"is_challenger": true/false` and
   `"model_revision"` β€” filter your feature log / dashboards on this to
   compare champion vs. challenger outcomes on identical live traffic.
5. Step up manually: 10 β†’ 25 β†’ 50 β†’ 100, checking confidence distribution
   and cascade-tier mix at each step. Drop back to `CANARY_PCT=0`
   immediately on any anomaly (confidence collapse, latency spike, error
   rate).
6. Once at 100% and stable for a chosen soak period, promote: set
   `MODEL_ID`'s default revision or `CHALLENGER_REVISION` as the new
   `main`, drop `CHALLENGER_MODEL_ID`, redeploy.

**What a real load balancer adds:** true percentage-based routing at the
network layer (not per-process randomness), automatic SLA-triggered
rollback without a human watching dashboards, and routing that doesn't
require both models resident in the same process's memory (relevant once a
challenger checkpoint is large enough that loading two at once strains the
free tier's RAM).

## 2. Blue-green (two Spaces)

**Mechanism:** a staging Space and a production Space; cut over by pointing
consumers at the new Space's URL; rollback is pointing back.

**How to use it:**
1. Duplicate the Space (HF Spaces supports this natively) as `<name>-staging`.
2. Deploy the challenger to staging, run your test suite (`pytest tests/`)
   and a manual smoke test against it directly.
3. Update the chat application's configured inference URL to the staging
   Space once satisfied.
4. Keep the old production Space running, untouched, for a rollback window
   β€” pointing the client config back is the entire rollback procedure.

**What a real blue-green setup adds:** automated traffic cutover (DNS/load
balancer level, not a client config change), health-check-gated automatic
cutover, and zero-downtime cutover for high-QPS services (a client config
change has a brief window where in-flight requests may hit the old URL
during redeploy).

## 3. Shadow deployment

**Mechanism:** a second free Space running the challenger, mirroring a
sample of real requests, logging output without ever returning it to the
user. Useful specifically for validating latency under real chat traffic
before a challenger goes anywhere near production.

**How to use it:**
1. Deploy the challenger to a separate Space (`<name>-shadow`).
2. In the chat application (not this repo β€” this is a client-side
   integration), fire a sampled fraction of real requests to both the
   production Space and the shadow Space, but only use the production
   Space's response.
3. Compare latency and confidence-tier distributions between the two logs.
   This is the cleanest way to validate `scripts/benchmark_latency.py`-style
   numbers under real traffic shape (codec mix, duration distribution)
   rather than synthetic sample clips.

**What this doesn't give you:** shadow traffic never influences what the
user sees, so it can't validate anything about response-system behavior β€”
only model-level latency and prediction distribution.

## Choosing between them

- **In-process canary** β€” cheapest, fastest to iterate, good default for
  comparing model quality on live traffic. Downside: both checkpoints share
  the same process's CPU/RAM.
- **Blue-green** β€” best when you want a hard, instant rollback path and
  don't need gradual traffic ramp.
- **Shadow** β€” best for latency/capacity validation before a challenger is
  trusted with any live traffic at all, especially after an architecture
  change (quantization, ONNX export, trimmed layers per Track 3.4).

In practice: shadow first (validate latency doesn't regress) β†’ in-process
canary (validate prediction quality on live traffic, ramp gradually) β†’
blue-green as the final cutover once canary is at 100% and stable.