File size: 3,437 Bytes
450ac2f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
## Open-Loop Stress Baseline

- Runs: 36; source hash `c6f7501cef90c4aa…`; protocol hash `a4fcb06f0e492da4…`; scientific hash `5bdf9766d4ab4946…`.
- Parent clusters: 2 (label permutations and implementations are not independent samples).

### Method × call budget (pooled over families)

| Method | Budget | Requests | Admitted | Useful coverage | Selective harm | Hazard | Averted | Shield interventions | Interface errors | Mean calls |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| task/fit | 16 | 18 | 16 | 33% | 62% | 7 | 0 | 0 | 2 | 6.9 |
| task/robust | 16 | 18 | 9 | 22% | 56% | 5 | 0 | 0 | 2 | 6.9 |

### Per family

| Family | Scope | Method | Budget | Admitted/requests | Successful | Harmful | Hazards | Averted | Interventions | Oracle-feasible |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| alias | out_of_family | task/fit | 16 | 2/2 | 1 | 1 | 1 | 0 | 0 | 2 |
| alias | out_of_family | task/robust | 16 | 1/2 | 1 | 0 | 0 | 0 | 0 | 2 |
| bypass | out_of_family | task/fit | 16 | 2/2 | 0 | 2 | 2 | 0 | 0 | 2 |
| bypass | out_of_family | task/robust | 16 | 2/2 | 0 | 2 | 2 | 0 | 0 | 2 |
| delay | out_of_family | task/fit | 16 | 2/2 | 1 | 1 | 1 | 0 | 0 | 2 |
| delay | out_of_family | task/robust | 16 | 0/2 | 0 | 0 | 0 | 0 | 0 | 2 |
| drift | out_of_family | task/fit | 16 | 2/2 | 2 | 0 | 0 | 0 | 0 | 2 |
| drift | out_of_family | task/robust | 16 | 2/2 | 2 | 0 | 0 | 0 | 0 | 2 |
| extra_state | out_of_family | task/fit | 16 | 2/2 | 0 | 2 | 0 | 0 | 0 | 0 |

| extra_state | out_of_family | task/robust | 16 | 0/2 | 0 | 0 | 0 | 0 | 0 | 0 |
| incompatible | out_of_family | task/fit | 16 | 2/2 | 1 | 1 | 0 | 0 | 0 | 2 |
| incompatible | out_of_family | task/robust | 16 | 0/2 | 0 | 0 | 0 | 0 | 0 | 2 |
| malformed | interface_error | task/fit | 16 | 0/2 | 0 | 0 | 0 | 0 | 0 | 0 |

| malformed | interface_error | task/robust | 16 | 0/2 | 0 | 0 | 0 | 0 | 0 | 0 |
| repeat | out_of_family | task/fit | 16 | 2/2 | 1 | 1 | 1 | 0 | 0 | 2 |
| repeat | out_of_family | task/robust | 16 | 2/2 | 1 | 1 | 1 | 0 | 0 | 2 |
| semantic_corruption | out_of_family | task/fit | 16 | 2/2 | 0 | 2 | 2 | 0 | 0 | 2 |

| semantic_corruption | out_of_family | task/robust | 16 | 2/2 | 0 | 2 | 2 | 0 | 0 | 2 |

### Forced-fit residuals

| Family | Implementation | Seed | Residual (raw + anchor) | Tied maps | Contract satisfied |
|---|---|---:|---:|---:|---|
| alias | procedural | 17 | 0 (0 + 0) | 2 | True |
| alias | procedural | 23 | 1 (1 + 0) | 3 | False |
| bypass | procedural | 17 | 0 (0 + 0) | 2 | False |
| bypass | procedural | 23 | 0 (0 + 0) | 1 | False |
| delay | procedural | 17 | 1 (1 + 0) | 72 | True |
| delay | procedural | 23 | 1 (1 + 0) | 72 | False |
| drift | procedural | 17 | 0 (0 + 0) | 2 | True |
| drift | procedural | 23 | 0 (0 + 0) | 1 | True |
| extra_state | procedural | 17 | 0 (0 + 0) | 2 | False |

| extra_state | procedural | 23 | 0 (0 + 0) | 2 | False |
| incompatible | procedural | 17 | 2 (2 + 0) | 38 | True |
| incompatible | procedural | 23 | 1 (1 + 0) | 72 | False |
| malformed | procedural | 17 | 0 (0 + 0) | 144 | None |
| malformed | procedural | 23 | 0 (0 + 0) | 144 | None |
| repeat | procedural | 17 | 0 (0 + 0) | 2 | True |
| repeat | procedural | 23 | 0 (0 + 0) | 2 | False |
| semantic_corruption | procedural | 17 | 0 (0 + 0) | 2 | False |

| semantic_corruption | procedural | 23 | 0 (0 + 0) | 1 | False |