Spaces:
Running
Running
File size: 3,834 Bytes
0d792fe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | # Da Yun blind calibration
This round adds a measurement harness and a separate broad trend signal. The
detailed `judgment.score` remains the structure/rule score; the free trend
preview and life curve prefer `judgment.trend_score`, which is a support-aware,
deduplicated ordinal heuristic. Neither `0..100` value is a probability.
## Protocol
1. Put pseudonymous chart inputs in
`data/accuracy/dayun_charts.template.json`. Do not put names, emails, or
other identifiers in the file.
2. Export scores before labels are collected:
```bash
PYTHONDONTWRITEBYTECODE=1 python dayun_prediction_export.py \
--charts data/accuracy/dayun_charts.template.json \
--output /tmp/dayun-predictions.json
```
3. Copy the period keys into a private label file based on dated evidence. Use
`data/accuracy/dayun_labels.template.json` as the format. Keep the scores
hidden from the person assigning the labels. Each label needs at least one
dated evidence item and a confidence level.
4. Evaluate only after the labels are sealed:
```bash
PYTHONDONTWRITEBYTECODE=1 python dayun_calibration.py \
--labels /path/to/private/dayun-labels.json \
--predictions /tmp/dayun-predictions.json
```
## Reported metrics
- Spearman rank correlation: whether higher model scores order periods better.
- Pairwise ranking accuracy: among periods with different observed labels, how
often the score ordering agrees; a prediction tie counts as half credit.
- Ordinal MAE: distance after mapping labels 0..4 and scores 0..100 to 0..1.
- Constant-50 baseline: the minimum comparator for a supposedly informative
score.
The prediction exporter includes both `score` and `trend_score` so they can be
evaluated separately. A trend score is intended to rank broad ten-year windows;
it must not be read as an event probability or a verdict on a particular
traditional school.
The evaluator reports warnings for small samples and unmatched periods. It
does not declare a rule “accurate” from a threshold. Any weight change must be
pre-registered, evaluated out-of-sample, and compared with the baseline.
## Test resolution
The focused fixtures and disposable mutation runner exercise stem alignment,
weak-storehouse penalty, relationship weight, and score clamping:
```bash
PYTHONDONTWRITEBYTECODE=1 python mutation_test_dayun.py
```
Each mutation must be detected. An intentionally changed weight that leaves
the test suite green is a test failure, not evidence that the rule is safe.
## Data boundary
The repository contains only empty templates and synthetic unit-test labels.
Real labels should remain private or be anonymized before sharing. The label
document rejects common input/identity fields and the prediction exporter does
not write birth details to its output.
## Cross-system discrepancy diagnostic
When another system appears to rate a period differently, run the
validation-only diagnostic before changing the production score:
```bash
PYTHONDONTWRITEBYTECODE=1 python dayun_cross_system_diagnostic.py \
--birth-date 1981-10-04 --birth-time 20:40 --gender female \
--current-age 44 --num-dayun 8 --target 壬寅 --target 癸卯 \
--output /tmp/dayun-cross-system.json
```
The report keeps four tracks separate: the production baseline, a diagnostic
that counts an identical natal branch signature once, a weak-body 扶抑
support track, and both counterfactuals combined. They are hypotheses, not
replacement outputs. For the 1981-10-04 chart the baseline is `壬寅=62` and
`癸卯=20`; the diagnostic exposes `癸卯=35` after relation de-duplication and
`75` on the support track. This spread is evidence of rule sensitivity, not
evidence that any counterfactual is correct. Use the blind label protocol
above to decide whether either change improves out-of-sample ranking.
|