File size: 3,578 Bytes
ef7481d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
"""C16B native Auto 0.7 adapter for the official Decision Index Engine API.

Uses the previously evaluated frozen native reader, not the released legacy
plain/Reasoning server. Supplies only state/questions to inference. No launch,
training, upload or power operations. Parallel-run timings are not leaderboard
single-request latency measurements.
"""
import hashlib
import importlib.util
import time
import json
import sys
from pathlib import Path
from decision_index.engines.base import Engine, Unsupported, validate

NATIVE_SHA = '764f3a683becd16b87b9ce9bfc01a5fe483ef26c9a3345a724b82780d73ea73a'
MODEL_SHA = '2859df99ed481592cdc3731fd768bf29ccd63273b2dab623642ad3b38b0e14c4'

class WaldNativeAuto(Engine):
    name = 'wald-v2-native-auto0.7'
    latency = 'Request wall time at declared concurrency; not official serial leaderboard latency.'

    def __init__(self, native_path, temperature_path, temperature_sha256,
                 deadline_epoch, endpoint='http://127.0.0.1:8371',
                 model='04400-c18', max_model_len=131072, reference_root=None, **options):
        super().__init__(**options)
        assert model == '04400-c18', 'This contract binds C16B only'
        path = Path(native_path)
        assert hashlib.sha256(path.read_bytes()).hexdigest() == NATIVE_SHA
        temp = Path(temperature_path)
        assert hashlib.sha256(temp.read_bytes()).hexdigest() == temperature_sha256
        self.deadline_epoch = float(deadline_epoch) if deadline_epoch is not None else None
        assert self.deadline_epoch is None or self.deadline_epoch > time.time()
        if reference_root:
            root = Path(reference_root)
            pins = json.loads((root / 'source-pins.json').read_text())['files_sha256']
            for rel, digest in pins.items():
                assert hashlib.sha256((root / rel).read_bytes()).hexdigest() == digest
            if str(root) not in sys.path:
                sys.path.insert(0, str(root))
        spec = importlib.util.spec_from_file_location('wald_frozen_native', path)
        self.native = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(self.native)
        self.client, self.sov = self.native.make_client(
            endpoint, model, int(max_model_len),
            time.monotonic() + self.deadline_epoch - time.time() if self.deadline_epoch is not None else None)
        self.table = self.sov.load_tables(temp).get('A')
        self.model = model
        self.provenance = {'model': model, 'model_sha256': MODEL_SHA,
            'native_reader_sha256': NATIVE_SHA,
            'temperature_sha256': temperature_sha256,
            'policy': 'native Auto 0.7; untempered A gate; at most 512 thought tokens; original option order',
            'prompt_format': 'repeat_state_plain in native chat',
            'wide': 'frozen onepass knockout; every supplied option receives probability',
            'context_limit': int(max_model_len), 'no_truncation': True,
            'new_training': False}

    def __call__(self, state, questions):
        if self.deadline_epoch is not None and time.time() >= self.deadline_epoch - 120:
            raise TimeoutError('Registered evaluation deadline reached')
        request = {'state': state, 'questions': questions}
        try:
            response = self.native.answer_task(self.client, self.sov,
                {'request': request, 'suite': 'di'}, self.table, 0.7, 512)
        except self.sov.Capacity as exc:
            raise Unsupported(str(exc)) from exc
        validate(questions, response)
        return response, response