Lamapi commited on
Commit
e187f2f
·
verified ·
1 Parent(s): 2f51feb

Publish Next-Echo model

Browse files
Files changed (10) hide show
  1. .gitattributes +1 -0
  2. README.md +61 -0
  3. echo_core.py +145 -0
  4. embedding.safetensors +3 -0
  5. evaluation.json +653 -0
  6. inference.py +62 -0
  7. model.json +0 -0
  8. nlrt.py +226 -0
  9. preview.png +3 -0
  10. selftest.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ preview.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-3.0
3
+ library_name: pytorch
4
+ tags:
5
+ - audio
6
+ - query-by-example
7
+ - sound-event-detection
8
+ - next-lab
9
+ ---
10
+
11
+ # Next Echo
12
+
13
+ Next Echo finds **every moment a given example sound occurs** in a longer recording — a clap, a knock, a bark — with no
14
+ fixed class list. A small CNN embeds 1-second log-mel windows; the recording is scanned against the example.
15
+ It is compared with plain log-mel similarity; the final score fuses both after per-recording normalisation.
16
+
17
+ **Space:** https://huggingface.co/spaces/thelamapi/Next-Echo — record a 2-second example with your microphone (or pick one), choose a recording, and listen to each hit. Runs in the browser.
18
+
19
+ ## Results on constructed 30 s test recordings (ESC-50 fold 5; 40 recordings per cell)
20
+
21
+ Event F1 (±0.5 s) / false alarms per minute — log-mel baseline vs Next Echo (better F1 in bold):
22
+
23
+ | Classes / condition | Log-mel similarity | Next Echo (learned) |
24
+ |---|---:|---:|
25
+ | seen/clean | **0.676** / 2.15 | 0.619 / 3.25 |
26
+ | seen/noisy | 0.606 / 1.45 | **0.630** / 2.30 |
27
+ | seen/overlap | **0.788** / 0.80 | 0.749 / 1.05 |
28
+ | seen/absent | – / 4.00 | – / 4.95 |
29
+ | unseen/clean | 0.617 / 2.10 | **0.658** / 3.30 |
30
+ | unseen/noisy | 0.667 / 1.10 | **0.717** / 2.05 |
31
+ | unseen/overlap | **0.790** / 0.85 | 0.767 / 1.70 |
32
+ | unseen/absent | – / 3.65 | – / 5.60 |
33
+ | seen/all-present | **0.690** / 1.47 | 0.662 / 2.20 |
34
+ | unseen/all-present | 0.691 / 1.35 | **0.712** / 2.35 |
35
+
36
+ *unseen* = the 10 classes never used in training (clapping, door_wood_knock, rooster, church_bells, glass_breaking, keyboard_typing, siren, cow, sneezing, can_opening). *absent* recordings contain no target, so
37
+ only false alarms count. Thresholds were chosen on validation recordings built from fold 4.
38
+
39
+ ![Example timeline and benchmark](preview.png)
40
+
41
+ ## Method
42
+ * Audio: 16 kHz mono, 512-point STFT (hop 256), 64 mel bands (50–8000 Hz), log.
43
+ * Embedding: 4 conv blocks + linear → 64-d, supervised contrastive loss (τ = 0.1) on random 1-s crops of 960 clips from 40 classes, with gain, mixing and noise augmentation, 5000 steps.
44
+ * Training windows mimic the test conditions: real waveform segments mixed with another sound and brown noise (0–30 dB SNR), alongside clean segments like the user's example.
45
+ * Query: the loudest 1-s windows of the example are embedded and averaged. Recording: windows every 0.256 s.
46
+ * Score: per-recording robust normalisation (median/MAD), then Next Echo = α·z(learned) + (1−α)·z(log-mel) with α = 0.75 chosen on validation recordings
47
+ (options α ∈ {1, 0.75, 0.5}). Consecutive windows above the threshold form one detection.
48
+ * Baseline: cosine between per-band mean/std of log-mel windows, raw or normalised — whichever validated better (raw).
49
+ * The browser computes the same log-mel (own FFT) and CNN with `nlrt.js`; Node parity on 3 real snippets: max abs difference 7.105427357601002e-15.
50
+
51
+ ## Usage
52
+ ```bash
53
+ python inference.py --query clap.wav --recording scene.wav # 16 kHz mono 16-bit WAV
54
+ python inference.py --selftest
55
+ ```
56
+
57
+ ## Limitations and license
58
+ * Recordings are mixtures of real clips, not naturally recorded scenes; real rooms add reverberation and far more varied backgrounds.
59
+ * Very short or very quiet examples give weak queries. Sounds that resemble each other (knock vs footsteps) cause false alarms.
60
+ * **Non-commercial:** ESC-50 is CC BY-NC 3.0, so this model, its demo audio and the Space are for non-commercial use only.
61
+ * Trained on NVIDIA L4; profile full. Failure cases are listed in `evaluation.json`.
echo_core.py ADDED
@@ -0,0 +1,145 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Next Echo: find where a short example sound occurs inside a longer recording.
2
+
3
+ Shared by the notebook, inference.py and (ported) web/core.js. 16 kHz mono audio.
4
+ * logmel: 512-point STFT (periodic Hann, hop 256) -> 64 mel bands -> log
5
+ * windows of 64 frames (~1.04 s) every 16 frames (~0.256 s)
6
+ * learned: CNN embedding (nlrt) of each window, cosine to the query embedding
7
+ * baseline: cosine between per-band mean/std statistics of log-mel windows
8
+ * robust_z: per-recording normalisation (median / MAD) so thresholds adapt to each recording's background
9
+ * final_scores: Next Echo score = alpha * z(learned) + (1 - alpha) * z(log-mel); baseline = log-mel (raw or z), chosen on validation
10
+ * detect: threshold the score curve, merge consecutive windows into intervals
11
+ """
12
+ import math
13
+
14
+ import numpy as np
15
+
16
+ from nlrt import forward
17
+
18
+ SR, N_FFT, HOP, N_MELS, WIN, STEP = 16000, 512, 256, 64, 64, 16
19
+ WIN_SEC = (WIN - 1) * HOP / SR + N_FFT / SR
20
+ STEP_SEC = STEP * HOP / SR
21
+
22
+
23
+ def mel_filterbank(n_mels=N_MELS, n_fft=N_FFT, sr=SR, fmin=50.0, fmax=8000.0):
24
+ hz2mel = lambda f: 2595.0 * math.log10(1.0 + f / 700.0) # noqa: E731
25
+ mel2hz = lambda m: 700.0 * (10 ** (m / 2595.0) - 1.0) # noqa: E731
26
+ mels = np.linspace(hz2mel(fmin), hz2mel(fmax), n_mels + 2)
27
+ hz = np.array([mel2hz(m) for m in mels])
28
+ bins = np.arange(n_fft // 2 + 1) * sr / n_fft
29
+ fb = np.zeros((n_mels, len(bins)))
30
+ for i in range(n_mels):
31
+ lo, c, hi = hz[i], hz[i + 1], hz[i + 2]
32
+ fb[i] = np.maximum(0, np.minimum((bins - lo) / (c - lo), (hi - bins) / (hi - c)))
33
+ return fb
34
+
35
+
36
+ def logmel(x, fb):
37
+ x = np.asarray(x, dtype=np.float64)
38
+ if len(x) < N_FFT:
39
+ x = np.concatenate([x, np.zeros(N_FFT - len(x))])
40
+ n = 1 + (len(x) - N_FFT) // HOP
41
+ w = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(N_FFT) / N_FFT)
42
+ idx = np.arange(N_FFT)[None, :] + HOP * np.arange(n)[:, None]
43
+ spec = np.fft.rfft(x[idx] * w, axis=1)
44
+ power = spec.real ** 2 + spec.imag ** 2
45
+ return np.log(fb @ power.T + 1e-6) # (n_mels, frames)
46
+
47
+
48
+ def pad_frames(lm, n):
49
+ if lm.shape[1] >= n:
50
+ return lm
51
+ return np.concatenate([lm, np.full((lm.shape[0], n - lm.shape[1]), lm.min())], axis=1)
52
+
53
+
54
+ def window_starts(n_frames):
55
+ return list(range(0, max(n_frames - WIN, 0) + 1, STEP))
56
+
57
+
58
+ def embed_windows(layers, lm, mu, sd, starts):
59
+ X = np.stack([(lm[:, s:s + WIN] - mu) / sd for s in starts])[:, None]
60
+ E = forward(layers, X)
61
+ return E / np.maximum(np.linalg.norm(E, axis=1, keepdims=True), 1e-12)
62
+
63
+
64
+ def stats_windows(lm, mu, sd, starts):
65
+ out = []
66
+ for s in starts:
67
+ z = (lm[:, s:s + WIN] - mu) / sd
68
+ v = np.concatenate([z.mean(axis=1), z.std(axis=1)])
69
+ v = v - v.mean()
70
+ out.append(v / max(float(np.linalg.norm(v)), 1e-12))
71
+ return np.stack(out)
72
+
73
+
74
+ def query_starts(lm):
75
+ """Windows of the query that carry the sound: mean log energy within 1.5 of the loudest window."""
76
+ lm = pad_frames(lm, WIN)
77
+ starts = list(range(0, lm.shape[1] - WIN + 1, 8))
78
+ e = [float(lm[:, s:s + WIN].mean()) for s in starts]
79
+ top = max(e)
80
+ return lm, [s for s, v in zip(starts, e) if v >= top - 1.5]
81
+
82
+
83
+ def query_vectors(model, q_audio):
84
+ fb, mu, sd = model["fb"], model["mu"], model["sd"]
85
+ lm, starts = query_starts(logmel(q_audio, fb))
86
+ qe = embed_windows(model["layers"], lm, mu, sd, starts).mean(axis=0)
87
+ qs = stats_windows(lm, mu, sd, starts).mean(axis=0)
88
+ return qe / max(float(np.linalg.norm(qe)), 1e-12), qs / max(float(np.linalg.norm(qs)), 1e-12)
89
+
90
+
91
+ def similarity_curves(model, q_audio, rec_audio):
92
+ qe, qs = query_vectors(model, q_audio)
93
+ lm = pad_frames(logmel(rec_audio, model["fb"]), WIN)
94
+ starts = window_starts(lm.shape[1])
95
+ learned = embed_windows(model["layers"], lm, model["mu"], model["sd"], starts) @ qe
96
+ base = stats_windows(lm, model["mu"], model["sd"], starts) @ qs
97
+ times = [s * HOP / SR for s in starts]
98
+ return times, learned, base
99
+
100
+
101
+ def robust_z(c):
102
+ c = np.asarray(c, dtype=np.float64)
103
+ med = float(np.median(c))
104
+ mad = float(np.median(np.abs(c - med))) + 1e-6
105
+ return (c - med) / (1.4826 * mad)
106
+
107
+
108
+ def final_scores(learned, base, cfg):
109
+ """cfg: {"alpha": float, "baseline_norm": "raw" | "z"} -> (next_echo_curve, baseline_curve)."""
110
+ a = cfg["alpha"]
111
+ nxt = a * robust_z(learned) + (1.0 - a) * robust_z(base)
112
+ bl = robust_z(base) if cfg["baseline_norm"] == "z" else np.asarray(base, dtype=np.float64)
113
+ return nxt, bl
114
+
115
+
116
+ def detect(times, sims, threshold):
117
+ out, cur = [], None
118
+ for t, s in zip(times, sims):
119
+ if s >= threshold:
120
+ if cur is None:
121
+ cur = [t, t + WIN_SEC, float(s)]
122
+ else:
123
+ cur[1] = t + WIN_SEC
124
+ cur[2] = max(cur[2], float(s))
125
+ elif cur is not None:
126
+ out.append(cur); cur = None
127
+ if cur is not None:
128
+ out.append(cur)
129
+ return out
130
+
131
+
132
+ def match_events(dets, gts, tol=0.5):
133
+ used, tp = set(), 0
134
+ for g0, g1 in gts:
135
+ for j, d in enumerate(dets):
136
+ if j not in used and d[0] < g1 + tol and d[1] > g0 - tol:
137
+ used.add(j); tp += 1
138
+ break
139
+ return tp, len(dets) - len(used), len(gts) - tp
140
+
141
+
142
+ def load_model(spec):
143
+ t = spec["tensors"]
144
+ return {"layers": spec["nets"]["embed"], "fb": t["mel_fb"], "mu": float(spec["meta"]["mu"]), "sd": float(spec["meta"]["sd"]),
145
+ "thresholds": spec["meta"]["thresholds"], "score": spec["meta"]["score"]}
embedding.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3788fb2afb593cead76c5729b3806f7cb088b653b52f666c902bebc3b1690d34
3
+ size 262552
evaluation.json ADDED
@@ -0,0 +1,653 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "project": "Next Echo",
3
+ "profile": "full",
4
+ "seed": 73,
5
+ "device": {
6
+ "python": "3.13.16",
7
+ "platform": "Linux-6.6.159+-x86_64-with-glibc2.39",
8
+ "cpu_count": 12,
9
+ "numpy": "2.4.6",
10
+ "torch": "2.14.1+cu130",
11
+ "cuda": true,
12
+ "gpu": "NVIDIA L4",
13
+ "colab": true
14
+ },
15
+ "data": {
16
+ "name": "ESC-50",
17
+ "source": "https://github.com/karoldvl/ESC-50/archive/master.zip",
18
+ "download": {
19
+ "url": "https://github.com/karoldvl/ESC-50/archive/master.zip",
20
+ "file": "esc50.zip",
21
+ "bytes": 645695005,
22
+ "sha256": "805afc618aff80eff0641d51311647871fead5807da68bb18e43a27db47a79ce",
23
+ "verified_against_pinned_sha256": false
24
+ },
25
+ "license": "CC BY-NC 3.0 (dataset); per-clip Freesound licenses in the archive",
26
+ "split": {
27
+ "embedding_training": "folds 1-3, 40 seen classes",
28
+ "validation_recordings": "fold 4, seen classes",
29
+ "test_recordings": "fold 5, seen and held-out classes"
30
+ },
31
+ "held_out_classes": [
32
+ "clapping",
33
+ "door_wood_knock",
34
+ "rooster",
35
+ "church_bells",
36
+ "glass_breaking",
37
+ "keyboard_typing",
38
+ "siren",
39
+ "cow",
40
+ "sneezing",
41
+ "can_opening"
42
+ ],
43
+ "resampling": "44.1 kHz -> 16 kHz (scipy resample_poly 160/441)"
44
+ },
45
+ "training": {
46
+ "steps": 5000,
47
+ "seconds": 1473.6,
48
+ "final_loss": 2.7861027669906617,
49
+ "clips": 960,
50
+ "mu": -3.852266374845604,
51
+ "sd": 5.0395802756944885
52
+ },
53
+ "thresholds": {
54
+ "learned": 1.553792940244492,
55
+ "baseline": 0.5790993811336966
56
+ },
57
+ "validation": {
58
+ "learned": {
59
+ "f1": 0.6113207547169811,
60
+ "precision": 0.54,
61
+ "recall": 0.7043478260869566,
62
+ "false_alarms_per_min": 2.3,
63
+ "tp": 81,
64
+ "fp": 69,
65
+ "fn": 34
66
+ },
67
+ "baseline": {
68
+ "f1": 0.5882352941176471,
69
+ "precision": 0.5357142857142857,
70
+ "recall": 0.6521739130434783,
71
+ "false_alarms_per_min": 2.1666666666666665,
72
+ "tp": 75,
73
+ "fp": 65,
74
+ "fn": 40
75
+ }
76
+ },
77
+ "protocol": {
78
+ "recording_seconds": 30,
79
+ "per_cell": 40,
80
+ "tolerance_s": 0.5,
81
+ "conditions": {
82
+ "clean": "30 dB SNR brown noise",
83
+ "noisy": "5 dB SNR",
84
+ "overlap": "distractors overlap target events",
85
+ "absent": "no target event"
86
+ },
87
+ "construction": "real ESC-50 event crops mixed at random times with 3-5 distractor clips"
88
+ },
89
+ "results": {
90
+ "seen/clean": {
91
+ "learned": {
92
+ "f1": 0.6186440677966102,
93
+ "precision": 0.5289855072463768,
94
+ "recall": 0.7448979591836735,
95
+ "false_alarms_per_min": 3.25,
96
+ "tp": 73,
97
+ "fp": 65,
98
+ "fn": 25
99
+ },
100
+ "baseline": {
101
+ "f1": 0.676056338028169,
102
+ "precision": 0.6260869565217392,
103
+ "recall": 0.7346938775510204,
104
+ "false_alarms_per_min": 2.15,
105
+ "tp": 72,
106
+ "fp": 43,
107
+ "fn": 26
108
+ }
109
+ },
110
+ "seen/noisy": {
111
+ "learned": {
112
+ "f1": 0.6296296296296297,
113
+ "precision": 0.5964912280701754,
114
+ "recall": 0.6666666666666666,
115
+ "false_alarms_per_min": 2.3,
116
+ "tp": 68,
117
+ "fp": 46,
118
+ "fn": 34
119
+ },
120
+ "baseline": {
121
+ "f1": 0.6063829787234043,
122
+ "precision": 0.6627906976744186,
123
+ "recall": 0.5588235294117647,
124
+ "false_alarms_per_min": 1.45,
125
+ "tp": 57,
126
+ "fp": 29,
127
+ "fn": 45
128
+ }
129
+ },
130
+ "seen/overlap": {
131
+ "learned": {
132
+ "f1": 0.7487179487179487,
133
+ "precision": 0.776595744680851,
134
+ "recall": 0.7227722772277227,
135
+ "false_alarms_per_min": 1.05,
136
+ "tp": 73,
137
+ "fp": 21,
138
+ "fn": 28
139
+ },
140
+ "baseline": {
141
+ "f1": 0.7875647668393783,
142
+ "precision": 0.8260869565217391,
143
+ "recall": 0.7524752475247525,
144
+ "false_alarms_per_min": 0.8,
145
+ "tp": 76,
146
+ "fp": 16,
147
+ "fn": 25
148
+ }
149
+ },
150
+ "seen/absent": {
151
+ "learned": {
152
+ "f1": 0.0,
153
+ "precision": 0.0,
154
+ "recall": 0.0,
155
+ "false_alarms_per_min": 4.95,
156
+ "tp": 0,
157
+ "fp": 99,
158
+ "fn": 0
159
+ },
160
+ "baseline": {
161
+ "f1": 0.0,
162
+ "precision": 0.0,
163
+ "recall": 0.0,
164
+ "false_alarms_per_min": 4.0,
165
+ "tp": 0,
166
+ "fp": 80,
167
+ "fn": 0
168
+ }
169
+ },
170
+ "unseen/clean": {
171
+ "learned": {
172
+ "f1": 0.6584362139917695,
173
+ "precision": 0.547945205479452,
174
+ "recall": 0.8247422680412371,
175
+ "false_alarms_per_min": 3.3,
176
+ "tp": 80,
177
+ "fp": 66,
178
+ "fn": 17
179
+ },
180
+ "baseline": {
181
+ "f1": 0.6169154228855721,
182
+ "precision": 0.5961538461538461,
183
+ "recall": 0.6391752577319587,
184
+ "false_alarms_per_min": 2.1,
185
+ "tp": 62,
186
+ "fp": 42,
187
+ "fn": 35
188
+ }
189
+ },
190
+ "unseen/noisy": {
191
+ "learned": {
192
+ "f1": 0.7168141592920354,
193
+ "precision": 0.6639344262295082,
194
+ "recall": 0.7788461538461539,
195
+ "false_alarms_per_min": 2.05,
196
+ "tp": 81,
197
+ "fp": 41,
198
+ "fn": 23
199
+ },
200
+ "baseline": {
201
+ "f1": 0.6666666666666666,
202
+ "precision": 0.7411764705882353,
203
+ "recall": 0.6057692307692307,
204
+ "false_alarms_per_min": 1.1,
205
+ "tp": 63,
206
+ "fp": 22,
207
+ "fn": 41
208
+ }
209
+ },
210
+ "unseen/overlap": {
211
+ "learned": {
212
+ "f1": 0.7671232876712328,
213
+ "precision": 0.711864406779661,
214
+ "recall": 0.8316831683168316,
215
+ "false_alarms_per_min": 1.7,
216
+ "tp": 84,
217
+ "fp": 34,
218
+ "fn": 17
219
+ },
220
+ "baseline": {
221
+ "f1": 0.7897435897435897,
222
+ "precision": 0.8191489361702128,
223
+ "recall": 0.7623762376237624,
224
+ "false_alarms_per_min": 0.85,
225
+ "tp": 77,
226
+ "fp": 17,
227
+ "fn": 24
228
+ }
229
+ },
230
+ "unseen/absent": {
231
+ "learned": {
232
+ "f1": 0.0,
233
+ "precision": 0.0,
234
+ "recall": 0.0,
235
+ "false_alarms_per_min": 5.6,
236
+ "tp": 0,
237
+ "fp": 112,
238
+ "fn": 0
239
+ },
240
+ "baseline": {
241
+ "f1": 0.0,
242
+ "precision": 0.0,
243
+ "recall": 0.0,
244
+ "false_alarms_per_min": 3.65,
245
+ "tp": 0,
246
+ "fp": 73,
247
+ "fn": 0
248
+ }
249
+ },
250
+ "seen/all-present": {
251
+ "learned": {
252
+ "f1": 0.6615146831530139,
253
+ "precision": 0.6184971098265896,
254
+ "recall": 0.7109634551495017,
255
+ "false_alarms_per_min": 2.2,
256
+ "tp": 214,
257
+ "fp": 132,
258
+ "fn": 87
259
+ },
260
+ "baseline": {
261
+ "f1": 0.6902356902356902,
262
+ "precision": 0.6996587030716723,
263
+ "recall": 0.6810631229235881,
264
+ "false_alarms_per_min": 1.4666666666666666,
265
+ "tp": 205,
266
+ "fp": 88,
267
+ "fn": 96
268
+ }
269
+ },
270
+ "unseen/all-present": {
271
+ "learned": {
272
+ "f1": 0.7122093023255814,
273
+ "precision": 0.6347150259067358,
274
+ "recall": 0.8112582781456954,
275
+ "false_alarms_per_min": 2.35,
276
+ "tp": 245,
277
+ "fp": 141,
278
+ "fn": 57
279
+ },
280
+ "baseline": {
281
+ "f1": 0.6905982905982906,
282
+ "precision": 0.7137809187279152,
283
+ "recall": 0.6688741721854304,
284
+ "false_alarms_per_min": 1.35,
285
+ "tp": 202,
286
+ "fp": 81,
287
+ "fn": 100
288
+ }
289
+ }
290
+ },
291
+ "failures_learned": [
292
+ {
293
+ "group": "seen",
294
+ "condition": "clean",
295
+ "target": "door_wood_creaks",
296
+ "missed": 2,
297
+ "false_alarms": 0,
298
+ "true_events": 2
299
+ },
300
+ {
301
+ "group": "seen",
302
+ "condition": "clean",
303
+ "target": "fireworks",
304
+ "missed": 1,
305
+ "false_alarms": 2,
306
+ "true_events": 3
307
+ },
308
+ {
309
+ "group": "seen",
310
+ "condition": "clean",
311
+ "target": "crow",
312
+ "missed": 1,
313
+ "false_alarms": 1,
314
+ "true_events": 3
315
+ },
316
+ {
317
+ "group": "seen",
318
+ "condition": "clean",
319
+ "target": "cat",
320
+ "missed": 1,
321
+ "false_alarms": 1,
322
+ "true_events": 3
323
+ },
324
+ {
325
+ "group": "seen",
326
+ "condition": "clean",
327
+ "target": "door_wood_creaks",
328
+ "missed": 2,
329
+ "false_alarms": 0,
330
+ "true_events": 2
331
+ },
332
+ {
333
+ "group": "seen",
334
+ "condition": "clean",
335
+ "target": "sea_waves",
336
+ "missed": 0,
337
+ "false_alarms": 2,
338
+ "true_events": 3
339
+ },
340
+ {
341
+ "group": "seen",
342
+ "condition": "clean",
343
+ "target": "snoring",
344
+ "missed": 0,
345
+ "false_alarms": 3,
346
+ "true_events": 2
347
+ },
348
+ {
349
+ "group": "seen",
350
+ "condition": "clean",
351
+ "target": "thunderstorm",
352
+ "missed": 1,
353
+ "false_alarms": 0,
354
+ "true_events": 2
355
+ },
356
+ {
357
+ "group": "seen",
358
+ "condition": "clean",
359
+ "target": "crow",
360
+ "missed": 1,
361
+ "false_alarms": 1,
362
+ "true_events": 3
363
+ },
364
+ {
365
+ "group": "seen",
366
+ "condition": "clean",
367
+ "target": "drinking_sipping",
368
+ "missed": 1,
369
+ "false_alarms": 2,
370
+ "true_events": 3
371
+ },
372
+ {
373
+ "group": "seen",
374
+ "condition": "clean",
375
+ "target": "drinking_sipping",
376
+ "missed": 1,
377
+ "false_alarms": 1,
378
+ "true_events": 2
379
+ },
380
+ {
381
+ "group": "seen",
382
+ "condition": "clean",
383
+ "target": "drinking_sipping",
384
+ "missed": 2,
385
+ "false_alarms": 0,
386
+ "true_events": 2
387
+ },
388
+ {
389
+ "group": "seen",
390
+ "condition": "clean",
391
+ "target": "water_drops",
392
+ "missed": 1,
393
+ "false_alarms": 0,
394
+ "true_events": 2
395
+ },
396
+ {
397
+ "group": "seen",
398
+ "condition": "clean",
399
+ "target": "breathing",
400
+ "missed": 0,
401
+ "false_alarms": 4,
402
+ "true_events": 2
403
+ },
404
+ {
405
+ "group": "seen",
406
+ "condition": "clean",
407
+ "target": "brushing_teeth",
408
+ "missed": 0,
409
+ "false_alarms": 2,
410
+ "true_events": 2
411
+ },
412
+ {
413
+ "group": "seen",
414
+ "condition": "clean",
415
+ "target": "thunderstorm",
416
+ "missed": 1,
417
+ "false_alarms": 1,
418
+ "true_events": 2
419
+ },
420
+ {
421
+ "group": "seen",
422
+ "condition": "clean",
423
+ "target": "breathing",
424
+ "missed": 0,
425
+ "false_alarms": 1,
426
+ "true_events": 3
427
+ },
428
+ {
429
+ "group": "seen",
430
+ "condition": "clean",
431
+ "target": "dog",
432
+ "missed": 0,
433
+ "false_alarms": 2,
434
+ "true_events": 2
435
+ },
436
+ {
437
+ "group": "seen",
438
+ "condition": "clean",
439
+ "target": "rain",
440
+ "missed": 0,
441
+ "false_alarms": 1,
442
+ "true_events": 2
443
+ },
444
+ {
445
+ "group": "seen",
446
+ "condition": "clean",
447
+ "target": "mouse_click",
448
+ "missed": 2,
449
+ "false_alarms": 0,
450
+ "true_events": 3
451
+ },
452
+ {
453
+ "group": "seen",
454
+ "condition": "clean",
455
+ "target": "crickets",
456
+ "missed": 1,
457
+ "false_alarms": 2,
458
+ "true_events": 2
459
+ },
460
+ {
461
+ "group": "seen",
462
+ "condition": "clean",
463
+ "target": "rain",
464
+ "missed": 1,
465
+ "false_alarms": 1,
466
+ "true_events": 3
467
+ },
468
+ {
469
+ "group": "seen",
470
+ "condition": "clean",
471
+ "target": "breathing",
472
+ "missed": 0,
473
+ "false_alarms": 6,
474
+ "true_events": 2
475
+ },
476
+ {
477
+ "group": "seen",
478
+ "condition": "clean",
479
+ "target": "crow",
480
+ "missed": 0,
481
+ "false_alarms": 2,
482
+ "true_events": 2
483
+ },
484
+ {
485
+ "group": "seen",
486
+ "condition": "clean",
487
+ "target": "engine",
488
+ "missed": 1,
489
+ "false_alarms": 0,
490
+ "true_events": 3
491
+ },
492
+ {
493
+ "group": "seen",
494
+ "condition": "clean",
495
+ "target": "car_horn",
496
+ "missed": 0,
497
+ "false_alarms": 4,
498
+ "true_events": 2
499
+ },
500
+ {
501
+ "group": "seen",
502
+ "condition": "clean",
503
+ "target": "insects",
504
+ "missed": 0,
505
+ "false_alarms": 1,
506
+ "true_events": 2
507
+ },
508
+ {
509
+ "group": "seen",
510
+ "condition": "clean",
511
+ "target": "toilet_flush",
512
+ "missed": 0,
513
+ "false_alarms": 1,
514
+ "true_events": 3
515
+ },
516
+ {
517
+ "group": "seen",
518
+ "condition": "clean",
519
+ "target": "crackling_fire",
520
+ "missed": 1,
521
+ "false_alarms": 3,
522
+ "true_events": 3
523
+ },
524
+ {
525
+ "group": "seen",
526
+ "condition": "clean",
527
+ "target": "hand_saw",
528
+ "missed": 1,
529
+ "false_alarms": 3,
530
+ "true_events": 3
531
+ },
532
+ {
533
+ "group": "seen",
534
+ "condition": "clean",
535
+ "target": "chainsaw",
536
+ "missed": 0,
537
+ "false_alarms": 3,
538
+ "true_events": 2
539
+ },
540
+ {
541
+ "group": "seen",
542
+ "condition": "clean",
543
+ "target": "crying_baby",
544
+ "missed": 0,
545
+ "false_alarms": 4,
546
+ "true_events": 2
547
+ },
548
+ {
549
+ "group": "seen",
550
+ "condition": "clean",
551
+ "target": "sea_waves",
552
+ "missed": 1,
553
+ "false_alarms": 2,
554
+ "true_events": 3
555
+ },
556
+ {
557
+ "group": "seen",
558
+ "condition": "clean",
559
+ "target": "engine",
560
+ "missed": 1,
561
+ "false_alarms": 2,
562
+ "true_events": 3
563
+ },
564
+ {
565
+ "group": "seen",
566
+ "condition": "clean",
567
+ "target": "helicopter",
568
+ "missed": 1,
569
+ "false_alarms": 2,
570
+ "true_events": 3
571
+ },
572
+ {
573
+ "group": "seen",
574
+ "condition": "clean",
575
+ "target": "hand_saw",
576
+ "missed": 0,
577
+ "false_alarms": 2,
578
+ "true_events": 2
579
+ },
580
+ {
581
+ "group": "seen",
582
+ "condition": "clean",
583
+ "target": "sheep",
584
+ "missed": 0,
585
+ "false_alarms": 3,
586
+ "true_events": 2
587
+ },
588
+ {
589
+ "group": "seen",
590
+ "condition": "noisy",
591
+ "target": "airplane",
592
+ "missed": 0,
593
+ "false_alarms": 2,
594
+ "true_events": 3
595
+ },
596
+ {
597
+ "group": "seen",
598
+ "condition": "noisy",
599
+ "target": "washing_machine",
600
+ "missed": 0,
601
+ "false_alarms": 1,
602
+ "true_events": 2
603
+ },
604
+ {
605
+ "group": "seen",
606
+ "condition": "noisy",
607
+ "target": "toilet_flush",
608
+ "missed": 0,
609
+ "false_alarms": 1,
610
+ "true_events": 3
611
+ }
612
+ ],
613
+ "n_recordings_with_errors_learned": 272,
614
+ "tests": {
615
+ "reload": {
616
+ "returncode": 0,
617
+ "seconds": 0.42,
618
+ "process": "fresh python -E subprocess, cwd = temporary copy of the model folder",
619
+ "passed": true,
620
+ "max_abs_diff": 0.0,
621
+ "windows": 28
622
+ },
623
+ "node_parity": {
624
+ "cases": 3,
625
+ "failed": 0,
626
+ "maxDiff": 7.105427357601002e-15,
627
+ "results": [
628
+ {
629
+ "name": "snippet0",
630
+ "maxDiff": 7.105427357601002e-15
631
+ },
632
+ {
633
+ "name": "snippet1",
634
+ "maxDiff": 7.771561172376096e-16
635
+ },
636
+ {
637
+ "name": "snippet2",
638
+ "maxDiff": 1.6653345369377348e-15
639
+ }
640
+ ],
641
+ "ms": 11322,
642
+ "runtime": "node v22.23.4",
643
+ "status": "passed"
644
+ },
645
+ "browser": {
646
+ "status": "skipped",
647
+ "reason": "playwright not installed"
648
+ }
649
+ },
650
+ "not_run": [
651
+ "No evaluation on naturally recorded long scenes; recordings are constructed mixtures."
652
+ ]
653
+ }
inference.py ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Next Echo inference (NumPy only): find occurrences of an example sound in a recording.
2
+
3
+ Usage:
4
+ python inference.py --query clap.wav --recording street.wav
5
+ python inference.py --selftest
6
+ Audio must be 16 kHz mono 16-bit WAV for the CLI (resample elsewhere if needed).
7
+ """
8
+ import argparse
9
+ import json
10
+ import os
11
+ import sys
12
+ import wave
13
+
14
+ HERE = os.path.dirname(os.path.abspath(__file__))
15
+ sys.path.insert(0, HERE)
16
+
17
+ import numpy as np # noqa: E402
18
+
19
+ import echo_core as ec # noqa: E402
20
+ from nlrt import load_model as _load # noqa: E402
21
+
22
+
23
+ def load(path=os.path.join(HERE, "model.json")):
24
+ return ec.load_model(_load(path))
25
+
26
+
27
+ def read_wav(path):
28
+ with wave.open(path, "rb") as w:
29
+ assert w.getframerate() == 16000 and w.getnchannels() == 1 and w.getsampwidth() == 2, "need 16 kHz mono 16-bit"
30
+ return np.frombuffer(w.readframes(w.getnframes()), dtype="<i2") / 32768.0
31
+
32
+
33
+ def find(model, query, recording, method="learned"):
34
+ times, learned, base = ec.similarity_curves(model, query, recording)
35
+ nxt, bl = ec.final_scores(learned, base, model["score"])
36
+ sims = nxt if method == "learned" else bl
37
+ th = model["thresholds"]["learned" if method == "learned" else "baseline"]
38
+ return [{"start": round(a, 3), "end": round(b, 3), "score": round(s, 4)} for a, b, s in ec.detect(times, sims, th)]
39
+
40
+
41
+ def _selftest():
42
+ model = load()
43
+ with open(os.path.join(HERE, "selftest.json"), encoding="utf-8") as f:
44
+ t = json.load(f)
45
+ q, r = np.asarray(t["query"]) / 32768.0, np.asarray(t["recording"]) / 32768.0
46
+ _, learned, base = ec.similarity_curves(model, q, r)
47
+ diff = max(float(np.abs(learned - np.asarray(t["learned"])).max()), float(np.abs(base - np.asarray(t["base"])).max()))
48
+ ok = diff < 1e-6
49
+ print(json.dumps({"passed": ok, "max_abs_diff": diff, "windows": len(learned)}))
50
+ return ok
51
+
52
+
53
+ if __name__ == "__main__":
54
+ ap = argparse.ArgumentParser()
55
+ ap.add_argument("--selftest", action="store_true")
56
+ ap.add_argument("--query")
57
+ ap.add_argument("--recording")
58
+ ap.add_argument("--method", default="learned", choices=["learned", "logmel"])
59
+ a = ap.parse_args()
60
+ if a.selftest:
61
+ sys.exit(0 if _selftest() else 1)
62
+ print(json.dumps(find(load(), read_wav(a.query), read_wav(a.recording), a.method), indent=1))
model.json ADDED
The diff for this file is too large to render. See raw diff
 
nlrt.py ADDED
@@ -0,0 +1,226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """nlrt: tiny portable runtime shared by Next Lab models.
2
+
3
+ The same layer list runs in NumPy (this file) and in the browser (nlrt.js).
4
+ Weights are stored as base64 little-endian float32 inside model.json.
5
+ Only NumPy is required to load and run a model.
6
+ """
7
+ import base64
8
+ import json
9
+ import math
10
+
11
+ import numpy as np
12
+
13
+ __all__ = ["b64e", "b64d", "forward", "load_model", "Mulberry32", "export_sequential"]
14
+
15
+
16
+ def b64e(a):
17
+ return base64.b64encode(np.ascontiguousarray(a, dtype="<f4").tobytes()).decode("ascii")
18
+
19
+
20
+ def b64d(s, shape=None):
21
+ a = np.frombuffer(base64.b64decode(s), dtype="<f4").astype(np.float64)
22
+ return a.reshape(shape) if shape is not None else a
23
+
24
+
25
+ def load_model(path):
26
+ with open(path, "r", encoding="utf-8") as f:
27
+ spec = json.load(f)
28
+ for net in spec.get("nets", {}).values():
29
+ _decode(net)
30
+ for k, v in list(spec.get("tensors", {}).items()):
31
+ spec["tensors"][k] = b64d(v["data"], v["shape"])
32
+ return spec
33
+
34
+
35
+ def _decode(layers):
36
+ for L in layers:
37
+ for key in ("w", "b"):
38
+ if key in L and isinstance(L[key], str):
39
+ L[key] = b64d(L[key], L.get(key + "_shape"))
40
+
41
+
42
+ class Mulberry32:
43
+ """Deterministic PRNG with an identical JavaScript twin in nlrt.js."""
44
+
45
+ def __init__(self, seed):
46
+ self.a = seed & 0xFFFFFFFF
47
+
48
+ def random(self):
49
+ self.a = (self.a + 0x6D2B79F5) & 0xFFFFFFFF
50
+ t = self.a
51
+ t = ((t ^ (t >> 15)) * (t | 1)) & 0xFFFFFFFF
52
+ t ^= (t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF
53
+ return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296.0
54
+
55
+ def uniform(self, lo, hi):
56
+ return lo + (hi - lo) * self.random()
57
+
58
+ def randint(self, n):
59
+ return int(math.floor(self.random() * n))
60
+
61
+ def normal(self):
62
+ u = max(self.random(), 1e-12)
63
+ v = self.random()
64
+ return math.sqrt(-2.0 * math.log(u)) * math.cos(2.0 * math.pi * v)
65
+
66
+
67
+ def _pair(v):
68
+ return list(v) if isinstance(v, (list, tuple)) else [v, v]
69
+
70
+
71
+ def _conv2d(x, w, b, stride, pad):
72
+ n, c, h, wd = x.shape
73
+ o, _, kh, kw = w.shape
74
+ sh, sw = stride
75
+ ph, pw = pad
76
+ xp = np.pad(x, ((0, 0), (0, 0), (ph, ph), (pw, pw)))
77
+ oh = (h + 2 * ph - kh) // sh + 1
78
+ ow = (wd + 2 * pw - kw) // sw + 1
79
+ cols = np.empty((n, c, kh, kw, oh, ow), dtype=x.dtype)
80
+ for i in range(kh):
81
+ for j in range(kw):
82
+ cols[:, :, i, j] = xp[:, :, i:i + sh * oh:sh, j:j + sw * ow:sw]
83
+ out = np.tensordot(cols, w, axes=([1, 2, 3], [1, 2, 3])) # n, oh, ow, o
84
+ out = out.transpose(0, 3, 1, 2)
85
+ if b is not None:
86
+ out = out + b[None, :, None, None]
87
+ return out
88
+
89
+
90
+ def _maxpool(x, k):
91
+ kh, kw = k
92
+ n, c, h, w = x.shape
93
+ oh, ow = h // kh, w // kw
94
+ x = x[:, :, :oh * kh, :ow * kw].reshape(n, c, oh, kh, ow, kw)
95
+ return x.max(axis=(3, 5))
96
+
97
+
98
+ def forward(layers, x):
99
+ """Run a layer list on a batch. x: (N, D) or (N, C, H, W) or (N, C, L)."""
100
+ x = np.asarray(x, dtype=np.float64)
101
+ for L in layers:
102
+ t = L["type"]
103
+ if t == "linear":
104
+ x = x @ L["w"].T + L["b"]
105
+ elif t == "conv2d":
106
+ squeeze = x.ndim == 3
107
+ if squeeze:
108
+ x = x[:, :, None, :]
109
+ x = _conv2d(x, L["w"], L["b"], _pair(L["stride"]), _pair(L["pad"]))
110
+ if squeeze:
111
+ x = x[:, :, 0, :]
112
+ elif t == "relu":
113
+ x = np.maximum(x, 0)
114
+ elif t == "tanh":
115
+ x = np.tanh(x)
116
+ elif t == "sigmoid":
117
+ x = 1 / (1 + np.exp(-x))
118
+ elif t == "maxpool":
119
+ squeeze = x.ndim == 3
120
+ if squeeze:
121
+ x = x[:, :, None, :]
122
+ x = _maxpool(x, _pair(L["k"]))
123
+ if squeeze:
124
+ x = x[:, :, 0, :]
125
+ elif t == "gap":
126
+ x = x.reshape(x.shape[0], x.shape[1], -1).mean(axis=2)
127
+ elif t == "flatten":
128
+ x = x.reshape(x.shape[0], -1)
129
+ elif t == "upsample":
130
+ x = x.repeat(L["s"], axis=-1)
131
+ if x.ndim == 4:
132
+ x = x.repeat(L["s"], axis=-2)
133
+ elif t == "l2norm":
134
+ x = x / np.maximum(np.linalg.norm(x, axis=1, keepdims=True), 1e-12)
135
+ elif t == "reshape":
136
+ x = x.reshape([x.shape[0]] + list(L["shape"]))
137
+ else:
138
+ raise ValueError("unknown layer " + t)
139
+ return x
140
+
141
+
142
+ def export_sequential(seq):
143
+ """Convert a torch nn.Sequential into a portable layer list (BatchNorm folded)."""
144
+ import torch.nn as nn
145
+
146
+ mods = list(seq.modules())[1:] if not isinstance(seq, (list, tuple)) else list(seq)
147
+ mods = [m for m in mods if not isinstance(m, nn.Sequential)]
148
+ layers = []
149
+
150
+ def tensor(t):
151
+ return t.detach().cpu().double().numpy()
152
+
153
+ for m in mods:
154
+ if isinstance(m, nn.Linear):
155
+ b = tensor(m.bias) if m.bias is not None else np.zeros(m.out_features)
156
+ layers.append({"type": "linear", "w": tensor(m.weight), "b": b})
157
+ elif isinstance(m, (nn.Conv2d, nn.Conv1d)):
158
+ w = tensor(m.weight)
159
+ b = tensor(m.bias) if m.bias is not None else np.zeros(w.shape[0])
160
+ if isinstance(m, nn.Conv1d):
161
+ w = w[:, :, None, :]
162
+ stride, pad = [1, m.stride[0]], [0, m.padding[0]]
163
+ else:
164
+ stride, pad = list(m.stride), list(m.padding)
165
+ layers.append({"type": "conv2d", "w": w, "b": b, "stride": stride, "pad": pad})
166
+ elif isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
167
+ prev = layers[-1]
168
+ assert prev["type"] in ("conv2d", "linear"), "BatchNorm must follow conv/linear"
169
+ scale = tensor(m.weight) / np.sqrt(tensor(m.running_var) + m.eps)
170
+ shift = tensor(m.bias) - tensor(m.running_mean) * scale
171
+ if prev["type"] == "conv2d":
172
+ prev["w"] = prev["w"] * scale[:, None, None, None]
173
+ else:
174
+ prev["w"] = prev["w"] * scale[:, None]
175
+ prev["b"] = prev["b"] * scale + shift
176
+ elif isinstance(m, nn.ReLU):
177
+ layers.append({"type": "relu"})
178
+ elif isinstance(m, nn.Tanh):
179
+ layers.append({"type": "tanh"})
180
+ elif isinstance(m, nn.Sigmoid):
181
+ layers.append({"type": "sigmoid"})
182
+ elif isinstance(m, nn.MaxPool2d):
183
+ layers.append({"type": "maxpool", "k": _pair(m.kernel_size)})
184
+ elif isinstance(m, nn.MaxPool1d):
185
+ k = m.kernel_size if isinstance(m, int) else m.kernel_size
186
+ layers.append({"type": "maxpool", "k": [1, k if isinstance(k, int) else k[0]]})
187
+ elif isinstance(m, (nn.AdaptiveAvgPool2d, nn.AdaptiveAvgPool1d)):
188
+ layers.append({"type": "gap"})
189
+ elif isinstance(m, nn.Flatten):
190
+ layers.append({"type": "flatten"})
191
+ elif isinstance(m, nn.Upsample):
192
+ layers.append({"type": "upsample", "s": int(m.scale_factor)})
193
+ elif isinstance(m, nn.Dropout):
194
+ continue
195
+ elif m.__class__.__name__ == "L2Norm":
196
+ layers.append({"type": "l2norm"})
197
+ elif m.__class__.__name__ == "Reshape":
198
+ layers.append({"type": "reshape", "shape": list(m.shape)})
199
+ else:
200
+ raise ValueError("cannot export " + m.__class__.__name__)
201
+ return layers
202
+
203
+
204
+ def encode_layers(layers):
205
+ out = []
206
+ for L in layers:
207
+ L2 = dict(L)
208
+ for key in ("w", "b"):
209
+ if key in L2:
210
+ arr = np.asarray(L2[key])
211
+ L2[key + "_shape"] = list(arr.shape)
212
+ L2[key] = b64e(arr)
213
+ out.append(L2)
214
+ return out
215
+
216
+
217
+ def save_model(path, nets=None, tensors=None, meta=None):
218
+ spec = {"format": "nlrt-1", "meta": meta or {}, "nets": {}, "tensors": {}}
219
+ for name, layers in (nets or {}).items():
220
+ spec["nets"][name] = encode_layers(layers)
221
+ for name, arr in (tensors or {}).items():
222
+ arr = np.asarray(arr)
223
+ spec["tensors"][name] = {"shape": list(arr.shape), "data": b64e(arr)}
224
+ with open(path, "w", encoding="utf-8") as f:
225
+ json.dump(spec, f, separators=(",", ":"))
226
+ return spec
preview.png ADDED

Git LFS Details

  • SHA256: 77276bae6da0e1e177964fcddd3772fc188358fb498cf9c5bf63adc2959f05c1
  • Pointer size: 131 Bytes
  • Size of remote file: 101 kB
selftest.json ADDED
The diff for this file is too large to render. See raw diff