jrrribeiro commited on
Commit
afa898b
·
1 Parent(s): 9e6db16

docs(ingestion): add HF segments upload architecture and rollout plan

Browse files
Files changed (1) hide show
  1. docs/HF_SEGMENTS_INGESTION_PLAN.md +194 -0
docs/HF_SEGMENTS_INGESTION_PLAN.md ADDED
@@ -0,0 +1,194 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # HF Segments Ingestion Plan
2
+
3
+ ## Objective
4
+ Build a dedicated ingestion tool (separate from validation UI) that accepts:
5
+ 1. A BirdNET detections CSV
6
+ 2. A root segments folder (including subfolders)
7
+
8
+ Then uploads both audio segments and a normalized metadata index to a Hugging Face dataset repo.
9
+
10
+ ## Why This Comes Next
11
+ - Validation UI is already stable for auth/admin/ACL.
12
+ - Real operations depend on real data availability.
13
+ - Ingestion first avoids UI rework over mock assumptions.
14
+
15
+ ## Real Dataset Observations (from local reference)
16
+ - Segments root folders (species folders): 293
17
+ - Segment files: 146562
18
+ - Extension profile: `.wav` only (plus one `_progress_log.txt`)
19
+ - Filename pattern sample:
20
+ - `Accipiter striatus/Catim_LS-1500_U10_20250221_060600_0.0-3.0s_85%.wav`
21
+ - `Cyanocorax cyanopogon/Aiuab_LN-0500_U16_20260121_153400_42.0-45.0s_98%.wav`
22
+
23
+ ## Input CSV (reference provided)
24
+ Observed columns include:
25
+ - `locality`
26
+ - `point`
27
+ - `date_folder`
28
+ - `source_file`
29
+ - `scientific_name`
30
+ - `common_name`
31
+ - `confidence`
32
+ - `start_time`
33
+ - `end_time`
34
+ - `exact_start`
35
+ - `exact_end`
36
+ - `min_freq`
37
+ - `max_freq`
38
+ - `box_source`
39
+ - `label`
40
+
41
+ ## Proposed HF Dataset Layout
42
+ - `audio/segments/<scientific_name>/<segment_filename>.wav`
43
+ - `index/shards/*.parquet` (normalized row index for fast reads)
44
+ - `manifest.json` (schema/version/counters)
45
+ - `audit/ingestion-runs/<run_id>.json` (run report)
46
+ - `audit/unmatched/<run_id>.parquet` (rows not linked to files)
47
+
48
+ ## Canonical Metadata Row (Normalized)
49
+ Required fields for each detection segment:
50
+ - `project_slug`
51
+ - `detection_key` (stable deterministic key)
52
+ - `audio_id` (source audio id, from `source_file` stem)
53
+ - `segment_filename`
54
+ - `segment_relpath`
55
+ - `scientific_name`
56
+ - `common_name`
57
+ - `confidence`
58
+ - `start_time`
59
+ - `end_time`
60
+ - `exact_start`
61
+ - `exact_end`
62
+ - `locality`
63
+ - `point`
64
+ - `date_folder`
65
+ - `min_freq`
66
+ - `max_freq`
67
+ - `box_source`
68
+ - `label`
69
+ - `source_file`
70
+ - `ingested_at`
71
+ - `schema_version`
72
+
73
+ Optional convenience fields:
74
+ - `segment_duration`
75
+ - `confidence_pct_rounded`
76
+ - `match_strategy`
77
+ - `match_score`
78
+
79
+ ## Matching Strategy (CSV <-> Segment Files)
80
+ Primary join (deterministic):
81
+ 1. species folder = `scientific_name`
82
+ 2. segment filename starts with `source_file` stem
83
+ 3. filename contains `<start_time>-<end_time>s`
84
+
85
+ Tolerance rules:
86
+ - Use decimal normalization (`3` vs `3.0`) for time token search.
87
+ - Confidence in filename (`85%`) is optional for join, used as tie-breaker only.
88
+
89
+ Tie-break policy:
90
+ - If more than one segment matches, pick exact time token first.
91
+ - If still ambiguous, pick closest confidence percentage.
92
+ - Persist ambiguity in audit file.
93
+
94
+ ## Deterministic detection_key
95
+ Generate with stable hash to satisfy model constraints and dedupe safely:
96
+ - input: `project_slug|source_file|scientific_name|start_time|end_time`
97
+ - `detection_key = sha256(input).hexdigest()[:24]`
98
+
99
+ Benefits:
100
+ - deterministic across reruns
101
+ - longer than minimum length constraints
102
+ - independent from local filesystem ordering
103
+
104
+ ## CLI Proposal (new command)
105
+ Add command to `cli/hf_dataset_cli.py`:
106
+ - `ingest-segments`
107
+
108
+ Arguments:
109
+ - `--project-slug`
110
+ - `--dataset-repo`
111
+ - `--detections-csv`
112
+ - `--segments-root`
113
+ - `--batch-size` (default 200)
114
+ - `--shard-size` (default 10000)
115
+ - `--dry-run`
116
+ - `--resume-state-file` (default `.ingest-segments-state.json`)
117
+ - `--max-retries` (default 3)
118
+ - `--retry-backoff-seconds` (default 1.0)
119
+
120
+ ## Command Behavior
121
+ Dry-run mode:
122
+ - validates input schema
123
+ - computes matching stats
124
+ - reports would-upload counts
125
+ - writes no remote files
126
+
127
+ Execute mode:
128
+ 1. ensure dataset scaffold
129
+ 2. parse CSV and normalize rows
130
+ 3. discover segments recursively
131
+ 4. match rows to segment files
132
+ 5. upload segment audio in batches with resume/retry
133
+ 6. build/upload index shards from matched rows
134
+ 7. update `manifest.json`
135
+ 8. write audit artifacts
136
+
137
+ ## Operational Report (stdout JSON)
138
+ Return key counters:
139
+ - `csv_rows_total`
140
+ - `segments_found_total`
141
+ - `matched_rows`
142
+ - `unmatched_rows`
143
+ - `ambiguous_rows`
144
+ - `uploaded_audio_now`
145
+ - `uploaded_audio_skipped_existing`
146
+ - `index_rows_written`
147
+ - `shards_written`
148
+ - `failed_uploads`
149
+ - `duration_seconds`
150
+
151
+ ## Failure Policy
152
+ Hard-fail:
153
+ - missing required CSV columns
154
+ - unreadable segments root
155
+ - HF auth/repo unreachable
156
+
157
+ Soft-fail with audit:
158
+ - unmatched rows
159
+ - ambiguous matches
160
+ - partial audio upload failures
161
+
162
+ Exit code:
163
+ - `0` when no upload failures (unmatched allowed with warning)
164
+ - `1` when upload/index write failures occur
165
+
166
+ ## Performance Notes
167
+ Given ~146k WAV files:
168
+ - keep file discovery cached in-memory map by species + source_file stem
169
+ - avoid O(N*M) matching scans
170
+ - process CSV in chunks where possible
171
+ - upload in batches and persist resume state after each batch
172
+
173
+ ## Security and Separation
174
+ - Ingestion tool remains separate from validation UI.
175
+ - Validation app reads index/audio already published to HF.
176
+ - ACL remains project-scoped at validation stage.
177
+
178
+ ## Implementation Phases
179
+ Phase 1 (MVP ingestion):
180
+ - `ingest-segments --dry-run`
181
+ - deterministic key generation
182
+ - CSV normalization + matching report
183
+
184
+ Phase 2 (publish):
185
+ - batched audio upload + resume/retry
186
+ - index shard generation and manifest update
187
+
188
+ Phase 3 (hardening):
189
+ - audit files for unmatched/ambiguous
190
+ - duplicate handling policy and idempotency checks
191
+ - integration tests with fixture CSV + synthetic segment tree
192
+
193
+ ## Suggested Next Action
194
+ Implement Phase 1 first (`dry-run`) so we can validate matching quality on your real CSV and folder before any remote upload.