Ryan.K commited on
Commit
31ec5d5
·
unverified ·
1 Parent(s): f9195f9

Evaluate public mixed test metrics

Browse files
Files changed (3) hide show
  1. README.md +38 -36
  2. src/bin/evaluate.rs +189 -14
  3. src/lib.rs +6 -12
README.md CHANGED
@@ -34,8 +34,8 @@ The current implementation is a first machine-learning baseline:
34
  - multi-label mood taxonomy
35
  - hashed text features
36
  - linear sigmoid classifier
37
- - SGD training on GoEmotions, optional external emotion datasets, and small
38
- DADOES seed augmentation
39
  - validation-loss early stopping
40
  - binary checkpoint save/load
41
  - JSON CLI output
@@ -119,29 +119,30 @@ Current default checkpoint:
119
  Training stopped at epoch 28, with the best validation checkpoint selected from
120
  epoch 24.
121
 
122
- These numbers are for the current linear hashed-feature baseline trained on
123
- GoEmotions mapped into the DADOES mood labels, public optional external sources
124
- prepared under `data/raw/external`, and small DADOES seed augmentation.
125
- They should be treated as a baseline, not a production ceiling.
126
-
127
- Per-mood numbers should be read against the mixed supervision source. GoEmotions
128
- does not directly annotate `lonely`, `bored`, or `tired`, but the public mixed
129
- checkpoint is not GoEmotions-only: optional external sources and DADOES seed
130
- examples add supervision for labels outside the GoEmotions taxonomy.
131
-
132
- Current label coverage:
133
-
134
- | Mood group | Supervision source |
135
- |---|---|
136
- | happy, satisfied, excited, curious, anxious, frustrated, sad, angry, hopeful, neutral | GoEmotions mapped labels plus optional public external emotion datasets |
137
- | lonely | Public loneliness data, optional non-commercial loneliness/dialogue data when enabled, and DADOES seed/domain records |
138
- | bored, tired | DADOES seed/domain records today; production-quality metrics require held-out reviewed examples |
139
-
140
- The aggregate public mixed metrics above are the supported local accuracy
141
- figures for the current checkpoint. Per-label metrics for the full mixed test
142
- set should be regenerated with a mixed-source evaluator before being published;
143
- showing GoEmotions-only rows for labels that GoEmotions never annotates would
144
- misrepresent the model's actual training and evaluation coverage.
 
145
 
146
  Recognition benchmark:
147
 
@@ -149,13 +150,13 @@ Recognition benchmark:
149
  |---|---:|
150
  | Build | `cargo run --release --bin evaluate` |
151
  | Platform | local `arm64`, `rustc 1.95.0` |
152
- | Model load time | 1.752 ms |
153
- | Test examples | 5,427 |
154
  | Repeats | 50 |
155
- | Total classifications | 271,350 |
156
- | Total classification time | 548.198 ms |
157
- | Throughput | 494,985.3 texts/s |
158
- | Mean classification latency | 2.020 us/text |
159
 
160
  Benchmark numbers measure classification after loading the model and should be
161
  treated as local-machine measurements, not portable latency guarantees.
@@ -172,7 +173,6 @@ Accuracy comparison:
172
  | Model | Evaluation Setup | Reported Result | Notes |
173
  |---|---|---:|---|
174
  | DADOES public mixed linear baseline | 13 DADOES labels, public mixed test | micro F1 0.6439 | Local result from `cargo run --release --bin train`; smaller label space than GoEmotions. |
175
- | DADOES public mixed linear baseline | GoEmotions test mapped to DADOES labels | GoEmotions-only diagnostic slice | This slice does not represent full public mixed coverage for DADOES-specific labels. |
176
  | [GoEmotions BERT baseline](https://research.google/pubs/goemotions-a-dataset-of-fine-grained-emotions/) | Original GoEmotions taxonomy | average F1 0.46 | Official GoEmotions paper baseline. |
177
  | [SamLowe/roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions) | GoEmotions 28-label test, threshold 0.5 | F1 0.450 | Model-card aggregate metric. |
178
  | [tasinhoque/distilbert-go-emotions](https://huggingface.co/tasinhoque/distilbert-go-emotions) | GoEmotions evaluation set | F1 0.4702 | Model-card aggregate metric. |
@@ -184,11 +184,11 @@ Recognition performance comparison:
184
 
185
  | Model | Runtime / Artifact | Reported Performance | Notes |
186
  |---|---|---:|---|
187
- | DADOES public mixed linear baseline | Rust native checkpoint, 1.6 MB | 494,985.3 texts/s; 2.020 us/text | Local `arm64` benchmark from `cargo run --release --bin evaluate`. |
188
 
189
  ## Train
190
 
191
- Download the GoEmotions split files into `data/raw/goemotions`, then run:
192
 
193
  ```bash
194
  cargo run --release --bin train
@@ -226,7 +226,8 @@ The default trainer:
226
 
227
  - uses `train.tsv` for gradient updates
228
  - uses `dev.tsv` for early stopping
229
- - evaluates `test.tsv` only after the best validation checkpoint is selected
 
230
  - repeats the local seed examples 4 times to cover DADOES labels that
231
  GoEmotions does not annotate, such as `lonely`, `bored`, and `tired`
232
  - mixes optional public external sources when prepared under `data/raw/external`
@@ -251,7 +252,8 @@ DADOES is licensed under the GNU Affero General Public License v3.0 only.
251
 
252
  ## Dataset Direction
253
 
254
- Use GoEmotions as the public foundation, then add curated DADOES-domain text.
 
255
  Keep the runtime labels smaller than GoEmotions:
256
 
257
  - happy
 
34
  - multi-label mood taxonomy
35
  - hashed text features
36
  - linear sigmoid classifier
37
+ - SGD training on a public mixed English emotion dataset assembled from
38
+ GoEmotions, optional external sources, and DADOES seed examples
39
  - validation-loss early stopping
40
  - binary checkpoint save/load
41
  - JSON CLI output
 
119
  Training stopped at epoch 28, with the best validation checkpoint selected from
120
  epoch 24.
121
 
122
+ These numbers are for the current linear hashed-feature baseline trained on the
123
+ public mixed dataset. GoEmotions is one source in that mix; optional external
124
+ sources under `data/raw/external` and DADOES seed examples are also used. They
125
+ should be treated as a baseline, not a production ceiling.
126
+
127
+ Mixed per-mood diagnostic metrics at threshold `0.35`:
128
+
129
+ | Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |
130
+ |---|---:|---:|---:|---:|---:|---:|---|
131
+ | happy | 9,879 | 3,534 | 0.8476 | 0.9135 | 0.6338 | 0.7484 | loaded mixed test |
132
+ | satisfied | 9,879 | 4,087 | 0.8198 | 0.8122 | 0.7343 | 0.7713 | loaded mixed test |
133
+ | excited | 9,879 | 1,504 | 0.8712 | 0.6543 | 0.3271 | 0.4362 | loaded mixed test |
134
+ | curious | 5,427 | 677 | 0.8841 | 0.5698 | 0.2895 | 0.3839 | loaded mixed test |
135
+ | anxious | 9,879 | 934 | 0.9288 | 0.6567 | 0.5182 | 0.5793 | loaded mixed test |
136
+ | frustrated | 5,427 | 699 | 0.8734 | 0.5319 | 0.1431 | 0.2255 | loaded mixed test |
137
+ | sad | 9,879 | 1,028 | 0.9167 | 0.7423 | 0.3054 | 0.4328 | loaded mixed test |
138
+ | angry | 9,879 | 1,063 | 0.9120 | 0.7245 | 0.2944 | 0.4187 | loaded mixed test |
139
+ | lonely | 153 | 89 | 0.7190 | 0.8286 | 0.6517 | 0.7296 | loaded mixed test |
140
+ | hopeful | 9,879 | 2,943 | 0.8631 | 0.8299 | 0.6799 | 0.7475 | loaded mixed test |
141
+ | neutral | 9,879 | 2,736 | 0.7973 | 0.6869 | 0.4931 | 0.5740 | loaded mixed test |
142
+
143
+ The loaded mixed test currently has no supervised examples for `bored` or
144
+ `tired`; those labels should not be published as `0/0` metrics. They require
145
+ held-out reviewed examples before a per-label accuracy can be reported.
146
 
147
  Recognition benchmark:
148
 
 
150
  |---|---:|
151
  | Build | `cargo run --release --bin evaluate` |
152
  | Platform | local `arm64`, `rustc 1.95.0` |
153
+ | Model load time | 1.356 ms |
154
+ | Test examples | 10,032 |
155
  | Repeats | 50 |
156
+ | Total classifications | 501,600 |
157
+ | Total classification time | 1,778.897 ms |
158
+ | Throughput | 281,972.5 texts/s |
159
+ | Mean classification latency | 3.546 us/text |
160
 
161
  Benchmark numbers measure classification after loading the model and should be
162
  treated as local-machine measurements, not portable latency guarantees.
 
173
  | Model | Evaluation Setup | Reported Result | Notes |
174
  |---|---|---:|---|
175
  | DADOES public mixed linear baseline | 13 DADOES labels, public mixed test | micro F1 0.6439 | Local result from `cargo run --release --bin train`; smaller label space than GoEmotions. |
 
176
  | [GoEmotions BERT baseline](https://research.google/pubs/goemotions-a-dataset-of-fine-grained-emotions/) | Original GoEmotions taxonomy | average F1 0.46 | Official GoEmotions paper baseline. |
177
  | [SamLowe/roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions) | GoEmotions 28-label test, threshold 0.5 | F1 0.450 | Model-card aggregate metric. |
178
  | [tasinhoque/distilbert-go-emotions](https://huggingface.co/tasinhoque/distilbert-go-emotions) | GoEmotions evaluation set | F1 0.4702 | Model-card aggregate metric. |
 
184
 
185
  | Model | Runtime / Artifact | Reported Performance | Notes |
186
  |---|---|---:|---|
187
+ | DADOES public mixed linear baseline | Rust native checkpoint, 1.6 MB | 281,972.5 texts/s; 3.546 us/text | Local `arm64` benchmark from `cargo run --release --bin evaluate`. |
188
 
189
  ## Train
190
 
191
+ Prepare the base GoEmotions split files in `data/raw/goemotions`, then run:
192
 
193
  ```bash
194
  cargo run --release --bin train
 
226
 
227
  - uses `train.tsv` for gradient updates
228
  - uses `dev.tsv` for early stopping
229
+ - starts from `test.tsv` and mixes optional external test splits before final
230
+ evaluation
231
  - repeats the local seed examples 4 times to cover DADOES labels that
232
  GoEmotions does not annotate, such as `lonely`, `bored`, and `tired`
233
  - mixes optional public external sources when prepared under `data/raw/external`
 
252
 
253
  ## Dataset Direction
254
 
255
+ Use a public mixed dataset assembled from GoEmotions and compatible external
256
+ sources, then add curated DADOES-domain text.
257
  Keep the runtime labels smaller than GoEmotions:
258
 
259
  - happy
src/bin/evaluate.rs CHANGED
@@ -8,10 +8,17 @@ use std::path::{Path, PathBuf};
8
  use std::process::ExitCode;
9
  use std::time::{Duration, Instant};
10
 
 
 
 
 
11
  use dadoes::goemotions::{GoEmotionsError, parse_split};
12
  use dadoes::{DadoesClassifier, EmotionClassifier, Mood, OwnedTrainingExample};
13
 
14
  const DEFAULT_TEST_SPLIT: &str = "data/raw/goemotions/test.tsv";
 
 
 
15
  const THRESHOLD: f32 = 0.35;
16
  const BENCHMARK_REPEATS: usize = 50;
17
 
@@ -28,7 +35,12 @@ fn main() -> ExitCode {
28
 
29
  fn run() -> Result<(), EvaluateCliError> {
30
  let args = EvaluateArgs::from_env()?;
31
- let examples = load_split(&args.test_split)?;
 
 
 
 
 
32
  let load_started = Instant::now();
33
  let classifier = DadoesClassifier::from_default_model()?;
34
  let load_duration = load_started.elapsed();
@@ -37,12 +49,36 @@ fn run() -> Result<(), EvaluateCliError> {
37
 
38
  println!("## Per-mood test metrics");
39
  println!();
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  println!(
41
  "| Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |"
42
  );
43
  println!("|---|---:|---:|---:|---:|---:|---:|---|");
44
  for metrics in &label_metrics {
45
- println!("{}", metrics.markdown_row());
 
 
 
 
 
 
 
 
 
 
46
  }
47
 
48
  println!();
@@ -59,18 +95,54 @@ fn run() -> Result<(), EvaluateCliError> {
59
  duration_ms(benchmark.duration)
60
  );
61
  println!(
62
- "| Throughput | {:.1} reports/s |",
63
- benchmark.reports_per_second()
64
  );
65
  println!(
66
- "| Mean classification latency | {:.3} us/report |",
67
- benchmark.micros_per_report()
68
  );
69
  println!("| Score checksum | {:.6} |", benchmark.score_checksum);
70
 
71
  Ok(())
72
  }
73
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
  fn load_split(path: &Path) -> Result<Vec<OwnedTrainingExample>, EvaluateCliError> {
75
  let input = fs::read_to_string(path).map_err(|source| EvaluateCliError::ReadFile {
76
  path: path.to_path_buf(),
@@ -207,12 +279,11 @@ struct LabelMetrics {
207
  }
208
 
209
  impl LabelMetrics {
 
 
 
 
210
  fn markdown_row(&self) -> String {
211
- let coverage = if self.supervised == 0 {
212
- "not covered by GoEmotions test"
213
- } else {
214
- "GoEmotions test"
215
- };
216
  format!(
217
  "| {} | {} | {} | {} | {} | {} | {} | {} |",
218
  self.mood.as_str(),
@@ -222,7 +293,7 @@ impl LabelMetrics {
222
  format_metric(self.precision),
223
  format_metric(self.recall),
224
  format_metric(self.f1),
225
- coverage
226
  )
227
  }
228
  }
@@ -236,11 +307,11 @@ struct BenchmarkMetrics {
236
  }
237
 
238
  impl BenchmarkMetrics {
239
- fn reports_per_second(&self) -> f64 {
240
  usize_to_f64(self.classifications) / self.duration.as_secs_f64()
241
  }
242
 
243
- fn micros_per_report(&self) -> f64 {
244
  duration_micros(self.duration) / usize_to_f64(self.classifications)
245
  }
246
  }
@@ -248,17 +319,27 @@ impl BenchmarkMetrics {
248
  #[derive(Debug, Clone, Eq, PartialEq)]
249
  struct EvaluateArgs {
250
  test_split: PathBuf,
 
 
251
  }
252
 
253
  impl EvaluateArgs {
254
  fn from_env() -> Result<Self, EvaluateCliError> {
255
  let args = std::env::args().skip(1).collect::<Vec<_>>();
 
 
 
 
256
  match args.as_slice() {
257
  [] => Ok(Self {
258
  test_split: PathBuf::from(DEFAULT_TEST_SPLIT),
 
 
259
  }),
260
  [test_split] => Ok(Self {
261
  test_split: PathBuf::from(test_split),
 
 
262
  }),
263
  _ => Err(EvaluateCliError::Usage),
264
  }
@@ -273,6 +354,10 @@ enum EvaluateCliError {
273
  source: std::io::Error,
274
  },
275
  ParseSplit(GoEmotionsError),
 
 
 
 
276
  Model(dadoes::ModelIoError),
277
  }
278
 
@@ -289,6 +374,13 @@ impl Display for EvaluateCliError {
289
  EvaluateCliError::ParseSplit(error) => {
290
  write!(formatter, "failed to parse split: {error}")
291
  }
 
 
 
 
 
 
 
292
  EvaluateCliError::Model(error) => write!(formatter, "failed to load model: {error}"),
293
  }
294
  }
@@ -326,6 +418,24 @@ fn format_metric(metric: Option<f32>) -> String {
326
  }
327
  }
328
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
329
  fn duration_ms(duration: Duration) -> f64 {
330
  duration.as_secs_f64() * 1_000.0
331
  }
@@ -341,3 +451,68 @@ fn usize_to_f32(value: usize) -> f32 {
341
  fn usize_to_f64(value: usize) -> f64 {
342
  value.to_string().parse::<f64>().unwrap_or(f64::INFINITY)
343
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  use std::process::ExitCode;
9
  use std::time::{Duration, Instant};
10
 
11
+ use dadoes::datasets::{
12
+ DatasetParseError, parse_dadoes_jsonl, parse_empathetic_dialogues_csv,
13
+ parse_fig_loneliness_jsonl, parse_loneliness_causes_csv, parse_um1neko_text_emotion_jsonl,
14
+ };
15
  use dadoes::goemotions::{GoEmotionsError, parse_split};
16
  use dadoes::{DadoesClassifier, EmotionClassifier, Mood, OwnedTrainingExample};
17
 
18
  const DEFAULT_TEST_SPLIT: &str = "data/raw/goemotions/test.tsv";
19
+ const DEFAULT_EXTERNAL_DATA_DIR: &str = "data/raw/external";
20
+ const EXTERNAL_DATA_DIR_ENV: &str = "DADOES_EXTERNAL_DATA_DIR";
21
+ const INCLUDE_NON_COMMERCIAL_ENV: &str = "DADOES_INCLUDE_NON_COMMERCIAL";
22
  const THRESHOLD: f32 = 0.35;
23
  const BENCHMARK_REPEATS: usize = 50;
24
 
 
35
 
36
  fn run() -> Result<(), EvaluateCliError> {
37
  let args = EvaluateArgs::from_env()?;
38
+ let mut examples = load_split(&args.test_split)?;
39
+ let external = load_external_test_mix(
40
+ &args.external_data_dir,
41
+ args.include_non_commercial,
42
+ &mut examples,
43
+ )?;
44
  let load_started = Instant::now();
45
  let classifier = DadoesClassifier::from_default_model()?;
46
  let load_duration = load_started.elapsed();
 
49
 
50
  println!("## Per-mood test metrics");
51
  println!();
52
+ println!(
53
+ "Loaded mixed test examples: {} (external files: {})",
54
+ examples.len(),
55
+ external.len()
56
+ );
57
+ for summary in &external {
58
+ println!(
59
+ "- {} examples={} path={}",
60
+ summary.name,
61
+ summary.examples,
62
+ summary.path.display()
63
+ );
64
+ }
65
+ println!();
66
  println!(
67
  "| Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |"
68
  );
69
  println!("|---|---:|---:|---:|---:|---:|---:|---|");
70
  for metrics in &label_metrics {
71
+ if metrics.has_supervision() {
72
+ println!("{}", metrics.markdown_row());
73
+ }
74
+ }
75
+ let missing_labels = labels_without_supervision(&label_metrics);
76
+ if !missing_labels.is_empty() {
77
+ println!();
78
+ println!(
79
+ "Labels without supervised examples in the loaded mixed test: {}",
80
+ missing_labels.join(", ")
81
+ );
82
  }
83
 
84
  println!();
 
95
  duration_ms(benchmark.duration)
96
  );
97
  println!(
98
+ "| Throughput | {:.1} texts/s |",
99
+ benchmark.texts_per_second()
100
  );
101
  println!(
102
+ "| Mean classification latency | {:.3} us/text |",
103
+ benchmark.micros_per_text()
104
  );
105
  println!("| Score checksum | {:.6} |", benchmark.score_checksum);
106
 
107
  Ok(())
108
  }
109
 
110
+ fn load_external_test_mix(
111
+ root: &Path,
112
+ include_non_commercial: bool,
113
+ test: &mut Vec<OwnedTrainingExample>,
114
+ ) -> Result<Vec<ExternalDatasetSummary>, EvaluateCliError> {
115
+ let mut summaries = Vec::new();
116
+ for file in EXTERNAL_TEST_DATASET_FILES {
117
+ if !file.license.allowed_by(include_non_commercial) {
118
+ continue;
119
+ }
120
+
121
+ let path = root.join(file.relative_path);
122
+ if !path.exists() {
123
+ continue;
124
+ }
125
+
126
+ let input = fs::read_to_string(&path).map_err(|source| EvaluateCliError::ReadFile {
127
+ path: path.clone(),
128
+ source,
129
+ })?;
130
+ let examples = (file.parser)(&input).map_err(|source| EvaluateCliError::ParseExternal {
131
+ path: path.clone(),
132
+ source,
133
+ })?;
134
+ let examples_count = examples.len();
135
+ test.extend(examples);
136
+ summaries.push(ExternalDatasetSummary {
137
+ name: file.name,
138
+ path,
139
+ examples: examples_count,
140
+ });
141
+ }
142
+
143
+ Ok(summaries)
144
+ }
145
+
146
  fn load_split(path: &Path) -> Result<Vec<OwnedTrainingExample>, EvaluateCliError> {
147
  let input = fs::read_to_string(path).map_err(|source| EvaluateCliError::ReadFile {
148
  path: path.to_path_buf(),
 
279
  }
280
 
281
  impl LabelMetrics {
282
+ fn has_supervision(&self) -> bool {
283
+ self.supervised > 0
284
+ }
285
+
286
  fn markdown_row(&self) -> String {
 
 
 
 
 
287
  format!(
288
  "| {} | {} | {} | {} | {} | {} | {} | {} |",
289
  self.mood.as_str(),
 
293
  format_metric(self.precision),
294
  format_metric(self.recall),
295
  format_metric(self.f1),
296
+ "loaded mixed test"
297
  )
298
  }
299
  }
 
307
  }
308
 
309
  impl BenchmarkMetrics {
310
+ fn texts_per_second(&self) -> f64 {
311
  usize_to_f64(self.classifications) / self.duration.as_secs_f64()
312
  }
313
 
314
+ fn micros_per_text(&self) -> f64 {
315
  duration_micros(self.duration) / usize_to_f64(self.classifications)
316
  }
317
  }
 
319
  #[derive(Debug, Clone, Eq, PartialEq)]
320
  struct EvaluateArgs {
321
  test_split: PathBuf,
322
+ external_data_dir: PathBuf,
323
+ include_non_commercial: bool,
324
  }
325
 
326
  impl EvaluateArgs {
327
  fn from_env() -> Result<Self, EvaluateCliError> {
328
  let args = std::env::args().skip(1).collect::<Vec<_>>();
329
+ let external_data_dir = std::env::var(EXTERNAL_DATA_DIR_ENV)
330
+ .map(PathBuf::from)
331
+ .unwrap_or_else(|_| PathBuf::from(DEFAULT_EXTERNAL_DATA_DIR));
332
+ let include_non_commercial = env_flag(INCLUDE_NON_COMMERCIAL_ENV);
333
  match args.as_slice() {
334
  [] => Ok(Self {
335
  test_split: PathBuf::from(DEFAULT_TEST_SPLIT),
336
+ external_data_dir,
337
+ include_non_commercial,
338
  }),
339
  [test_split] => Ok(Self {
340
  test_split: PathBuf::from(test_split),
341
+ external_data_dir,
342
+ include_non_commercial,
343
  }),
344
  _ => Err(EvaluateCliError::Usage),
345
  }
 
354
  source: std::io::Error,
355
  },
356
  ParseSplit(GoEmotionsError),
357
+ ParseExternal {
358
+ path: PathBuf,
359
+ source: DatasetParseError,
360
+ },
361
  Model(dadoes::ModelIoError),
362
  }
363
 
 
374
  EvaluateCliError::ParseSplit(error) => {
375
  write!(formatter, "failed to parse split: {error}")
376
  }
377
+ EvaluateCliError::ParseExternal { path, source } => {
378
+ write!(
379
+ formatter,
380
+ "failed to parse external dataset {}: {source}",
381
+ path.display()
382
+ )
383
+ }
384
  EvaluateCliError::Model(error) => write!(formatter, "failed to load model: {error}"),
385
  }
386
  }
 
418
  }
419
  }
420
 
421
+ fn labels_without_supervision(metrics: &[LabelMetrics]) -> Vec<&'static str> {
422
+ metrics
423
+ .iter()
424
+ .filter(|metrics| !metrics.has_supervision())
425
+ .map(|metrics| metrics.mood.as_str())
426
+ .collect()
427
+ }
428
+
429
+ fn env_flag(name: &str) -> bool {
430
+ match std::env::var(name) {
431
+ Ok(value) => matches!(
432
+ value.trim().to_ascii_lowercase().as_str(),
433
+ "1" | "true" | "yes" | "on"
434
+ ),
435
+ Err(_) => false,
436
+ }
437
+ }
438
+
439
  fn duration_ms(duration: Duration) -> f64 {
440
  duration.as_secs_f64() * 1_000.0
441
  }
 
451
  fn usize_to_f64(value: usize) -> f64 {
452
  value.to_string().parse::<f64>().unwrap_or(f64::INFINITY)
453
  }
454
+
455
+ type ExternalParser = fn(&str) -> Result<Vec<OwnedTrainingExample>, DatasetParseError>;
456
+
457
+ #[derive(Debug, Copy, Clone, Eq, PartialEq)]
458
+ enum LicenseClass {
459
+ Public,
460
+ NonCommercial,
461
+ }
462
+
463
+ impl LicenseClass {
464
+ fn allowed_by(self, include_non_commercial: bool) -> bool {
465
+ match self {
466
+ LicenseClass::Public => true,
467
+ LicenseClass::NonCommercial => include_non_commercial,
468
+ }
469
+ }
470
+ }
471
+
472
+ #[derive(Debug, Copy, Clone)]
473
+ struct ExternalDatasetFile {
474
+ name: &'static str,
475
+ license: LicenseClass,
476
+ relative_path: &'static str,
477
+ parser: ExternalParser,
478
+ }
479
+
480
+ #[derive(Debug, Clone)]
481
+ struct ExternalDatasetSummary {
482
+ name: &'static str,
483
+ path: PathBuf,
484
+ examples: usize,
485
+ }
486
+
487
+ const EXTERNAL_TEST_DATASET_FILES: &[ExternalDatasetFile] = &[
488
+ ExternalDatasetFile {
489
+ name: "dadoes-domain",
490
+ license: LicenseClass::Public,
491
+ relative_path: "dadoes-domain/test.jsonl",
492
+ parser: parse_dadoes_jsonl,
493
+ },
494
+ ExternalDatasetFile {
495
+ name: "facebook/empathetic_dialogues",
496
+ license: LicenseClass::NonCommercial,
497
+ relative_path: "empathetic_dialogues/test.csv",
498
+ parser: parse_empathetic_dialogues_csv,
499
+ },
500
+ ExternalDatasetFile {
501
+ name: "FIG-Loneliness/FIG-Loneliness",
502
+ license: LicenseClass::NonCommercial,
503
+ relative_path: "fig_loneliness/test.jsonl",
504
+ parser: parse_fig_loneliness_jsonl,
505
+ },
506
+ ExternalDatasetFile {
507
+ name: "yael-katsman/Loneliness-Causes-and-Intensity",
508
+ license: LicenseClass::Public,
509
+ relative_path: "loneliness_causes/test_data.csv",
510
+ parser: parse_loneliness_causes_csv,
511
+ },
512
+ ExternalDatasetFile {
513
+ name: "Um1neko/text_emotion",
514
+ license: LicenseClass::Public,
515
+ relative_path: "um1neko/test.json",
516
+ parser: parse_um1neko_text_emotion_jsonl,
517
+ },
518
+ ];
src/lib.rs CHANGED
@@ -1,11 +1,8 @@
1
  #![deny(missing_docs)]
2
  //! DADOES: Do Androids Dream of Electric Sheep.
3
  //!
4
- //! The crate starts with a pure Rust mood classifier for English daily reports.
5
- //! The first model is intentionally small: a hashed bag-of-words encoder and a
6
- //! multi-label linear classifier trained with binary cross entropy. It gives the
7
- //! project a testable machine-learning boundary with a small Rust-native
8
- //! runtime.
9
 
10
  use std::collections::BTreeMap;
11
  use std::collections::hash_map::DefaultHasher;
@@ -164,7 +161,7 @@ pub struct MoodScore {
164
  pub score: f32,
165
  }
166
 
167
- /// Complete classifier output for one report.
168
  #[derive(Debug, Clone, PartialEq)]
169
  pub struct MoodAnalysis {
170
  scores: Vec<MoodScore>,
@@ -201,16 +198,16 @@ impl MoodAnalysis {
201
  }
202
  }
203
 
204
- /// A report classifier that maps text to a multi-label mood analysis.
205
  pub trait EmotionClassifier {
206
- /// Classifies one English daily report.
207
  fn classify(&self, text: &str) -> MoodAnalysis;
208
  }
209
 
210
  /// One supervised training example.
211
  #[derive(Debug, Copy, Clone)]
212
  pub struct TrainingExample<'a> {
213
- /// The English daily report text.
214
  pub text: &'a str,
215
  /// The moods present in the report.
216
  ///
@@ -356,9 +353,6 @@ pub enum ModelError {
356
  /// A dataset row had empty report text.
357
  EmptyExampleText,
358
  /// A dataset row had no mood labels.
359
- ///
360
- /// This is retained for checkpoint/API compatibility. Partial-supervision
361
- /// records may now have no positive labels.
362
  EmptyLabels,
363
  /// A dataset row had no supervised labels.
364
  EmptySupervisedLabels,
 
1
  #![deny(missing_docs)]
2
  //! DADOES: Do Androids Dream of Electric Sheep.
3
  //!
4
+ //! The crate provides a small Rust-native mood classifier for English text,
5
+ //! using hashed lexical features and a multi-label linear model.
 
 
 
6
 
7
  use std::collections::BTreeMap;
8
  use std::collections::hash_map::DefaultHasher;
 
161
  pub score: f32,
162
  }
163
 
164
+ /// Complete classifier output for one text.
165
  #[derive(Debug, Clone, PartialEq)]
166
  pub struct MoodAnalysis {
167
  scores: Vec<MoodScore>,
 
198
  }
199
  }
200
 
201
+ /// A classifier that maps English text to a multi-label mood analysis.
202
  pub trait EmotionClassifier {
203
+ /// Classifies one English text.
204
  fn classify(&self, text: &str) -> MoodAnalysis;
205
  }
206
 
207
  /// One supervised training example.
208
  #[derive(Debug, Copy, Clone)]
209
  pub struct TrainingExample<'a> {
210
+ /// The English text.
211
  pub text: &'a str,
212
  /// The moods present in the report.
213
  ///
 
353
  /// A dataset row had empty report text.
354
  EmptyExampleText,
355
  /// A dataset row had no mood labels.
 
 
 
356
  EmptyLabels,
357
  /// A dataset row had no supervised labels.
358
  EmptySupervisedLabels,