Ryan.K commited on
Evaluate public mixed test metrics
Browse files- README.md +38 -36
- src/bin/evaluate.rs +189 -14
- src/lib.rs +6 -12
README.md
CHANGED
|
@@ -34,8 +34,8 @@ The current implementation is a first machine-learning baseline:
|
|
| 34 |
- multi-label mood taxonomy
|
| 35 |
- hashed text features
|
| 36 |
- linear sigmoid classifier
|
| 37 |
-
- SGD training on
|
| 38 |
-
DADOES seed
|
| 39 |
- validation-loss early stopping
|
| 40 |
- binary checkpoint save/load
|
| 41 |
- JSON CLI output
|
|
@@ -119,29 +119,30 @@ Current default checkpoint:
|
|
| 119 |
Training stopped at epoch 28, with the best validation checkpoint selected from
|
| 120 |
epoch 24.
|
| 121 |
|
| 122 |
-
These numbers are for the current linear hashed-feature baseline trained on
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
|
| 135 |
-
|
|
| 136 |
-
|
|
| 137 |
-
|
|
| 138 |
-
|
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
|
|
|
| 145 |
|
| 146 |
Recognition benchmark:
|
| 147 |
|
|
@@ -149,13 +150,13 @@ Recognition benchmark:
|
|
| 149 |
|---|---:|
|
| 150 |
| Build | `cargo run --release --bin evaluate` |
|
| 151 |
| Platform | local `arm64`, `rustc 1.95.0` |
|
| 152 |
-
| Model load time | 1.
|
| 153 |
-
| Test examples |
|
| 154 |
| Repeats | 50 |
|
| 155 |
-
| Total classifications |
|
| 156 |
-
| Total classification time |
|
| 157 |
-
| Throughput |
|
| 158 |
-
| Mean classification latency |
|
| 159 |
|
| 160 |
Benchmark numbers measure classification after loading the model and should be
|
| 161 |
treated as local-machine measurements, not portable latency guarantees.
|
|
@@ -172,7 +173,6 @@ Accuracy comparison:
|
|
| 172 |
| Model | Evaluation Setup | Reported Result | Notes |
|
| 173 |
|---|---|---:|---|
|
| 174 |
| DADOES public mixed linear baseline | 13 DADOES labels, public mixed test | micro F1 0.6439 | Local result from `cargo run --release --bin train`; smaller label space than GoEmotions. |
|
| 175 |
-
| DADOES public mixed linear baseline | GoEmotions test mapped to DADOES labels | GoEmotions-only diagnostic slice | This slice does not represent full public mixed coverage for DADOES-specific labels. |
|
| 176 |
| [GoEmotions BERT baseline](https://research.google/pubs/goemotions-a-dataset-of-fine-grained-emotions/) | Original GoEmotions taxonomy | average F1 0.46 | Official GoEmotions paper baseline. |
|
| 177 |
| [SamLowe/roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions) | GoEmotions 28-label test, threshold 0.5 | F1 0.450 | Model-card aggregate metric. |
|
| 178 |
| [tasinhoque/distilbert-go-emotions](https://huggingface.co/tasinhoque/distilbert-go-emotions) | GoEmotions evaluation set | F1 0.4702 | Model-card aggregate metric. |
|
|
@@ -184,11 +184,11 @@ Recognition performance comparison:
|
|
| 184 |
|
| 185 |
| Model | Runtime / Artifact | Reported Performance | Notes |
|
| 186 |
|---|---|---:|---|
|
| 187 |
-
| DADOES public mixed linear baseline | Rust native checkpoint, 1.6 MB |
|
| 188 |
|
| 189 |
## Train
|
| 190 |
|
| 191 |
-
|
| 192 |
|
| 193 |
```bash
|
| 194 |
cargo run --release --bin train
|
|
@@ -226,7 +226,8 @@ The default trainer:
|
|
| 226 |
|
| 227 |
- uses `train.tsv` for gradient updates
|
| 228 |
- uses `dev.tsv` for early stopping
|
| 229 |
-
-
|
|
|
|
| 230 |
- repeats the local seed examples 4 times to cover DADOES labels that
|
| 231 |
GoEmotions does not annotate, such as `lonely`, `bored`, and `tired`
|
| 232 |
- mixes optional public external sources when prepared under `data/raw/external`
|
|
@@ -251,7 +252,8 @@ DADOES is licensed under the GNU Affero General Public License v3.0 only.
|
|
| 251 |
|
| 252 |
## Dataset Direction
|
| 253 |
|
| 254 |
-
Use
|
|
|
|
| 255 |
Keep the runtime labels smaller than GoEmotions:
|
| 256 |
|
| 257 |
- happy
|
|
|
|
| 34 |
- multi-label mood taxonomy
|
| 35 |
- hashed text features
|
| 36 |
- linear sigmoid classifier
|
| 37 |
+
- SGD training on a public mixed English emotion dataset assembled from
|
| 38 |
+
GoEmotions, optional external sources, and DADOES seed examples
|
| 39 |
- validation-loss early stopping
|
| 40 |
- binary checkpoint save/load
|
| 41 |
- JSON CLI output
|
|
|
|
| 119 |
Training stopped at epoch 28, with the best validation checkpoint selected from
|
| 120 |
epoch 24.
|
| 121 |
|
| 122 |
+
These numbers are for the current linear hashed-feature baseline trained on the
|
| 123 |
+
public mixed dataset. GoEmotions is one source in that mix; optional external
|
| 124 |
+
sources under `data/raw/external` and DADOES seed examples are also used. They
|
| 125 |
+
should be treated as a baseline, not a production ceiling.
|
| 126 |
+
|
| 127 |
+
Mixed per-mood diagnostic metrics at threshold `0.35`:
|
| 128 |
+
|
| 129 |
+
| Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |
|
| 130 |
+
|---|---:|---:|---:|---:|---:|---:|---|
|
| 131 |
+
| happy | 9,879 | 3,534 | 0.8476 | 0.9135 | 0.6338 | 0.7484 | loaded mixed test |
|
| 132 |
+
| satisfied | 9,879 | 4,087 | 0.8198 | 0.8122 | 0.7343 | 0.7713 | loaded mixed test |
|
| 133 |
+
| excited | 9,879 | 1,504 | 0.8712 | 0.6543 | 0.3271 | 0.4362 | loaded mixed test |
|
| 134 |
+
| curious | 5,427 | 677 | 0.8841 | 0.5698 | 0.2895 | 0.3839 | loaded mixed test |
|
| 135 |
+
| anxious | 9,879 | 934 | 0.9288 | 0.6567 | 0.5182 | 0.5793 | loaded mixed test |
|
| 136 |
+
| frustrated | 5,427 | 699 | 0.8734 | 0.5319 | 0.1431 | 0.2255 | loaded mixed test |
|
| 137 |
+
| sad | 9,879 | 1,028 | 0.9167 | 0.7423 | 0.3054 | 0.4328 | loaded mixed test |
|
| 138 |
+
| angry | 9,879 | 1,063 | 0.9120 | 0.7245 | 0.2944 | 0.4187 | loaded mixed test |
|
| 139 |
+
| lonely | 153 | 89 | 0.7190 | 0.8286 | 0.6517 | 0.7296 | loaded mixed test |
|
| 140 |
+
| hopeful | 9,879 | 2,943 | 0.8631 | 0.8299 | 0.6799 | 0.7475 | loaded mixed test |
|
| 141 |
+
| neutral | 9,879 | 2,736 | 0.7973 | 0.6869 | 0.4931 | 0.5740 | loaded mixed test |
|
| 142 |
+
|
| 143 |
+
The loaded mixed test currently has no supervised examples for `bored` or
|
| 144 |
+
`tired`; those labels should not be published as `0/0` metrics. They require
|
| 145 |
+
held-out reviewed examples before a per-label accuracy can be reported.
|
| 146 |
|
| 147 |
Recognition benchmark:
|
| 148 |
|
|
|
|
| 150 |
|---|---:|
|
| 151 |
| Build | `cargo run --release --bin evaluate` |
|
| 152 |
| Platform | local `arm64`, `rustc 1.95.0` |
|
| 153 |
+
| Model load time | 1.356 ms |
|
| 154 |
+
| Test examples | 10,032 |
|
| 155 |
| Repeats | 50 |
|
| 156 |
+
| Total classifications | 501,600 |
|
| 157 |
+
| Total classification time | 1,778.897 ms |
|
| 158 |
+
| Throughput | 281,972.5 texts/s |
|
| 159 |
+
| Mean classification latency | 3.546 us/text |
|
| 160 |
|
| 161 |
Benchmark numbers measure classification after loading the model and should be
|
| 162 |
treated as local-machine measurements, not portable latency guarantees.
|
|
|
|
| 173 |
| Model | Evaluation Setup | Reported Result | Notes |
|
| 174 |
|---|---|---:|---|
|
| 175 |
| DADOES public mixed linear baseline | 13 DADOES labels, public mixed test | micro F1 0.6439 | Local result from `cargo run --release --bin train`; smaller label space than GoEmotions. |
|
|
|
|
| 176 |
| [GoEmotions BERT baseline](https://research.google/pubs/goemotions-a-dataset-of-fine-grained-emotions/) | Original GoEmotions taxonomy | average F1 0.46 | Official GoEmotions paper baseline. |
|
| 177 |
| [SamLowe/roberta-base-go_emotions](https://huggingface.co/SamLowe/roberta-base-go_emotions) | GoEmotions 28-label test, threshold 0.5 | F1 0.450 | Model-card aggregate metric. |
|
| 178 |
| [tasinhoque/distilbert-go-emotions](https://huggingface.co/tasinhoque/distilbert-go-emotions) | GoEmotions evaluation set | F1 0.4702 | Model-card aggregate metric. |
|
|
|
|
| 184 |
|
| 185 |
| Model | Runtime / Artifact | Reported Performance | Notes |
|
| 186 |
|---|---|---:|---|
|
| 187 |
+
| DADOES public mixed linear baseline | Rust native checkpoint, 1.6 MB | 281,972.5 texts/s; 3.546 us/text | Local `arm64` benchmark from `cargo run --release --bin evaluate`. |
|
| 188 |
|
| 189 |
## Train
|
| 190 |
|
| 191 |
+
Prepare the base GoEmotions split files in `data/raw/goemotions`, then run:
|
| 192 |
|
| 193 |
```bash
|
| 194 |
cargo run --release --bin train
|
|
|
|
| 226 |
|
| 227 |
- uses `train.tsv` for gradient updates
|
| 228 |
- uses `dev.tsv` for early stopping
|
| 229 |
+
- starts from `test.tsv` and mixes optional external test splits before final
|
| 230 |
+
evaluation
|
| 231 |
- repeats the local seed examples 4 times to cover DADOES labels that
|
| 232 |
GoEmotions does not annotate, such as `lonely`, `bored`, and `tired`
|
| 233 |
- mixes optional public external sources when prepared under `data/raw/external`
|
|
|
|
| 252 |
|
| 253 |
## Dataset Direction
|
| 254 |
|
| 255 |
+
Use a public mixed dataset assembled from GoEmotions and compatible external
|
| 256 |
+
sources, then add curated DADOES-domain text.
|
| 257 |
Keep the runtime labels smaller than GoEmotions:
|
| 258 |
|
| 259 |
- happy
|
src/bin/evaluate.rs
CHANGED
|
@@ -8,10 +8,17 @@ use std::path::{Path, PathBuf};
|
|
| 8 |
use std::process::ExitCode;
|
| 9 |
use std::time::{Duration, Instant};
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
use dadoes::goemotions::{GoEmotionsError, parse_split};
|
| 12 |
use dadoes::{DadoesClassifier, EmotionClassifier, Mood, OwnedTrainingExample};
|
| 13 |
|
| 14 |
const DEFAULT_TEST_SPLIT: &str = "data/raw/goemotions/test.tsv";
|
|
|
|
|
|
|
|
|
|
| 15 |
const THRESHOLD: f32 = 0.35;
|
| 16 |
const BENCHMARK_REPEATS: usize = 50;
|
| 17 |
|
|
@@ -28,7 +35,12 @@ fn main() -> ExitCode {
|
|
| 28 |
|
| 29 |
fn run() -> Result<(), EvaluateCliError> {
|
| 30 |
let args = EvaluateArgs::from_env()?;
|
| 31 |
-
let examples = load_split(&args.test_split)?;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
let load_started = Instant::now();
|
| 33 |
let classifier = DadoesClassifier::from_default_model()?;
|
| 34 |
let load_duration = load_started.elapsed();
|
|
@@ -37,12 +49,36 @@ fn run() -> Result<(), EvaluateCliError> {
|
|
| 37 |
|
| 38 |
println!("## Per-mood test metrics");
|
| 39 |
println!();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
println!(
|
| 41 |
"| Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |"
|
| 42 |
);
|
| 43 |
println!("|---|---:|---:|---:|---:|---:|---:|---|");
|
| 44 |
for metrics in &label_metrics {
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
}
|
| 47 |
|
| 48 |
println!();
|
|
@@ -59,18 +95,54 @@ fn run() -> Result<(), EvaluateCliError> {
|
|
| 59 |
duration_ms(benchmark.duration)
|
| 60 |
);
|
| 61 |
println!(
|
| 62 |
-
"| Throughput | {:.1}
|
| 63 |
-
benchmark.
|
| 64 |
);
|
| 65 |
println!(
|
| 66 |
-
"| Mean classification latency | {:.3} us/
|
| 67 |
-
benchmark.
|
| 68 |
);
|
| 69 |
println!("| Score checksum | {:.6} |", benchmark.score_checksum);
|
| 70 |
|
| 71 |
Ok(())
|
| 72 |
}
|
| 73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
fn load_split(path: &Path) -> Result<Vec<OwnedTrainingExample>, EvaluateCliError> {
|
| 75 |
let input = fs::read_to_string(path).map_err(|source| EvaluateCliError::ReadFile {
|
| 76 |
path: path.to_path_buf(),
|
|
@@ -207,12 +279,11 @@ struct LabelMetrics {
|
|
| 207 |
}
|
| 208 |
|
| 209 |
impl LabelMetrics {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
fn markdown_row(&self) -> String {
|
| 211 |
-
let coverage = if self.supervised == 0 {
|
| 212 |
-
"not covered by GoEmotions test"
|
| 213 |
-
} else {
|
| 214 |
-
"GoEmotions test"
|
| 215 |
-
};
|
| 216 |
format!(
|
| 217 |
"| {} | {} | {} | {} | {} | {} | {} | {} |",
|
| 218 |
self.mood.as_str(),
|
|
@@ -222,7 +293,7 @@ impl LabelMetrics {
|
|
| 222 |
format_metric(self.precision),
|
| 223 |
format_metric(self.recall),
|
| 224 |
format_metric(self.f1),
|
| 225 |
-
|
| 226 |
)
|
| 227 |
}
|
| 228 |
}
|
|
@@ -236,11 +307,11 @@ struct BenchmarkMetrics {
|
|
| 236 |
}
|
| 237 |
|
| 238 |
impl BenchmarkMetrics {
|
| 239 |
-
fn
|
| 240 |
usize_to_f64(self.classifications) / self.duration.as_secs_f64()
|
| 241 |
}
|
| 242 |
|
| 243 |
-
fn
|
| 244 |
duration_micros(self.duration) / usize_to_f64(self.classifications)
|
| 245 |
}
|
| 246 |
}
|
|
@@ -248,17 +319,27 @@ impl BenchmarkMetrics {
|
|
| 248 |
#[derive(Debug, Clone, Eq, PartialEq)]
|
| 249 |
struct EvaluateArgs {
|
| 250 |
test_split: PathBuf,
|
|
|
|
|
|
|
| 251 |
}
|
| 252 |
|
| 253 |
impl EvaluateArgs {
|
| 254 |
fn from_env() -> Result<Self, EvaluateCliError> {
|
| 255 |
let args = std::env::args().skip(1).collect::<Vec<_>>();
|
|
|
|
|
|
|
|
|
|
|
|
|
| 256 |
match args.as_slice() {
|
| 257 |
[] => Ok(Self {
|
| 258 |
test_split: PathBuf::from(DEFAULT_TEST_SPLIT),
|
|
|
|
|
|
|
| 259 |
}),
|
| 260 |
[test_split] => Ok(Self {
|
| 261 |
test_split: PathBuf::from(test_split),
|
|
|
|
|
|
|
| 262 |
}),
|
| 263 |
_ => Err(EvaluateCliError::Usage),
|
| 264 |
}
|
|
@@ -273,6 +354,10 @@ enum EvaluateCliError {
|
|
| 273 |
source: std::io::Error,
|
| 274 |
},
|
| 275 |
ParseSplit(GoEmotionsError),
|
|
|
|
|
|
|
|
|
|
|
|
|
| 276 |
Model(dadoes::ModelIoError),
|
| 277 |
}
|
| 278 |
|
|
@@ -289,6 +374,13 @@ impl Display for EvaluateCliError {
|
|
| 289 |
EvaluateCliError::ParseSplit(error) => {
|
| 290 |
write!(formatter, "failed to parse split: {error}")
|
| 291 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 292 |
EvaluateCliError::Model(error) => write!(formatter, "failed to load model: {error}"),
|
| 293 |
}
|
| 294 |
}
|
|
@@ -326,6 +418,24 @@ fn format_metric(metric: Option<f32>) -> String {
|
|
| 326 |
}
|
| 327 |
}
|
| 328 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 329 |
fn duration_ms(duration: Duration) -> f64 {
|
| 330 |
duration.as_secs_f64() * 1_000.0
|
| 331 |
}
|
|
@@ -341,3 +451,68 @@ fn usize_to_f32(value: usize) -> f32 {
|
|
| 341 |
fn usize_to_f64(value: usize) -> f64 {
|
| 342 |
value.to_string().parse::<f64>().unwrap_or(f64::INFINITY)
|
| 343 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
use std::process::ExitCode;
|
| 9 |
use std::time::{Duration, Instant};
|
| 10 |
|
| 11 |
+
use dadoes::datasets::{
|
| 12 |
+
DatasetParseError, parse_dadoes_jsonl, parse_empathetic_dialogues_csv,
|
| 13 |
+
parse_fig_loneliness_jsonl, parse_loneliness_causes_csv, parse_um1neko_text_emotion_jsonl,
|
| 14 |
+
};
|
| 15 |
use dadoes::goemotions::{GoEmotionsError, parse_split};
|
| 16 |
use dadoes::{DadoesClassifier, EmotionClassifier, Mood, OwnedTrainingExample};
|
| 17 |
|
| 18 |
const DEFAULT_TEST_SPLIT: &str = "data/raw/goemotions/test.tsv";
|
| 19 |
+
const DEFAULT_EXTERNAL_DATA_DIR: &str = "data/raw/external";
|
| 20 |
+
const EXTERNAL_DATA_DIR_ENV: &str = "DADOES_EXTERNAL_DATA_DIR";
|
| 21 |
+
const INCLUDE_NON_COMMERCIAL_ENV: &str = "DADOES_INCLUDE_NON_COMMERCIAL";
|
| 22 |
const THRESHOLD: f32 = 0.35;
|
| 23 |
const BENCHMARK_REPEATS: usize = 50;
|
| 24 |
|
|
|
|
| 35 |
|
| 36 |
fn run() -> Result<(), EvaluateCliError> {
|
| 37 |
let args = EvaluateArgs::from_env()?;
|
| 38 |
+
let mut examples = load_split(&args.test_split)?;
|
| 39 |
+
let external = load_external_test_mix(
|
| 40 |
+
&args.external_data_dir,
|
| 41 |
+
args.include_non_commercial,
|
| 42 |
+
&mut examples,
|
| 43 |
+
)?;
|
| 44 |
let load_started = Instant::now();
|
| 45 |
let classifier = DadoesClassifier::from_default_model()?;
|
| 46 |
let load_duration = load_started.elapsed();
|
|
|
|
| 49 |
|
| 50 |
println!("## Per-mood test metrics");
|
| 51 |
println!();
|
| 52 |
+
println!(
|
| 53 |
+
"Loaded mixed test examples: {} (external files: {})",
|
| 54 |
+
examples.len(),
|
| 55 |
+
external.len()
|
| 56 |
+
);
|
| 57 |
+
for summary in &external {
|
| 58 |
+
println!(
|
| 59 |
+
"- {} examples={} path={}",
|
| 60 |
+
summary.name,
|
| 61 |
+
summary.examples,
|
| 62 |
+
summary.path.display()
|
| 63 |
+
);
|
| 64 |
+
}
|
| 65 |
+
println!();
|
| 66 |
println!(
|
| 67 |
"| Mood | Supervised Examples | Positives | Accuracy | Precision | Recall | F1 | Coverage |"
|
| 68 |
);
|
| 69 |
println!("|---|---:|---:|---:|---:|---:|---:|---|");
|
| 70 |
for metrics in &label_metrics {
|
| 71 |
+
if metrics.has_supervision() {
|
| 72 |
+
println!("{}", metrics.markdown_row());
|
| 73 |
+
}
|
| 74 |
+
}
|
| 75 |
+
let missing_labels = labels_without_supervision(&label_metrics);
|
| 76 |
+
if !missing_labels.is_empty() {
|
| 77 |
+
println!();
|
| 78 |
+
println!(
|
| 79 |
+
"Labels without supervised examples in the loaded mixed test: {}",
|
| 80 |
+
missing_labels.join(", ")
|
| 81 |
+
);
|
| 82 |
}
|
| 83 |
|
| 84 |
println!();
|
|
|
|
| 95 |
duration_ms(benchmark.duration)
|
| 96 |
);
|
| 97 |
println!(
|
| 98 |
+
"| Throughput | {:.1} texts/s |",
|
| 99 |
+
benchmark.texts_per_second()
|
| 100 |
);
|
| 101 |
println!(
|
| 102 |
+
"| Mean classification latency | {:.3} us/text |",
|
| 103 |
+
benchmark.micros_per_text()
|
| 104 |
);
|
| 105 |
println!("| Score checksum | {:.6} |", benchmark.score_checksum);
|
| 106 |
|
| 107 |
Ok(())
|
| 108 |
}
|
| 109 |
|
| 110 |
+
fn load_external_test_mix(
|
| 111 |
+
root: &Path,
|
| 112 |
+
include_non_commercial: bool,
|
| 113 |
+
test: &mut Vec<OwnedTrainingExample>,
|
| 114 |
+
) -> Result<Vec<ExternalDatasetSummary>, EvaluateCliError> {
|
| 115 |
+
let mut summaries = Vec::new();
|
| 116 |
+
for file in EXTERNAL_TEST_DATASET_FILES {
|
| 117 |
+
if !file.license.allowed_by(include_non_commercial) {
|
| 118 |
+
continue;
|
| 119 |
+
}
|
| 120 |
+
|
| 121 |
+
let path = root.join(file.relative_path);
|
| 122 |
+
if !path.exists() {
|
| 123 |
+
continue;
|
| 124 |
+
}
|
| 125 |
+
|
| 126 |
+
let input = fs::read_to_string(&path).map_err(|source| EvaluateCliError::ReadFile {
|
| 127 |
+
path: path.clone(),
|
| 128 |
+
source,
|
| 129 |
+
})?;
|
| 130 |
+
let examples = (file.parser)(&input).map_err(|source| EvaluateCliError::ParseExternal {
|
| 131 |
+
path: path.clone(),
|
| 132 |
+
source,
|
| 133 |
+
})?;
|
| 134 |
+
let examples_count = examples.len();
|
| 135 |
+
test.extend(examples);
|
| 136 |
+
summaries.push(ExternalDatasetSummary {
|
| 137 |
+
name: file.name,
|
| 138 |
+
path,
|
| 139 |
+
examples: examples_count,
|
| 140 |
+
});
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
Ok(summaries)
|
| 144 |
+
}
|
| 145 |
+
|
| 146 |
fn load_split(path: &Path) -> Result<Vec<OwnedTrainingExample>, EvaluateCliError> {
|
| 147 |
let input = fs::read_to_string(path).map_err(|source| EvaluateCliError::ReadFile {
|
| 148 |
path: path.to_path_buf(),
|
|
|
|
| 279 |
}
|
| 280 |
|
| 281 |
impl LabelMetrics {
|
| 282 |
+
fn has_supervision(&self) -> bool {
|
| 283 |
+
self.supervised > 0
|
| 284 |
+
}
|
| 285 |
+
|
| 286 |
fn markdown_row(&self) -> String {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 287 |
format!(
|
| 288 |
"| {} | {} | {} | {} | {} | {} | {} | {} |",
|
| 289 |
self.mood.as_str(),
|
|
|
|
| 293 |
format_metric(self.precision),
|
| 294 |
format_metric(self.recall),
|
| 295 |
format_metric(self.f1),
|
| 296 |
+
"loaded mixed test"
|
| 297 |
)
|
| 298 |
}
|
| 299 |
}
|
|
|
|
| 307 |
}
|
| 308 |
|
| 309 |
impl BenchmarkMetrics {
|
| 310 |
+
fn texts_per_second(&self) -> f64 {
|
| 311 |
usize_to_f64(self.classifications) / self.duration.as_secs_f64()
|
| 312 |
}
|
| 313 |
|
| 314 |
+
fn micros_per_text(&self) -> f64 {
|
| 315 |
duration_micros(self.duration) / usize_to_f64(self.classifications)
|
| 316 |
}
|
| 317 |
}
|
|
|
|
| 319 |
#[derive(Debug, Clone, Eq, PartialEq)]
|
| 320 |
struct EvaluateArgs {
|
| 321 |
test_split: PathBuf,
|
| 322 |
+
external_data_dir: PathBuf,
|
| 323 |
+
include_non_commercial: bool,
|
| 324 |
}
|
| 325 |
|
| 326 |
impl EvaluateArgs {
|
| 327 |
fn from_env() -> Result<Self, EvaluateCliError> {
|
| 328 |
let args = std::env::args().skip(1).collect::<Vec<_>>();
|
| 329 |
+
let external_data_dir = std::env::var(EXTERNAL_DATA_DIR_ENV)
|
| 330 |
+
.map(PathBuf::from)
|
| 331 |
+
.unwrap_or_else(|_| PathBuf::from(DEFAULT_EXTERNAL_DATA_DIR));
|
| 332 |
+
let include_non_commercial = env_flag(INCLUDE_NON_COMMERCIAL_ENV);
|
| 333 |
match args.as_slice() {
|
| 334 |
[] => Ok(Self {
|
| 335 |
test_split: PathBuf::from(DEFAULT_TEST_SPLIT),
|
| 336 |
+
external_data_dir,
|
| 337 |
+
include_non_commercial,
|
| 338 |
}),
|
| 339 |
[test_split] => Ok(Self {
|
| 340 |
test_split: PathBuf::from(test_split),
|
| 341 |
+
external_data_dir,
|
| 342 |
+
include_non_commercial,
|
| 343 |
}),
|
| 344 |
_ => Err(EvaluateCliError::Usage),
|
| 345 |
}
|
|
|
|
| 354 |
source: std::io::Error,
|
| 355 |
},
|
| 356 |
ParseSplit(GoEmotionsError),
|
| 357 |
+
ParseExternal {
|
| 358 |
+
path: PathBuf,
|
| 359 |
+
source: DatasetParseError,
|
| 360 |
+
},
|
| 361 |
Model(dadoes::ModelIoError),
|
| 362 |
}
|
| 363 |
|
|
|
|
| 374 |
EvaluateCliError::ParseSplit(error) => {
|
| 375 |
write!(formatter, "failed to parse split: {error}")
|
| 376 |
}
|
| 377 |
+
EvaluateCliError::ParseExternal { path, source } => {
|
| 378 |
+
write!(
|
| 379 |
+
formatter,
|
| 380 |
+
"failed to parse external dataset {}: {source}",
|
| 381 |
+
path.display()
|
| 382 |
+
)
|
| 383 |
+
}
|
| 384 |
EvaluateCliError::Model(error) => write!(formatter, "failed to load model: {error}"),
|
| 385 |
}
|
| 386 |
}
|
|
|
|
| 418 |
}
|
| 419 |
}
|
| 420 |
|
| 421 |
+
fn labels_without_supervision(metrics: &[LabelMetrics]) -> Vec<&'static str> {
|
| 422 |
+
metrics
|
| 423 |
+
.iter()
|
| 424 |
+
.filter(|metrics| !metrics.has_supervision())
|
| 425 |
+
.map(|metrics| metrics.mood.as_str())
|
| 426 |
+
.collect()
|
| 427 |
+
}
|
| 428 |
+
|
| 429 |
+
fn env_flag(name: &str) -> bool {
|
| 430 |
+
match std::env::var(name) {
|
| 431 |
+
Ok(value) => matches!(
|
| 432 |
+
value.trim().to_ascii_lowercase().as_str(),
|
| 433 |
+
"1" | "true" | "yes" | "on"
|
| 434 |
+
),
|
| 435 |
+
Err(_) => false,
|
| 436 |
+
}
|
| 437 |
+
}
|
| 438 |
+
|
| 439 |
fn duration_ms(duration: Duration) -> f64 {
|
| 440 |
duration.as_secs_f64() * 1_000.0
|
| 441 |
}
|
|
|
|
| 451 |
fn usize_to_f64(value: usize) -> f64 {
|
| 452 |
value.to_string().parse::<f64>().unwrap_or(f64::INFINITY)
|
| 453 |
}
|
| 454 |
+
|
| 455 |
+
type ExternalParser = fn(&str) -> Result<Vec<OwnedTrainingExample>, DatasetParseError>;
|
| 456 |
+
|
| 457 |
+
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
|
| 458 |
+
enum LicenseClass {
|
| 459 |
+
Public,
|
| 460 |
+
NonCommercial,
|
| 461 |
+
}
|
| 462 |
+
|
| 463 |
+
impl LicenseClass {
|
| 464 |
+
fn allowed_by(self, include_non_commercial: bool) -> bool {
|
| 465 |
+
match self {
|
| 466 |
+
LicenseClass::Public => true,
|
| 467 |
+
LicenseClass::NonCommercial => include_non_commercial,
|
| 468 |
+
}
|
| 469 |
+
}
|
| 470 |
+
}
|
| 471 |
+
|
| 472 |
+
#[derive(Debug, Copy, Clone)]
|
| 473 |
+
struct ExternalDatasetFile {
|
| 474 |
+
name: &'static str,
|
| 475 |
+
license: LicenseClass,
|
| 476 |
+
relative_path: &'static str,
|
| 477 |
+
parser: ExternalParser,
|
| 478 |
+
}
|
| 479 |
+
|
| 480 |
+
#[derive(Debug, Clone)]
|
| 481 |
+
struct ExternalDatasetSummary {
|
| 482 |
+
name: &'static str,
|
| 483 |
+
path: PathBuf,
|
| 484 |
+
examples: usize,
|
| 485 |
+
}
|
| 486 |
+
|
| 487 |
+
const EXTERNAL_TEST_DATASET_FILES: &[ExternalDatasetFile] = &[
|
| 488 |
+
ExternalDatasetFile {
|
| 489 |
+
name: "dadoes-domain",
|
| 490 |
+
license: LicenseClass::Public,
|
| 491 |
+
relative_path: "dadoes-domain/test.jsonl",
|
| 492 |
+
parser: parse_dadoes_jsonl,
|
| 493 |
+
},
|
| 494 |
+
ExternalDatasetFile {
|
| 495 |
+
name: "facebook/empathetic_dialogues",
|
| 496 |
+
license: LicenseClass::NonCommercial,
|
| 497 |
+
relative_path: "empathetic_dialogues/test.csv",
|
| 498 |
+
parser: parse_empathetic_dialogues_csv,
|
| 499 |
+
},
|
| 500 |
+
ExternalDatasetFile {
|
| 501 |
+
name: "FIG-Loneliness/FIG-Loneliness",
|
| 502 |
+
license: LicenseClass::NonCommercial,
|
| 503 |
+
relative_path: "fig_loneliness/test.jsonl",
|
| 504 |
+
parser: parse_fig_loneliness_jsonl,
|
| 505 |
+
},
|
| 506 |
+
ExternalDatasetFile {
|
| 507 |
+
name: "yael-katsman/Loneliness-Causes-and-Intensity",
|
| 508 |
+
license: LicenseClass::Public,
|
| 509 |
+
relative_path: "loneliness_causes/test_data.csv",
|
| 510 |
+
parser: parse_loneliness_causes_csv,
|
| 511 |
+
},
|
| 512 |
+
ExternalDatasetFile {
|
| 513 |
+
name: "Um1neko/text_emotion",
|
| 514 |
+
license: LicenseClass::Public,
|
| 515 |
+
relative_path: "um1neko/test.json",
|
| 516 |
+
parser: parse_um1neko_text_emotion_jsonl,
|
| 517 |
+
},
|
| 518 |
+
];
|
src/lib.rs
CHANGED
|
@@ -1,11 +1,8 @@
|
|
| 1 |
#![deny(missing_docs)]
|
| 2 |
//! DADOES: Do Androids Dream of Electric Sheep.
|
| 3 |
//!
|
| 4 |
-
//! The crate
|
| 5 |
-
//!
|
| 6 |
-
//! multi-label linear classifier trained with binary cross entropy. It gives the
|
| 7 |
-
//! project a testable machine-learning boundary with a small Rust-native
|
| 8 |
-
//! runtime.
|
| 9 |
|
| 10 |
use std::collections::BTreeMap;
|
| 11 |
use std::collections::hash_map::DefaultHasher;
|
|
@@ -164,7 +161,7 @@ pub struct MoodScore {
|
|
| 164 |
pub score: f32,
|
| 165 |
}
|
| 166 |
|
| 167 |
-
/// Complete classifier output for one
|
| 168 |
#[derive(Debug, Clone, PartialEq)]
|
| 169 |
pub struct MoodAnalysis {
|
| 170 |
scores: Vec<MoodScore>,
|
|
@@ -201,16 +198,16 @@ impl MoodAnalysis {
|
|
| 201 |
}
|
| 202 |
}
|
| 203 |
|
| 204 |
-
/// A
|
| 205 |
pub trait EmotionClassifier {
|
| 206 |
-
/// Classifies one English
|
| 207 |
fn classify(&self, text: &str) -> MoodAnalysis;
|
| 208 |
}
|
| 209 |
|
| 210 |
/// One supervised training example.
|
| 211 |
#[derive(Debug, Copy, Clone)]
|
| 212 |
pub struct TrainingExample<'a> {
|
| 213 |
-
/// The English
|
| 214 |
pub text: &'a str,
|
| 215 |
/// The moods present in the report.
|
| 216 |
///
|
|
@@ -356,9 +353,6 @@ pub enum ModelError {
|
|
| 356 |
/// A dataset row had empty report text.
|
| 357 |
EmptyExampleText,
|
| 358 |
/// A dataset row had no mood labels.
|
| 359 |
-
///
|
| 360 |
-
/// This is retained for checkpoint/API compatibility. Partial-supervision
|
| 361 |
-
/// records may now have no positive labels.
|
| 362 |
EmptyLabels,
|
| 363 |
/// A dataset row had no supervised labels.
|
| 364 |
EmptySupervisedLabels,
|
|
|
|
| 1 |
#![deny(missing_docs)]
|
| 2 |
//! DADOES: Do Androids Dream of Electric Sheep.
|
| 3 |
//!
|
| 4 |
+
//! The crate provides a small Rust-native mood classifier for English text,
|
| 5 |
+
//! using hashed lexical features and a multi-label linear model.
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
use std::collections::BTreeMap;
|
| 8 |
use std::collections::hash_map::DefaultHasher;
|
|
|
|
| 161 |
pub score: f32,
|
| 162 |
}
|
| 163 |
|
| 164 |
+
/// Complete classifier output for one text.
|
| 165 |
#[derive(Debug, Clone, PartialEq)]
|
| 166 |
pub struct MoodAnalysis {
|
| 167 |
scores: Vec<MoodScore>,
|
|
|
|
| 198 |
}
|
| 199 |
}
|
| 200 |
|
| 201 |
+
/// A classifier that maps English text to a multi-label mood analysis.
|
| 202 |
pub trait EmotionClassifier {
|
| 203 |
+
/// Classifies one English text.
|
| 204 |
fn classify(&self, text: &str) -> MoodAnalysis;
|
| 205 |
}
|
| 206 |
|
| 207 |
/// One supervised training example.
|
| 208 |
#[derive(Debug, Copy, Clone)]
|
| 209 |
pub struct TrainingExample<'a> {
|
| 210 |
+
/// The English text.
|
| 211 |
pub text: &'a str,
|
| 212 |
/// The moods present in the report.
|
| 213 |
///
|
|
|
|
| 353 |
/// A dataset row had empty report text.
|
| 354 |
EmptyExampleText,
|
| 355 |
/// A dataset row had no mood labels.
|
|
|
|
|
|
|
|
|
|
| 356 |
EmptyLabels,
|
| 357 |
/// A dataset row had no supervised labels.
|
| 358 |
EmptySupervisedLabels,
|