Next-Ask / README.md
Lamapi's picture
Update README.md
a1de9e6 verified
|
Raw History Blame Contribute Delete
3.35 kB
---
license: cc-by-4.0
library_name: numpy
tags:
- active-learning
- tabular-classification
- next-lab
---
![ask](https://cdn-uploads.huggingface.co/production/uploads/67d46bc5fe6ad6f6511d6f44/2kNVbLXV3h5bzAUnMSsOe.png)
# Next Ask
Next Ask decides **which unlabeled example to send to a person next** when labels are expensive. It ranks the pool by
classifier uncertainty, shortlists the 40 least certain items, then picks items that are both uncertain and far from
everything already labeled. A small softmax classifier retrains after each batch.
**Space:** https://huggingface.co/spaces/thelamapi/Next-Ask — label the digits the selector asks for; the classifier retrains in your browser.
## Results (10 seeds, 360 held-out test digits, batches of 10, budget 200)
| Strategy | Area under learning curve | Final accuracy | Labels to reach 90% (median) |
|---|---:|---:|---:|
| Uncertainty + diversity (Next Ask) | 0.8897 ± 0.0066 | 0.9583 | 70 (10/10 seeds) |
| Uncertainty only | 0.8837 ± 0.0135 | 0.9569 | 70 (10/10 seeds) |
| Random | 0.8508 ± 0.0115 | 0.9283 | 115 (10/10 seeds) |
Mean test accuracy at selected label counts:
| Labels | Random | Uncertainty | Uncertainty + diversity |
|---:|---:|---:|---:|
| 20 | 0.6508 | 0.6608 | 0.7036 |
| 50 | 0.8386 | 0.8650 | 0.8736 |
| 100 | 0.8864 | 0.9294 | 0.9314 |
| 150 | 0.9131 | 0.9447 | 0.9492 |
| 200 | 0.9283 | 0.9569 | 0.9583 |
Paired by seed (same initial labels): hybrid − random AULC = +0.0388 ± 0.0113
(hybrid better in 10/10 seeds); hybrid − uncertainty = +0.0060 ± 0.0104
(6/10 seeds).
![Learning curves](preview.png)
## Method
* Classifier: multinomial logistic regression on the 64 pixel intensities, full-batch gradient descent from zero weights
(steps=150, lr=1.0, l2=0.001, chosen on a 200-example validation split, never on test).
* Uncertainty: margin between the two most probable classes.
* Diversity: farthest-first selection on the uncertain shortlist, score = (1 − margin) × distance to the nearest labeled or already chosen example.
* The browser runs a line-by-line JavaScript port. Node parity on 4 fixed cases: max abs difference 1.1102230246251565e-16.
## Usage
```python
from inference import load, predict, next_queries
spec = load() # model.json, NumPy only
probs = predict(spec, pixels) # pixels: (N, 64) ints 0..16
picks = next_queries(spec, pool_pixels, labeled_idx, labels, strategy="hybrid", batch=5)
```
`python inference.py --selftest` reloads the model and checks stored outputs.
## Data
Optical Recognition of Handwritten Digits (8x8) — UCI ML Repository id 80 via sklearn.datasets.load_digits, license CC BY 4.0. Split: 1237 pool /
200 validation / 360 test, stratified, seed 73.
## Limitations
* 8×8 digits are an easy, balanced benchmark; gains from active learning are usually smaller on noisy, imbalanced real data.
* The simulation assumes a perfect annotator (dataset labels). The Space lets people label, but no human study was run.
* Final model in `model.json` is the seed-0 hybrid run at the full budget, kept for demonstration (17 of 360 test digits wrong).
* Hardware for the recorded run: NVIDIA L4. Profile: full.
Files: `model.json` (weights), `ask_core.py` (selectors and training), `inference.py`, `nlrt.py`, `config.json`, `evaluation.json` (all numbers, seeds, failures, tests).