Fuzzy ARTMAP vs MLP and EWC on Split-Digits: Preliminary Data and Results

Author: Aliyu Alege · Status: small preliminary benchmark, not peer reviewed · Code: not included

A small class-incremental benchmark comparing three learners as five two-class tasks arrive one after another: a plain one-hidden-layer MLP trained with SGD, the same MLP with Elastic Weight Consolidation (EWC), and a from-scratch Simplified Fuzzy ARTMAP. The question is how much each one forgets, and what Fuzzy ARTMAP pays for its stability.

Read this first.

• The files here were regenerated on 5 October 2026 by re-running the original implementation. The raw outputs of the original run were not kept. The headline numbers of the original run reproduce exactly (see Reproduction).
• The benchmark is tiny (8x8 digit images), the baselines are untuned, and there is no replay baseline. It illustrates a mechanism and its cost. It is not a state-of-the-art comparison.
• The EWC baseline in this implementation does not work: it either has no effect or becomes numerically unstable. See EWC sensitivity. Do not read the MLP+EWC results as evidence about well-tuned EWC.

Headline results

Class-incremental Split-Digits: after each task, every method is tested on all tasks seen so far using a single shared 10-way output, with no task identity at test time.

Forgetting for a task is its best accuracy at any point minus its accuracy after the last task, averaged over the first four tasks.

The cost. Fuzzy ARTMAP stored 523 categories for 1,257 training samples in the seed-42 run (2.40 samples per category; 533 ± 2 categories across 10 seeds, range 529 to 537). That is about 0.42 stored prototypes per training example, so its stability comes with almost no compression. Categories grew steadily with each task: 89, 195, 306, 409, 523.

Vigilance controls that cost. Fitting on all 1,797 images in one epoch, the category count rises from 67 at ρ = 0.5 to 303, 738, 1,694 and finally 1,797 at ρ = 0.99, where every sample becomes its own category (results/vigilance_sweep.csv).

What is in this repository

data/
  split_digits.csv                    all 1,797 images with task and train/test assignment
results/
  reproduction_seed42/                the original-configuration run
    accuracy_long.csv                 accuracy of every task after every training stage, per method
    summary.json                      configuration, final accuracy, forgetting, category counts
    art_categories_per_class.csv      Fuzzy ARTMAP categories per digit class
    original_script_stdout.txt        printed output of the original script, unmodified
    forgetting_curve.png              average accuracy vs tasks trained
  multi_seed/                         10 seeds (split, initialisation and shuffling vary)
    accuracy_long.csv, per_seed_summary.csv, aggregate_summary.csv
  ewc_lambda_sensitivity/             EWC strength sweep, 5 seeds
    per_seed.csv, aggregate.csv
  vigilance_sweep.csv                 Fuzzy ARTMAP category count vs vigilance

Data

The images are scikit-learn's built-in copy of the UCI "Optical Recognition of Handwritten Digits" test set (1,797 images, 8x8 pixels, values 0 to 16), originally from Alpaydin and Kaynak. Please check the UCI repository for the terms of the underlying data. No download is needed to reproduce the split.

data/split_digits.csv columns: sklearn_index (position in the scikit-learn dataset), label (digit 0 to 9), task (1 to 5), split_seed42 (train or test), and pixel_0 to pixel_63 (raw values 0 to 16; the models use these divided by 16).

Tasks are (0,1), (2,3), (4,5), (6,7), (8,9). Each task is split 70/30, stratified by class, with seed 42:

Results files

accuracy_long.csv has one row per (method, after_task, eval_task) giving the test accuracy on eval_task after training through after_task. The multi-seed file adds a seed column. per_seed_summary.csv gives final average accuracy and forgetting per method and seed, plus category counts for Fuzzy ARTMAP.

Methods

• Plain MLP. 64-64-10, ReLU, softmax output, SGD with learning rate 0.1, batch size 32, 60 epochs per task, trained sequentially with no defence against forgetting.
• MLP + EWC. The same network with a quadratic penalty λ · Fisher · (θ − θ*) for every earlier task, where the diagonal Fisher information is estimated from labels sampled from the model's own predictions, and a snapshot is taken after each task. Main results use λ = 2.
• Fuzzy ARTMAP. Simplified Fuzzy ARTMAP with complement coding, choice parameter α = 0.001, learning rate β = 1 (fast learning), match tracking with ε = 0.001, vigilance ρ = 0.85, 5 shuffled epochs per task. A new category is created whenever no existing category both passes the vigilance test and predicts the right label.
• Metrics. Test accuracy on all tasks seen so far after each task; final average accuracy over all five tasks; average forgetting over the first four.
• Seeds. The seed-42 run uses split seed 42 and network initialisation seed 0. The multi-seed run uses seeds 1 to 10 for the split, initialisation, shuffling and EWC sampling together.

Reproduction of the original run

Re-running the original script unmodified gives the same headline numbers as the original write-up: plain MLP 0.196 / 0.998, Fuzzy ARTMAP 0.978 / 0.005, and 523 categories. The one difference is in EWC's intermediate rows, for example task 1's accuracy after task 2 (0.31 in the original printout, 0.05 in original_script_stdout.txt). The original EWC Fisher estimate samples labels without a fixed seed, so these rows vary between runs. The final accuracy and forgetting do not. The seed-42 results in reproduction_seed42/ use a fixed seed for that sampling, so they are deterministic.

EWC strength sensitivity

Five seeds per setting, same network and training as above. "Diverged" means training produced NaN probabilities and stopped.

At every setting where training completed, EWC did no better than the plain MLP. At larger λ the penalty made training numerically unstable, probably because the penalty's step size exceeds the stable range at this learning rate. No gradient clipping or learning-rate tuning was tried. So this EWC baseline shows only that this implementation, untuned, does not rescue the MLP here. It does not show how a carefully tuned EWC would behave.

Limitations

• Tiny data: 1,257 training and 540 test images, 8x8 pixels, five tasks. Accuracies are not comparable to published Split-MNIST or Split-CIFAR numbers.
• Baselines are weak. EWC is untuned and ineffective as implemented, and there is no replay baseline, which is the standard strong approach in the class-incremental setting.
• Fuzzy ARTMAP's stability costs about as much memory as the data itself (about 0.42 categories per training sample). Accuracy was not compared at matched memory.
• Fuzzy ARTMAP was run at one vigilance (ρ = 0.85) for the headline results. The vigilance sweep reports category counts only, not accuracy.
• Single architecture and hyperparameter setting for each method. Seed variation (10 seeds) covers the split, initialisation and shuffling, not hyperparameters.
• The code is not included, so the runs cannot be repeated from this repository alone. The data, split and every result are included for inspection and re-analysis.

References

• Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., & Rosen, D. B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3(5), 698-713.
• Kasuba, T. (1993). Simplified Fuzzy ARTMAP. AI Expert, 8(11), 18-25.
• Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13), 3521-3526.

Citation

Alege, A. (2026). Fuzzy ARTMAP vs MLP and EWC on Split-Digits: Preliminary data and results.
Hugging Face (aaaEpalea/ART-Continual-Learning-Preliminary-Data-and-Results).