tahamajs commited on
Commit
56d43bc
·
verified ·
1 Parent(s): cc3730b

Upload NeuralOps showcase checkpoint (smoothquant) -> tahamajs/bert-comp-smoothquant-sst2

Browse files
README.md ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: mit
5
+ license_link: https://github.com/tahamajs/NeuralOps/blob/main/LICENSE
6
+ library_name: neuralops
7
+ base_model: google-bert/bert-base-uncased
8
+ datasets:
9
+ - nyu-mll/glue
10
+ metrics:
11
+ - accuracy
12
+ tags:
13
+ - neuralops
14
+ - bert
15
+ - model-compression
16
+ - compressor-zoo
17
+ - lora-healing
18
+ - glue-sst2
19
+ - quantization
20
+ - smoothquant
21
+ pipeline_tag: text-classification
22
+ model-index:
23
+ - name: bert-comp-smoothquant-sst2
24
+ results:
25
+ - task:
26
+ type: text-classification
27
+ name: Sentiment Classification
28
+ dataset:
29
+ type: nyu-mll/glue
30
+ name: GLUE SST-2
31
+ config: sst2
32
+ split: validation
33
+ metrics:
34
+ - type: accuracy
35
+ value: 91.74
36
+ name: GLUE SST-2 Validation Accuracy
37
+ ---
38
+
39
+ # BERT-Base (smoothquant) — NeuralOps Compressed & LoRA-Healed Model Zoo
40
+
41
+ [![SST-2 Acc](https://img.shields.io/badge/GLUE_SST--2_Acc-91.74%25-brightgreen)](#-multi-rank-pareto-evaluation-k256-128-64-32)
42
+ [![Teacher Baseline](https://img.shields.io/badge/Teacher_Baseline-91.63%25-blue)](#-multi-rank-pareto-evaluation-k256-128-64-32)
43
+ [![Compression Ratio](https://img.shields.io/badge/Compression-1.00x_(0.0%25_Saved)-purple)](#-compression--architecture-profile)
44
+ [![Showcase Rank](https://img.shields.io/badge/Showcase_Rank-k=32-teal)](#-multi-rank-pareto-evaluation-k256-128-64-32)
45
+ [![Weights](https://img.shields.io/badge/Weights-291.0_MB-indigo)](#-packaged-artifacts)
46
+ [![License](https://img.shields.io/badge/License-MIT-green)](https://github.com/tahamajs/NeuralOps/blob/main/LICENSE)
47
+
48
+ This repository provides a physically compressed and LoRA-healed checkpoint of **`google-bert/bert-base-uncased`** fine-tuned on **GLUE SST-2**, compressed using the **`smoothquant`** (`SmoothQuantCompressor`) engine from the [NeuralOps Library](https://github.com/tahamajs/NeuralOps) at Pareto-selected rank **`k=32`** (`1.00x` compression ratio, `-0.0%` parameter reduction).
49
+
50
+ ---
51
+
52
+ ## Compression & Architecture Profile
53
+
54
+ > **SmoothQuantCompressor**: Unified BaseCompressor wrapper for SmoothQuant W8A8 with compressed-tensors support.
55
+
56
+ | Property | Specification |
57
+ |---|---|
58
+ | **Registry Key (`COMPRESSOR_REGISTRY`)** | `smoothquant` |
59
+ | **NeuralOps Class & Family** | `SmoothQuantCompressor` (`quantization`) |
60
+ | **Source Implementation** | [`neuralops/compression/quantization/smoothquant.py`](https://github.com/tahamajs/NeuralOps/blob/main/neuralops/compression/quantization/smoothquant.py) |
61
+ | **Base Architecture** | [`google-bert/bert-base-uncased`](https://huggingface.co/google-bert/bert-base-uncased) (`109,483,778` parameters, 12 layers, $D=768$) |
62
+ | **Showcase Rank (`k`)** | **`32`** (`LoRA Recovery Applied: False`) |
63
+ | **Compressed Parameter Count** | **`109,483,778`** (`-0.00%` reduction, **`1.000x`** physical compression ratio) |
64
+ | **Teacher Baseline SST-2 Acc** | `91.63%` |
65
+ | **Showcase Final SST-2 Acc** | **`91.74%`** (Zero-Shot: `91.74%`, Parity Delta: `+0.11%`) |
66
+
67
+ ---
68
+
69
+ ## Multi-Rank Pareto Evaluation (`k=256, 128, 64, 32`)
70
+
71
+ Evaluated physically on **NVIDIA GeForce RTX 3090 (24GB)** on the GLUE SST-2 validation split (`872` samples). Whenever zero-shot compression drops accuracy by `> 2.0%` alongside `> 1.10x` reduction, NeuralOps applies targeted `r=8` LoRA healing (`50` steps, `lr=2e-4`, 45s budget):
72
+
73
+ | Rank (`k`) | Physical Ratio | Compressed Params (Reduction) | Zero-Shot Acc | Healed Acc | Parity vs Teacher (`91.63%`) |
74
+ |:---:|:---:|---:|---:|---:|---:|
75
+ | **`256`** | `1.00x` | `109,483,778` (`-0.0%`) | `91.74%` | **`91.74%`** | `+0.11%` |
76
+ | **`128`** | `1.00x` | `109,483,778` (`-0.0%`) | `91.74%` | **`91.74%`** | `+0.11%` |
77
+ | **`64`** | `1.00x` | `109,483,778` (`-0.0%`) | `91.74%` | **`91.74%`** | `+0.11%` |
78
+ | **`32` ⭐** | `1.00x` | `109,483,778` (`-0.0%`) | `91.74%` | **`91.74%`** | `+0.11%` |
79
+
80
+ *(⭐ indicates the serialized checkpoint packaged in this Hugging Face repository.)*
81
+
82
+ ---
83
+
84
+ ## Packaged Artifacts
85
+
86
+ - **`pytorch_model.bin`**: Full serialized compressed BERT state dictionary (with low-rank factors / quantization buffers / LoRA healing adapters baked in).
87
+ - **`config.json`**: Model configuration tagged with `compression_method: "smoothquant"` and `compression_rank: 32`.
88
+ - **`compression_summary.json`**: Complete machine-readable telemetry across all evaluated ranks `[256, 128, 64, 32]`.
89
+ - **Tokenizer Assets**: Standard `bert-base-uncased` vocabulary (`vocab.txt`, `tokenizer.json`, `tokenizer_config.json`).
90
+
91
+ ---
92
+
93
+ ## Quick Start (Reconstruction & Inference)
94
+
95
+ ```python
96
+ import torch
97
+ from huggingface_hub import hf_hub_download
98
+ from transformers import AutoTokenizer, BertForSequenceClassification
99
+ from neuralops.core.compression_pipeline import compress_model
100
+
101
+ repo_id = "tahamajs/bert-comp-smoothquant-sst2"
102
+ tokenizer = AutoTokenizer.from_pretrained(repo_id)
103
+
104
+ # Reconstruct compressed NeuralOps topology and load serialized compressed + healed weights
105
+ model = BertForSequenceClassification.from_pretrained("google-bert/bert-base-uncased", num_labels=2)
106
+ model = compress_model(model, {"compressors": [{"method": "smoothquant", "rank": 32}]})
107
+
108
+ ckpt_path = hf_hub_download(repo_id=repo_id, filename="pytorch_model.bin")
109
+ state_dict = torch.load(ckpt_path, map_location="cpu")
110
+ model.load_state_dict(state_dict, strict=False)
111
+ model.eval()
112
+
113
+ inputs = tokenizer("NeuralOps achieves strong compression without sacrificing downstream accuracy.", return_tensors="pt")
114
+ with torch.no_grad():
115
+ logits = model(**inputs).logits
116
+ pred = logits.argmax(dim=-1).item()
117
+ print("Predicted SST-2 Sentiment (0=Negative, 1=Positive):", pred)
118
+ ```
119
+
120
+ ---
121
+
122
+ ## Citation
123
+
124
+ ```bibtex
125
+ @software{neuralops2026compressorzoo,
126
+ title = {NeuralOps: Multi-Rank Compressed BERT-SST2 Zoo with Pareto LoRA Recovery},
127
+ author = {Majlesi, Taha},
128
+ year = {2026},
129
+ url = {https://github.com/tahamajs/NeuralOps}
130
+ }
131
+ ```
compression_summary.json ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "compressor": "smoothquant",
3
+ "showcase_rank": 32,
4
+ "showcase_metrics": {
5
+ "compressor": "smoothquant",
6
+ "rank": 32,
7
+ "ratio": 1.0,
8
+ "comp_params": 109483778,
9
+ "orig_params": 109483778,
10
+ "zero_shot_acc": 91.74,
11
+ "zero_shot_loss": 0.2429,
12
+ "delta_zero": 0.11,
13
+ "lora_healed": false,
14
+ "healed_acc": 91.74,
15
+ "timestamp": "2026-11-10T19:54:10.581460Z",
16
+ "status": "PASSED"
17
+ },
18
+ "all_ranks": [
19
+ {
20
+ "compressor": "smoothquant",
21
+ "rank": 256,
22
+ "ratio": 1.0,
23
+ "comp_params": 109483778,
24
+ "orig_params": 109483778,
25
+ "zero_shot_acc": 91.74,
26
+ "zero_shot_loss": 0.2429,
27
+ "delta_zero": 0.11,
28
+ "lora_healed": false,
29
+ "healed_acc": 91.74,
30
+ "timestamp": "2026-11-10T19:53:59.025154Z",
31
+ "status": "PASSED"
32
+ },
33
+ {
34
+ "compressor": "smoothquant",
35
+ "rank": 128,
36
+ "ratio": 1.0,
37
+ "comp_params": 109483778,
38
+ "orig_params": 109483778,
39
+ "zero_shot_acc": 91.74,
40
+ "zero_shot_loss": 0.2429,
41
+ "delta_zero": 0.11,
42
+ "lora_healed": false,
43
+ "healed_acc": 91.74,
44
+ "timestamp": "2026-11-10T19:54:04.585256Z",
45
+ "status": "PASSED"
46
+ },
47
+ {
48
+ "compressor": "smoothquant",
49
+ "rank": 64,
50
+ "ratio": 1.0,
51
+ "comp_params": 109483778,
52
+ "orig_params": 109483778,
53
+ "zero_shot_acc": 91.74,
54
+ "zero_shot_loss": 0.2429,
55
+ "delta_zero": 0.11,
56
+ "lora_healed": false,
57
+ "healed_acc": 91.74,
58
+ "timestamp": "2026-11-10T19:54:08.056739Z",
59
+ "status": "PASSED"
60
+ },
61
+ {
62
+ "compressor": "smoothquant",
63
+ "rank": 32,
64
+ "ratio": 1.0,
65
+ "comp_params": 109483778,
66
+ "orig_params": 109483778,
67
+ "zero_shot_acc": 91.74,
68
+ "zero_shot_loss": 0.2429,
69
+ "delta_zero": 0.11,
70
+ "lora_healed": false,
71
+ "healed_acc": 91.74,
72
+ "timestamp": "2026-11-10T19:54:10.581460Z",
73
+ "status": "PASSED"
74
+ }
75
+ ],
76
+ "timestamp": "2026-11-10T22:14:39.895339Z"
77
+ }
config.json ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "return_dict": true,
3
+ "output_hidden_states": false,
4
+ "output_attentions": false,
5
+ "torchscript": false,
6
+ "torch_dtype": "float32",
7
+ "use_bfloat16": false,
8
+ "tf_legacy_loss": false,
9
+ "pruned_heads": {},
10
+ "tie_word_embeddings": true,
11
+ "chunk_size_feed_forward": 0,
12
+ "is_encoder_decoder": false,
13
+ "is_decoder": false,
14
+ "cross_attention_hidden_size": null,
15
+ "add_cross_attention": false,
16
+ "tie_encoder_decoder": false,
17
+ "max_length": 20,
18
+ "min_length": 0,
19
+ "do_sample": false,
20
+ "early_stopping": false,
21
+ "num_beams": 1,
22
+ "num_beam_groups": 1,
23
+ "diversity_penalty": 0.0,
24
+ "temperature": 1.0,
25
+ "top_k": 50,
26
+ "top_p": 1.0,
27
+ "typical_p": 1.0,
28
+ "repetition_penalty": 1.0,
29
+ "length_penalty": 1.0,
30
+ "no_repeat_ngram_size": 0,
31
+ "encoder_no_repeat_ngram_size": 0,
32
+ "bad_words_ids": null,
33
+ "num_return_sequences": 1,
34
+ "output_scores": false,
35
+ "return_dict_in_generate": false,
36
+ "forced_bos_token_id": null,
37
+ "forced_eos_token_id": null,
38
+ "remove_invalid_values": false,
39
+ "exponential_decay_length_penalty": null,
40
+ "suppress_tokens": null,
41
+ "begin_suppress_tokens": null,
42
+ "architectures": [
43
+ "BertForSequenceClassification"
44
+ ],
45
+ "finetuning_task": null,
46
+ "id2label": {
47
+ "0": "LABEL_0",
48
+ "1": "LABEL_1"
49
+ },
50
+ "label2id": {
51
+ "LABEL_0": 0,
52
+ "LABEL_1": 1
53
+ },
54
+ "tokenizer_class": null,
55
+ "prefix": null,
56
+ "bos_token_id": null,
57
+ "pad_token_id": 0,
58
+ "eos_token_id": null,
59
+ "sep_token_id": null,
60
+ "decoder_start_token_id": null,
61
+ "task_specific_params": null,
62
+ "problem_type": null,
63
+ "_name_or_path": "google-bert/bert-base-uncased",
64
+ "transformers_version": "4.45.2",
65
+ "gradient_checkpointing": false,
66
+ "model_type": "bert",
67
+ "vocab_size": 30522,
68
+ "hidden_size": 768,
69
+ "num_hidden_layers": 12,
70
+ "num_attention_heads": 12,
71
+ "hidden_act": "gelu",
72
+ "intermediate_size": 3072,
73
+ "hidden_dropout_prob": 0.1,
74
+ "attention_probs_dropout_prob": 0.1,
75
+ "max_position_embeddings": 512,
76
+ "type_vocab_size": 2,
77
+ "initializer_range": 0.02,
78
+ "layer_norm_eps": 1e-12,
79
+ "position_embedding_type": "absolute",
80
+ "use_cache": true,
81
+ "classifier_dropout": null,
82
+ "base_model_name_or_path": "google-bert/bert-base-uncased",
83
+ "license": "mit",
84
+ "compression_method": "smoothquant",
85
+ "compression_rank": 32,
86
+ "compression_ratio": 1.0
87
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3c267d51678eb7021ce769118fe6ae68ad82cab462f6be3e59d3476d91d146e7
3
+ size 305173279
special_tokens_map.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": {
3
+ "content": "[CLS]",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "mask_token": {
10
+ "content": "[MASK]",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "[PAD]",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "sep_token": {
24
+ "content": "[SEP]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "unk_token": {
31
+ "content": "[UNK]",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ }
37
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "100": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "101": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "102": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "103": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_lower_case": true,
47
+ "mask_token": "[MASK]",
48
+ "max_length": 128,
49
+ "model_max_length": 512,
50
+ "pad_to_multiple_of": null,
51
+ "pad_token": "[PAD]",
52
+ "pad_token_type_id": 0,
53
+ "padding_side": "right",
54
+ "sep_token": "[SEP]",
55
+ "stride": 0,
56
+ "strip_accents": null,
57
+ "tokenize_chinese_chars": true,
58
+ "tokenizer_class": "BertTokenizer",
59
+ "truncation_side": "right",
60
+ "truncation_strategy": "longest_first",
61
+ "unk_token": "[UNK]"
62
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff