Angshul commited on
Commit
a42d9ad
·
verified ·
1 Parent(s): 4a47ec0

Upload 14 files

Browse files
Dockerfile ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM python:3.12-slim-bookworm
2
+ ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 PYTHONDONTWRITEBYTECODE=1
3
+ RUN apt-get update \
4
+ && apt-get install -y --no-install-recommends g++ \
5
+ && rm -rf /var/lib/apt/lists/* \
6
+ && useradd -m -u 1000 user \
7
+ && mkdir -p /models /home/user/work \
8
+ && chown -R user:user /home/user/work
9
+ WORKDIR /app
10
+ COPY pyproject.toml README.md space_app.py ./
11
+ COPY tensor_pruning/ ./tensor_pruning/
12
+ RUN python -m pip install '.[space]'
13
+ USER user
14
+ EXPOSE 7860
15
+ CMD ["python", "-m", "uvicorn", "space_app:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
README.md CHANGED
@@ -1,11 +1,79 @@
1
  ---
2
- title: Tensor Pruning
3
- emoji: 🏃
4
- colorFrom: indigo
5
- colorTo: yellow
6
  sdk: docker
7
- pinned: false
8
- license: apache-2.0
9
  ---
10
 
11
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: Tensor Pruning API
3
+ emoji: ✂️
4
+ colorFrom: blue
5
+ colorTo: purple
6
  sdk: docker
7
+ app_port: 7860
 
8
  ---
9
 
10
+ # Tensor Pruning API
11
+
12
+ A Docker Space for [the Tensor Pruning API](https://github.com/AngshulMajumdar/Tensor-Pruning-API). It selects query-weight masks for dense Llama-layout checkpoints with a single retention count across all layers. The CPU selector uses the nested weighted \((2,4,8)\) residual objective from the API. A job publishes one Boolean NumPy mask per query projection, with axes `[head, channel, input]`, plus the tensor names and retained counts in its status response.
13
+
14
+ ## Run on Hugging Face
15
+
16
+ Unzip the release and run the following from the directory containing `publish.py`. Log in through the Hugging Face CLI using an account with permission to create Spaces. Omit the model flag if you will attach the model later.
17
+
18
+ ```sh
19
+ python -m pip install -U huggingface_hub
20
+ hf auth login
21
+ python publish.py --repo Angshul/Tensor-Pruning-API --model-repo OWNER/MODEL
22
+ ```
23
+
24
+ The helper creates a private Docker Space with the chosen model mounted read-only at `/models`. If the repository already exists, review its contents before uploading the deployment files with the Hugging Face UI or CLI. Use a model that your account can access. If `config.json` and the weight files are in a mounted subdirectory, send that relative directory as `checkpoint`.
25
+
26
+ In Space Settings, set a secret named `TENSOR_PRUNING_API_TOKEN` to an operator-chosen random token. Configure `PRUNING_THREADS` to match the assigned CPUs if necessary. Keep `PRUNING_WORK_ROOT` on local storage that supports memory mapping and file preallocation. The default is `/home/user/work`. A model mount supplies the checkpoint. The job's query staging files, index scratch, and masks occupy local disk.
27
+
28
+ ## Submit and retrieve a job
29
+
30
+ Open the Space's direct app URL from its Hugging Face page. The root opens the API documentation at `/docs`. The health route is `/health`. The example uses a private Space, so it sends a Hugging Face access token for the proxy and a separate pruning token for the app.
31
+
32
+ ```sh
33
+ curl -fsS "$APP_URL/v1/checkpoint-jobs" \
34
+ -H "Authorization: Bearer $HF_TOKEN" \
35
+ -H "X-Tensor-Pruning-Token: $PRUNING_TOKEN" \
36
+ -H 'Content-Type: application/json' \
37
+ -d '{"checkpoint":".","keep":1610612736,"seed":0}'
38
+
39
+ curl -fsS "$APP_URL/v1/jobs/$JOB_ID" \
40
+ -H "Authorization: Bearer $HF_TOKEN" \
41
+ -H "X-Tensor-Pruning-Token: $PRUNING_TOKEN"
42
+
43
+ curl -fsS "$APP_URL/v1/jobs/$JOB_ID/masks/0" \
44
+ -H "Authorization: Bearer $HF_TOKEN" \
45
+ -H "X-Tensor-Pruning-Token: $PRUNING_TOKEN" \
46
+ -o mask-0000.npy
47
+ ```
48
+
49
+ The example retention count is 30% of the Llama 3.1 70B query slots. The service runs one job at a time and bounds the waiting queue. Status and mask routes also serve the original staged NumPy-manifest API at `POST /v1/jobs`. Download completed masks before a Space restart, which can clear local storage. Clear obsolete job directories when the service is stopped to recover disk space. Interrupted jobs can be resubmitted.
50
+
51
+ The checkpoint endpoint accepts the standard `model.safetensors` file or `model.safetensors.index.json` and its shards. It reads one `model.layers.N.self_attn.q_proj.weight` tensor per layer. It supports dense FP16, BF16, FP32, and FP64 weights. The worker leaves model weights unchanged. Mask application is a downstream step.
52
+
53
+ The default objective uses unit input moments. To use activation weighting, supply `moments_manifest` as a relative path to a JSON object mapping each eligible tensor name to a corresponding NumPy vector file. Stage the JSON and vectors under `PRUNING_INPUT_ROOT`, which defaults to `/home/user/work/inputs`, or mount a separate read-only volume there. Dense-model calibration is performed externally.
54
+
55
+ ## Size a Llama 3.1 70B job
56
+
57
+ The standard layout has 80 layers and \(8192\times8192\) query matrices, yielding 5,368,709,120 eligible slots. For FP16 or BF16 and an interior retention budget:
58
+
59
+ | Local file use | Bytes per query slot | Approximate decimal GB |
60
+ | --- | ---: | ---: |
61
+ | Staged query weights | 2 | 10.74 |
62
+ | Channel-local partition indices | 4 | 21.47 |
63
+ | Output Boolean masks | 1 | 5.37 |
64
+ | Total | 7 | 37.58 |
65
+
66
+ The API checks available disk before staging and requires roughly 38.66 GB free for this job, including its reserve. Older output masks, the image, and other local files consume additional disk space. The model repository must be mounted separately for the sizing above. The worker reads only the query byte ranges from the mounted shards and memory-maps local working files. The figures are disk usage estimates. RAM and elapsed time depend on local I/O, page caching, and CPU allocation. The 70B layout was checked with synthetic metadata. A full 70B selection run remains to be measured.
67
+
68
+ ## Run locally
69
+
70
+ ```sh
71
+ docker build -t tensor-pruning-space .
72
+ docker run --rm -p 7860:7860 \
73
+ -v "$CHECKPOINT_DIR:/models:ro" \
74
+ -e TENSOR_PRUNING_API_TOKEN \
75
+ -e PRUNING_THREADS=4 \
76
+ tensor-pruning-space
77
+ ```
78
+
79
+ Use `http://localhost:7860/docs` with the pruning token. The local container needs sufficient free disk or a writable local bind mount for the work root. The Python integration tests can be run with `python -m pip install '.[space,test]'` followed by `python -m unittest discover -s tests -v`.
publish.py ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Create the private Docker Space and upload this deployment source tree."""
2
+
3
+ import argparse
4
+ from pathlib import Path
5
+
6
+ from huggingface_hub import HfApi, Volume
7
+
8
+
9
+ def main():
10
+ parser = argparse.ArgumentParser(description=__doc__)
11
+ parser.add_argument("--repo", default="Angshul/Tensor-Pruning-API")
12
+ parser.add_argument("--model-repo", help="Optionally mount a model repo read-only at /models")
13
+ args = parser.parse_args()
14
+ api = HfApi()
15
+ options = {}
16
+ if args.model_repo:
17
+ options["space_volumes"] = [
18
+ Volume(type="model", source=args.model_repo, mount_path="/models", read_only=True)
19
+ ]
20
+ api.create_repo(args.repo, repo_type="space", space_sdk="docker", private=True, **options)
21
+ api.upload_folder(repo_id=args.repo, repo_type="space", folder_path=Path(__file__).parent,
22
+ allow_patterns=["README.md", "Dockerfile", "space_app.py",
23
+ "pyproject.toml", "tensor_pruning/*.py",
24
+ "tensor_pruning/kernel.cpp"],
25
+ commit_message="Deploy the CPU tensor-pruning API")
26
+ print(f"https://huggingface.co/spaces/{args.repo}")
27
+
28
+
29
+ if __name__ == "__main__":
30
+ main()
pyproject.toml ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [build-system]
2
+ requires = ["setuptools>=68"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "tensor-pruning"
7
+ version = "1.0.0"
8
+ description = "Training-free global-budget tensor pruning for LLM query projections"
9
+ readme = "README.md"
10
+ requires-python = ">=3.10"
11
+ dependencies = ["numpy>=1.24"]
12
+
13
+ [project.optional-dependencies]
14
+ http = ["fastapi>=0.115,<1", "uvicorn>=0.30,<1"]
15
+ space = ["fastapi>=0.115,<1", "uvicorn>=0.30,<1", "ml-dtypes>=0.4,<1"]
16
+ test = ["httpx>=0.27,<1"]
17
+
18
+ [project.scripts]
19
+ tensor-prune = "tensor_pruning.cli:main"
20
+
21
+ [tool.setuptools.packages.find]
22
+ include = ["tensor_pruning*"]
23
+
24
+ [tool.setuptools.package-data]
25
+ tensor_pruning = ["kernel.cpp"]
space_app.py ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Configure the Tensor Pruning API for a single Docker Space replica."""
2
+
3
+ import os
4
+ from pathlib import Path
5
+
6
+ from fastapi.responses import RedirectResponse
7
+
8
+ from tensor_pruning.service import create_app
9
+
10
+
11
+ work = Path(os.getenv("PRUNING_WORK_ROOT", "/home/user/work"))
12
+ inputs = Path(os.getenv("PRUNING_INPUT_ROOT", str(work / "inputs")))
13
+ inputs.mkdir(parents=True, exist_ok=True)
14
+
15
+ app = create_app(
16
+ inputs,
17
+ work / "jobs",
18
+ workers=1,
19
+ queue_size=int(os.getenv("PRUNING_QUEUE_SIZE", "1")),
20
+ threads_per_job=int(os.getenv("PRUNING_THREADS", str(min(2, len(os.sched_getaffinity(0)))))),
21
+ max_weights=int(os.getenv("PRUNING_MAX_WEIGHTS", "6000000000")),
22
+ token=os.getenv("TENSOR_PRUNING_API_TOKEN"),
23
+ require_token=True,
24
+ checkpoint_root=os.getenv("PRUNING_CHECKPOINT_ROOT", "/models"),
25
+ )
26
+
27
+
28
+ @app.get("/", include_in_schema=False)
29
+ def index():
30
+ return RedirectResponse("/docs")
tensor_pruning/__init__.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ """Training-free, global-budget tensor pruning for LLM query projections."""
2
+
3
+ from .core import prune_queries
4
+ from .files import prune_manifest
5
+
6
+ __all__ = ["prune_queries", "prune_manifest"]
tensor_pruning/checkpoint.py ADDED
@@ -0,0 +1,176 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Stage Llama-layout query ranges from safetensors and select one global mask."""
2
+
3
+ from dataclasses import dataclass
4
+ import json
5
+ from pathlib import Path
6
+ import shutil
7
+ import struct
8
+ import tempfile
9
+
10
+ import ml_dtypes
11
+ import numpy as np
12
+
13
+ from .files import _input_path, _open_npy, _publish_masks
14
+
15
+
16
+ _DTYPES = {"F16": np.dtype("<f2"), "BF16": np.dtype(ml_dtypes.bfloat16),
17
+ "F32": np.dtype("<f4"), "F64": np.dtype("<f8")}
18
+ _CHUNK = 8 * 1024**2
19
+ _HEADROOM = 1024**3
20
+
21
+
22
+ def _object(pairs):
23
+ result = {}
24
+ for key, value in pairs:
25
+ if key in result:
26
+ raise ValueError(f"Duplicate JSON key: {key}")
27
+ result[key] = value
28
+ return result
29
+
30
+
31
+ def _json_object(raw):
32
+ value = json.loads(raw, object_pairs_hook=_object)
33
+ if not isinstance(value, dict):
34
+ raise ValueError("Expected a JSON object")
35
+ return value
36
+
37
+
38
+ def _json_file(path, limit):
39
+ with path.open("rb") as stream:
40
+ raw = stream.read(limit + 1)
41
+ if len(raw) > limit:
42
+ raise ValueError(f"{path.name} exceeds the supported metadata size")
43
+ return _json_object(raw)
44
+
45
+
46
+ def _layout(config):
47
+ layers = config.get("num_hidden_layers")
48
+ heads = config.get("num_attention_heads")
49
+ width = config.get("hidden_size")
50
+ if any(type(n) is not int or n < 1 for n in (layers, heads, width)) or layers > 1024:
51
+ raise ValueError("Expected valid Llama-layout layer, head and hidden sizes")
52
+ channels = config.get("head_dim")
53
+ if channels is None:
54
+ channels, extra = divmod(width, heads)
55
+ if extra:
56
+ raise ValueError("hidden_size must be divisible by num_attention_heads")
57
+ if type(channels) is not int or channels < 1:
58
+ raise ValueError("Invalid head_dim")
59
+ return layers, (heads, channels, width)
60
+
61
+
62
+ def _header(path):
63
+ with path.open("rb") as stream:
64
+ prefix = stream.read(8)
65
+ if len(prefix) != 8:
66
+ raise ValueError(f"Incomplete safetensors header: {path.name}")
67
+ size, = struct.unpack("<Q", prefix)
68
+ if not 2 <= size <= 100_000_000:
69
+ raise ValueError(f"Invalid safetensors header length: {path.name}")
70
+ raw = stream.read(size)
71
+ if len(raw) != size or not raw.startswith(b"{"):
72
+ raise ValueError(f"Invalid safetensors header: {path.name}")
73
+ start = 8 + size
74
+ return _json_object(raw), start, path.stat().st_size - start
75
+
76
+
77
+ @dataclass(frozen=True)
78
+ class QueryRange:
79
+ name: str
80
+ shard: Path
81
+ offset: int
82
+ length: int
83
+ shape: tuple[int, int, int]
84
+ dtype: np.dtype
85
+
86
+
87
+ def _queries(checkpoint, max_weights):
88
+ root = Path(checkpoint).resolve(strict=True)
89
+ layers, shape = _layout(_json_file(_input_path(root, "config.json", ".json"), 1024**2))
90
+ heads, channels, width = shape
91
+ total = layers * heads * channels * width
92
+ if total > max_weights:
93
+ raise ValueError("Checkpoint exceeds the configured eligible weight limit")
94
+ names = [f"model.layers.{layer}.self_attn.q_proj.weight" for layer in range(layers)]
95
+ index = root / "model.safetensors.index.json"
96
+ if index.exists():
97
+ mapping = _json_file(_input_path(root, index.name, ".json"), 16 * 1024**2).get("weight_map")
98
+ if not isinstance(mapping, dict) or any(name not in mapping for name in names):
99
+ raise ValueError("Expected one named Llama-layout query tensor for every layer")
100
+ else:
101
+ mapping = dict.fromkeys(names, "model.safetensors")
102
+
103
+ headers = {}
104
+ queries = []
105
+ ranges = {}
106
+ for name in names:
107
+ shard = _input_path(root, mapping[name], ".safetensors")
108
+ if shard not in headers:
109
+ headers[shard] = _header(shard)
110
+ header, start, payload_size = headers[shard]
111
+ item = header.get(name)
112
+ if not isinstance(item, dict) or item.get("shape") != [heads * channels, width]:
113
+ raise ValueError(f"Unexpected query shape: {name}")
114
+ dtype = _DTYPES.get(item.get("dtype"))
115
+ if dtype is None:
116
+ raise ValueError(f"Expected dense floating-point query weights: {name}")
117
+ offsets = item.get("data_offsets")
118
+ length = heads * channels * width * dtype.itemsize
119
+ if (not isinstance(offsets, list) or len(offsets) != 2
120
+ or any(type(n) is not int for n in offsets)):
121
+ raise ValueError(f"Invalid safetensors offsets: {name}")
122
+ low, high = offsets
123
+ if not 0 <= low < high <= payload_size or high - low != length:
124
+ raise ValueError(f"Invalid safetensors byte range: {name}")
125
+ ranges.setdefault(shard, []).append((low, high))
126
+ queries.append(QueryRange(name, shard, start + low, length, shape, dtype))
127
+ for shard, spans in ranges.items():
128
+ ordered = sorted(spans)
129
+ if any(left[1] > right[0] for left, right in zip(ordered, ordered[1:])):
130
+ raise ValueError(f"Overlapping query byte ranges: {shard.name}")
131
+ return queries, total
132
+
133
+
134
+ def _copy_query(query, path):
135
+ with query.shard.open("rb") as src, path.open("wb") as dst:
136
+ src.seek(query.offset)
137
+ remaining = query.length
138
+ while remaining:
139
+ block = src.read(min(remaining, _CHUNK))
140
+ if not block:
141
+ raise OSError(f"Shard ended during staging: {query.shard.name}")
142
+ dst.write(block)
143
+ remaining -= len(block)
144
+ return np.memmap(path, dtype=query.dtype, mode="r", shape=query.shape)
145
+
146
+
147
+ def prune_checkpoint(checkpoint, output_dir, *, keep, workspace, max_weights,
148
+ input_root=None, moments_manifest=None, threads=1, seed=0):
149
+ """Read only the eligible query bytes and write per-layer Boolean masks."""
150
+ queries, total = _queries(checkpoint, max_weights)
151
+ if not 0 <= keep <= total:
152
+ raise ValueError("keep must be between zero and the eligible query slot count")
153
+ names = [query.name for query in queries]
154
+ moments = None
155
+ if moments_manifest is not None:
156
+ root = Path(input_root).resolve(strict=True)
157
+ sources = _json_file(_input_path(root, moments_manifest, ".json"), 1024**2)
158
+ if set(sources) != set(names):
159
+ raise ValueError("Moment manifest must map every eligible query name to an .npy vector")
160
+ moments = [_open_npy(root, sources[name]) for name in names]
161
+ workspace = Path(workspace)
162
+ output_dir = Path(output_dir)
163
+ if workspace.stat().st_dev != output_dir.parent.stat().st_dev:
164
+ raise ValueError("Checkpoint work and output must use the same local filesystem")
165
+ needed = sum(query.length for query in queries) + total + 4096 * len(queries)
166
+ if 0 < keep < total:
167
+ needed += 4 * total
168
+ if shutil.disk_usage(workspace).free < needed + _HEADROOM:
169
+ raise OSError(f"Insufficient local disk: need {needed + _HEADROOM:,} bytes free")
170
+ with tempfile.TemporaryDirectory(prefix="query-staging-", dir=workspace) as stage:
171
+ staged = {}
172
+ for index, query in sorted(enumerate(queries), key=lambda row: (row[1].shard, row[1].offset)):
173
+ staged[index] = _copy_query(query, Path(stage) / f"{index:04d}.bin")
174
+ arrays = [staged[index] for index in range(len(queries))]
175
+ return _publish_masks(names, arrays, moments, keep, output_dir,
176
+ scratch_dir=workspace, threads=threads, seed=seed)
tensor_pruning/cli.py ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Command-line access to the file API and HTTP service."""
2
+
3
+ import argparse
4
+ import json
5
+ import os
6
+ from pathlib import Path
7
+
8
+ from .files import prune_manifest
9
+
10
+
11
+ def main(argv=None):
12
+ parser = argparse.ArgumentParser(prog="tensor-prune", description=__doc__)
13
+ commands = parser.add_subparsers(dest="command", required=True)
14
+ prune = commands.add_parser("prune", help="prune a memory-mapped manifest")
15
+ prune.add_argument("--input-root", required=True, type=Path)
16
+ prune.add_argument("--manifest", required=True)
17
+ prune.add_argument("--output", required=True, type=Path)
18
+ prune.add_argument("--scratch-dir", type=Path)
19
+ prune.add_argument("--threads", type=int, default=1)
20
+ prune.add_argument("--seed", type=int, default=0)
21
+
22
+ serve = commands.add_parser("serve", help="run the bounded HTTP job service")
23
+ serve.add_argument("--input-root", required=True, type=Path)
24
+ serve.add_argument("--job-root", required=True, type=Path)
25
+ serve.add_argument("--host", default="127.0.0.1")
26
+ serve.add_argument("--port", type=int, default=8080)
27
+ serve.add_argument("--workers", type=int, default=1, help="simultaneous jobs in this process")
28
+ serve.add_argument("--queue-size", type=int, default=2)
29
+ serve.add_argument("--threads-per-job", type=int, default=1)
30
+ serve.add_argument("--max-weights", type=int, default=100_000_000)
31
+
32
+ args = parser.parse_args(argv)
33
+ if args.command == "prune":
34
+ result = prune_manifest(args.input_root, args.manifest, args.output,
35
+ scratch_dir=args.scratch_dir, threads=args.threads, seed=args.seed)
36
+ print(json.dumps(result))
37
+ return
38
+
39
+ try:
40
+ import uvicorn
41
+ from .service import create_app
42
+ except ImportError:
43
+ parser.error("install the HTTP extras with pip install '.[http]'")
44
+ token = os.environ.get("TENSOR_PRUNING_API_TOKEN")
45
+ if args.host not in {"127.0.0.1", "::1", "localhost"} and not token:
46
+ parser.error("set TENSOR_PRUNING_API_TOKEN before binding to a non-loopback address")
47
+ app = create_app(args.input_root, args.job_root, workers=args.workers,
48
+ queue_size=args.queue_size, threads_per_job=args.threads_per_job,
49
+ max_weights=args.max_weights, token=token)
50
+ uvicorn.run(app, host=args.host, port=args.port)
51
+
52
+
53
+ if __name__ == "__main__":
54
+ main()
tensor_pruning/core.py ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """NumPy interface to the OpenMP selector."""
2
+
3
+ from contextlib import contextmanager
4
+ import ctypes as ct
5
+ from functools import lru_cache
6
+ import mmap
7
+ import operator
8
+ import os
9
+ from pathlib import Path
10
+ import shlex
11
+ import subprocess
12
+ import tempfile
13
+ from threading import Lock
14
+
15
+ import numpy as np
16
+
17
+
18
+ class Projection(ct.Structure):
19
+ _fields_ = [
20
+ ("weights", ct.c_void_p), ("moments", ct.c_void_p), ("mask", ct.c_void_p),
21
+ ("heads", ct.c_size_t), ("channels", ct.c_size_t), ("inputs", ct.c_size_t),
22
+ ("dtype", ct.c_int),
23
+ ]
24
+
25
+
26
+ _BUILD_LOCK = Lock()
27
+ _DTYPES = {"float16": 2, "bfloat16": 3, "float32": 4, "float64": 8}
28
+
29
+
30
+ @lru_cache(maxsize=1)
31
+ def _compile():
32
+ compiler = shlex.split(os.environ.get("CXX", "g++"))
33
+ source = Path(__file__).with_name("kernel.cpp")
34
+ with tempfile.TemporaryDirectory(prefix="tensor-pruning-build-") as directory:
35
+ library = Path(directory) / "kernel.so"
36
+ command = [*compiler, "-std=c++17", "-O3", "-fopenmp", "-shared", "-fPIC",
37
+ str(source), "-o", str(library)]
38
+ try:
39
+ subprocess.run(command, check=True, capture_output=True, text=True)
40
+ except (OSError, subprocess.CalledProcessError) as exc:
41
+ message = getattr(exc, "stderr", None) or str(exc)
42
+ raise RuntimeError(f"An OpenMP C++17 compiler is required. {message}") from exc
43
+ function = ct.CDLL(str(library)).tensor_prune_many
44
+ function.argtypes = [ct.POINTER(Projection), ct.c_size_t, ct.c_void_p,
45
+ ct.c_size_t, ct.c_int, ct.c_uint64]
46
+ function.restype = ct.c_int
47
+ return function
48
+
49
+
50
+ def _kernel():
51
+ with _BUILD_LOCK:
52
+ return _compile()
53
+
54
+
55
+ def _inputs(projections, input_second_moments):
56
+ arrays = tuple(np.asarray(w) for w in projections)
57
+ if not arrays:
58
+ raise ValueError("at least one query projection is required")
59
+ moments = (None,) * len(arrays) if input_second_moments is None else tuple(input_second_moments)
60
+ if len(moments) != len(arrays):
61
+ raise ValueError("provide one input-second-moment vector per projection")
62
+ checked = []
63
+ for w, v in zip(arrays, moments):
64
+ if w.ndim != 3 or not all(w.shape):
65
+ raise ValueError("projections must have nonempty [head, channel, input] shape")
66
+ if w.shape[-1] > np.iinfo(np.uint32).max:
67
+ raise ValueError("input axis exceeds the 32-bit channel index limit")
68
+ if w.dtype.name not in _DTYPES or not w.dtype.isnative:
69
+ raise TypeError("use native-endian float16, bfloat16, float32, or float64 weights")
70
+ if not w.flags.c_contiguous or not w.flags.aligned:
71
+ raise ValueError("projections must be C-contiguous and aligned")
72
+ if v is None:
73
+ v = np.ones(w.shape[-1], dtype=np.float64)
74
+ else:
75
+ v = np.asarray(v)
76
+ if v.dtype.kind not in "fiu" or v.shape != (w.shape[-1],):
77
+ raise ValueError("each moment vector must be real with shape [input]")
78
+ v = np.require(v, dtype=np.float64, requirements=["C", "A"])
79
+ if not np.all(np.isfinite(v) & (v >= 0)):
80
+ raise ValueError("input second moments must be finite and nonnegative")
81
+ checked.append(v)
82
+ return arrays, checked
83
+
84
+
85
+ def _options(total, keep, threads, seed):
86
+ keep = operator.index(keep)
87
+ if not 0 <= keep <= total:
88
+ raise ValueError("keep must be between zero and the total number of weights")
89
+ threads = operator.index(threads)
90
+ if not 1 <= threads <= np.iinfo(np.int32).max:
91
+ raise ValueError("threads must be a positive integer")
92
+ return keep, threads, operator.index(seed) % (1 << 64)
93
+
94
+
95
+ @contextmanager
96
+ def _partition_workspace(count, scratch_dir):
97
+ if count == 0:
98
+ yield None
99
+ elif scratch_dir is None:
100
+ indices = np.empty(count, dtype=np.uint32)
101
+ yield indices.ctypes.data
102
+ else:
103
+ with tempfile.TemporaryFile(dir=scratch_dir) as file:
104
+ size = count * np.dtype(np.uint32).itemsize
105
+ os.posix_fallocate(file.fileno(), 0, size)
106
+ with mmap.mmap(file.fileno(), size) as region:
107
+ yield ct.addressof(ct.c_char.from_buffer(region))
108
+
109
+
110
+ def _run(arrays, moments, masks, total, keep, threads, seed, scratch_dir):
111
+ descriptors = (Projection * len(arrays))(*[
112
+ Projection(w.ctypes.data, v.ctypes.data, mask.ctypes.data, *w.shape, _DTYPES[w.dtype.name])
113
+ for w, v, mask in zip(arrays, moments, masks)
114
+ ])
115
+ native = _kernel()
116
+ with _partition_workspace(total if 0 < keep < total else 0, scratch_dir) as scratch:
117
+ status = native(descriptors, len(descriptors), scratch, keep, threads, seed)
118
+ if status == 1:
119
+ raise ValueError("weights and intermediate energies must be finite")
120
+ if status == 2:
121
+ raise MemoryError("native partition allocation failed")
122
+ if status:
123
+ raise RuntimeError(f"native selector failed with status {status}")
124
+
125
+
126
+ def prune_queries(projections, keep, *, input_second_moments=None, threads=1,
127
+ seed=0, scratch_dir=None):
128
+ """Return one boolean mask per [head, channel, input] projection.
129
+
130
+ All projections participate in a single retention budget, even if they
131
+ have different shapes or dtypes. Masks contain exactly keep true entries.
132
+ Existing zeros count as eligible slots. Inputs can be read-only memmaps.
133
+ Set scratch_dir to use a disk-backed partition workspace.
134
+ """
135
+ arrays, moments = _inputs(projections, input_second_moments)
136
+ total = sum(w.size for w in arrays)
137
+ keep, threads, seed = _options(total, keep, threads, seed)
138
+ masks = tuple(np.empty(w.shape, dtype=bool) for w in arrays)
139
+ _run(arrays, moments, masks, total, keep, threads, seed, scratch_dir)
140
+ return masks
tensor_pruning/files.py ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Memory-mapped checkpoint manifests and atomic publication of masks."""
2
+
3
+ import fcntl
4
+ import json
5
+ import os
6
+ from pathlib import Path
7
+ import shutil
8
+ import tempfile
9
+
10
+ import numpy as np
11
+
12
+ from .core import _inputs, _options, _run
13
+
14
+
15
+ def _input_path(root, relative, suffix):
16
+ if not isinstance(relative, str) or Path(relative).is_absolute():
17
+ raise ValueError("manifest paths must be relative to the input root")
18
+ path = (root / relative).resolve(strict=True)
19
+ if not path.is_relative_to(root) or not path.is_file() or path.suffix != suffix:
20
+ raise ValueError(f"expected a {suffix} file inside the input root")
21
+ return path
22
+
23
+
24
+ def _open_npy(root, name):
25
+ return np.load(_input_path(root, name, ".npy"), mmap_mode="r", allow_pickle=False)
26
+
27
+
28
+ def _read_manifest(root, manifest, max_weights):
29
+ path = _input_path(root, manifest, ".json")
30
+ with path.open("rb") as file:
31
+ raw = file.read(1_048_577)
32
+ if len(raw) > 1_048_576:
33
+ raise ValueError("manifest exceeds 1 MiB")
34
+ spec = json.loads(raw)
35
+ if not isinstance(spec, dict) or set(spec) != {"keep", "projections"}:
36
+ raise ValueError("manifest must contain keep and projections")
37
+ items = spec["projections"]
38
+ if not isinstance(items, list) or not 1 <= len(items) <= 1024:
39
+ raise ValueError("provide 1 to 1024 projection records")
40
+ if type(spec["keep"]) is not int:
41
+ raise ValueError("manifest keep must be an integer")
42
+ names, weights, moments = [], [], []
43
+ seen = set()
44
+ total = 0
45
+ for item in items:
46
+ if (not isinstance(item, dict) or not {"name", "weights"} <= item.keys()
47
+ or item.keys() - {"name", "weights", "input_second_moment"}):
48
+ raise ValueError("projection records require name and weights")
49
+ name = item["name"]
50
+ if not isinstance(name, str) or not 1 <= len(name) <= 256 or name in seen:
51
+ raise ValueError("projection names must be nonempty and unique")
52
+ seen.add(name)
53
+ w = _open_npy(root, item["weights"])
54
+ total += w.size
55
+ if max_weights is not None and total > max_weights:
56
+ raise ValueError("manifest exceeds the configured weight limit")
57
+ names.append(name)
58
+ weights.append(w)
59
+ moments.append(_open_npy(root, item["input_second_moment"])
60
+ if "input_second_moment" in item else None)
61
+ return names, weights, moments, spec["keep"]
62
+
63
+
64
+ def prune_manifest(input_root, manifest, output_dir, *, scratch_dir=None,
65
+ threads=1, seed=0, max_weights=None):
66
+ """Publish disk-backed .npy masks and result.json to a new directory."""
67
+ root = Path(input_root).resolve(strict=True)
68
+ names, weights, moments, keep = _read_manifest(root, manifest, max_weights)
69
+ return _publish_masks(names, weights, moments, keep, output_dir,
70
+ scratch_dir=scratch_dir, threads=threads, seed=seed)
71
+
72
+
73
+ def _publish_masks(names, weights, moments, keep, output_dir, *,
74
+ scratch_dir, threads, seed):
75
+ output = Path(output_dir).absolute()
76
+ arrays, moments = _inputs(weights, moments)
77
+ total = sum(w.size for w in arrays)
78
+ keep, threads, seed = _options(total, keep, threads, seed)
79
+
80
+ output.parent.mkdir(parents=True, exist_ok=True)
81
+ lock_path = output.parent / f".{output.name}.lock"
82
+ with lock_path.open("a+b") as lock:
83
+ fcntl.flock(lock, fcntl.LOCK_EX)
84
+ if output.exists() or output.is_symlink():
85
+ raise FileExistsError(output)
86
+ stage = Path(tempfile.mkdtemp(prefix=f".{output.name}-", dir=output.parent))
87
+ try:
88
+ paths = [stage / f"mask-{j:04d}.npy" for j in range(len(arrays))]
89
+ masks = [np.lib.format.open_memmap(path, mode="w+", dtype=bool, shape=w.shape)
90
+ for path, w in zip(paths, arrays)]
91
+ for path in paths:
92
+ with path.open("r+b") as file:
93
+ os.posix_fallocate(file.fileno(), 0, path.stat().st_size)
94
+ scratch = Path(scratch_dir) if scratch_dir is not None else stage
95
+ _run(arrays, moments, masks, total, keep, threads, seed, scratch)
96
+ entries = []
97
+ for name, path, mask in zip(names, paths, masks):
98
+ mask.flush()
99
+ entries.append(dict(name=name, mask=path.name, shape=list(mask.shape),
100
+ kept_slots=int(np.count_nonzero(mask))))
101
+ result = dict(weights=total, kept_slots=keep, projections=entries)
102
+ if sum(item["kept_slots"] for item in entries) != keep:
103
+ raise RuntimeError("native selector returned an incorrect retention budget")
104
+ (stage / "result.json").write_text(json.dumps(result, indent=2) + "\n")
105
+ stage.rename(output)
106
+ return result
107
+ finally:
108
+ if stage.exists():
109
+ shutil.rmtree(stage)
tensor_pruning/kernel.cpp ADDED
@@ -0,0 +1,438 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <algorithm>
2
+ #include <cmath>
3
+ #include <cstddef>
4
+ #include <cstdint>
5
+ #include <cstring>
6
+ #include <limits>
7
+ #include <new>
8
+ #include <exception>
9
+ #include <random>
10
+ #include <utility>
11
+ #include <vector>
12
+ #include <omp.h>
13
+
14
+ namespace {
15
+ using Real = long double;
16
+ using Index = std::size_t;
17
+ using Input = std::uint32_t;
18
+ using RNG = std::mt19937_64;
19
+
20
+ struct Key {
21
+ Real score;
22
+ Index id;
23
+ };
24
+
25
+ bool less(Key a, Key b) {
26
+ return a.score < b.score || (a.score == b.score && a.id < b.id);
27
+ }
28
+
29
+ struct Band {
30
+ Index first = 0, last = 0;
31
+ Real before = 0, mass = 0;
32
+
33
+ Index size() const { return last - first; }
34
+ Real gains() const { return mass * (2 * before + mass); }
35
+ };
36
+
37
+ // The two nonempty bands are adjacent in both array position and score order.
38
+ Band join(Band a, Band b) {
39
+ if (!a.size()) return b;
40
+ if (!b.size()) return a;
41
+ return {a.first, b.last, a.before, a.mass + b.mass};
42
+ }
43
+
44
+ struct Parts {
45
+ Band low, equal, high;
46
+ };
47
+
48
+ struct RowBand {
49
+ Index channel;
50
+ Band band;
51
+ };
52
+
53
+ struct HeadBand {
54
+ std::vector<RowBand> rows;
55
+ Index count = 0;
56
+ Real before = 0; // Channel-gain mass below this interval in the full head.
57
+ Real mass = 0;
58
+
59
+ void add(Index channel, Band band) {
60
+ if (band.size()) {
61
+ rows.push_back({channel, band});
62
+ count += band.size();
63
+ mass += band.gains();
64
+ }
65
+ }
66
+ };
67
+
68
+ // Merge metadata only. Each channel's partitioned indices stay in place.
69
+ HeadBand join(HeadBand a, HeadBand b) {
70
+ if (!a.count) return b;
71
+ if (!b.count) return a;
72
+ HeadBand result;
73
+ result.before = a.before;
74
+ result.rows.reserve(a.rows.size() + b.rows.size());
75
+ Index i = 0, j = 0;
76
+ while (i < a.rows.size() && j < b.rows.size()) {
77
+ const auto& left = a.rows[i];
78
+ const auto& right = b.rows[j];
79
+ if (left.channel < right.channel) {
80
+ result.add(left.channel, left.band);
81
+ ++i;
82
+ } else if (right.channel < left.channel) {
83
+ result.add(right.channel, right.band);
84
+ ++j;
85
+ } else {
86
+ result.add(left.channel, join(left.band, right.band));
87
+ ++i;
88
+ ++j;
89
+ }
90
+ }
91
+ for (; i < a.rows.size(); ++i) result.add(a.rows[i].channel, a.rows[i].band);
92
+ for (; j < b.rows.size(); ++j) result.add(b.rows[j].channel, b.rows[j].band);
93
+ return result;
94
+ }
95
+
96
+ struct HeadParts {
97
+ HeadBand low, equal, high;
98
+ };
99
+
100
+ struct Pivot {
101
+ Index row_slot;
102
+ Input input;
103
+ };
104
+
105
+ Index random_index(RNG& rng, Index size) {
106
+ return std::uniform_int_distribution<Index>(0, size - 1)(rng);
107
+ }
108
+
109
+ struct Projection {
110
+ const void* weights;
111
+ const double* moments;
112
+ std::uint8_t* mask;
113
+ Index heads, channels, inputs;
114
+ int dtype;
115
+ };
116
+
117
+ Real weight_at(const unsigned char* weights, int dtype, Index i) {
118
+ if (dtype == 4) {
119
+ float x;
120
+ std::memcpy(&x, weights + 4*i, sizeof(x));
121
+ return x;
122
+ }
123
+ if (dtype == 8) {
124
+ double x;
125
+ std::memcpy(&x, weights + 8*i, sizeof(x));
126
+ return x;
127
+ }
128
+ std::uint16_t bits;
129
+ std::memcpy(&bits, weights + 2*i, sizeof(bits));
130
+ if (dtype == 3) {
131
+ const std::uint32_t wide = static_cast<std::uint32_t>(bits) << 16;
132
+ float x;
133
+ std::memcpy(&x, &wide, sizeof(x));
134
+ return x;
135
+ }
136
+ #ifdef __FLT16_MANT_DIG__
137
+ _Float16 x;
138
+ std::memcpy(&x, &bits, sizeof(x));
139
+ return static_cast<float>(x);
140
+ #else
141
+ const unsigned exponent = (bits >> 10) & 31;
142
+ const unsigned fraction = bits & 1023;
143
+ Real x;
144
+ if (exponent == 31) {
145
+ x = fraction ? std::numeric_limits<Real>::quiet_NaN()
146
+ : std::numeric_limits<Real>::infinity();
147
+ } else if (exponent) {
148
+ x = std::ldexp(static_cast<Real>(1024 + fraction), static_cast<int>(exponent) - 25);
149
+ } else {
150
+ x = std::ldexp(static_cast<Real>(fraction), -24);
151
+ }
152
+ return bits & 32768 ? -x : x;
153
+ #endif
154
+ }
155
+
156
+ struct Channel {
157
+ const unsigned char* weights;
158
+ const double* moments;
159
+ Input* indices;
160
+ std::uint8_t* mask;
161
+ Index base;
162
+ int dtype;
163
+
164
+ Key key(Input input) const {
165
+ const Real w = weight_at(weights, dtype, input);
166
+ return {w * w * static_cast<Real>(moments[input]), base + input};
167
+ }
168
+
169
+ Parts split_raw(Band band, Input pivot) const {
170
+ const Key cut = key(pivot);
171
+ Index lower = band.first, pos = band.first, upper = band.last;
172
+ Real low_mass = 0, high_mass = 0;
173
+ while (pos < upper) {
174
+ const Key value = key(indices[pos]);
175
+ if (less(value, cut)) {
176
+ low_mass += value.score;
177
+ std::swap(indices[lower++], indices[pos++]);
178
+ } else if (less(cut, value)) {
179
+ high_mass += value.score;
180
+ std::swap(indices[pos], indices[--upper]);
181
+ } else {
182
+ ++pos;
183
+ }
184
+ }
185
+ return {
186
+ {band.first, lower, band.before, low_mass},
187
+ {lower, upper, band.before + low_mass, cut.score},
188
+ {upper, band.last, band.before + low_mass + cut.score, high_mass}
189
+ };
190
+ }
191
+
192
+ Parts split_gain(Band active, Key cutoff, RNG& rng) const {
193
+ Band low, high;
194
+ while (active.size()) {
195
+ const Input pivot = indices[active.first + random_index(rng, active.size())];
196
+ const Parts parts = split_raw(active, pivot);
197
+ const Key gain{parts.equal.gains(), base + pivot};
198
+ if (less(gain, cutoff)) {
199
+ low = join(join(low, parts.low), parts.equal);
200
+ active = parts.high;
201
+ } else if (less(cutoff, gain)) {
202
+ high = join(join(parts.equal, parts.high), high);
203
+ active = parts.low;
204
+ } else {
205
+ return {join(low, parts.low), parts.equal, join(parts.high, high)};
206
+ }
207
+ }
208
+ return {low, {}, high};
209
+ }
210
+ };
211
+
212
+ struct HeadView {
213
+ const unsigned char* weights;
214
+ const double* moments;
215
+ std::uint8_t* mask;
216
+ Input* indices;
217
+ Index base, channels, inputs, row_base;
218
+ int dtype;
219
+ };
220
+
221
+ struct Tensor {
222
+ std::vector<HeadView> heads;
223
+
224
+ Channel row(Index head, Index channel) {
225
+ const auto& view = heads[head];
226
+ const Index offset = channel * view.inputs;
227
+ const Index item_size = view.dtype == 3 ? 2 : view.dtype;
228
+ return {view.weights + offset * item_size, view.moments,
229
+ view.indices ? view.indices + offset : nullptr, view.mask + offset,
230
+ view.base + offset, view.dtype};
231
+ }
232
+
233
+ Pivot pick(Index head, const HeadBand& active, Index rank) {
234
+ Index slot = 0;
235
+ for (; rank >= active.rows[slot].band.size(); ++slot) {
236
+ rank -= active.rows[slot].band.size();
237
+ }
238
+ const auto& item = active.rows[slot];
239
+ return {slot, row(head, item.channel).indices[item.band.first + rank]};
240
+ }
241
+
242
+ std::pair<HeadParts, Key> split_pivot(
243
+ Index head, const HeadBand& active, Pivot pivot, RNG& rng
244
+ ) {
245
+ const auto& pivot_row = active.rows[pivot.row_slot];
246
+ const auto channel = row(head, pivot_row.channel);
247
+ const Parts pivot_parts = channel.split_raw(pivot_row.band, pivot.input);
248
+ const Key channel_gain{pivot_parts.equal.gains(), channel.base + pivot.input};
249
+
250
+ HeadParts result;
251
+ result.low.before = active.before;
252
+ result.low.rows.reserve(active.rows.size());
253
+ result.high.rows.reserve(active.rows.size());
254
+ for (Index j = 0; j < active.rows.size(); ++j) {
255
+ const auto& item = active.rows[j];
256
+ const Parts parts = j == pivot.row_slot ? pivot_parts
257
+ : row(head, item.channel).split_gain(item.band, channel_gain, rng);
258
+ result.low.add(item.channel, parts.low);
259
+ result.high.add(item.channel, parts.high);
260
+ }
261
+ result.equal.before = active.before + result.low.mass;
262
+ result.equal.add(pivot_row.channel, pivot_parts.equal);
263
+ result.high.before = result.equal.before + channel_gain.score;
264
+ const Key final_gain{
265
+ channel_gain.score * (2 * result.equal.before + channel_gain.score),
266
+ channel_gain.id
267
+ };
268
+ return {std::move(result), final_gain};
269
+ }
270
+
271
+ HeadParts split_head(Index head, const HeadBand& active, Key cutoff, RNG& rng) {
272
+ if (!active.count) return {};
273
+ const Pivot pivot = pick(head, active, random_index(rng, active.count));
274
+ auto [parts, gain] = split_pivot(head, active, pivot, rng);
275
+ if (less(gain, cutoff)) {
276
+ HeadParts next = split_head(head, parts.high, cutoff, rng);
277
+ next.low = join(join(std::move(parts.low), std::move(parts.equal)),
278
+ std::move(next.low));
279
+ return next;
280
+ }
281
+ if (less(cutoff, gain)) {
282
+ HeadParts next = split_head(head, parts.low, cutoff, rng);
283
+ next.high = join(join(std::move(next.high), std::move(parts.equal)),
284
+ std::move(parts.high));
285
+ return next;
286
+ }
287
+ return parts;
288
+ }
289
+
290
+ void keep(Index head, const HeadBand& band) {
291
+ for (const auto& item : band.rows) {
292
+ const auto channel = row(head, item.channel);
293
+ for (Index j = item.band.first; j < item.band.last; ++j) {
294
+ channel.mask[channel.indices[j]] = 1;
295
+ }
296
+ }
297
+ }
298
+ };
299
+
300
+ int prune(const Projection* projections, Index count, Input* indices,
301
+ Index budget, int threads, std::uint64_t seed) {
302
+ Tensor tensor;
303
+ Index total = 0, total_rows = 0;
304
+ std::vector<std::pair<Index, Index>> rows;
305
+ for (Index p = 0; p < count; ++p) {
306
+ const auto& view = projections[p];
307
+ const Index head_size = view.channels * view.inputs;
308
+ const Index item_size = view.dtype == 3 ? 2 : view.dtype;
309
+ const auto* weights = static_cast<const unsigned char*>(view.weights);
310
+ for (Index h = 0; h < view.heads; ++h) {
311
+ const Index offset = h * head_size;
312
+ for (Index c = 0; c < view.channels; ++c) rows.emplace_back(tensor.heads.size(), c);
313
+ tensor.heads.push_back({weights + offset * item_size, view.moments,
314
+ view.mask + offset, indices ? indices + total + offset : nullptr,
315
+ total + offset, view.channels, view.inputs, total_rows, view.dtype});
316
+ total_rows += view.channels;
317
+ }
318
+ total += view.heads * head_size;
319
+ }
320
+
321
+ std::vector<Real> masses(indices ? rows.size() : 0);
322
+ int invalid = 0;
323
+ const int workers = threads ? threads : omp_get_max_threads();
324
+ const int row_workers = static_cast<int>(std::min<Index>(workers, rows.size()));
325
+ #pragma omp parallel for schedule(static) num_threads(row_workers) if(row_workers > 1 && total >= 8192) reduction(|:invalid)
326
+ for (Index r = 0; r < rows.size(); ++r) {
327
+ const auto [h, c] = rows[r];
328
+ const auto channel = tensor.row(h, c);
329
+ const auto inputs = tensor.heads[h].inputs;
330
+ Real mass = 0;
331
+ for (Index i = 0; i < inputs; ++i) {
332
+ const Real w = weight_at(channel.weights, channel.dtype, i);
333
+ if (!std::isfinite(w)) invalid = 1;
334
+ if (indices) {
335
+ channel.indices[i] = static_cast<Input>(i);
336
+ mass += w * w * static_cast<Real>(channel.moments[i]);
337
+ }
338
+ }
339
+ std::fill_n(channel.mask, inputs, budget == total ? 1 : 0);
340
+ if (indices) {
341
+ masses[r] = mass;
342
+ if (!std::isfinite(mass) || !std::isfinite(mass * mass * mass * mass)) invalid = 1;
343
+ }
344
+ }
345
+ if (invalid) return 1;
346
+ if (!indices) return 0;
347
+
348
+ struct ActiveHead {
349
+ Index id;
350
+ HeadBand band;
351
+ RNG rng;
352
+ };
353
+ std::vector<ActiveHead> active;
354
+ active.reserve(tensor.heads.size());
355
+ for (Index h = 0; h < tensor.heads.size(); ++h) {
356
+ const auto& view = tensor.heads[h];
357
+ active.push_back({h, {}, RNG(seed + h + 1)});
358
+ auto& band = active.back().band;
359
+ band.rows.reserve(view.channels);
360
+ for (Index c = 0; c < view.channels; ++c) {
361
+ band.add(c, {0, view.inputs, 0, masses[view.row_base + c]});
362
+ }
363
+ if (!std::isfinite(band.mass * band.mass)) return 1;
364
+ }
365
+
366
+ RNG rng(seed);
367
+ Index remaining = total;
368
+ while (budget && remaining) {
369
+ if (budget == remaining) {
370
+ const int team = static_cast<int>(std::min<Index>(workers, active.size()));
371
+ #pragma omp parallel for schedule(static) num_threads(team) if(team > 1 && remaining >= 8192)
372
+ for (Index j = 0; j < active.size(); ++j) tensor.keep(active[j].id, active[j].band);
373
+ break;
374
+ }
375
+ Index rank = random_index(rng, remaining), chosen = 0;
376
+ for (; rank >= active[chosen].band.count; ++chosen) rank -= active[chosen].band.count;
377
+ auto& anchor = active[chosen];
378
+ const Pivot pivot = tensor.pick(anchor.id, anchor.band, rank);
379
+ auto [anchor_parts, cutoff] = tensor.split_pivot(anchor.id, anchor.band, pivot, anchor.rng);
380
+
381
+ std::vector<HeadParts> parts(active.size());
382
+ std::vector<std::exception_ptr> errors(active.size());
383
+ parts[chosen] = std::move(anchor_parts);
384
+ const int team = static_cast<int>(std::min<Index>(workers, active.size() - 1));
385
+ const int split_workers = std::max(team, 1);
386
+ #pragma omp parallel for schedule(dynamic, 1) num_threads(split_workers) if(split_workers > 1 && remaining >= 8192)
387
+ for (Index j = 0; j < active.size(); ++j) {
388
+ if (j == chosen) continue;
389
+ try {
390
+ auto& head = active[j];
391
+ parts[j] = tensor.split_head(head.id, head.band, cutoff, head.rng);
392
+ } catch (...) {
393
+ errors[j] = std::current_exception();
394
+ }
395
+ }
396
+ Index above = 0;
397
+ for (Index j = 0; j < parts.size(); ++j) {
398
+ if (errors[j]) std::rethrow_exception(errors[j]);
399
+ above += parts[j].high.count;
400
+ }
401
+
402
+ const bool choose_upper = budget <= above;
403
+ if (!choose_upper) {
404
+ const int team = static_cast<int>(std::min<Index>(workers, active.size()));
405
+ #pragma omp parallel for schedule(static) num_threads(team) if(team > 1 && remaining >= 8192)
406
+ for (Index j = 0; j < active.size(); ++j) {
407
+ tensor.keep(active[j].id, parts[j].high);
408
+ tensor.keep(active[j].id, parts[j].equal);
409
+ }
410
+ budget -= above + 1; // Global index makes the pivot key unique.
411
+ }
412
+ remaining = 0;
413
+ Index next = 0;
414
+ for (Index j = 0; j < active.size(); ++j) {
415
+ HeadBand& band = choose_upper ? parts[j].high : parts[j].low;
416
+ if (band.count) {
417
+ remaining += band.count;
418
+ active[j].band = std::move(band);
419
+ if (next != j) active[next] = std::move(active[j]);
420
+ ++next;
421
+ }
422
+ }
423
+ active.resize(next);
424
+ }
425
+ return 0;
426
+ }
427
+ } // namespace
428
+
429
+ extern "C" int tensor_prune_many(const Projection* projections, Index count, Input* indices,
430
+ Index budget, int threads, std::uint64_t seed) {
431
+ try {
432
+ return prune(projections, count, indices, budget, threads, seed);
433
+ } catch (const std::bad_alloc&) {
434
+ return 2;
435
+ } catch (...) {
436
+ return 3;
437
+ }
438
+ }
tensor_pruning/service.py ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Bounded HTTP job service for local checkpoint manifests."""
2
+
3
+ from concurrent.futures import ThreadPoolExecutor
4
+ from contextlib import asynccontextmanager
5
+ import fcntl
6
+ import json
7
+ import logging
8
+ from pathlib import Path
9
+ import secrets
10
+ from threading import BoundedSemaphore
11
+ from uuid import UUID, uuid4
12
+
13
+ from fastapi import Depends, FastAPI, HTTPException, Request
14
+ from fastapi.responses import FileResponse
15
+ from fastapi.security import APIKeyHeader, HTTPAuthorizationCredentials, HTTPBearer
16
+ from pydantic import BaseModel, ConfigDict, Field
17
+
18
+ from .core import _kernel
19
+ from .files import prune_manifest
20
+
21
+
22
+ class JobRequest(BaseModel):
23
+ model_config = ConfigDict(extra="forbid")
24
+ manifest: str = Field(min_length=1, max_length=1024)
25
+ seed: int = Field(default=0, ge=0, lt=1 << 64)
26
+
27
+
28
+ class CheckpointJobRequest(BaseModel):
29
+ model_config = ConfigDict(extra="forbid")
30
+ checkpoint: str = Field(default=".", min_length=1, max_length=1024)
31
+ keep: int = Field(strict=True, ge=0, lt=1 << 63)
32
+ moments_manifest: str | None = Field(default=None, min_length=1, max_length=1024)
33
+ seed: int = Field(default=0, ge=0, lt=1 << 64)
34
+
35
+
36
+ def _write_state(directory, run_id, state, **fields):
37
+ path = directory / "state.json"
38
+ tmp = directory / ".state.json.tmp"
39
+ tmp.write_text(json.dumps(dict(state=state, run_id=run_id, **fields)))
40
+ tmp.replace(path)
41
+
42
+
43
+ def create_app(input_root, job_root, *, workers=1, queue_size=2, threads_per_job=1,
44
+ max_weights=100_000_000, token=None, require_token=False,
45
+ checkpoint_root=None):
46
+ """Create a single-process HTTP service with configurable CPU and queue limits."""
47
+ if min(workers, threads_per_job, max_weights) < 1 or queue_size < 0:
48
+ raise ValueError("invalid worker, queue, thread, or weight limit")
49
+ inputs = Path(input_root).resolve(strict=True)
50
+ jobs = Path(job_root).resolve()
51
+ checkpoints = Path(checkpoint_root).resolve(strict=True) if checkpoint_root is not None else None
52
+ if jobs.is_relative_to(inputs) or inputs.is_relative_to(jobs):
53
+ raise ValueError("input and job roots must be separate directories")
54
+
55
+ @asynccontextmanager
56
+ async def lifespan(app):
57
+ jobs.mkdir(parents=True, exist_ok=True)
58
+ with (jobs / ".server.lock").open("a+b") as lock:
59
+ try:
60
+ fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
61
+ except BlockingIOError as exc:
62
+ raise RuntimeError("another service already owns this job directory") from exc
63
+ _kernel()
64
+ app.state.run_id = uuid4().hex
65
+ app.state.slots = BoundedSemaphore(workers + queue_size)
66
+ with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="prune") as pool:
67
+ app.state.pool = pool
68
+ yield
69
+
70
+ app = FastAPI(title="Tensor Pruning", version="1.0.0", lifespan=lifespan)
71
+
72
+ bearer = HTTPBearer(auto_error=False)
73
+ api_key = APIKeyHeader(name="X-Tensor-Pruning-Token", auto_error=False)
74
+
75
+ def authenticate(credentials: HTTPAuthorizationCredentials | None = Depends(bearer),
76
+ space_token: str | None = Depends(api_key)):
77
+ if not token:
78
+ raise HTTPException(503, "The operator must configure the API token")
79
+ supplied = space_token if space_token is not None else (credentials.credentials if credentials else "")
80
+ if not secrets.compare_digest(supplied.encode(), token.encode()):
81
+ raise HTTPException(401, "A valid bearer token is required",
82
+ headers={"WWW-Authenticate": "Bearer"})
83
+
84
+ protected = [Depends(authenticate)] if token or require_token else []
85
+
86
+ @app.get("/health")
87
+ def health():
88
+ return {"status": "ok"}
89
+
90
+ def read_job(job_id, request):
91
+ directory = jobs / str(job_id)
92
+ try:
93
+ state = json.loads((directory / "state.json").read_text())
94
+ except FileNotFoundError as exc:
95
+ raise HTTPException(404, "Unknown job") from exc
96
+ run_id = state.pop("run_id")
97
+ if run_id != request.app.state.run_id and state["state"] in {"queued", "running"}:
98
+ state["state"] = "interrupted"
99
+ return directory, state
100
+
101
+ def enqueue(request, operation):
102
+ state = request.app.state
103
+ if not state.slots.acquire(blocking=False):
104
+ raise HTTPException(429, "Pruning queue is full", headers={"Retry-After": "1"})
105
+ job_id = uuid4()
106
+ directory = jobs / str(job_id)
107
+
108
+ def run():
109
+ try:
110
+ _write_state(directory, state.run_id, "running")
111
+ result = operation(directory)
112
+ _write_state(directory, state.run_id, "completed", result=result)
113
+ except (ValueError, TypeError, OSError, MemoryError) as exc:
114
+ _write_state(directory, state.run_id, "failed", error=str(exc))
115
+ except Exception:
116
+ logging.getLogger(__name__).exception("Pruning job failed")
117
+ _write_state(directory, state.run_id, "failed", error="Internal pruning failure")
118
+ finally:
119
+ state.slots.release()
120
+
121
+ try:
122
+ directory.mkdir()
123
+ _write_state(directory, state.run_id, "queued")
124
+ state.pool.submit(run)
125
+ except Exception:
126
+ state.slots.release()
127
+ raise
128
+ return {"id": str(job_id), "state": "queued"}
129
+
130
+ @app.post("/v1/jobs", status_code=202, dependencies=protected)
131
+ def submit(body: JobRequest, request: Request):
132
+ manifest = Path(body.manifest)
133
+ if manifest.is_absolute() or not (inputs / manifest).resolve().is_relative_to(inputs):
134
+ raise HTTPException(400, "Manifest must be inside the input root")
135
+ return enqueue(request, lambda directory: prune_manifest(
136
+ inputs, body.manifest, directory / "output", threads=threads_per_job,
137
+ seed=body.seed, max_weights=max_weights))
138
+
139
+ if checkpoints is not None:
140
+ from .checkpoint import prune_checkpoint
141
+
142
+ @app.post("/v1/checkpoint-jobs", status_code=202, dependencies=protected)
143
+ def submit_checkpoint(body: CheckpointJobRequest, request: Request):
144
+ relative = Path(body.checkpoint)
145
+ checkpoint = (checkpoints / relative).resolve()
146
+ if relative.is_absolute() or not checkpoint.is_relative_to(checkpoints):
147
+ raise HTTPException(400, "Checkpoint must be inside the model mount")
148
+ if not checkpoint.is_dir():
149
+ raise HTTPException(404, "Mounted checkpoint directory not found")
150
+ return enqueue(request, lambda directory: prune_checkpoint(
151
+ checkpoint, directory / "output", keep=body.keep, workspace=directory,
152
+ max_weights=max_weights, input_root=inputs,
153
+ moments_manifest=body.moments_manifest, threads=threads_per_job, seed=body.seed))
154
+
155
+ @app.get("/v1/jobs/{job_id}", dependencies=protected)
156
+ def status(job_id: UUID, request: Request):
157
+ _, state = read_job(job_id, request)
158
+ return {"id": str(job_id), **state}
159
+
160
+ @app.get("/v1/jobs/{job_id}/masks/{index}", dependencies=protected,
161
+ response_class=FileResponse, responses={
162
+ 200: {"content": {"application/octet-stream": {
163
+ "schema": {"type": "string", "format": "binary"}
164
+ }}}
165
+ })
166
+ def mask(job_id: UUID, index: int, request: Request):
167
+ directory, state = read_job(job_id, request)
168
+ if state["state"] != "completed":
169
+ raise HTTPException(409, "The job has no completed masks")
170
+ projections = state["result"]["projections"]
171
+ if not 0 <= index < len(projections):
172
+ raise HTTPException(404, "Unknown projection index")
173
+ filename = projections[index]["mask"]
174
+ return FileResponse(directory / "output" / filename,
175
+ filename=filename, media_type="application/octet-stream")
176
+
177
+ return app
tests/test_checkpoint.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import io
2
+ import json
3
+ from pathlib import Path
4
+ from types import SimpleNamespace
5
+ import struct
6
+ import tempfile
7
+ import time
8
+ import unittest
9
+ from unittest.mock import patch
10
+
11
+ from fastapi.testclient import TestClient
12
+ import ml_dtypes
13
+ import numpy as np
14
+
15
+ from tensor_pruning import prune_queries
16
+ from tensor_pruning.checkpoint import _queries, prune_checkpoint
17
+ from tensor_pruning.service import create_app
18
+
19
+
20
+ def _name(layer):
21
+ return f"model.layers.{layer}.self_attn.q_proj.weight"
22
+
23
+
24
+ def _shard(path, entries, *, sparse=False):
25
+ header, offset = {}, 0
26
+ for name, dtype, shape, length, _ in entries:
27
+ header[name] = dict(dtype=dtype, shape=shape, data_offsets=[offset, offset + length])
28
+ offset += length
29
+ raw = json.dumps(header).encode()
30
+ raw += b" " * (-len(raw) % 8)
31
+ with path.open("wb") as f:
32
+ f.write(struct.pack("<Q", len(raw)))
33
+ f.write(raw)
34
+ if sparse:
35
+ f.truncate(8 + len(raw) + offset)
36
+ else:
37
+ for _, _, _, _, payload in entries:
38
+ f.write(payload)
39
+
40
+
41
+ class CheckpointTests(unittest.TestCase):
42
+ def setUp(self):
43
+ self.temp = tempfile.TemporaryDirectory()
44
+ self.addCleanup(self.temp.cleanup)
45
+ self.root = Path(self.temp.name)
46
+ self.inputs = self.root / "inputs"
47
+ self.models = self.root / "models"
48
+ self.inputs.mkdir()
49
+ self.models.mkdir()
50
+ self.jobs = self.root / "jobs"
51
+
52
+ def fixture(self):
53
+ dims = dict(num_hidden_layers=2, num_attention_heads=2, hidden_size=4)
54
+ (self.models / "config.json").write_text(json.dumps(dims))
55
+ arrays = [np.arange(16, dtype=np.float16).reshape(2, 2, 4) - 4,
56
+ np.asarray(np.arange(16) / 3, dtype=ml_dtypes.bfloat16).reshape(2, 2, 4)]
57
+ mapping = {}
58
+ moments = [np.array([1, 2, 3, 4]), np.array([4, 3, 2, 1])]
59
+ moment_names = {}
60
+ for i, (array, dtype) in enumerate(zip(arrays, ("F16", "BF16"))):
61
+ name = _name(i)
62
+ filename = f"shard-{1-i}.safetensors"
63
+ mapping[name] = filename
64
+ noise = ("unused", "F16", [2], 4, b"\xff" * 4)
65
+ query = (name, dtype, [4, 4], array.nbytes, array.tobytes())
66
+ _shard(self.models / filename, [noise, query])
67
+ moment_names[name] = f"moments-{i}.npy"
68
+ np.save(self.inputs / moment_names[name], moments[i])
69
+ (self.models / "model.safetensors.index.json").write_text(json.dumps({"weight_map": mapping}))
70
+ (self.inputs / "moments.json").write_text(json.dumps(moment_names))
71
+ return arrays, moments
72
+
73
+ def test_service_selects_one_global_mask_from_fp16_and_bf16(self):
74
+ arrays, moments = self.fixture()
75
+ expected = prune_queries(arrays, 11, input_second_moments=moments, seed=47)
76
+ app = create_app(self.inputs, self.jobs, checkpoint_root=self.models,
77
+ token="example", require_token=True)
78
+ headers = {"X-Tensor-Pruning-Token": "example"}
79
+ body = {"keep": 11, "seed": 47, "moments_manifest": "moments.json"}
80
+ with TestClient(app) as client:
81
+ self.assertEqual(client.post("/v1/checkpoint-jobs", json=body).status_code, 401)
82
+ self.assertEqual(client.post("/v1/checkpoint-jobs", headers=headers,
83
+ json={**body, "checkpoint": "../"}).status_code, 400)
84
+ created = client.post("/v1/checkpoint-jobs", headers=headers, json=body)
85
+ self.assertEqual(created.status_code, 202, created.text)
86
+ url = "/v1/jobs/" + created.json()["id"]
87
+ for _ in range(500):
88
+ status = client.get(url, headers=headers).json()
89
+ if status["state"] in {"completed", "failed"}:
90
+ break
91
+ time.sleep(.01)
92
+ self.assertEqual(status["state"], "completed", status)
93
+ self.assertEqual(status["result"]["kept_slots"], 11)
94
+ self.assertEqual([p["name"] for p in status["result"]["projections"]],
95
+ [_name(0), _name(1)])
96
+ for i, want in enumerate(expected):
97
+ got = client.get(f"{url}/masks/{i}", headers=headers)
98
+ self.assertEqual(got.status_code, 200, got.text if got.status_code != 200 else "")
99
+ np.testing.assert_array_equal(np.load(io.BytesIO(got.content)), want)
100
+ self.assertEqual(list(self.jobs.glob("*/query-staging-*")), [])
101
+
102
+ def test_job_submission_fails_closed_without_token(self):
103
+ app = create_app(self.inputs, self.jobs, checkpoint_root=self.models, require_token=True)
104
+ with TestClient(app) as client:
105
+ self.assertEqual(client.get("/health").status_code, 200)
106
+ self.assertEqual(client.post("/v1/checkpoint-jobs", json={"keep": 1}).status_code, 503)
107
+
108
+ def test_disk_preflight_rejects_before_staging(self):
109
+ self.fixture()
110
+ usage = SimpleNamespace(total=100, used=100, free=0)
111
+ with patch("tensor_pruning.checkpoint.shutil.disk_usage", return_value=usage):
112
+ with self.assertRaisesRegex(OSError, "Insufficient local disk"):
113
+ prune_checkpoint(self.models, self.root / "out", keep=11,
114
+ workspace=self.root, max_weights=100)
115
+ self.assertEqual(list(self.root.glob("query-staging-*")), [])
116
+
117
+ def test_llama_70b_layout_exceeds_32_bit_global_indices(self):
118
+ shape = [8192, 8192]
119
+ byte_length = 2 * shape[0] * shape[1]
120
+ config = dict(num_hidden_layers=80, num_attention_heads=64, hidden_size=8192)
121
+ (self.models / "config.json").write_text(json.dumps(config))
122
+ entries = [(_name(i), "BF16", shape, byte_length, None) for i in range(80)]
123
+ _shard(self.models / "model.safetensors", entries, sparse=True)
124
+ query, total = _queries(self.models, 6_000_000_000)
125
+ self.assertEqual(total, 5_368_709_120)
126
+ self.assertGreater(total, 1 << 32)
127
+ self.assertEqual(query[-1].shape, (64, 128, 8192))
128
+ self.assertGreater(query[-1].offset, 1 << 33)
129
+ with self.assertRaisesRegex(ValueError, "eligible weight limit"):
130
+ _queries(self.models, 1 << 32)
131
+
132
+
tests/test_pruning.py ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from concurrent.futures import ThreadPoolExecutor
2
+ import io
3
+ import itertools
4
+ import json
5
+ from pathlib import Path
6
+ import tempfile
7
+ from threading import Event
8
+ import time
9
+ import unittest
10
+ from unittest.mock import patch
11
+
12
+ import numpy as np
13
+
14
+ from tensor_pruning import prune_manifest, prune_queries
15
+ from tensor_pruning.service import create_app
16
+ from tensor_pruning.core import Projection, _kernel
17
+
18
+ from fastapi.testclient import TestClient
19
+
20
+
21
+ def reference(arrays, keep, moments):
22
+ gains = []
23
+ offset = 0
24
+ for w, v in zip(arrays, moments):
25
+ energies = w.astype(np.longdouble)**2 * v
26
+ for head in energies:
27
+ intermediate = []
28
+ for channel in head:
29
+ mass = 0
30
+ for u, i in sorted((u, offset+i) for i, u in enumerate(channel)):
31
+ intermediate.append((u * (2*mass+u), i))
32
+ mass += u
33
+ offset += len(channel)
34
+ mass = 0
35
+ for d, i in sorted(intermediate):
36
+ gains.append((d * (2*mass+d), i))
37
+ mass += d
38
+ keep_ids = {i for _, i in sorted(gains, reverse=True)[:keep]}
39
+ offset = 0
40
+ masks = []
41
+ for w in arrays:
42
+ masks.append(np.array([i in keep_ids for i in range(offset, offset+w.size)]).reshape(w.shape))
43
+ offset += w.size
44
+ return masks
45
+
46
+
47
+ def objective(arrays, masks, moments):
48
+ return sum(np.sum(np.sum(np.sum(w.astype(np.longdouble)**2 * v * ~mask, axis=-1)**2,
49
+ axis=-1)**2)
50
+ for w, mask, v in zip(arrays, masks, moments))
51
+
52
+
53
+ class SelectorTest(unittest.TestCase):
54
+ def test_mixed_shapes_and_ties_against_reference(self):
55
+ rng = np.random.default_rng(17)
56
+ for i in range(100):
57
+ shapes = [(rng.integers(1, 6), rng.integers(1, 7), rng.integers(1, 14)) for _ in range(3)]
58
+ weights = [rng.integers(-4, 5, size=shape).astype(dtype)
59
+ for shape, dtype in zip(shapes, (np.float16, np.float32, np.float64))]
60
+ moments = [rng.integers(0, 6, size=w.shape[-1]) for w in weights]
61
+ total = sum(w.size for w in weights)
62
+ keep = int(rng.integers(0, total+1))
63
+ actual = prune_queries(weights, keep, input_second_moments=moments, threads=2, seed=i)
64
+ self.assertEqual(sum(np.count_nonzero(mask) for mask in actual), keep)
65
+ for mask, expected in zip(actual, reference(weights, keep, moments)):
66
+ np.testing.assert_array_equal(mask, expected)
67
+
68
+ def test_exhaustive_global_optimum(self):
69
+ rng = np.random.default_rng(18)
70
+ for _ in range(18):
71
+ weights = [rng.integers(-3, 4, size=shape).astype(np.float32)
72
+ for shape in ((1, 2, 3), (2, 1, 2))]
73
+ moments = [np.arange(1, w.shape[-1]+1) for w in weights]
74
+ keep = int(rng.integers(0, 11))
75
+ got = prune_queries(weights, keep, input_second_moments=moments)
76
+ best = float("inf")
77
+ for ids in itertools.combinations(range(10), keep):
78
+ flat = np.zeros(10, dtype=bool)
79
+ flat[list(ids)] = True
80
+ trial = [flat[:6].reshape(weights[0].shape), flat[6:].reshape(weights[1].shape)]
81
+ best = min(best, objective(weights, trial, moments))
82
+ self.assertEqual(objective(weights, got, moments), best)
83
+
84
+ def test_concurrent_calls_and_disk_workspace(self):
85
+ rng = np.random.default_rng(19)
86
+ weights = [rng.normal(size=shape).astype(dtype)
87
+ for shape, dtype in [((12, 8, 65), np.float32), ((6, 5, 73), np.float64)]]
88
+ moments = [rng.uniform(0.2, 2, size=w.shape[-1]) for w in weights]
89
+ with tempfile.TemporaryDirectory() as scratch, ThreadPoolExecutor(3) as pool:
90
+ def run(keep):
91
+ return prune_queries(weights, keep, input_second_moments=moments,
92
+ threads=2, scratch_dir=scratch)
93
+ for keep, result in zip((2145, 3941, 6158), pool.map(run, (2145, 3941, 6158))):
94
+ for mask, expected in zip(result, reference(weights, keep, moments)):
95
+ np.testing.assert_array_equal(mask, expected)
96
+
97
+ def test_native_bfloat16_bits(self):
98
+ x = np.linspace(-6, 5, 72, dtype=np.float32).reshape(3, 4, 6)
99
+ bits = (x.view(np.uint32) >> 16).astype(np.uint16)
100
+ float_equivalent = (bits.astype(np.uint32) << 16).view(np.float32)
101
+ moments = np.linspace(0.25, 1.25, x.shape[-1], dtype=np.float64)
102
+ workspace = np.empty(x.size, dtype=np.uint32)
103
+ mask = np.empty(x.shape, dtype=bool)
104
+ desc = Projection(bits.ctypes.data, moments.ctypes.data, mask.ctypes.data, *x.shape, 3)
105
+ self.assertEqual(_kernel()(desc, 1, workspace.ctypes.data, 29, 1, 0), 0)
106
+ expected = prune_queries([float_equivalent], 29, input_second_moments=[moments])[0]
107
+ np.testing.assert_array_equal(mask, expected)
108
+
109
+ def test_zero_and_full_budget(self):
110
+ w = np.ones((1, 2, 3), dtype=np.float16)
111
+ self.assertEqual(prune_queries([w], 0)[0].sum(), 0)
112
+ self.assertEqual(prune_queries([w], 6)[0].sum(), 6)
113
+ with self.assertRaises(ValueError):
114
+ prune_queries([w], 7)
115
+ with self.assertRaises(ValueError):
116
+ prune_queries([np.full_like(w, np.inf)], 0)
117
+ with self.assertRaises(ValueError):
118
+ prune_queries([w[..., ::-1]], 3)
119
+ with self.assertRaises(ValueError):
120
+ prune_queries([w], 3, input_second_moments=[[-1, 2, 3]])
121
+ with self.assertRaises(TypeError):
122
+ prune_queries([np.ones((1, 2, 3), dtype=np.int16)], 3)
123
+
124
+
125
+ class FileAndServiceTest(unittest.TestCase):
126
+ def setUp(self):
127
+ self.temp = tempfile.TemporaryDirectory()
128
+ self.addCleanup(self.temp.cleanup)
129
+ self.root = Path(self.temp.name)
130
+ self.inputs = self.root / "data"
131
+ self.inputs.mkdir()
132
+ self.weights = np.arange(24, dtype=np.float32).reshape(2, 3, 4)
133
+ np.save(self.inputs / "a.npy", self.weights)
134
+ np.save(self.inputs / "v.npy", np.array([0.5, 1., 2., 3.]))
135
+ self.spec = dict(keep=7, projections=[dict(name="q", weights="a.npy",
136
+ input_second_moment="v.npy")])
137
+ (self.inputs / "job.json").write_text(json.dumps(self.spec))
138
+
139
+ def test_manifest_publishes_exact_masks(self):
140
+ output = self.root / "results"
141
+ result = prune_manifest(self.inputs, "job.json", output, threads=2)
142
+ self.assertEqual(result["kept_slots"], 7)
143
+ mask = np.load(output / "mask-0000.npy", allow_pickle=False)
144
+ expected = prune_queries([self.weights], 7, input_second_moments=[[.5, 1., 2., 3.]])[0]
145
+ np.testing.assert_array_equal(mask, expected)
146
+ self.assertEqual(json.loads((output / "result.json").read_text()), result)
147
+ with self.assertRaises(FileExistsError):
148
+ prune_manifest(self.inputs, "job.json", output)
149
+ with self.assertRaises(ValueError):
150
+ prune_manifest(self.inputs, "job.json", self.root / "large", max_weights=12)
151
+
152
+ def test_rejects_symlink_and_failed_native_run_leaves_no_output(self):
153
+ external = self.root / "outside.npy"
154
+ np.save(external, self.weights)
155
+ (self.inputs / "escaped.npy").symlink_to(external)
156
+ self.spec["projections"][0]["weights"] = "escaped.npy"
157
+ (self.inputs / "escape.json").write_text(json.dumps(self.spec))
158
+ with self.assertRaises(ValueError):
159
+ prune_manifest(self.inputs, "escape.json", self.root / "out")
160
+ np.save(self.inputs / "a.npy", np.full_like(self.weights, np.nan))
161
+ with self.assertRaises(ValueError):
162
+ prune_manifest(self.inputs, "job.json", self.root / "out")
163
+ self.assertFalse((self.root / "out").exists())
164
+
165
+ def test_http_job_mask_and_auth(self):
166
+ app = create_app(self.inputs, self.root / "jobs", token="test-token", threads_per_job=2)
167
+ headers = {"Authorization": "Bearer test-token"}
168
+ with TestClient(app) as client:
169
+ self.assertEqual(client.get("/health").status_code, 200)
170
+ self.assertEqual(client.post("/v1/jobs", json={"manifest": "job.json"}).status_code, 401)
171
+ escape = client.post("/v1/jobs", json={"manifest": "../out.json"}, headers=headers)
172
+ self.assertEqual(escape.status_code, 400)
173
+ response = client.post("/v1/jobs", json={"manifest": "job.json"}, headers=headers)
174
+ self.assertEqual(response.status_code, 202)
175
+ endpoint = f"/v1/jobs/{response.json()['id']}"
176
+ for _ in range(150):
177
+ status = client.get(endpoint, headers=headers).json()
178
+ if status["state"] in {"completed", "failed"}:
179
+ break
180
+ time.sleep(0.01)
181
+ self.assertEqual(status["state"], "completed", status)
182
+ binary = client.get(endpoint + "/masks/0", headers=headers)
183
+ self.assertEqual(binary.status_code, 200)
184
+ self.assertEqual(np.load(io.BytesIO(binary.content)).sum(), 7)
185
+ self.assertEqual(client.get(endpoint + "/masks/1", headers=headers).status_code, 404)
186
+
187
+ def test_queue_backpressure(self):
188
+ started, proceed = Event(), Event()
189
+
190
+ def slow(*args, **kwargs):
191
+ started.set()
192
+ if not proceed.wait(5):
193
+ raise RuntimeError("test timeout")
194
+ return prune_manifest(*args, **kwargs)
195
+
196
+ app = create_app(self.inputs, self.root / "jobs", workers=1, queue_size=0)
197
+ with patch("tensor_pruning.service.prune_manifest", slow), TestClient(app) as client:
198
+ try:
199
+ self.assertEqual(client.post("/v1/jobs", json={"manifest": "job.json"}).status_code, 202)
200
+ self.assertTrue(started.wait(2))
201
+ response = client.post("/v1/jobs", json={"manifest": "job.json"})
202
+ self.assertEqual(response.status_code, 429)
203
+ self.assertEqual(response.headers["Retry-After"], "1")
204
+ finally:
205
+ proceed.set()
206
+
207
+
208
+ if __name__ == "__main__":
209
+ unittest.main()