Spaces:
Sleeping
Sleeping
Upload 14 files
Browse files- Dockerfile +15 -0
- README.md +75 -7
- publish.py +30 -0
- pyproject.toml +25 -0
- space_app.py +30 -0
- tensor_pruning/__init__.py +6 -0
- tensor_pruning/checkpoint.py +176 -0
- tensor_pruning/cli.py +54 -0
- tensor_pruning/core.py +140 -0
- tensor_pruning/files.py +109 -0
- tensor_pruning/kernel.cpp +438 -0
- tensor_pruning/service.py +177 -0
- tests/test_checkpoint.py +132 -0
- tests/test_pruning.py +209 -0
Dockerfile
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM python:3.12-slim-bookworm
|
| 2 |
+
ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 PYTHONDONTWRITEBYTECODE=1
|
| 3 |
+
RUN apt-get update \
|
| 4 |
+
&& apt-get install -y --no-install-recommends g++ \
|
| 5 |
+
&& rm -rf /var/lib/apt/lists/* \
|
| 6 |
+
&& useradd -m -u 1000 user \
|
| 7 |
+
&& mkdir -p /models /home/user/work \
|
| 8 |
+
&& chown -R user:user /home/user/work
|
| 9 |
+
WORKDIR /app
|
| 10 |
+
COPY pyproject.toml README.md space_app.py ./
|
| 11 |
+
COPY tensor_pruning/ ./tensor_pruning/
|
| 12 |
+
RUN python -m pip install '.[space]'
|
| 13 |
+
USER user
|
| 14 |
+
EXPOSE 7860
|
| 15 |
+
CMD ["python", "-m", "uvicorn", "space_app:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
|
README.md
CHANGED
|
@@ -1,11 +1,79 @@
|
|
| 1 |
---
|
| 2 |
-
title: Tensor Pruning
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: docker
|
| 7 |
-
|
| 8 |
-
license: apache-2.0
|
| 9 |
---
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: Tensor Pruning API
|
| 3 |
+
emoji: ✂️
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: purple
|
| 6 |
sdk: docker
|
| 7 |
+
app_port: 7860
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# Tensor Pruning API
|
| 11 |
+
|
| 12 |
+
A Docker Space for [the Tensor Pruning API](https://github.com/AngshulMajumdar/Tensor-Pruning-API). It selects query-weight masks for dense Llama-layout checkpoints with a single retention count across all layers. The CPU selector uses the nested weighted \((2,4,8)\) residual objective from the API. A job publishes one Boolean NumPy mask per query projection, with axes `[head, channel, input]`, plus the tensor names and retained counts in its status response.
|
| 13 |
+
|
| 14 |
+
## Run on Hugging Face
|
| 15 |
+
|
| 16 |
+
Unzip the release and run the following from the directory containing `publish.py`. Log in through the Hugging Face CLI using an account with permission to create Spaces. Omit the model flag if you will attach the model later.
|
| 17 |
+
|
| 18 |
+
```sh
|
| 19 |
+
python -m pip install -U huggingface_hub
|
| 20 |
+
hf auth login
|
| 21 |
+
python publish.py --repo Angshul/Tensor-Pruning-API --model-repo OWNER/MODEL
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
The helper creates a private Docker Space with the chosen model mounted read-only at `/models`. If the repository already exists, review its contents before uploading the deployment files with the Hugging Face UI or CLI. Use a model that your account can access. If `config.json` and the weight files are in a mounted subdirectory, send that relative directory as `checkpoint`.
|
| 25 |
+
|
| 26 |
+
In Space Settings, set a secret named `TENSOR_PRUNING_API_TOKEN` to an operator-chosen random token. Configure `PRUNING_THREADS` to match the assigned CPUs if necessary. Keep `PRUNING_WORK_ROOT` on local storage that supports memory mapping and file preallocation. The default is `/home/user/work`. A model mount supplies the checkpoint. The job's query staging files, index scratch, and masks occupy local disk.
|
| 27 |
+
|
| 28 |
+
## Submit and retrieve a job
|
| 29 |
+
|
| 30 |
+
Open the Space's direct app URL from its Hugging Face page. The root opens the API documentation at `/docs`. The health route is `/health`. The example uses a private Space, so it sends a Hugging Face access token for the proxy and a separate pruning token for the app.
|
| 31 |
+
|
| 32 |
+
```sh
|
| 33 |
+
curl -fsS "$APP_URL/v1/checkpoint-jobs" \
|
| 34 |
+
-H "Authorization: Bearer $HF_TOKEN" \
|
| 35 |
+
-H "X-Tensor-Pruning-Token: $PRUNING_TOKEN" \
|
| 36 |
+
-H 'Content-Type: application/json' \
|
| 37 |
+
-d '{"checkpoint":".","keep":1610612736,"seed":0}'
|
| 38 |
+
|
| 39 |
+
curl -fsS "$APP_URL/v1/jobs/$JOB_ID" \
|
| 40 |
+
-H "Authorization: Bearer $HF_TOKEN" \
|
| 41 |
+
-H "X-Tensor-Pruning-Token: $PRUNING_TOKEN"
|
| 42 |
+
|
| 43 |
+
curl -fsS "$APP_URL/v1/jobs/$JOB_ID/masks/0" \
|
| 44 |
+
-H "Authorization: Bearer $HF_TOKEN" \
|
| 45 |
+
-H "X-Tensor-Pruning-Token: $PRUNING_TOKEN" \
|
| 46 |
+
-o mask-0000.npy
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
The example retention count is 30% of the Llama 3.1 70B query slots. The service runs one job at a time and bounds the waiting queue. Status and mask routes also serve the original staged NumPy-manifest API at `POST /v1/jobs`. Download completed masks before a Space restart, which can clear local storage. Clear obsolete job directories when the service is stopped to recover disk space. Interrupted jobs can be resubmitted.
|
| 50 |
+
|
| 51 |
+
The checkpoint endpoint accepts the standard `model.safetensors` file or `model.safetensors.index.json` and its shards. It reads one `model.layers.N.self_attn.q_proj.weight` tensor per layer. It supports dense FP16, BF16, FP32, and FP64 weights. The worker leaves model weights unchanged. Mask application is a downstream step.
|
| 52 |
+
|
| 53 |
+
The default objective uses unit input moments. To use activation weighting, supply `moments_manifest` as a relative path to a JSON object mapping each eligible tensor name to a corresponding NumPy vector file. Stage the JSON and vectors under `PRUNING_INPUT_ROOT`, which defaults to `/home/user/work/inputs`, or mount a separate read-only volume there. Dense-model calibration is performed externally.
|
| 54 |
+
|
| 55 |
+
## Size a Llama 3.1 70B job
|
| 56 |
+
|
| 57 |
+
The standard layout has 80 layers and \(8192\times8192\) query matrices, yielding 5,368,709,120 eligible slots. For FP16 or BF16 and an interior retention budget:
|
| 58 |
+
|
| 59 |
+
| Local file use | Bytes per query slot | Approximate decimal GB |
|
| 60 |
+
| --- | ---: | ---: |
|
| 61 |
+
| Staged query weights | 2 | 10.74 |
|
| 62 |
+
| Channel-local partition indices | 4 | 21.47 |
|
| 63 |
+
| Output Boolean masks | 1 | 5.37 |
|
| 64 |
+
| Total | 7 | 37.58 |
|
| 65 |
+
|
| 66 |
+
The API checks available disk before staging and requires roughly 38.66 GB free for this job, including its reserve. Older output masks, the image, and other local files consume additional disk space. The model repository must be mounted separately for the sizing above. The worker reads only the query byte ranges from the mounted shards and memory-maps local working files. The figures are disk usage estimates. RAM and elapsed time depend on local I/O, page caching, and CPU allocation. The 70B layout was checked with synthetic metadata. A full 70B selection run remains to be measured.
|
| 67 |
+
|
| 68 |
+
## Run locally
|
| 69 |
+
|
| 70 |
+
```sh
|
| 71 |
+
docker build -t tensor-pruning-space .
|
| 72 |
+
docker run --rm -p 7860:7860 \
|
| 73 |
+
-v "$CHECKPOINT_DIR:/models:ro" \
|
| 74 |
+
-e TENSOR_PRUNING_API_TOKEN \
|
| 75 |
+
-e PRUNING_THREADS=4 \
|
| 76 |
+
tensor-pruning-space
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Use `http://localhost:7860/docs` with the pruning token. The local container needs sufficient free disk or a writable local bind mount for the work root. The Python integration tests can be run with `python -m pip install '.[space,test]'` followed by `python -m unittest discover -s tests -v`.
|
publish.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Create the private Docker Space and upload this deployment source tree."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
from huggingface_hub import HfApi, Volume
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def main():
|
| 10 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 11 |
+
parser.add_argument("--repo", default="Angshul/Tensor-Pruning-API")
|
| 12 |
+
parser.add_argument("--model-repo", help="Optionally mount a model repo read-only at /models")
|
| 13 |
+
args = parser.parse_args()
|
| 14 |
+
api = HfApi()
|
| 15 |
+
options = {}
|
| 16 |
+
if args.model_repo:
|
| 17 |
+
options["space_volumes"] = [
|
| 18 |
+
Volume(type="model", source=args.model_repo, mount_path="/models", read_only=True)
|
| 19 |
+
]
|
| 20 |
+
api.create_repo(args.repo, repo_type="space", space_sdk="docker", private=True, **options)
|
| 21 |
+
api.upload_folder(repo_id=args.repo, repo_type="space", folder_path=Path(__file__).parent,
|
| 22 |
+
allow_patterns=["README.md", "Dockerfile", "space_app.py",
|
| 23 |
+
"pyproject.toml", "tensor_pruning/*.py",
|
| 24 |
+
"tensor_pruning/kernel.cpp"],
|
| 25 |
+
commit_message="Deploy the CPU tensor-pruning API")
|
| 26 |
+
print(f"https://huggingface.co/spaces/{args.repo}")
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
if __name__ == "__main__":
|
| 30 |
+
main()
|
pyproject.toml
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[build-system]
|
| 2 |
+
requires = ["setuptools>=68"]
|
| 3 |
+
build-backend = "setuptools.build_meta"
|
| 4 |
+
|
| 5 |
+
[project]
|
| 6 |
+
name = "tensor-pruning"
|
| 7 |
+
version = "1.0.0"
|
| 8 |
+
description = "Training-free global-budget tensor pruning for LLM query projections"
|
| 9 |
+
readme = "README.md"
|
| 10 |
+
requires-python = ">=3.10"
|
| 11 |
+
dependencies = ["numpy>=1.24"]
|
| 12 |
+
|
| 13 |
+
[project.optional-dependencies]
|
| 14 |
+
http = ["fastapi>=0.115,<1", "uvicorn>=0.30,<1"]
|
| 15 |
+
space = ["fastapi>=0.115,<1", "uvicorn>=0.30,<1", "ml-dtypes>=0.4,<1"]
|
| 16 |
+
test = ["httpx>=0.27,<1"]
|
| 17 |
+
|
| 18 |
+
[project.scripts]
|
| 19 |
+
tensor-prune = "tensor_pruning.cli:main"
|
| 20 |
+
|
| 21 |
+
[tool.setuptools.packages.find]
|
| 22 |
+
include = ["tensor_pruning*"]
|
| 23 |
+
|
| 24 |
+
[tool.setuptools.package-data]
|
| 25 |
+
tensor_pruning = ["kernel.cpp"]
|
space_app.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Configure the Tensor Pruning API for a single Docker Space replica."""
|
| 2 |
+
|
| 3 |
+
import os
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
from fastapi.responses import RedirectResponse
|
| 7 |
+
|
| 8 |
+
from tensor_pruning.service import create_app
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
work = Path(os.getenv("PRUNING_WORK_ROOT", "/home/user/work"))
|
| 12 |
+
inputs = Path(os.getenv("PRUNING_INPUT_ROOT", str(work / "inputs")))
|
| 13 |
+
inputs.mkdir(parents=True, exist_ok=True)
|
| 14 |
+
|
| 15 |
+
app = create_app(
|
| 16 |
+
inputs,
|
| 17 |
+
work / "jobs",
|
| 18 |
+
workers=1,
|
| 19 |
+
queue_size=int(os.getenv("PRUNING_QUEUE_SIZE", "1")),
|
| 20 |
+
threads_per_job=int(os.getenv("PRUNING_THREADS", str(min(2, len(os.sched_getaffinity(0)))))),
|
| 21 |
+
max_weights=int(os.getenv("PRUNING_MAX_WEIGHTS", "6000000000")),
|
| 22 |
+
token=os.getenv("TENSOR_PRUNING_API_TOKEN"),
|
| 23 |
+
require_token=True,
|
| 24 |
+
checkpoint_root=os.getenv("PRUNING_CHECKPOINT_ROOT", "/models"),
|
| 25 |
+
)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
@app.get("/", include_in_schema=False)
|
| 29 |
+
def index():
|
| 30 |
+
return RedirectResponse("/docs")
|
tensor_pruning/__init__.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Training-free, global-budget tensor pruning for LLM query projections."""
|
| 2 |
+
|
| 3 |
+
from .core import prune_queries
|
| 4 |
+
from .files import prune_manifest
|
| 5 |
+
|
| 6 |
+
__all__ = ["prune_queries", "prune_manifest"]
|
tensor_pruning/checkpoint.py
ADDED
|
@@ -0,0 +1,176 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Stage Llama-layout query ranges from safetensors and select one global mask."""
|
| 2 |
+
|
| 3 |
+
from dataclasses import dataclass
|
| 4 |
+
import json
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
import shutil
|
| 7 |
+
import struct
|
| 8 |
+
import tempfile
|
| 9 |
+
|
| 10 |
+
import ml_dtypes
|
| 11 |
+
import numpy as np
|
| 12 |
+
|
| 13 |
+
from .files import _input_path, _open_npy, _publish_masks
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
_DTYPES = {"F16": np.dtype("<f2"), "BF16": np.dtype(ml_dtypes.bfloat16),
|
| 17 |
+
"F32": np.dtype("<f4"), "F64": np.dtype("<f8")}
|
| 18 |
+
_CHUNK = 8 * 1024**2
|
| 19 |
+
_HEADROOM = 1024**3
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _object(pairs):
|
| 23 |
+
result = {}
|
| 24 |
+
for key, value in pairs:
|
| 25 |
+
if key in result:
|
| 26 |
+
raise ValueError(f"Duplicate JSON key: {key}")
|
| 27 |
+
result[key] = value
|
| 28 |
+
return result
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _json_object(raw):
|
| 32 |
+
value = json.loads(raw, object_pairs_hook=_object)
|
| 33 |
+
if not isinstance(value, dict):
|
| 34 |
+
raise ValueError("Expected a JSON object")
|
| 35 |
+
return value
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _json_file(path, limit):
|
| 39 |
+
with path.open("rb") as stream:
|
| 40 |
+
raw = stream.read(limit + 1)
|
| 41 |
+
if len(raw) > limit:
|
| 42 |
+
raise ValueError(f"{path.name} exceeds the supported metadata size")
|
| 43 |
+
return _json_object(raw)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _layout(config):
|
| 47 |
+
layers = config.get("num_hidden_layers")
|
| 48 |
+
heads = config.get("num_attention_heads")
|
| 49 |
+
width = config.get("hidden_size")
|
| 50 |
+
if any(type(n) is not int or n < 1 for n in (layers, heads, width)) or layers > 1024:
|
| 51 |
+
raise ValueError("Expected valid Llama-layout layer, head and hidden sizes")
|
| 52 |
+
channels = config.get("head_dim")
|
| 53 |
+
if channels is None:
|
| 54 |
+
channels, extra = divmod(width, heads)
|
| 55 |
+
if extra:
|
| 56 |
+
raise ValueError("hidden_size must be divisible by num_attention_heads")
|
| 57 |
+
if type(channels) is not int or channels < 1:
|
| 58 |
+
raise ValueError("Invalid head_dim")
|
| 59 |
+
return layers, (heads, channels, width)
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def _header(path):
|
| 63 |
+
with path.open("rb") as stream:
|
| 64 |
+
prefix = stream.read(8)
|
| 65 |
+
if len(prefix) != 8:
|
| 66 |
+
raise ValueError(f"Incomplete safetensors header: {path.name}")
|
| 67 |
+
size, = struct.unpack("<Q", prefix)
|
| 68 |
+
if not 2 <= size <= 100_000_000:
|
| 69 |
+
raise ValueError(f"Invalid safetensors header length: {path.name}")
|
| 70 |
+
raw = stream.read(size)
|
| 71 |
+
if len(raw) != size or not raw.startswith(b"{"):
|
| 72 |
+
raise ValueError(f"Invalid safetensors header: {path.name}")
|
| 73 |
+
start = 8 + size
|
| 74 |
+
return _json_object(raw), start, path.stat().st_size - start
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
@dataclass(frozen=True)
|
| 78 |
+
class QueryRange:
|
| 79 |
+
name: str
|
| 80 |
+
shard: Path
|
| 81 |
+
offset: int
|
| 82 |
+
length: int
|
| 83 |
+
shape: tuple[int, int, int]
|
| 84 |
+
dtype: np.dtype
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def _queries(checkpoint, max_weights):
|
| 88 |
+
root = Path(checkpoint).resolve(strict=True)
|
| 89 |
+
layers, shape = _layout(_json_file(_input_path(root, "config.json", ".json"), 1024**2))
|
| 90 |
+
heads, channels, width = shape
|
| 91 |
+
total = layers * heads * channels * width
|
| 92 |
+
if total > max_weights:
|
| 93 |
+
raise ValueError("Checkpoint exceeds the configured eligible weight limit")
|
| 94 |
+
names = [f"model.layers.{layer}.self_attn.q_proj.weight" for layer in range(layers)]
|
| 95 |
+
index = root / "model.safetensors.index.json"
|
| 96 |
+
if index.exists():
|
| 97 |
+
mapping = _json_file(_input_path(root, index.name, ".json"), 16 * 1024**2).get("weight_map")
|
| 98 |
+
if not isinstance(mapping, dict) or any(name not in mapping for name in names):
|
| 99 |
+
raise ValueError("Expected one named Llama-layout query tensor for every layer")
|
| 100 |
+
else:
|
| 101 |
+
mapping = dict.fromkeys(names, "model.safetensors")
|
| 102 |
+
|
| 103 |
+
headers = {}
|
| 104 |
+
queries = []
|
| 105 |
+
ranges = {}
|
| 106 |
+
for name in names:
|
| 107 |
+
shard = _input_path(root, mapping[name], ".safetensors")
|
| 108 |
+
if shard not in headers:
|
| 109 |
+
headers[shard] = _header(shard)
|
| 110 |
+
header, start, payload_size = headers[shard]
|
| 111 |
+
item = header.get(name)
|
| 112 |
+
if not isinstance(item, dict) or item.get("shape") != [heads * channels, width]:
|
| 113 |
+
raise ValueError(f"Unexpected query shape: {name}")
|
| 114 |
+
dtype = _DTYPES.get(item.get("dtype"))
|
| 115 |
+
if dtype is None:
|
| 116 |
+
raise ValueError(f"Expected dense floating-point query weights: {name}")
|
| 117 |
+
offsets = item.get("data_offsets")
|
| 118 |
+
length = heads * channels * width * dtype.itemsize
|
| 119 |
+
if (not isinstance(offsets, list) or len(offsets) != 2
|
| 120 |
+
or any(type(n) is not int for n in offsets)):
|
| 121 |
+
raise ValueError(f"Invalid safetensors offsets: {name}")
|
| 122 |
+
low, high = offsets
|
| 123 |
+
if not 0 <= low < high <= payload_size or high - low != length:
|
| 124 |
+
raise ValueError(f"Invalid safetensors byte range: {name}")
|
| 125 |
+
ranges.setdefault(shard, []).append((low, high))
|
| 126 |
+
queries.append(QueryRange(name, shard, start + low, length, shape, dtype))
|
| 127 |
+
for shard, spans in ranges.items():
|
| 128 |
+
ordered = sorted(spans)
|
| 129 |
+
if any(left[1] > right[0] for left, right in zip(ordered, ordered[1:])):
|
| 130 |
+
raise ValueError(f"Overlapping query byte ranges: {shard.name}")
|
| 131 |
+
return queries, total
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
def _copy_query(query, path):
|
| 135 |
+
with query.shard.open("rb") as src, path.open("wb") as dst:
|
| 136 |
+
src.seek(query.offset)
|
| 137 |
+
remaining = query.length
|
| 138 |
+
while remaining:
|
| 139 |
+
block = src.read(min(remaining, _CHUNK))
|
| 140 |
+
if not block:
|
| 141 |
+
raise OSError(f"Shard ended during staging: {query.shard.name}")
|
| 142 |
+
dst.write(block)
|
| 143 |
+
remaining -= len(block)
|
| 144 |
+
return np.memmap(path, dtype=query.dtype, mode="r", shape=query.shape)
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def prune_checkpoint(checkpoint, output_dir, *, keep, workspace, max_weights,
|
| 148 |
+
input_root=None, moments_manifest=None, threads=1, seed=0):
|
| 149 |
+
"""Read only the eligible query bytes and write per-layer Boolean masks."""
|
| 150 |
+
queries, total = _queries(checkpoint, max_weights)
|
| 151 |
+
if not 0 <= keep <= total:
|
| 152 |
+
raise ValueError("keep must be between zero and the eligible query slot count")
|
| 153 |
+
names = [query.name for query in queries]
|
| 154 |
+
moments = None
|
| 155 |
+
if moments_manifest is not None:
|
| 156 |
+
root = Path(input_root).resolve(strict=True)
|
| 157 |
+
sources = _json_file(_input_path(root, moments_manifest, ".json"), 1024**2)
|
| 158 |
+
if set(sources) != set(names):
|
| 159 |
+
raise ValueError("Moment manifest must map every eligible query name to an .npy vector")
|
| 160 |
+
moments = [_open_npy(root, sources[name]) for name in names]
|
| 161 |
+
workspace = Path(workspace)
|
| 162 |
+
output_dir = Path(output_dir)
|
| 163 |
+
if workspace.stat().st_dev != output_dir.parent.stat().st_dev:
|
| 164 |
+
raise ValueError("Checkpoint work and output must use the same local filesystem")
|
| 165 |
+
needed = sum(query.length for query in queries) + total + 4096 * len(queries)
|
| 166 |
+
if 0 < keep < total:
|
| 167 |
+
needed += 4 * total
|
| 168 |
+
if shutil.disk_usage(workspace).free < needed + _HEADROOM:
|
| 169 |
+
raise OSError(f"Insufficient local disk: need {needed + _HEADROOM:,} bytes free")
|
| 170 |
+
with tempfile.TemporaryDirectory(prefix="query-staging-", dir=workspace) as stage:
|
| 171 |
+
staged = {}
|
| 172 |
+
for index, query in sorted(enumerate(queries), key=lambda row: (row[1].shard, row[1].offset)):
|
| 173 |
+
staged[index] = _copy_query(query, Path(stage) / f"{index:04d}.bin")
|
| 174 |
+
arrays = [staged[index] for index in range(len(queries))]
|
| 175 |
+
return _publish_masks(names, arrays, moments, keep, output_dir,
|
| 176 |
+
scratch_dir=workspace, threads=threads, seed=seed)
|
tensor_pruning/cli.py
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Command-line access to the file API and HTTP service."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
|
| 8 |
+
from .files import prune_manifest
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def main(argv=None):
|
| 12 |
+
parser = argparse.ArgumentParser(prog="tensor-prune", description=__doc__)
|
| 13 |
+
commands = parser.add_subparsers(dest="command", required=True)
|
| 14 |
+
prune = commands.add_parser("prune", help="prune a memory-mapped manifest")
|
| 15 |
+
prune.add_argument("--input-root", required=True, type=Path)
|
| 16 |
+
prune.add_argument("--manifest", required=True)
|
| 17 |
+
prune.add_argument("--output", required=True, type=Path)
|
| 18 |
+
prune.add_argument("--scratch-dir", type=Path)
|
| 19 |
+
prune.add_argument("--threads", type=int, default=1)
|
| 20 |
+
prune.add_argument("--seed", type=int, default=0)
|
| 21 |
+
|
| 22 |
+
serve = commands.add_parser("serve", help="run the bounded HTTP job service")
|
| 23 |
+
serve.add_argument("--input-root", required=True, type=Path)
|
| 24 |
+
serve.add_argument("--job-root", required=True, type=Path)
|
| 25 |
+
serve.add_argument("--host", default="127.0.0.1")
|
| 26 |
+
serve.add_argument("--port", type=int, default=8080)
|
| 27 |
+
serve.add_argument("--workers", type=int, default=1, help="simultaneous jobs in this process")
|
| 28 |
+
serve.add_argument("--queue-size", type=int, default=2)
|
| 29 |
+
serve.add_argument("--threads-per-job", type=int, default=1)
|
| 30 |
+
serve.add_argument("--max-weights", type=int, default=100_000_000)
|
| 31 |
+
|
| 32 |
+
args = parser.parse_args(argv)
|
| 33 |
+
if args.command == "prune":
|
| 34 |
+
result = prune_manifest(args.input_root, args.manifest, args.output,
|
| 35 |
+
scratch_dir=args.scratch_dir, threads=args.threads, seed=args.seed)
|
| 36 |
+
print(json.dumps(result))
|
| 37 |
+
return
|
| 38 |
+
|
| 39 |
+
try:
|
| 40 |
+
import uvicorn
|
| 41 |
+
from .service import create_app
|
| 42 |
+
except ImportError:
|
| 43 |
+
parser.error("install the HTTP extras with pip install '.[http]'")
|
| 44 |
+
token = os.environ.get("TENSOR_PRUNING_API_TOKEN")
|
| 45 |
+
if args.host not in {"127.0.0.1", "::1", "localhost"} and not token:
|
| 46 |
+
parser.error("set TENSOR_PRUNING_API_TOKEN before binding to a non-loopback address")
|
| 47 |
+
app = create_app(args.input_root, args.job_root, workers=args.workers,
|
| 48 |
+
queue_size=args.queue_size, threads_per_job=args.threads_per_job,
|
| 49 |
+
max_weights=args.max_weights, token=token)
|
| 50 |
+
uvicorn.run(app, host=args.host, port=args.port)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
if __name__ == "__main__":
|
| 54 |
+
main()
|
tensor_pruning/core.py
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""NumPy interface to the OpenMP selector."""
|
| 2 |
+
|
| 3 |
+
from contextlib import contextmanager
|
| 4 |
+
import ctypes as ct
|
| 5 |
+
from functools import lru_cache
|
| 6 |
+
import mmap
|
| 7 |
+
import operator
|
| 8 |
+
import os
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
import shlex
|
| 11 |
+
import subprocess
|
| 12 |
+
import tempfile
|
| 13 |
+
from threading import Lock
|
| 14 |
+
|
| 15 |
+
import numpy as np
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
class Projection(ct.Structure):
|
| 19 |
+
_fields_ = [
|
| 20 |
+
("weights", ct.c_void_p), ("moments", ct.c_void_p), ("mask", ct.c_void_p),
|
| 21 |
+
("heads", ct.c_size_t), ("channels", ct.c_size_t), ("inputs", ct.c_size_t),
|
| 22 |
+
("dtype", ct.c_int),
|
| 23 |
+
]
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
_BUILD_LOCK = Lock()
|
| 27 |
+
_DTYPES = {"float16": 2, "bfloat16": 3, "float32": 4, "float64": 8}
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
@lru_cache(maxsize=1)
|
| 31 |
+
def _compile():
|
| 32 |
+
compiler = shlex.split(os.environ.get("CXX", "g++"))
|
| 33 |
+
source = Path(__file__).with_name("kernel.cpp")
|
| 34 |
+
with tempfile.TemporaryDirectory(prefix="tensor-pruning-build-") as directory:
|
| 35 |
+
library = Path(directory) / "kernel.so"
|
| 36 |
+
command = [*compiler, "-std=c++17", "-O3", "-fopenmp", "-shared", "-fPIC",
|
| 37 |
+
str(source), "-o", str(library)]
|
| 38 |
+
try:
|
| 39 |
+
subprocess.run(command, check=True, capture_output=True, text=True)
|
| 40 |
+
except (OSError, subprocess.CalledProcessError) as exc:
|
| 41 |
+
message = getattr(exc, "stderr", None) or str(exc)
|
| 42 |
+
raise RuntimeError(f"An OpenMP C++17 compiler is required. {message}") from exc
|
| 43 |
+
function = ct.CDLL(str(library)).tensor_prune_many
|
| 44 |
+
function.argtypes = [ct.POINTER(Projection), ct.c_size_t, ct.c_void_p,
|
| 45 |
+
ct.c_size_t, ct.c_int, ct.c_uint64]
|
| 46 |
+
function.restype = ct.c_int
|
| 47 |
+
return function
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def _kernel():
|
| 51 |
+
with _BUILD_LOCK:
|
| 52 |
+
return _compile()
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def _inputs(projections, input_second_moments):
|
| 56 |
+
arrays = tuple(np.asarray(w) for w in projections)
|
| 57 |
+
if not arrays:
|
| 58 |
+
raise ValueError("at least one query projection is required")
|
| 59 |
+
moments = (None,) * len(arrays) if input_second_moments is None else tuple(input_second_moments)
|
| 60 |
+
if len(moments) != len(arrays):
|
| 61 |
+
raise ValueError("provide one input-second-moment vector per projection")
|
| 62 |
+
checked = []
|
| 63 |
+
for w, v in zip(arrays, moments):
|
| 64 |
+
if w.ndim != 3 or not all(w.shape):
|
| 65 |
+
raise ValueError("projections must have nonempty [head, channel, input] shape")
|
| 66 |
+
if w.shape[-1] > np.iinfo(np.uint32).max:
|
| 67 |
+
raise ValueError("input axis exceeds the 32-bit channel index limit")
|
| 68 |
+
if w.dtype.name not in _DTYPES or not w.dtype.isnative:
|
| 69 |
+
raise TypeError("use native-endian float16, bfloat16, float32, or float64 weights")
|
| 70 |
+
if not w.flags.c_contiguous or not w.flags.aligned:
|
| 71 |
+
raise ValueError("projections must be C-contiguous and aligned")
|
| 72 |
+
if v is None:
|
| 73 |
+
v = np.ones(w.shape[-1], dtype=np.float64)
|
| 74 |
+
else:
|
| 75 |
+
v = np.asarray(v)
|
| 76 |
+
if v.dtype.kind not in "fiu" or v.shape != (w.shape[-1],):
|
| 77 |
+
raise ValueError("each moment vector must be real with shape [input]")
|
| 78 |
+
v = np.require(v, dtype=np.float64, requirements=["C", "A"])
|
| 79 |
+
if not np.all(np.isfinite(v) & (v >= 0)):
|
| 80 |
+
raise ValueError("input second moments must be finite and nonnegative")
|
| 81 |
+
checked.append(v)
|
| 82 |
+
return arrays, checked
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _options(total, keep, threads, seed):
|
| 86 |
+
keep = operator.index(keep)
|
| 87 |
+
if not 0 <= keep <= total:
|
| 88 |
+
raise ValueError("keep must be between zero and the total number of weights")
|
| 89 |
+
threads = operator.index(threads)
|
| 90 |
+
if not 1 <= threads <= np.iinfo(np.int32).max:
|
| 91 |
+
raise ValueError("threads must be a positive integer")
|
| 92 |
+
return keep, threads, operator.index(seed) % (1 << 64)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
@contextmanager
|
| 96 |
+
def _partition_workspace(count, scratch_dir):
|
| 97 |
+
if count == 0:
|
| 98 |
+
yield None
|
| 99 |
+
elif scratch_dir is None:
|
| 100 |
+
indices = np.empty(count, dtype=np.uint32)
|
| 101 |
+
yield indices.ctypes.data
|
| 102 |
+
else:
|
| 103 |
+
with tempfile.TemporaryFile(dir=scratch_dir) as file:
|
| 104 |
+
size = count * np.dtype(np.uint32).itemsize
|
| 105 |
+
os.posix_fallocate(file.fileno(), 0, size)
|
| 106 |
+
with mmap.mmap(file.fileno(), size) as region:
|
| 107 |
+
yield ct.addressof(ct.c_char.from_buffer(region))
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def _run(arrays, moments, masks, total, keep, threads, seed, scratch_dir):
|
| 111 |
+
descriptors = (Projection * len(arrays))(*[
|
| 112 |
+
Projection(w.ctypes.data, v.ctypes.data, mask.ctypes.data, *w.shape, _DTYPES[w.dtype.name])
|
| 113 |
+
for w, v, mask in zip(arrays, moments, masks)
|
| 114 |
+
])
|
| 115 |
+
native = _kernel()
|
| 116 |
+
with _partition_workspace(total if 0 < keep < total else 0, scratch_dir) as scratch:
|
| 117 |
+
status = native(descriptors, len(descriptors), scratch, keep, threads, seed)
|
| 118 |
+
if status == 1:
|
| 119 |
+
raise ValueError("weights and intermediate energies must be finite")
|
| 120 |
+
if status == 2:
|
| 121 |
+
raise MemoryError("native partition allocation failed")
|
| 122 |
+
if status:
|
| 123 |
+
raise RuntimeError(f"native selector failed with status {status}")
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def prune_queries(projections, keep, *, input_second_moments=None, threads=1,
|
| 127 |
+
seed=0, scratch_dir=None):
|
| 128 |
+
"""Return one boolean mask per [head, channel, input] projection.
|
| 129 |
+
|
| 130 |
+
All projections participate in a single retention budget, even if they
|
| 131 |
+
have different shapes or dtypes. Masks contain exactly keep true entries.
|
| 132 |
+
Existing zeros count as eligible slots. Inputs can be read-only memmaps.
|
| 133 |
+
Set scratch_dir to use a disk-backed partition workspace.
|
| 134 |
+
"""
|
| 135 |
+
arrays, moments = _inputs(projections, input_second_moments)
|
| 136 |
+
total = sum(w.size for w in arrays)
|
| 137 |
+
keep, threads, seed = _options(total, keep, threads, seed)
|
| 138 |
+
masks = tuple(np.empty(w.shape, dtype=bool) for w in arrays)
|
| 139 |
+
_run(arrays, moments, masks, total, keep, threads, seed, scratch_dir)
|
| 140 |
+
return masks
|
tensor_pruning/files.py
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Memory-mapped checkpoint manifests and atomic publication of masks."""
|
| 2 |
+
|
| 3 |
+
import fcntl
|
| 4 |
+
import json
|
| 5 |
+
import os
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
import shutil
|
| 8 |
+
import tempfile
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
|
| 12 |
+
from .core import _inputs, _options, _run
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def _input_path(root, relative, suffix):
|
| 16 |
+
if not isinstance(relative, str) or Path(relative).is_absolute():
|
| 17 |
+
raise ValueError("manifest paths must be relative to the input root")
|
| 18 |
+
path = (root / relative).resolve(strict=True)
|
| 19 |
+
if not path.is_relative_to(root) or not path.is_file() or path.suffix != suffix:
|
| 20 |
+
raise ValueError(f"expected a {suffix} file inside the input root")
|
| 21 |
+
return path
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _open_npy(root, name):
|
| 25 |
+
return np.load(_input_path(root, name, ".npy"), mmap_mode="r", allow_pickle=False)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def _read_manifest(root, manifest, max_weights):
|
| 29 |
+
path = _input_path(root, manifest, ".json")
|
| 30 |
+
with path.open("rb") as file:
|
| 31 |
+
raw = file.read(1_048_577)
|
| 32 |
+
if len(raw) > 1_048_576:
|
| 33 |
+
raise ValueError("manifest exceeds 1 MiB")
|
| 34 |
+
spec = json.loads(raw)
|
| 35 |
+
if not isinstance(spec, dict) or set(spec) != {"keep", "projections"}:
|
| 36 |
+
raise ValueError("manifest must contain keep and projections")
|
| 37 |
+
items = spec["projections"]
|
| 38 |
+
if not isinstance(items, list) or not 1 <= len(items) <= 1024:
|
| 39 |
+
raise ValueError("provide 1 to 1024 projection records")
|
| 40 |
+
if type(spec["keep"]) is not int:
|
| 41 |
+
raise ValueError("manifest keep must be an integer")
|
| 42 |
+
names, weights, moments = [], [], []
|
| 43 |
+
seen = set()
|
| 44 |
+
total = 0
|
| 45 |
+
for item in items:
|
| 46 |
+
if (not isinstance(item, dict) or not {"name", "weights"} <= item.keys()
|
| 47 |
+
or item.keys() - {"name", "weights", "input_second_moment"}):
|
| 48 |
+
raise ValueError("projection records require name and weights")
|
| 49 |
+
name = item["name"]
|
| 50 |
+
if not isinstance(name, str) or not 1 <= len(name) <= 256 or name in seen:
|
| 51 |
+
raise ValueError("projection names must be nonempty and unique")
|
| 52 |
+
seen.add(name)
|
| 53 |
+
w = _open_npy(root, item["weights"])
|
| 54 |
+
total += w.size
|
| 55 |
+
if max_weights is not None and total > max_weights:
|
| 56 |
+
raise ValueError("manifest exceeds the configured weight limit")
|
| 57 |
+
names.append(name)
|
| 58 |
+
weights.append(w)
|
| 59 |
+
moments.append(_open_npy(root, item["input_second_moment"])
|
| 60 |
+
if "input_second_moment" in item else None)
|
| 61 |
+
return names, weights, moments, spec["keep"]
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def prune_manifest(input_root, manifest, output_dir, *, scratch_dir=None,
|
| 65 |
+
threads=1, seed=0, max_weights=None):
|
| 66 |
+
"""Publish disk-backed .npy masks and result.json to a new directory."""
|
| 67 |
+
root = Path(input_root).resolve(strict=True)
|
| 68 |
+
names, weights, moments, keep = _read_manifest(root, manifest, max_weights)
|
| 69 |
+
return _publish_masks(names, weights, moments, keep, output_dir,
|
| 70 |
+
scratch_dir=scratch_dir, threads=threads, seed=seed)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def _publish_masks(names, weights, moments, keep, output_dir, *,
|
| 74 |
+
scratch_dir, threads, seed):
|
| 75 |
+
output = Path(output_dir).absolute()
|
| 76 |
+
arrays, moments = _inputs(weights, moments)
|
| 77 |
+
total = sum(w.size for w in arrays)
|
| 78 |
+
keep, threads, seed = _options(total, keep, threads, seed)
|
| 79 |
+
|
| 80 |
+
output.parent.mkdir(parents=True, exist_ok=True)
|
| 81 |
+
lock_path = output.parent / f".{output.name}.lock"
|
| 82 |
+
with lock_path.open("a+b") as lock:
|
| 83 |
+
fcntl.flock(lock, fcntl.LOCK_EX)
|
| 84 |
+
if output.exists() or output.is_symlink():
|
| 85 |
+
raise FileExistsError(output)
|
| 86 |
+
stage = Path(tempfile.mkdtemp(prefix=f".{output.name}-", dir=output.parent))
|
| 87 |
+
try:
|
| 88 |
+
paths = [stage / f"mask-{j:04d}.npy" for j in range(len(arrays))]
|
| 89 |
+
masks = [np.lib.format.open_memmap(path, mode="w+", dtype=bool, shape=w.shape)
|
| 90 |
+
for path, w in zip(paths, arrays)]
|
| 91 |
+
for path in paths:
|
| 92 |
+
with path.open("r+b") as file:
|
| 93 |
+
os.posix_fallocate(file.fileno(), 0, path.stat().st_size)
|
| 94 |
+
scratch = Path(scratch_dir) if scratch_dir is not None else stage
|
| 95 |
+
_run(arrays, moments, masks, total, keep, threads, seed, scratch)
|
| 96 |
+
entries = []
|
| 97 |
+
for name, path, mask in zip(names, paths, masks):
|
| 98 |
+
mask.flush()
|
| 99 |
+
entries.append(dict(name=name, mask=path.name, shape=list(mask.shape),
|
| 100 |
+
kept_slots=int(np.count_nonzero(mask))))
|
| 101 |
+
result = dict(weights=total, kept_slots=keep, projections=entries)
|
| 102 |
+
if sum(item["kept_slots"] for item in entries) != keep:
|
| 103 |
+
raise RuntimeError("native selector returned an incorrect retention budget")
|
| 104 |
+
(stage / "result.json").write_text(json.dumps(result, indent=2) + "\n")
|
| 105 |
+
stage.rename(output)
|
| 106 |
+
return result
|
| 107 |
+
finally:
|
| 108 |
+
if stage.exists():
|
| 109 |
+
shutil.rmtree(stage)
|
tensor_pruning/kernel.cpp
ADDED
|
@@ -0,0 +1,438 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#include <algorithm>
|
| 2 |
+
#include <cmath>
|
| 3 |
+
#include <cstddef>
|
| 4 |
+
#include <cstdint>
|
| 5 |
+
#include <cstring>
|
| 6 |
+
#include <limits>
|
| 7 |
+
#include <new>
|
| 8 |
+
#include <exception>
|
| 9 |
+
#include <random>
|
| 10 |
+
#include <utility>
|
| 11 |
+
#include <vector>
|
| 12 |
+
#include <omp.h>
|
| 13 |
+
|
| 14 |
+
namespace {
|
| 15 |
+
using Real = long double;
|
| 16 |
+
using Index = std::size_t;
|
| 17 |
+
using Input = std::uint32_t;
|
| 18 |
+
using RNG = std::mt19937_64;
|
| 19 |
+
|
| 20 |
+
struct Key {
|
| 21 |
+
Real score;
|
| 22 |
+
Index id;
|
| 23 |
+
};
|
| 24 |
+
|
| 25 |
+
bool less(Key a, Key b) {
|
| 26 |
+
return a.score < b.score || (a.score == b.score && a.id < b.id);
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
struct Band {
|
| 30 |
+
Index first = 0, last = 0;
|
| 31 |
+
Real before = 0, mass = 0;
|
| 32 |
+
|
| 33 |
+
Index size() const { return last - first; }
|
| 34 |
+
Real gains() const { return mass * (2 * before + mass); }
|
| 35 |
+
};
|
| 36 |
+
|
| 37 |
+
// The two nonempty bands are adjacent in both array position and score order.
|
| 38 |
+
Band join(Band a, Band b) {
|
| 39 |
+
if (!a.size()) return b;
|
| 40 |
+
if (!b.size()) return a;
|
| 41 |
+
return {a.first, b.last, a.before, a.mass + b.mass};
|
| 42 |
+
}
|
| 43 |
+
|
| 44 |
+
struct Parts {
|
| 45 |
+
Band low, equal, high;
|
| 46 |
+
};
|
| 47 |
+
|
| 48 |
+
struct RowBand {
|
| 49 |
+
Index channel;
|
| 50 |
+
Band band;
|
| 51 |
+
};
|
| 52 |
+
|
| 53 |
+
struct HeadBand {
|
| 54 |
+
std::vector<RowBand> rows;
|
| 55 |
+
Index count = 0;
|
| 56 |
+
Real before = 0; // Channel-gain mass below this interval in the full head.
|
| 57 |
+
Real mass = 0;
|
| 58 |
+
|
| 59 |
+
void add(Index channel, Band band) {
|
| 60 |
+
if (band.size()) {
|
| 61 |
+
rows.push_back({channel, band});
|
| 62 |
+
count += band.size();
|
| 63 |
+
mass += band.gains();
|
| 64 |
+
}
|
| 65 |
+
}
|
| 66 |
+
};
|
| 67 |
+
|
| 68 |
+
// Merge metadata only. Each channel's partitioned indices stay in place.
|
| 69 |
+
HeadBand join(HeadBand a, HeadBand b) {
|
| 70 |
+
if (!a.count) return b;
|
| 71 |
+
if (!b.count) return a;
|
| 72 |
+
HeadBand result;
|
| 73 |
+
result.before = a.before;
|
| 74 |
+
result.rows.reserve(a.rows.size() + b.rows.size());
|
| 75 |
+
Index i = 0, j = 0;
|
| 76 |
+
while (i < a.rows.size() && j < b.rows.size()) {
|
| 77 |
+
const auto& left = a.rows[i];
|
| 78 |
+
const auto& right = b.rows[j];
|
| 79 |
+
if (left.channel < right.channel) {
|
| 80 |
+
result.add(left.channel, left.band);
|
| 81 |
+
++i;
|
| 82 |
+
} else if (right.channel < left.channel) {
|
| 83 |
+
result.add(right.channel, right.band);
|
| 84 |
+
++j;
|
| 85 |
+
} else {
|
| 86 |
+
result.add(left.channel, join(left.band, right.band));
|
| 87 |
+
++i;
|
| 88 |
+
++j;
|
| 89 |
+
}
|
| 90 |
+
}
|
| 91 |
+
for (; i < a.rows.size(); ++i) result.add(a.rows[i].channel, a.rows[i].band);
|
| 92 |
+
for (; j < b.rows.size(); ++j) result.add(b.rows[j].channel, b.rows[j].band);
|
| 93 |
+
return result;
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
struct HeadParts {
|
| 97 |
+
HeadBand low, equal, high;
|
| 98 |
+
};
|
| 99 |
+
|
| 100 |
+
struct Pivot {
|
| 101 |
+
Index row_slot;
|
| 102 |
+
Input input;
|
| 103 |
+
};
|
| 104 |
+
|
| 105 |
+
Index random_index(RNG& rng, Index size) {
|
| 106 |
+
return std::uniform_int_distribution<Index>(0, size - 1)(rng);
|
| 107 |
+
}
|
| 108 |
+
|
| 109 |
+
struct Projection {
|
| 110 |
+
const void* weights;
|
| 111 |
+
const double* moments;
|
| 112 |
+
std::uint8_t* mask;
|
| 113 |
+
Index heads, channels, inputs;
|
| 114 |
+
int dtype;
|
| 115 |
+
};
|
| 116 |
+
|
| 117 |
+
Real weight_at(const unsigned char* weights, int dtype, Index i) {
|
| 118 |
+
if (dtype == 4) {
|
| 119 |
+
float x;
|
| 120 |
+
std::memcpy(&x, weights + 4*i, sizeof(x));
|
| 121 |
+
return x;
|
| 122 |
+
}
|
| 123 |
+
if (dtype == 8) {
|
| 124 |
+
double x;
|
| 125 |
+
std::memcpy(&x, weights + 8*i, sizeof(x));
|
| 126 |
+
return x;
|
| 127 |
+
}
|
| 128 |
+
std::uint16_t bits;
|
| 129 |
+
std::memcpy(&bits, weights + 2*i, sizeof(bits));
|
| 130 |
+
if (dtype == 3) {
|
| 131 |
+
const std::uint32_t wide = static_cast<std::uint32_t>(bits) << 16;
|
| 132 |
+
float x;
|
| 133 |
+
std::memcpy(&x, &wide, sizeof(x));
|
| 134 |
+
return x;
|
| 135 |
+
}
|
| 136 |
+
#ifdef __FLT16_MANT_DIG__
|
| 137 |
+
_Float16 x;
|
| 138 |
+
std::memcpy(&x, &bits, sizeof(x));
|
| 139 |
+
return static_cast<float>(x);
|
| 140 |
+
#else
|
| 141 |
+
const unsigned exponent = (bits >> 10) & 31;
|
| 142 |
+
const unsigned fraction = bits & 1023;
|
| 143 |
+
Real x;
|
| 144 |
+
if (exponent == 31) {
|
| 145 |
+
x = fraction ? std::numeric_limits<Real>::quiet_NaN()
|
| 146 |
+
: std::numeric_limits<Real>::infinity();
|
| 147 |
+
} else if (exponent) {
|
| 148 |
+
x = std::ldexp(static_cast<Real>(1024 + fraction), static_cast<int>(exponent) - 25);
|
| 149 |
+
} else {
|
| 150 |
+
x = std::ldexp(static_cast<Real>(fraction), -24);
|
| 151 |
+
}
|
| 152 |
+
return bits & 32768 ? -x : x;
|
| 153 |
+
#endif
|
| 154 |
+
}
|
| 155 |
+
|
| 156 |
+
struct Channel {
|
| 157 |
+
const unsigned char* weights;
|
| 158 |
+
const double* moments;
|
| 159 |
+
Input* indices;
|
| 160 |
+
std::uint8_t* mask;
|
| 161 |
+
Index base;
|
| 162 |
+
int dtype;
|
| 163 |
+
|
| 164 |
+
Key key(Input input) const {
|
| 165 |
+
const Real w = weight_at(weights, dtype, input);
|
| 166 |
+
return {w * w * static_cast<Real>(moments[input]), base + input};
|
| 167 |
+
}
|
| 168 |
+
|
| 169 |
+
Parts split_raw(Band band, Input pivot) const {
|
| 170 |
+
const Key cut = key(pivot);
|
| 171 |
+
Index lower = band.first, pos = band.first, upper = band.last;
|
| 172 |
+
Real low_mass = 0, high_mass = 0;
|
| 173 |
+
while (pos < upper) {
|
| 174 |
+
const Key value = key(indices[pos]);
|
| 175 |
+
if (less(value, cut)) {
|
| 176 |
+
low_mass += value.score;
|
| 177 |
+
std::swap(indices[lower++], indices[pos++]);
|
| 178 |
+
} else if (less(cut, value)) {
|
| 179 |
+
high_mass += value.score;
|
| 180 |
+
std::swap(indices[pos], indices[--upper]);
|
| 181 |
+
} else {
|
| 182 |
+
++pos;
|
| 183 |
+
}
|
| 184 |
+
}
|
| 185 |
+
return {
|
| 186 |
+
{band.first, lower, band.before, low_mass},
|
| 187 |
+
{lower, upper, band.before + low_mass, cut.score},
|
| 188 |
+
{upper, band.last, band.before + low_mass + cut.score, high_mass}
|
| 189 |
+
};
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
Parts split_gain(Band active, Key cutoff, RNG& rng) const {
|
| 193 |
+
Band low, high;
|
| 194 |
+
while (active.size()) {
|
| 195 |
+
const Input pivot = indices[active.first + random_index(rng, active.size())];
|
| 196 |
+
const Parts parts = split_raw(active, pivot);
|
| 197 |
+
const Key gain{parts.equal.gains(), base + pivot};
|
| 198 |
+
if (less(gain, cutoff)) {
|
| 199 |
+
low = join(join(low, parts.low), parts.equal);
|
| 200 |
+
active = parts.high;
|
| 201 |
+
} else if (less(cutoff, gain)) {
|
| 202 |
+
high = join(join(parts.equal, parts.high), high);
|
| 203 |
+
active = parts.low;
|
| 204 |
+
} else {
|
| 205 |
+
return {join(low, parts.low), parts.equal, join(parts.high, high)};
|
| 206 |
+
}
|
| 207 |
+
}
|
| 208 |
+
return {low, {}, high};
|
| 209 |
+
}
|
| 210 |
+
};
|
| 211 |
+
|
| 212 |
+
struct HeadView {
|
| 213 |
+
const unsigned char* weights;
|
| 214 |
+
const double* moments;
|
| 215 |
+
std::uint8_t* mask;
|
| 216 |
+
Input* indices;
|
| 217 |
+
Index base, channels, inputs, row_base;
|
| 218 |
+
int dtype;
|
| 219 |
+
};
|
| 220 |
+
|
| 221 |
+
struct Tensor {
|
| 222 |
+
std::vector<HeadView> heads;
|
| 223 |
+
|
| 224 |
+
Channel row(Index head, Index channel) {
|
| 225 |
+
const auto& view = heads[head];
|
| 226 |
+
const Index offset = channel * view.inputs;
|
| 227 |
+
const Index item_size = view.dtype == 3 ? 2 : view.dtype;
|
| 228 |
+
return {view.weights + offset * item_size, view.moments,
|
| 229 |
+
view.indices ? view.indices + offset : nullptr, view.mask + offset,
|
| 230 |
+
view.base + offset, view.dtype};
|
| 231 |
+
}
|
| 232 |
+
|
| 233 |
+
Pivot pick(Index head, const HeadBand& active, Index rank) {
|
| 234 |
+
Index slot = 0;
|
| 235 |
+
for (; rank >= active.rows[slot].band.size(); ++slot) {
|
| 236 |
+
rank -= active.rows[slot].band.size();
|
| 237 |
+
}
|
| 238 |
+
const auto& item = active.rows[slot];
|
| 239 |
+
return {slot, row(head, item.channel).indices[item.band.first + rank]};
|
| 240 |
+
}
|
| 241 |
+
|
| 242 |
+
std::pair<HeadParts, Key> split_pivot(
|
| 243 |
+
Index head, const HeadBand& active, Pivot pivot, RNG& rng
|
| 244 |
+
) {
|
| 245 |
+
const auto& pivot_row = active.rows[pivot.row_slot];
|
| 246 |
+
const auto channel = row(head, pivot_row.channel);
|
| 247 |
+
const Parts pivot_parts = channel.split_raw(pivot_row.band, pivot.input);
|
| 248 |
+
const Key channel_gain{pivot_parts.equal.gains(), channel.base + pivot.input};
|
| 249 |
+
|
| 250 |
+
HeadParts result;
|
| 251 |
+
result.low.before = active.before;
|
| 252 |
+
result.low.rows.reserve(active.rows.size());
|
| 253 |
+
result.high.rows.reserve(active.rows.size());
|
| 254 |
+
for (Index j = 0; j < active.rows.size(); ++j) {
|
| 255 |
+
const auto& item = active.rows[j];
|
| 256 |
+
const Parts parts = j == pivot.row_slot ? pivot_parts
|
| 257 |
+
: row(head, item.channel).split_gain(item.band, channel_gain, rng);
|
| 258 |
+
result.low.add(item.channel, parts.low);
|
| 259 |
+
result.high.add(item.channel, parts.high);
|
| 260 |
+
}
|
| 261 |
+
result.equal.before = active.before + result.low.mass;
|
| 262 |
+
result.equal.add(pivot_row.channel, pivot_parts.equal);
|
| 263 |
+
result.high.before = result.equal.before + channel_gain.score;
|
| 264 |
+
const Key final_gain{
|
| 265 |
+
channel_gain.score * (2 * result.equal.before + channel_gain.score),
|
| 266 |
+
channel_gain.id
|
| 267 |
+
};
|
| 268 |
+
return {std::move(result), final_gain};
|
| 269 |
+
}
|
| 270 |
+
|
| 271 |
+
HeadParts split_head(Index head, const HeadBand& active, Key cutoff, RNG& rng) {
|
| 272 |
+
if (!active.count) return {};
|
| 273 |
+
const Pivot pivot = pick(head, active, random_index(rng, active.count));
|
| 274 |
+
auto [parts, gain] = split_pivot(head, active, pivot, rng);
|
| 275 |
+
if (less(gain, cutoff)) {
|
| 276 |
+
HeadParts next = split_head(head, parts.high, cutoff, rng);
|
| 277 |
+
next.low = join(join(std::move(parts.low), std::move(parts.equal)),
|
| 278 |
+
std::move(next.low));
|
| 279 |
+
return next;
|
| 280 |
+
}
|
| 281 |
+
if (less(cutoff, gain)) {
|
| 282 |
+
HeadParts next = split_head(head, parts.low, cutoff, rng);
|
| 283 |
+
next.high = join(join(std::move(next.high), std::move(parts.equal)),
|
| 284 |
+
std::move(parts.high));
|
| 285 |
+
return next;
|
| 286 |
+
}
|
| 287 |
+
return parts;
|
| 288 |
+
}
|
| 289 |
+
|
| 290 |
+
void keep(Index head, const HeadBand& band) {
|
| 291 |
+
for (const auto& item : band.rows) {
|
| 292 |
+
const auto channel = row(head, item.channel);
|
| 293 |
+
for (Index j = item.band.first; j < item.band.last; ++j) {
|
| 294 |
+
channel.mask[channel.indices[j]] = 1;
|
| 295 |
+
}
|
| 296 |
+
}
|
| 297 |
+
}
|
| 298 |
+
};
|
| 299 |
+
|
| 300 |
+
int prune(const Projection* projections, Index count, Input* indices,
|
| 301 |
+
Index budget, int threads, std::uint64_t seed) {
|
| 302 |
+
Tensor tensor;
|
| 303 |
+
Index total = 0, total_rows = 0;
|
| 304 |
+
std::vector<std::pair<Index, Index>> rows;
|
| 305 |
+
for (Index p = 0; p < count; ++p) {
|
| 306 |
+
const auto& view = projections[p];
|
| 307 |
+
const Index head_size = view.channels * view.inputs;
|
| 308 |
+
const Index item_size = view.dtype == 3 ? 2 : view.dtype;
|
| 309 |
+
const auto* weights = static_cast<const unsigned char*>(view.weights);
|
| 310 |
+
for (Index h = 0; h < view.heads; ++h) {
|
| 311 |
+
const Index offset = h * head_size;
|
| 312 |
+
for (Index c = 0; c < view.channels; ++c) rows.emplace_back(tensor.heads.size(), c);
|
| 313 |
+
tensor.heads.push_back({weights + offset * item_size, view.moments,
|
| 314 |
+
view.mask + offset, indices ? indices + total + offset : nullptr,
|
| 315 |
+
total + offset, view.channels, view.inputs, total_rows, view.dtype});
|
| 316 |
+
total_rows += view.channels;
|
| 317 |
+
}
|
| 318 |
+
total += view.heads * head_size;
|
| 319 |
+
}
|
| 320 |
+
|
| 321 |
+
std::vector<Real> masses(indices ? rows.size() : 0);
|
| 322 |
+
int invalid = 0;
|
| 323 |
+
const int workers = threads ? threads : omp_get_max_threads();
|
| 324 |
+
const int row_workers = static_cast<int>(std::min<Index>(workers, rows.size()));
|
| 325 |
+
#pragma omp parallel for schedule(static) num_threads(row_workers) if(row_workers > 1 && total >= 8192) reduction(|:invalid)
|
| 326 |
+
for (Index r = 0; r < rows.size(); ++r) {
|
| 327 |
+
const auto [h, c] = rows[r];
|
| 328 |
+
const auto channel = tensor.row(h, c);
|
| 329 |
+
const auto inputs = tensor.heads[h].inputs;
|
| 330 |
+
Real mass = 0;
|
| 331 |
+
for (Index i = 0; i < inputs; ++i) {
|
| 332 |
+
const Real w = weight_at(channel.weights, channel.dtype, i);
|
| 333 |
+
if (!std::isfinite(w)) invalid = 1;
|
| 334 |
+
if (indices) {
|
| 335 |
+
channel.indices[i] = static_cast<Input>(i);
|
| 336 |
+
mass += w * w * static_cast<Real>(channel.moments[i]);
|
| 337 |
+
}
|
| 338 |
+
}
|
| 339 |
+
std::fill_n(channel.mask, inputs, budget == total ? 1 : 0);
|
| 340 |
+
if (indices) {
|
| 341 |
+
masses[r] = mass;
|
| 342 |
+
if (!std::isfinite(mass) || !std::isfinite(mass * mass * mass * mass)) invalid = 1;
|
| 343 |
+
}
|
| 344 |
+
}
|
| 345 |
+
if (invalid) return 1;
|
| 346 |
+
if (!indices) return 0;
|
| 347 |
+
|
| 348 |
+
struct ActiveHead {
|
| 349 |
+
Index id;
|
| 350 |
+
HeadBand band;
|
| 351 |
+
RNG rng;
|
| 352 |
+
};
|
| 353 |
+
std::vector<ActiveHead> active;
|
| 354 |
+
active.reserve(tensor.heads.size());
|
| 355 |
+
for (Index h = 0; h < tensor.heads.size(); ++h) {
|
| 356 |
+
const auto& view = tensor.heads[h];
|
| 357 |
+
active.push_back({h, {}, RNG(seed + h + 1)});
|
| 358 |
+
auto& band = active.back().band;
|
| 359 |
+
band.rows.reserve(view.channels);
|
| 360 |
+
for (Index c = 0; c < view.channels; ++c) {
|
| 361 |
+
band.add(c, {0, view.inputs, 0, masses[view.row_base + c]});
|
| 362 |
+
}
|
| 363 |
+
if (!std::isfinite(band.mass * band.mass)) return 1;
|
| 364 |
+
}
|
| 365 |
+
|
| 366 |
+
RNG rng(seed);
|
| 367 |
+
Index remaining = total;
|
| 368 |
+
while (budget && remaining) {
|
| 369 |
+
if (budget == remaining) {
|
| 370 |
+
const int team = static_cast<int>(std::min<Index>(workers, active.size()));
|
| 371 |
+
#pragma omp parallel for schedule(static) num_threads(team) if(team > 1 && remaining >= 8192)
|
| 372 |
+
for (Index j = 0; j < active.size(); ++j) tensor.keep(active[j].id, active[j].band);
|
| 373 |
+
break;
|
| 374 |
+
}
|
| 375 |
+
Index rank = random_index(rng, remaining), chosen = 0;
|
| 376 |
+
for (; rank >= active[chosen].band.count; ++chosen) rank -= active[chosen].band.count;
|
| 377 |
+
auto& anchor = active[chosen];
|
| 378 |
+
const Pivot pivot = tensor.pick(anchor.id, anchor.band, rank);
|
| 379 |
+
auto [anchor_parts, cutoff] = tensor.split_pivot(anchor.id, anchor.band, pivot, anchor.rng);
|
| 380 |
+
|
| 381 |
+
std::vector<HeadParts> parts(active.size());
|
| 382 |
+
std::vector<std::exception_ptr> errors(active.size());
|
| 383 |
+
parts[chosen] = std::move(anchor_parts);
|
| 384 |
+
const int team = static_cast<int>(std::min<Index>(workers, active.size() - 1));
|
| 385 |
+
const int split_workers = std::max(team, 1);
|
| 386 |
+
#pragma omp parallel for schedule(dynamic, 1) num_threads(split_workers) if(split_workers > 1 && remaining >= 8192)
|
| 387 |
+
for (Index j = 0; j < active.size(); ++j) {
|
| 388 |
+
if (j == chosen) continue;
|
| 389 |
+
try {
|
| 390 |
+
auto& head = active[j];
|
| 391 |
+
parts[j] = tensor.split_head(head.id, head.band, cutoff, head.rng);
|
| 392 |
+
} catch (...) {
|
| 393 |
+
errors[j] = std::current_exception();
|
| 394 |
+
}
|
| 395 |
+
}
|
| 396 |
+
Index above = 0;
|
| 397 |
+
for (Index j = 0; j < parts.size(); ++j) {
|
| 398 |
+
if (errors[j]) std::rethrow_exception(errors[j]);
|
| 399 |
+
above += parts[j].high.count;
|
| 400 |
+
}
|
| 401 |
+
|
| 402 |
+
const bool choose_upper = budget <= above;
|
| 403 |
+
if (!choose_upper) {
|
| 404 |
+
const int team = static_cast<int>(std::min<Index>(workers, active.size()));
|
| 405 |
+
#pragma omp parallel for schedule(static) num_threads(team) if(team > 1 && remaining >= 8192)
|
| 406 |
+
for (Index j = 0; j < active.size(); ++j) {
|
| 407 |
+
tensor.keep(active[j].id, parts[j].high);
|
| 408 |
+
tensor.keep(active[j].id, parts[j].equal);
|
| 409 |
+
}
|
| 410 |
+
budget -= above + 1; // Global index makes the pivot key unique.
|
| 411 |
+
}
|
| 412 |
+
remaining = 0;
|
| 413 |
+
Index next = 0;
|
| 414 |
+
for (Index j = 0; j < active.size(); ++j) {
|
| 415 |
+
HeadBand& band = choose_upper ? parts[j].high : parts[j].low;
|
| 416 |
+
if (band.count) {
|
| 417 |
+
remaining += band.count;
|
| 418 |
+
active[j].band = std::move(band);
|
| 419 |
+
if (next != j) active[next] = std::move(active[j]);
|
| 420 |
+
++next;
|
| 421 |
+
}
|
| 422 |
+
}
|
| 423 |
+
active.resize(next);
|
| 424 |
+
}
|
| 425 |
+
return 0;
|
| 426 |
+
}
|
| 427 |
+
} // namespace
|
| 428 |
+
|
| 429 |
+
extern "C" int tensor_prune_many(const Projection* projections, Index count, Input* indices,
|
| 430 |
+
Index budget, int threads, std::uint64_t seed) {
|
| 431 |
+
try {
|
| 432 |
+
return prune(projections, count, indices, budget, threads, seed);
|
| 433 |
+
} catch (const std::bad_alloc&) {
|
| 434 |
+
return 2;
|
| 435 |
+
} catch (...) {
|
| 436 |
+
return 3;
|
| 437 |
+
}
|
| 438 |
+
}
|
tensor_pruning/service.py
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Bounded HTTP job service for local checkpoint manifests."""
|
| 2 |
+
|
| 3 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 4 |
+
from contextlib import asynccontextmanager
|
| 5 |
+
import fcntl
|
| 6 |
+
import json
|
| 7 |
+
import logging
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
import secrets
|
| 10 |
+
from threading import BoundedSemaphore
|
| 11 |
+
from uuid import UUID, uuid4
|
| 12 |
+
|
| 13 |
+
from fastapi import Depends, FastAPI, HTTPException, Request
|
| 14 |
+
from fastapi.responses import FileResponse
|
| 15 |
+
from fastapi.security import APIKeyHeader, HTTPAuthorizationCredentials, HTTPBearer
|
| 16 |
+
from pydantic import BaseModel, ConfigDict, Field
|
| 17 |
+
|
| 18 |
+
from .core import _kernel
|
| 19 |
+
from .files import prune_manifest
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
class JobRequest(BaseModel):
|
| 23 |
+
model_config = ConfigDict(extra="forbid")
|
| 24 |
+
manifest: str = Field(min_length=1, max_length=1024)
|
| 25 |
+
seed: int = Field(default=0, ge=0, lt=1 << 64)
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class CheckpointJobRequest(BaseModel):
|
| 29 |
+
model_config = ConfigDict(extra="forbid")
|
| 30 |
+
checkpoint: str = Field(default=".", min_length=1, max_length=1024)
|
| 31 |
+
keep: int = Field(strict=True, ge=0, lt=1 << 63)
|
| 32 |
+
moments_manifest: str | None = Field(default=None, min_length=1, max_length=1024)
|
| 33 |
+
seed: int = Field(default=0, ge=0, lt=1 << 64)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _write_state(directory, run_id, state, **fields):
|
| 37 |
+
path = directory / "state.json"
|
| 38 |
+
tmp = directory / ".state.json.tmp"
|
| 39 |
+
tmp.write_text(json.dumps(dict(state=state, run_id=run_id, **fields)))
|
| 40 |
+
tmp.replace(path)
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def create_app(input_root, job_root, *, workers=1, queue_size=2, threads_per_job=1,
|
| 44 |
+
max_weights=100_000_000, token=None, require_token=False,
|
| 45 |
+
checkpoint_root=None):
|
| 46 |
+
"""Create a single-process HTTP service with configurable CPU and queue limits."""
|
| 47 |
+
if min(workers, threads_per_job, max_weights) < 1 or queue_size < 0:
|
| 48 |
+
raise ValueError("invalid worker, queue, thread, or weight limit")
|
| 49 |
+
inputs = Path(input_root).resolve(strict=True)
|
| 50 |
+
jobs = Path(job_root).resolve()
|
| 51 |
+
checkpoints = Path(checkpoint_root).resolve(strict=True) if checkpoint_root is not None else None
|
| 52 |
+
if jobs.is_relative_to(inputs) or inputs.is_relative_to(jobs):
|
| 53 |
+
raise ValueError("input and job roots must be separate directories")
|
| 54 |
+
|
| 55 |
+
@asynccontextmanager
|
| 56 |
+
async def lifespan(app):
|
| 57 |
+
jobs.mkdir(parents=True, exist_ok=True)
|
| 58 |
+
with (jobs / ".server.lock").open("a+b") as lock:
|
| 59 |
+
try:
|
| 60 |
+
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
| 61 |
+
except BlockingIOError as exc:
|
| 62 |
+
raise RuntimeError("another service already owns this job directory") from exc
|
| 63 |
+
_kernel()
|
| 64 |
+
app.state.run_id = uuid4().hex
|
| 65 |
+
app.state.slots = BoundedSemaphore(workers + queue_size)
|
| 66 |
+
with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="prune") as pool:
|
| 67 |
+
app.state.pool = pool
|
| 68 |
+
yield
|
| 69 |
+
|
| 70 |
+
app = FastAPI(title="Tensor Pruning", version="1.0.0", lifespan=lifespan)
|
| 71 |
+
|
| 72 |
+
bearer = HTTPBearer(auto_error=False)
|
| 73 |
+
api_key = APIKeyHeader(name="X-Tensor-Pruning-Token", auto_error=False)
|
| 74 |
+
|
| 75 |
+
def authenticate(credentials: HTTPAuthorizationCredentials | None = Depends(bearer),
|
| 76 |
+
space_token: str | None = Depends(api_key)):
|
| 77 |
+
if not token:
|
| 78 |
+
raise HTTPException(503, "The operator must configure the API token")
|
| 79 |
+
supplied = space_token if space_token is not None else (credentials.credentials if credentials else "")
|
| 80 |
+
if not secrets.compare_digest(supplied.encode(), token.encode()):
|
| 81 |
+
raise HTTPException(401, "A valid bearer token is required",
|
| 82 |
+
headers={"WWW-Authenticate": "Bearer"})
|
| 83 |
+
|
| 84 |
+
protected = [Depends(authenticate)] if token or require_token else []
|
| 85 |
+
|
| 86 |
+
@app.get("/health")
|
| 87 |
+
def health():
|
| 88 |
+
return {"status": "ok"}
|
| 89 |
+
|
| 90 |
+
def read_job(job_id, request):
|
| 91 |
+
directory = jobs / str(job_id)
|
| 92 |
+
try:
|
| 93 |
+
state = json.loads((directory / "state.json").read_text())
|
| 94 |
+
except FileNotFoundError as exc:
|
| 95 |
+
raise HTTPException(404, "Unknown job") from exc
|
| 96 |
+
run_id = state.pop("run_id")
|
| 97 |
+
if run_id != request.app.state.run_id and state["state"] in {"queued", "running"}:
|
| 98 |
+
state["state"] = "interrupted"
|
| 99 |
+
return directory, state
|
| 100 |
+
|
| 101 |
+
def enqueue(request, operation):
|
| 102 |
+
state = request.app.state
|
| 103 |
+
if not state.slots.acquire(blocking=False):
|
| 104 |
+
raise HTTPException(429, "Pruning queue is full", headers={"Retry-After": "1"})
|
| 105 |
+
job_id = uuid4()
|
| 106 |
+
directory = jobs / str(job_id)
|
| 107 |
+
|
| 108 |
+
def run():
|
| 109 |
+
try:
|
| 110 |
+
_write_state(directory, state.run_id, "running")
|
| 111 |
+
result = operation(directory)
|
| 112 |
+
_write_state(directory, state.run_id, "completed", result=result)
|
| 113 |
+
except (ValueError, TypeError, OSError, MemoryError) as exc:
|
| 114 |
+
_write_state(directory, state.run_id, "failed", error=str(exc))
|
| 115 |
+
except Exception:
|
| 116 |
+
logging.getLogger(__name__).exception("Pruning job failed")
|
| 117 |
+
_write_state(directory, state.run_id, "failed", error="Internal pruning failure")
|
| 118 |
+
finally:
|
| 119 |
+
state.slots.release()
|
| 120 |
+
|
| 121 |
+
try:
|
| 122 |
+
directory.mkdir()
|
| 123 |
+
_write_state(directory, state.run_id, "queued")
|
| 124 |
+
state.pool.submit(run)
|
| 125 |
+
except Exception:
|
| 126 |
+
state.slots.release()
|
| 127 |
+
raise
|
| 128 |
+
return {"id": str(job_id), "state": "queued"}
|
| 129 |
+
|
| 130 |
+
@app.post("/v1/jobs", status_code=202, dependencies=protected)
|
| 131 |
+
def submit(body: JobRequest, request: Request):
|
| 132 |
+
manifest = Path(body.manifest)
|
| 133 |
+
if manifest.is_absolute() or not (inputs / manifest).resolve().is_relative_to(inputs):
|
| 134 |
+
raise HTTPException(400, "Manifest must be inside the input root")
|
| 135 |
+
return enqueue(request, lambda directory: prune_manifest(
|
| 136 |
+
inputs, body.manifest, directory / "output", threads=threads_per_job,
|
| 137 |
+
seed=body.seed, max_weights=max_weights))
|
| 138 |
+
|
| 139 |
+
if checkpoints is not None:
|
| 140 |
+
from .checkpoint import prune_checkpoint
|
| 141 |
+
|
| 142 |
+
@app.post("/v1/checkpoint-jobs", status_code=202, dependencies=protected)
|
| 143 |
+
def submit_checkpoint(body: CheckpointJobRequest, request: Request):
|
| 144 |
+
relative = Path(body.checkpoint)
|
| 145 |
+
checkpoint = (checkpoints / relative).resolve()
|
| 146 |
+
if relative.is_absolute() or not checkpoint.is_relative_to(checkpoints):
|
| 147 |
+
raise HTTPException(400, "Checkpoint must be inside the model mount")
|
| 148 |
+
if not checkpoint.is_dir():
|
| 149 |
+
raise HTTPException(404, "Mounted checkpoint directory not found")
|
| 150 |
+
return enqueue(request, lambda directory: prune_checkpoint(
|
| 151 |
+
checkpoint, directory / "output", keep=body.keep, workspace=directory,
|
| 152 |
+
max_weights=max_weights, input_root=inputs,
|
| 153 |
+
moments_manifest=body.moments_manifest, threads=threads_per_job, seed=body.seed))
|
| 154 |
+
|
| 155 |
+
@app.get("/v1/jobs/{job_id}", dependencies=protected)
|
| 156 |
+
def status(job_id: UUID, request: Request):
|
| 157 |
+
_, state = read_job(job_id, request)
|
| 158 |
+
return {"id": str(job_id), **state}
|
| 159 |
+
|
| 160 |
+
@app.get("/v1/jobs/{job_id}/masks/{index}", dependencies=protected,
|
| 161 |
+
response_class=FileResponse, responses={
|
| 162 |
+
200: {"content": {"application/octet-stream": {
|
| 163 |
+
"schema": {"type": "string", "format": "binary"}
|
| 164 |
+
}}}
|
| 165 |
+
})
|
| 166 |
+
def mask(job_id: UUID, index: int, request: Request):
|
| 167 |
+
directory, state = read_job(job_id, request)
|
| 168 |
+
if state["state"] != "completed":
|
| 169 |
+
raise HTTPException(409, "The job has no completed masks")
|
| 170 |
+
projections = state["result"]["projections"]
|
| 171 |
+
if not 0 <= index < len(projections):
|
| 172 |
+
raise HTTPException(404, "Unknown projection index")
|
| 173 |
+
filename = projections[index]["mask"]
|
| 174 |
+
return FileResponse(directory / "output" / filename,
|
| 175 |
+
filename=filename, media_type="application/octet-stream")
|
| 176 |
+
|
| 177 |
+
return app
|
tests/test_checkpoint.py
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import io
|
| 2 |
+
import json
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
from types import SimpleNamespace
|
| 5 |
+
import struct
|
| 6 |
+
import tempfile
|
| 7 |
+
import time
|
| 8 |
+
import unittest
|
| 9 |
+
from unittest.mock import patch
|
| 10 |
+
|
| 11 |
+
from fastapi.testclient import TestClient
|
| 12 |
+
import ml_dtypes
|
| 13 |
+
import numpy as np
|
| 14 |
+
|
| 15 |
+
from tensor_pruning import prune_queries
|
| 16 |
+
from tensor_pruning.checkpoint import _queries, prune_checkpoint
|
| 17 |
+
from tensor_pruning.service import create_app
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def _name(layer):
|
| 21 |
+
return f"model.layers.{layer}.self_attn.q_proj.weight"
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _shard(path, entries, *, sparse=False):
|
| 25 |
+
header, offset = {}, 0
|
| 26 |
+
for name, dtype, shape, length, _ in entries:
|
| 27 |
+
header[name] = dict(dtype=dtype, shape=shape, data_offsets=[offset, offset + length])
|
| 28 |
+
offset += length
|
| 29 |
+
raw = json.dumps(header).encode()
|
| 30 |
+
raw += b" " * (-len(raw) % 8)
|
| 31 |
+
with path.open("wb") as f:
|
| 32 |
+
f.write(struct.pack("<Q", len(raw)))
|
| 33 |
+
f.write(raw)
|
| 34 |
+
if sparse:
|
| 35 |
+
f.truncate(8 + len(raw) + offset)
|
| 36 |
+
else:
|
| 37 |
+
for _, _, _, _, payload in entries:
|
| 38 |
+
f.write(payload)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class CheckpointTests(unittest.TestCase):
|
| 42 |
+
def setUp(self):
|
| 43 |
+
self.temp = tempfile.TemporaryDirectory()
|
| 44 |
+
self.addCleanup(self.temp.cleanup)
|
| 45 |
+
self.root = Path(self.temp.name)
|
| 46 |
+
self.inputs = self.root / "inputs"
|
| 47 |
+
self.models = self.root / "models"
|
| 48 |
+
self.inputs.mkdir()
|
| 49 |
+
self.models.mkdir()
|
| 50 |
+
self.jobs = self.root / "jobs"
|
| 51 |
+
|
| 52 |
+
def fixture(self):
|
| 53 |
+
dims = dict(num_hidden_layers=2, num_attention_heads=2, hidden_size=4)
|
| 54 |
+
(self.models / "config.json").write_text(json.dumps(dims))
|
| 55 |
+
arrays = [np.arange(16, dtype=np.float16).reshape(2, 2, 4) - 4,
|
| 56 |
+
np.asarray(np.arange(16) / 3, dtype=ml_dtypes.bfloat16).reshape(2, 2, 4)]
|
| 57 |
+
mapping = {}
|
| 58 |
+
moments = [np.array([1, 2, 3, 4]), np.array([4, 3, 2, 1])]
|
| 59 |
+
moment_names = {}
|
| 60 |
+
for i, (array, dtype) in enumerate(zip(arrays, ("F16", "BF16"))):
|
| 61 |
+
name = _name(i)
|
| 62 |
+
filename = f"shard-{1-i}.safetensors"
|
| 63 |
+
mapping[name] = filename
|
| 64 |
+
noise = ("unused", "F16", [2], 4, b"\xff" * 4)
|
| 65 |
+
query = (name, dtype, [4, 4], array.nbytes, array.tobytes())
|
| 66 |
+
_shard(self.models / filename, [noise, query])
|
| 67 |
+
moment_names[name] = f"moments-{i}.npy"
|
| 68 |
+
np.save(self.inputs / moment_names[name], moments[i])
|
| 69 |
+
(self.models / "model.safetensors.index.json").write_text(json.dumps({"weight_map": mapping}))
|
| 70 |
+
(self.inputs / "moments.json").write_text(json.dumps(moment_names))
|
| 71 |
+
return arrays, moments
|
| 72 |
+
|
| 73 |
+
def test_service_selects_one_global_mask_from_fp16_and_bf16(self):
|
| 74 |
+
arrays, moments = self.fixture()
|
| 75 |
+
expected = prune_queries(arrays, 11, input_second_moments=moments, seed=47)
|
| 76 |
+
app = create_app(self.inputs, self.jobs, checkpoint_root=self.models,
|
| 77 |
+
token="example", require_token=True)
|
| 78 |
+
headers = {"X-Tensor-Pruning-Token": "example"}
|
| 79 |
+
body = {"keep": 11, "seed": 47, "moments_manifest": "moments.json"}
|
| 80 |
+
with TestClient(app) as client:
|
| 81 |
+
self.assertEqual(client.post("/v1/checkpoint-jobs", json=body).status_code, 401)
|
| 82 |
+
self.assertEqual(client.post("/v1/checkpoint-jobs", headers=headers,
|
| 83 |
+
json={**body, "checkpoint": "../"}).status_code, 400)
|
| 84 |
+
created = client.post("/v1/checkpoint-jobs", headers=headers, json=body)
|
| 85 |
+
self.assertEqual(created.status_code, 202, created.text)
|
| 86 |
+
url = "/v1/jobs/" + created.json()["id"]
|
| 87 |
+
for _ in range(500):
|
| 88 |
+
status = client.get(url, headers=headers).json()
|
| 89 |
+
if status["state"] in {"completed", "failed"}:
|
| 90 |
+
break
|
| 91 |
+
time.sleep(.01)
|
| 92 |
+
self.assertEqual(status["state"], "completed", status)
|
| 93 |
+
self.assertEqual(status["result"]["kept_slots"], 11)
|
| 94 |
+
self.assertEqual([p["name"] for p in status["result"]["projections"]],
|
| 95 |
+
[_name(0), _name(1)])
|
| 96 |
+
for i, want in enumerate(expected):
|
| 97 |
+
got = client.get(f"{url}/masks/{i}", headers=headers)
|
| 98 |
+
self.assertEqual(got.status_code, 200, got.text if got.status_code != 200 else "")
|
| 99 |
+
np.testing.assert_array_equal(np.load(io.BytesIO(got.content)), want)
|
| 100 |
+
self.assertEqual(list(self.jobs.glob("*/query-staging-*")), [])
|
| 101 |
+
|
| 102 |
+
def test_job_submission_fails_closed_without_token(self):
|
| 103 |
+
app = create_app(self.inputs, self.jobs, checkpoint_root=self.models, require_token=True)
|
| 104 |
+
with TestClient(app) as client:
|
| 105 |
+
self.assertEqual(client.get("/health").status_code, 200)
|
| 106 |
+
self.assertEqual(client.post("/v1/checkpoint-jobs", json={"keep": 1}).status_code, 503)
|
| 107 |
+
|
| 108 |
+
def test_disk_preflight_rejects_before_staging(self):
|
| 109 |
+
self.fixture()
|
| 110 |
+
usage = SimpleNamespace(total=100, used=100, free=0)
|
| 111 |
+
with patch("tensor_pruning.checkpoint.shutil.disk_usage", return_value=usage):
|
| 112 |
+
with self.assertRaisesRegex(OSError, "Insufficient local disk"):
|
| 113 |
+
prune_checkpoint(self.models, self.root / "out", keep=11,
|
| 114 |
+
workspace=self.root, max_weights=100)
|
| 115 |
+
self.assertEqual(list(self.root.glob("query-staging-*")), [])
|
| 116 |
+
|
| 117 |
+
def test_llama_70b_layout_exceeds_32_bit_global_indices(self):
|
| 118 |
+
shape = [8192, 8192]
|
| 119 |
+
byte_length = 2 * shape[0] * shape[1]
|
| 120 |
+
config = dict(num_hidden_layers=80, num_attention_heads=64, hidden_size=8192)
|
| 121 |
+
(self.models / "config.json").write_text(json.dumps(config))
|
| 122 |
+
entries = [(_name(i), "BF16", shape, byte_length, None) for i in range(80)]
|
| 123 |
+
_shard(self.models / "model.safetensors", entries, sparse=True)
|
| 124 |
+
query, total = _queries(self.models, 6_000_000_000)
|
| 125 |
+
self.assertEqual(total, 5_368_709_120)
|
| 126 |
+
self.assertGreater(total, 1 << 32)
|
| 127 |
+
self.assertEqual(query[-1].shape, (64, 128, 8192))
|
| 128 |
+
self.assertGreater(query[-1].offset, 1 << 33)
|
| 129 |
+
with self.assertRaisesRegex(ValueError, "eligible weight limit"):
|
| 130 |
+
_queries(self.models, 1 << 32)
|
| 131 |
+
|
| 132 |
+
|
tests/test_pruning.py
ADDED
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 2 |
+
import io
|
| 3 |
+
import itertools
|
| 4 |
+
import json
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
import tempfile
|
| 7 |
+
from threading import Event
|
| 8 |
+
import time
|
| 9 |
+
import unittest
|
| 10 |
+
from unittest.mock import patch
|
| 11 |
+
|
| 12 |
+
import numpy as np
|
| 13 |
+
|
| 14 |
+
from tensor_pruning import prune_manifest, prune_queries
|
| 15 |
+
from tensor_pruning.service import create_app
|
| 16 |
+
from tensor_pruning.core import Projection, _kernel
|
| 17 |
+
|
| 18 |
+
from fastapi.testclient import TestClient
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def reference(arrays, keep, moments):
|
| 22 |
+
gains = []
|
| 23 |
+
offset = 0
|
| 24 |
+
for w, v in zip(arrays, moments):
|
| 25 |
+
energies = w.astype(np.longdouble)**2 * v
|
| 26 |
+
for head in energies:
|
| 27 |
+
intermediate = []
|
| 28 |
+
for channel in head:
|
| 29 |
+
mass = 0
|
| 30 |
+
for u, i in sorted((u, offset+i) for i, u in enumerate(channel)):
|
| 31 |
+
intermediate.append((u * (2*mass+u), i))
|
| 32 |
+
mass += u
|
| 33 |
+
offset += len(channel)
|
| 34 |
+
mass = 0
|
| 35 |
+
for d, i in sorted(intermediate):
|
| 36 |
+
gains.append((d * (2*mass+d), i))
|
| 37 |
+
mass += d
|
| 38 |
+
keep_ids = {i for _, i in sorted(gains, reverse=True)[:keep]}
|
| 39 |
+
offset = 0
|
| 40 |
+
masks = []
|
| 41 |
+
for w in arrays:
|
| 42 |
+
masks.append(np.array([i in keep_ids for i in range(offset, offset+w.size)]).reshape(w.shape))
|
| 43 |
+
offset += w.size
|
| 44 |
+
return masks
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def objective(arrays, masks, moments):
|
| 48 |
+
return sum(np.sum(np.sum(np.sum(w.astype(np.longdouble)**2 * v * ~mask, axis=-1)**2,
|
| 49 |
+
axis=-1)**2)
|
| 50 |
+
for w, mask, v in zip(arrays, masks, moments))
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
class SelectorTest(unittest.TestCase):
|
| 54 |
+
def test_mixed_shapes_and_ties_against_reference(self):
|
| 55 |
+
rng = np.random.default_rng(17)
|
| 56 |
+
for i in range(100):
|
| 57 |
+
shapes = [(rng.integers(1, 6), rng.integers(1, 7), rng.integers(1, 14)) for _ in range(3)]
|
| 58 |
+
weights = [rng.integers(-4, 5, size=shape).astype(dtype)
|
| 59 |
+
for shape, dtype in zip(shapes, (np.float16, np.float32, np.float64))]
|
| 60 |
+
moments = [rng.integers(0, 6, size=w.shape[-1]) for w in weights]
|
| 61 |
+
total = sum(w.size for w in weights)
|
| 62 |
+
keep = int(rng.integers(0, total+1))
|
| 63 |
+
actual = prune_queries(weights, keep, input_second_moments=moments, threads=2, seed=i)
|
| 64 |
+
self.assertEqual(sum(np.count_nonzero(mask) for mask in actual), keep)
|
| 65 |
+
for mask, expected in zip(actual, reference(weights, keep, moments)):
|
| 66 |
+
np.testing.assert_array_equal(mask, expected)
|
| 67 |
+
|
| 68 |
+
def test_exhaustive_global_optimum(self):
|
| 69 |
+
rng = np.random.default_rng(18)
|
| 70 |
+
for _ in range(18):
|
| 71 |
+
weights = [rng.integers(-3, 4, size=shape).astype(np.float32)
|
| 72 |
+
for shape in ((1, 2, 3), (2, 1, 2))]
|
| 73 |
+
moments = [np.arange(1, w.shape[-1]+1) for w in weights]
|
| 74 |
+
keep = int(rng.integers(0, 11))
|
| 75 |
+
got = prune_queries(weights, keep, input_second_moments=moments)
|
| 76 |
+
best = float("inf")
|
| 77 |
+
for ids in itertools.combinations(range(10), keep):
|
| 78 |
+
flat = np.zeros(10, dtype=bool)
|
| 79 |
+
flat[list(ids)] = True
|
| 80 |
+
trial = [flat[:6].reshape(weights[0].shape), flat[6:].reshape(weights[1].shape)]
|
| 81 |
+
best = min(best, objective(weights, trial, moments))
|
| 82 |
+
self.assertEqual(objective(weights, got, moments), best)
|
| 83 |
+
|
| 84 |
+
def test_concurrent_calls_and_disk_workspace(self):
|
| 85 |
+
rng = np.random.default_rng(19)
|
| 86 |
+
weights = [rng.normal(size=shape).astype(dtype)
|
| 87 |
+
for shape, dtype in [((12, 8, 65), np.float32), ((6, 5, 73), np.float64)]]
|
| 88 |
+
moments = [rng.uniform(0.2, 2, size=w.shape[-1]) for w in weights]
|
| 89 |
+
with tempfile.TemporaryDirectory() as scratch, ThreadPoolExecutor(3) as pool:
|
| 90 |
+
def run(keep):
|
| 91 |
+
return prune_queries(weights, keep, input_second_moments=moments,
|
| 92 |
+
threads=2, scratch_dir=scratch)
|
| 93 |
+
for keep, result in zip((2145, 3941, 6158), pool.map(run, (2145, 3941, 6158))):
|
| 94 |
+
for mask, expected in zip(result, reference(weights, keep, moments)):
|
| 95 |
+
np.testing.assert_array_equal(mask, expected)
|
| 96 |
+
|
| 97 |
+
def test_native_bfloat16_bits(self):
|
| 98 |
+
x = np.linspace(-6, 5, 72, dtype=np.float32).reshape(3, 4, 6)
|
| 99 |
+
bits = (x.view(np.uint32) >> 16).astype(np.uint16)
|
| 100 |
+
float_equivalent = (bits.astype(np.uint32) << 16).view(np.float32)
|
| 101 |
+
moments = np.linspace(0.25, 1.25, x.shape[-1], dtype=np.float64)
|
| 102 |
+
workspace = np.empty(x.size, dtype=np.uint32)
|
| 103 |
+
mask = np.empty(x.shape, dtype=bool)
|
| 104 |
+
desc = Projection(bits.ctypes.data, moments.ctypes.data, mask.ctypes.data, *x.shape, 3)
|
| 105 |
+
self.assertEqual(_kernel()(desc, 1, workspace.ctypes.data, 29, 1, 0), 0)
|
| 106 |
+
expected = prune_queries([float_equivalent], 29, input_second_moments=[moments])[0]
|
| 107 |
+
np.testing.assert_array_equal(mask, expected)
|
| 108 |
+
|
| 109 |
+
def test_zero_and_full_budget(self):
|
| 110 |
+
w = np.ones((1, 2, 3), dtype=np.float16)
|
| 111 |
+
self.assertEqual(prune_queries([w], 0)[0].sum(), 0)
|
| 112 |
+
self.assertEqual(prune_queries([w], 6)[0].sum(), 6)
|
| 113 |
+
with self.assertRaises(ValueError):
|
| 114 |
+
prune_queries([w], 7)
|
| 115 |
+
with self.assertRaises(ValueError):
|
| 116 |
+
prune_queries([np.full_like(w, np.inf)], 0)
|
| 117 |
+
with self.assertRaises(ValueError):
|
| 118 |
+
prune_queries([w[..., ::-1]], 3)
|
| 119 |
+
with self.assertRaises(ValueError):
|
| 120 |
+
prune_queries([w], 3, input_second_moments=[[-1, 2, 3]])
|
| 121 |
+
with self.assertRaises(TypeError):
|
| 122 |
+
prune_queries([np.ones((1, 2, 3), dtype=np.int16)], 3)
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
class FileAndServiceTest(unittest.TestCase):
|
| 126 |
+
def setUp(self):
|
| 127 |
+
self.temp = tempfile.TemporaryDirectory()
|
| 128 |
+
self.addCleanup(self.temp.cleanup)
|
| 129 |
+
self.root = Path(self.temp.name)
|
| 130 |
+
self.inputs = self.root / "data"
|
| 131 |
+
self.inputs.mkdir()
|
| 132 |
+
self.weights = np.arange(24, dtype=np.float32).reshape(2, 3, 4)
|
| 133 |
+
np.save(self.inputs / "a.npy", self.weights)
|
| 134 |
+
np.save(self.inputs / "v.npy", np.array([0.5, 1., 2., 3.]))
|
| 135 |
+
self.spec = dict(keep=7, projections=[dict(name="q", weights="a.npy",
|
| 136 |
+
input_second_moment="v.npy")])
|
| 137 |
+
(self.inputs / "job.json").write_text(json.dumps(self.spec))
|
| 138 |
+
|
| 139 |
+
def test_manifest_publishes_exact_masks(self):
|
| 140 |
+
output = self.root / "results"
|
| 141 |
+
result = prune_manifest(self.inputs, "job.json", output, threads=2)
|
| 142 |
+
self.assertEqual(result["kept_slots"], 7)
|
| 143 |
+
mask = np.load(output / "mask-0000.npy", allow_pickle=False)
|
| 144 |
+
expected = prune_queries([self.weights], 7, input_second_moments=[[.5, 1., 2., 3.]])[0]
|
| 145 |
+
np.testing.assert_array_equal(mask, expected)
|
| 146 |
+
self.assertEqual(json.loads((output / "result.json").read_text()), result)
|
| 147 |
+
with self.assertRaises(FileExistsError):
|
| 148 |
+
prune_manifest(self.inputs, "job.json", output)
|
| 149 |
+
with self.assertRaises(ValueError):
|
| 150 |
+
prune_manifest(self.inputs, "job.json", self.root / "large", max_weights=12)
|
| 151 |
+
|
| 152 |
+
def test_rejects_symlink_and_failed_native_run_leaves_no_output(self):
|
| 153 |
+
external = self.root / "outside.npy"
|
| 154 |
+
np.save(external, self.weights)
|
| 155 |
+
(self.inputs / "escaped.npy").symlink_to(external)
|
| 156 |
+
self.spec["projections"][0]["weights"] = "escaped.npy"
|
| 157 |
+
(self.inputs / "escape.json").write_text(json.dumps(self.spec))
|
| 158 |
+
with self.assertRaises(ValueError):
|
| 159 |
+
prune_manifest(self.inputs, "escape.json", self.root / "out")
|
| 160 |
+
np.save(self.inputs / "a.npy", np.full_like(self.weights, np.nan))
|
| 161 |
+
with self.assertRaises(ValueError):
|
| 162 |
+
prune_manifest(self.inputs, "job.json", self.root / "out")
|
| 163 |
+
self.assertFalse((self.root / "out").exists())
|
| 164 |
+
|
| 165 |
+
def test_http_job_mask_and_auth(self):
|
| 166 |
+
app = create_app(self.inputs, self.root / "jobs", token="test-token", threads_per_job=2)
|
| 167 |
+
headers = {"Authorization": "Bearer test-token"}
|
| 168 |
+
with TestClient(app) as client:
|
| 169 |
+
self.assertEqual(client.get("/health").status_code, 200)
|
| 170 |
+
self.assertEqual(client.post("/v1/jobs", json={"manifest": "job.json"}).status_code, 401)
|
| 171 |
+
escape = client.post("/v1/jobs", json={"manifest": "../out.json"}, headers=headers)
|
| 172 |
+
self.assertEqual(escape.status_code, 400)
|
| 173 |
+
response = client.post("/v1/jobs", json={"manifest": "job.json"}, headers=headers)
|
| 174 |
+
self.assertEqual(response.status_code, 202)
|
| 175 |
+
endpoint = f"/v1/jobs/{response.json()['id']}"
|
| 176 |
+
for _ in range(150):
|
| 177 |
+
status = client.get(endpoint, headers=headers).json()
|
| 178 |
+
if status["state"] in {"completed", "failed"}:
|
| 179 |
+
break
|
| 180 |
+
time.sleep(0.01)
|
| 181 |
+
self.assertEqual(status["state"], "completed", status)
|
| 182 |
+
binary = client.get(endpoint + "/masks/0", headers=headers)
|
| 183 |
+
self.assertEqual(binary.status_code, 200)
|
| 184 |
+
self.assertEqual(np.load(io.BytesIO(binary.content)).sum(), 7)
|
| 185 |
+
self.assertEqual(client.get(endpoint + "/masks/1", headers=headers).status_code, 404)
|
| 186 |
+
|
| 187 |
+
def test_queue_backpressure(self):
|
| 188 |
+
started, proceed = Event(), Event()
|
| 189 |
+
|
| 190 |
+
def slow(*args, **kwargs):
|
| 191 |
+
started.set()
|
| 192 |
+
if not proceed.wait(5):
|
| 193 |
+
raise RuntimeError("test timeout")
|
| 194 |
+
return prune_manifest(*args, **kwargs)
|
| 195 |
+
|
| 196 |
+
app = create_app(self.inputs, self.root / "jobs", workers=1, queue_size=0)
|
| 197 |
+
with patch("tensor_pruning.service.prune_manifest", slow), TestClient(app) as client:
|
| 198 |
+
try:
|
| 199 |
+
self.assertEqual(client.post("/v1/jobs", json={"manifest": "job.json"}).status_code, 202)
|
| 200 |
+
self.assertTrue(started.wait(2))
|
| 201 |
+
response = client.post("/v1/jobs", json={"manifest": "job.json"})
|
| 202 |
+
self.assertEqual(response.status_code, 429)
|
| 203 |
+
self.assertEqual(response.headers["Retry-After"], "1")
|
| 204 |
+
finally:
|
| 205 |
+
proceed.set()
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
if __name__ == "__main__":
|
| 209 |
+
unittest.main()
|