docs/miner_setup.md

# Running Miner

A high-performance decentralized video processing miner that forwards validator requests to containerized video services. The current miner framework supports:

- **Video2X upscaling** through the `upscaling-video2x` Docker Compose profile.
- **FFmpeg upscaling** through the `upscaling-ffmpeg` Docker Compose profile.
- **FFmpeg compression** through the `compression` Docker Compose service.

`neurons/miner.py` no longer needs the old PM2-managed `services/upscaling/server.py` or `services/compress/server.py` workers. It forwards directly to the service container ports.

---

## Machine Requirements

To achieve optimal results, we recommend the following setup:

- **Operating System**: Ubuntu 24.04 LTS or higher  
  [Learn more about Ubuntu 24.04 LTS](https://ubuntu.com/blog/tag/ubuntu-24-04-lts)
- **GPU**: NVIDIA RTX 4090 or higher
- **VRAM**: At least 16 GB per GPU
- **Docker** and **Docker Compose**
- **NVIDIA drivers** and `nvidia-container-toolkit`
- **Python**: Version 3.10 or higher for running the miner process

---

## Install System Dependencies

Install common host dependencies:

```bash
sudo apt update
sudo apt install -y git curl npm python3 python3-venv python3-pip
```

Install Docker and Docker Compose using the official Docker instructions for your host, then install NVIDIA Container Toolkit so Compose services can access the GPU.

Verify Docker can see the GPU:

```bash
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```

---

## Install PM2

PM2 is optional for the processing containers, but useful for managing the miner process itself.

```bash
sudo npm install pm2 -g
pm2 update
```

---

## Install Project Dependencies

### 1. Clone the Repository

```bash
git clone https://github.com/vidaio-subnet/vidaio-subnet.git
cd vidaio-subnet
```

### 2. Set Up a Virtual Environment

```bash
python3 -m venv venv
source venv/bin/activate
```

### 3. Install the Package

```bash
pip install -e .
```

---

## Configure Storage

The miner process downloads validator payload URLs, places local files in a shared work directory, forwards local paths to the processing containers, uploads processed outputs to S3-compatible storage, and returns presigned URLs. The processing containers run without egress.

Create a miner service environment file:

```bash
cp miner/.env.template miner/.env
```

Edit `miner/.env` with your storage credentials:

```env
COMPOSE_PROJECT_NAME=miner
MINER_SHARED_DIR=/tmp/vidaio-miner-video-tmp
MINER_STORAGE_PROVIDER=backblaze
MINER_STORAGE_S3_ACCESS_KEY_ID=your-access-key
MINER_STORAGE_S3_SECRET_ACCESS_KEY=your-secret-key
MINER_STORAGE_S3_REGION=us-east-005
MINER_STORAGE_S3_BUCKET_NAME=your-bucket-name
MINER_STORAGE_S3_ENDPOINT_URL=https://s3.us-east-005.backblazeb2.com
```

For AWS S3, use the same variable names and leave `MINER_STORAGE_S3_ENDPOINT_URL` blank.

Create the shared work directory before starting the miner and containers:

```bash
mkdir -p /tmp/vidaio-miner-video-tmp
chmod 777 /tmp/vidaio-miner-video-tmp
```

---

## Start Processing Services

Before starting a service, choose which inference task this miner will advertise
to validators. In `neurons/miner.py`, use:

```python
warrant_task = TaskType.UPSCALING
```

for upscaling, or:

```python
warrant_task = TaskType.COMPRESSION
```

for compression. Starting the compression container does **not** select
compression inference automatically. `MINER_MODES` must also include
`inference`. Use the full
[`miner_inference_checklist.md`](miner_inference_checklist.md) before going live.

Run Docker Compose commands from the `miner` directory:

```bash
cd miner
```

### Option A: Video2X Upscaling Miner

Use this profile if you want Video2X-based upscaling. Complete the host/container setup first:

```bash
cat upscaling/SETUP.md 
```

After running all the commands listed above, start the Video2X upscaling service:

```bash
docker compose --profile upscaling-video2x up -d upscaling-video2x
```

The service is exposed on `http://localhost:8003`.

### Option B: FFmpeg Upscaling Miner

Use this profile if you want FFmpeg-based upscaling instead of Video2X:

```bash
docker compose --profile upscaling-ffmpeg up -d upscaling-ffmpeg
```

The FFmpeg upscaling service is exposed on `http://localhost:8005`.

Before starting `neurons/miner.py`, point the miner at that port:

```bash
export MINER_UPSCALING_SERVICE_URL=http://localhost:8005
```

### Option C: Compression Miner

Start the compression service:

```bash
docker compose up -d compression
```

The compression service is exposed on `http://localhost:8004`.

Return to the repository root before launching the miner:

```bash
cd ..
```

---

## Miner Forwarding Configuration

`neurons/miner.py` forwards requests to these service URLs:

```bash
export MINER_UPSCALING_SERVICE_URL=http://localhost:8003
export MINER_COMPRESSION_SERVICE_URL=http://localhost:8004
export MINER_SHARED_VOLUME_PATH=/tmp/vidaio-miner-video-tmp
```

Defaults:

- `MINER_UPSCALING_SERVICE_URL` defaults to `http://localhost:8003` for Video2X upscaling.
- `MINER_COMPRESSION_SERVICE_URL` defaults to `http://localhost:8004`.
- `MINER_SHARED_VOLUME_PATH` defaults to `MINER_SHARED_DIR`, then `/tmp/vidaio-miner-video-tmp`.
- For FFmpeg upscaling, override `MINER_UPSCALING_SERVICE_URL` to `http://localhost:8005`.

Useful service checks:

```bash
curl -sf http://localhost:8003/health
curl -sf http://localhost:8004/health
curl -sf http://localhost:8005/health
```

Only the services you started need to pass health checks.

---

## Run the Miner

Start the miner from the repository root:

```bash
python3 neurons/miner.py \
  --wallet.name [Your_Wallet_Name] \
  --wallet.hotkey [Your_Hotkey_Name] \
  --subtensor.network finney \
  --netuid 85 \
  --axon.port [port] \
  --logging.debug
```

### Run the Miner with PM2

Export any `MINER_*_SERVICE_URL` overrides in the same shell before starting PM2. If you change those variables later, restart with `--update-env`.

```bash
pm2 start "python3 neurons/miner.py --wallet.name [Your_Wallet_Name] --wallet.hotkey [Your_Hotkey_Name] --subtensor.network finney --netuid 85 --axon.port [port] --logging.debug" --name video-miner
```

### Parameters

- `--wallet.name`: Replace `[Your_Wallet_Name]` with your wallet name.
- `--wallet.hotkey`: Replace `[Your_Hotkey_Name]` with your hotkey name.
- `--subtensor.network`: Specify the target network, usually `finney`.
- `--netuid`: Specify the network UID, currently `85`.
- `--axon.port`: Replace `[port]` with the desired public axon port.
- `--logging.debug`: Enables debug-level logging.

---

## Managing Services

Processing containers:

```bash
cd miner
docker compose ps
docker compose logs -f upscaling-video2x
docker compose logs -f upscaling-ffmpeg
docker compose logs -f compression
docker compose down
```

Miner process:

```bash
pm2 logs video-miner
pm2 restart video-miner --update-env
pm2 stop video-miner
```

---

## Additional Notes

- For ordinary validator inference, complete
  [`miner_inference_checklist.md`](miner_inference_checklist.md). Compression
  miners must advertise `TaskType.COMPRESSION` through `warrant_task` in
  `neurons/miner.py`.
- For a new compression competition, complete
  [`miner_competition_checklist.md`](miner_competition_checklist.md). The full
  competition contract is in [`competitions.md`](competitions.md).
- Run either `upscaling-video2x` or `upscaling-ffmpeg` for an upscaling miner. They expose different host ports but both implement `/upscale`.
- Run `compression` for a compression miner. It implements `/compress`.
- Processing containers publish localhost-only service ports and have `DISABLE_REMOTE_IO=true`; they only process local files from the shared work directory by default.
- The old PM2 service scripts for upscaling and compression are redundant with this container framework.
- For Video2X-specific preparation, use `miner/upscaling/SETUP.md`.
- For FFmpeg upscaling build/runtime notes, use `miner/upscaling/ffmpeg/SETUP.md`.