Hugging Face's logo

silvanus0930
/
vidaio-subnet

Model card Files
xet
 Community
vidaio-subnet / docs /miner_setup.md
Arpan Tripathi
add inference miners' checklist in docs (#193)
d3c84b1 unverified
preview code
|
Raw History Blame Contribute Delete
8.03 kB

Running Miner

A high-performance decentralized video processing miner that forwards validator requests to containerized video services. The current miner framework supports:

  • Video2X upscaling through the upscaling-video2x Docker Compose profile.
  • FFmpeg upscaling through the upscaling-ffmpeg Docker Compose profile.
  • FFmpeg compression through the compression Docker Compose service.

neurons/miner.py no longer needs the old PM2-managed services/upscaling/server.py or services/compress/server.py workers. It forwards directly to the service container ports.

Machine Requirements

To achieve optimal results, we recommend the following setup:

  • Operating System: Ubuntu 24.04 LTS or higher
    Learn more about Ubuntu 24.04 LTS
  • GPU: NVIDIA RTX 4090 or higher
  • VRAM: At least 16 GB per GPU
  • Docker and Docker Compose
  • NVIDIA drivers and nvidia-container-toolkit
  • Python: Version 3.10 or higher for running the miner process

Install System Dependencies

Install common host dependencies:

sudo apt update
sudo apt install -y git curl npm python3 python3-venv python3-pip

Install Docker and Docker Compose using the official Docker instructions for your host, then install NVIDIA Container Toolkit so Compose services can access the GPU.

Verify Docker can see the GPU:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Install PM2

PM2 is optional for the processing containers, but useful for managing the miner process itself.

sudo npm install pm2 -g
pm2 update

Install Project Dependencies

1. Clone the Repository 

git clone https://github.com/vidaio-subnet/vidaio-subnet.git
cd vidaio-subnet

2. Set Up a Virtual Environment 

python3 -m venv venv
source venv/bin/activate

3. Install the Package 

pip install -e .

Configure Storage

The miner process downloads validator payload URLs, places local files in a shared work directory, forwards local paths to the processing containers, uploads processed outputs to S3-compatible storage, and returns presigned URLs. The processing containers run without egress.

Create a miner service environment file:

cp miner/.env.template miner/.env

Edit miner/.env with your storage credentials:

COMPOSE_PROJECT_NAME=miner
MINER_SHARED_DIR=/tmp/vidaio-miner-video-tmp
MINER_STORAGE_PROVIDER=backblaze
MINER_STORAGE_S3_ACCESS_KEY_ID=your-access-key
MINER_STORAGE_S3_SECRET_ACCESS_KEY=your-secret-key
MINER_STORAGE_S3_REGION=us-east-005
MINER_STORAGE_S3_BUCKET_NAME=your-bucket-name
MINER_STORAGE_S3_ENDPOINT_URL=https://s3.us-east-005.backblazeb2.com

For AWS S3, use the same variable names and leave MINER_STORAGE_S3_ENDPOINT_URL blank.

Create the shared work directory before starting the miner and containers:

mkdir -p /tmp/vidaio-miner-video-tmp
chmod 777 /tmp/vidaio-miner-video-tmp

Start Processing Services

Before starting a service, choose which inference task this miner will advertise to validators. In neurons/miner.py, use:

warrant_task = TaskType.UPSCALING

for upscaling, or:

warrant_task = TaskType.COMPRESSION

for compression. Starting the compression container does not select compression inference automatically. MINER_MODES must also include inference. Use the full miner_inference_checklist.md before going live.

Run Docker Compose commands from the miner directory:

cd miner

Option A: Video2X Upscaling Miner

Use this profile if you want Video2X-based upscaling. Complete the host/container setup first:

cat upscaling/SETUP.md

After running all the commands listed above, start the Video2X upscaling service:

docker compose --profile upscaling-video2x up -d upscaling-video2x

The service is exposed on http://localhost:8003.

Option B: FFmpeg Upscaling Miner

Use this profile if you want FFmpeg-based upscaling instead of Video2X:

docker compose --profile upscaling-ffmpeg up -d upscaling-ffmpeg

The FFmpeg upscaling service is exposed on http://localhost:8005.

Before starting neurons/miner.py, point the miner at that port:

export MINER_UPSCALING_SERVICE_URL=http://localhost:8005

Option C: Compression Miner

Start the compression service:

docker compose up -d compression

The compression service is exposed on http://localhost:8004.

Return to the repository root before launching the miner:

cd ..

Miner Forwarding Configuration

neurons/miner.py forwards requests to these service URLs:

export MINER_UPSCALING_SERVICE_URL=http://localhost:8003
export MINER_COMPRESSION_SERVICE_URL=http://localhost:8004
export MINER_SHARED_VOLUME_PATH=/tmp/vidaio-miner-video-tmp

Defaults:

  • MINER_UPSCALING_SERVICE_URL defaults to http://localhost:8003 for Video2X upscaling.
  • MINER_COMPRESSION_SERVICE_URL defaults to http://localhost:8004.
  • MINER_SHARED_VOLUME_PATH defaults to MINER_SHARED_DIR, then /tmp/vidaio-miner-video-tmp.
  • For FFmpeg upscaling, override MINER_UPSCALING_SERVICE_URL to http://localhost:8005.

Useful service checks:

curl -sf http://localhost:8003/health
curl -sf http://localhost:8004/health
curl -sf http://localhost:8005/health

Only the services you started need to pass health checks.

Run the Miner

Start the miner from the repository root:

python3 neurons/miner.py \
  --wallet.name [Your_Wallet_Name] \
  --wallet.hotkey [Your_Hotkey_Name] \
  --subtensor.network finney \
  --netuid 85 \
  --axon.port [port] \
  --logging.debug

Run the Miner with PM2

Export any MINER_*_SERVICE_URL overrides in the same shell before starting PM2. If you change those variables later, restart with --update-env.

pm2 start "python3 neurons/miner.py --wallet.name [Your_Wallet_Name] --wallet.hotkey [Your_Hotkey_Name] --subtensor.network finney --netuid 85 --axon.port [port] --logging.debug" --name video-miner

Parameters

  • --wallet.name: Replace [Your_Wallet_Name] with your wallet name.
  • --wallet.hotkey: Replace [Your_Hotkey_Name] with your hotkey name.
  • --subtensor.network: Specify the target network, usually finney.
  • --netuid: Specify the network UID, currently 85.
  • --axon.port: Replace [port] with the desired public axon port.
  • --logging.debug: Enables debug-level logging.

Managing Services

Processing containers:

cd miner
docker compose ps
docker compose logs -f upscaling-video2x
docker compose logs -f upscaling-ffmpeg
docker compose logs -f compression
docker compose down

Miner process:

pm2 logs video-miner
pm2 restart video-miner --update-env
pm2 stop video-miner

Additional Notes

  • For ordinary validator inference, complete miner_inference_checklist.md. Compression miners must advertise TaskType.COMPRESSION through warrant_task in neurons/miner.py.
  • For a new compression competition, complete miner_competition_checklist.md. The full competition contract is in competitions.md.
  • Run either upscaling-video2x or upscaling-ffmpeg for an upscaling miner. They expose different host ports but both implement /upscale.
  • Run compression for a compression miner. It implements /compress.
  • Processing containers publish localhost-only service ports and have DISABLE_REMOTE_IO=true; they only process local files from the shared work directory by default.
  • The old PM2 service scripts for upscaling and compression are redundant with this container framework.
  • For Video2X-specific preparation, use miner/upscaling/SETUP.md.
  • For FFmpeg upscaling build/runtime notes, use miner/upscaling/ffmpeg/SETUP.md.