AffKernel: IIT-AFF affordance segmentation (R50vd, stride-2 + deep supervision)

Single-pass, NMS-free affordance segmentation. AffKernel couples RT-DETR object queries with a CondInst-style per-query dynamic-convolution kernel that decodes per-object affordance masks from one shared high-resolution affordance map. A detector tells a robot what an object is; AffKernel also tells it where to act on it, at a latency a control loop can afford.

Code: https://github.com/anh0001/affkernel
Reproduction guide: https://github.com/anh0001/affkernel/blob/main/docs/reproduction.md
Architecture: RT-DETR-R50vd + affordance branch with stride-2 readout and deep supervision
Training data: IIT-AFF (Nguyen et al., IROS 2017), 6,184 train images
Weights: last-epoch (epoch 72) EMA parameters, fp32, 174 MB

Results

Evaluated on the IIT-AFF test split (2,651 images) with the Margolin weighted F-measure.

Metric	Value
`F_beta^w` (beta^2 = 1), this checkpoint	0.8685
`F_beta^w` (beta^2 = 0.3), this checkpoint	0.8582
`F_beta^w` (beta^2 = 1), mean of 3 seeds	0.8675 ± 0.0009
Mask quality on detected instances	0.8933
Latency, 640x640, fp32, RTX 6000 Ada	16.3 ms median
Throughput	61.5 img/s

Against published baselines on the same benchmark: Mask R-CNN 0.844 at 45 ms, deterministic Swin-T 0.883 at 42 ms, Bayesian Swin-T deep ensemble 0.906 at roughly 1015 ms. AffKernel beats the Mask R-CNN baseline at about a third of its latency and trails the deterministic Swin-T by 1.55 points at 39% of its latency, with fully deterministic single-pass inference.

On seed selection. This is seed 42, which is both the primary anchor seed used throughout the paper and the highest scoring of the three seeds trained (seed 7: 0.8668, seed 123: 0.8673). IIT-AFF provides no validation split, so the epoch was fixed a priori (last epoch, EMA weights) rather than selected on the test set.

On the beta convention. Two conventions circulate in this literature. The AffordanceNet lineage reports beta^2 = 0.3; recent transformer baselines report beta^2 = 1. Both are given above so that comparisons can be made at a matched convention. Do not compare a beta^2 = 1 number against a beta^2 = 0.3 number.

Usage

git clone https://github.com/anh0001/affkernel.git
cd affkernel
pip install -r requirements.txt

pip install huggingface_hub
hf download anhrisn/affkernel-iit-aff \
  affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/

python tools/infer.py \
  -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \
  -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth \
  --input path/to/image.jpg \
  --output outputs/

--input takes a single image or a directory. --output is a directory; results are written as <stem>_aff.png. Add --device cpu to run without a GPU. Verified working on GPU and CPU, for both single-image and folder input.

Reproduce the reported metric (requires the IIT-AFF dataset; see docs/datasets.md):

python tools/decompose_fbw_gap.py \
  -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \
  -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth --beta2 1.0

Checkpoint format

A single file containing the EMA weights only:

{"ema": {"module": <OrderedDict of 760 tensors>, "updates": 111312}}

The evaluation path uses the EMA parameters, so the raw (non-EMA) weights and the optimizer state from training are not included. Do not repack this into a bare {"model": ...} dict: under an EMA-enabled config the solver would then evaluate a freshly initialised EMA module and score near zero.

The first run downloads ImageNet-pretrained ResNet-50vd backbone weights from the RT-DETR release artefacts, so it needs network access. For an offline deployment, set PResNet.pretrained: False in the config.

Deployment

All rows below use these same weights — fp16, CUDA graphs, BatchNorm folding and the TensorRT backbone are runtime configurations, not separate checkpoints.

RTX 6000 Ada, 640x640, batch 1, end to end (preprocess, forward, mask decode, device-to-host copy):

Inference path	End to end	FPS	Peak GPU	`F_beta^w` (0.3 / 1)
fp32, as released	35.7 ms	28.0	0.43 GiB	0.8582 / 0.8685
fp16 + CUDA graph + BN fold	16.0 ms	62.7	0.29 GiB	0.8577 / 0.8680

Jetson AGX Orin Developer Kit (64 GB), JetPack 6.2, MAXN:

Inference path	End to end	FPS	Peak GPU	`F_beta^w` (0.3 / 1)
fp32, as released	199.5 ms	5.0	1.29 GiB	0.8598 / 0.8680
`--half --cudagraph`	53.2 ms	18.8	0.21 GiB	0.8596 / 0.8678
`--trt-backbone`	43.4 ms	23.1	0.23 GiB	0.8591 / 0.8673

The fastest path costs 0.0007 F_beta^w at beta^2=1 — below the ±0.0009 seed-to-seed spread of the training run itself. Dropping the input to 512x512 is not worth it by comparison: it costs 0.007, ten times more, to save less time.

`backbone_fp16.plan` — prebuilt TensorRT engine

This repository also ships an fp16 TensorRT engine for the backbone only. The dynamic-kernel affordance head cannot be exported to ONNX, but the backbone is plain convolutions, exports cleanly, and is the largest single term of the forward pass: 17.7 ms → 5.3 ms. The encoder, decoder and affordance head stay in PyTorch, CUDA-graphed against the engine's output buffers.


Built on	Jetson AGX Orin (64 GB), JetPack 6.2 (L4T R36.4.7)
TensorRT / CUDA	10.3.0 / 12.6
Input	`images` — `(1, 3, 640, 640)`, fp16, fixed
Outputs	`feat0..feat3` — fp16
Size	46.4 MiB

Engine files are not portable. A .plan is compiled for one specific GPU, TensorRT version and input shape. On any other device, JetPack/TensorRT version or input size it will fail to deserialize — or load and behave incorrectly. It is published here as a convenience for setups identical to the table above. If yours differs in any respect, build your own — one command, a few minutes. That is the supported path.

# use the prebuilt engine (identical setups only)
hf download anhrisn/affkernel-iit-aff backbone_fp16.plan --local-dir weights/

# or build your own (recommended)
python tools/build_trt_backbone.py \
  -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \
  -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth \
  --out weights/backbone_fp16.plan

python tools/infer.py \
  -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \
  -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth \
  --input path/to/image.jpg --output outputs/ \
  --trt-backbone weights/backbone_fp16.plan --gpu-preprocess

Keep the checkpoint either way — the engine replaces the backbone at inference only, and training is unaffected.

Files

File	What it is
`affkernel_iit_r50vd_stride2_deepsup_seed42.pth`	The model. fp32 EMA weights, 174 MB.
`backbone_fp16.plan`	Optional deployment artifact. TensorRT fp16 backbone engine for AGX Orin / JetPack 6.2 / TensorRT 10.3 @ 640x640. Not portable; not usable on its own.

Classes

Objects (10): bowl, tvm, pan, hammer, knife, cup, drill, racket, spatula, bottle

Affordances (9): contain, cut, display, engine, grasp, hit, pound, support, w-grasp

Intended use and limitations

Intended for research on affordance perception and perception-guided grasping.

Trained on a closed vocabulary of 10 object and 9 affordance classes; it will not generalise to unseen categories.
IIT-AFF is a tabletop dataset. Performance under other viewpoints, lighting or clutter regimes is untested.
The residual error is dominated by missed detections, not by mask quality. On instances the detector does fire on, mask quality (0.893) already exceeds the deterministic Swin-T baseline's overall score.
The headline latency is fp32 on an RTX 6000 Ada. fp16, CUDA graphs and a TensorRT backbone are now characterised on both a workstation and a Jetson AGX Orin — see Deployment — and cost at most 0.0007 F_beta^w.
Not validated for safety-critical deployment. A predicted grasp region is a perception cue, not a guarantee of a safe grasp.

License and attribution

These weights are released under Apache-2.0, matching the licence of the RT-DETR components and the ImageNet-pretrained backbone they were initialised from. The AffKernel repository's own source contributions are MIT licensed; see THIRD_PARTY_LICENSES.md.

The IIT-AFF dataset is not redistributed here or in the code repository. It states no licence; its authors request citation of the original paper. Obtain it from https://sites.google.com/site/iitaffdataset/.

Citation

@article{risnumawan2026affkernel,
  title   = {AffKernel: Single-Pass Affordance Segmentation with Per-Query
             Dynamic Convolution for Real-Time Robotic Manipulation},
  author  = {Risnumawan, Anhar},
  journal = {IEEE Access},
  year    = {2026},
  note    = {Under review}
}

@inproceedings{nguyen2017object,
  title     = {Object-Based Affordances Detection with Convolutional Neural
               Networks and Dense Conditional Random Fields},
  author    = {Nguyen, Anh and Kanoulas, Dimitrios and Caldwell, Darwin G. and
               Tsagarakis, Nikos G.},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and
               Systems (IROS)},
  year      = {2017}
}

@inproceedings{lv2024detrs,
  title     = {DETRs Beat YOLOs on Real-time Object Detection},
  author    = {Lv, Wenyu and Zhao, Yian and Xu, Shangliang and Wei, Jinman and
               Wang, Guanzhong and Cui, Cheng and Du, Yuning and Dang, Qingqing
               and Liu, Yi},
  booktitle = {CVPR},
  year      = {2024}
}

Downloads last month: -; Downloads are not tracked for this model. How to track