Zainulabedin1/cornell-grasp-detection

GraspCNN — Cornell Grasp Detection

A small convolutional neural network trained from scratch to predict a grasp rectangle (x, y, w, h, angle) for a parallel-jaw gripper from a single RGB image, evaluated with the standard Cornell rectangle metric (correct iff IoU > 0.25 with a human-labelled grasp and angle error < 30°).

Results (image-wise split, held-out test images)

Model	Rectangle-metric accuracy	Mean IoU	Mean angle error
GraspCNN (this model)	36.2%	0.23	7.9°
Baseline: dataset-mean rectangle	16.0%	0.20	31.5°
Baseline: image-centre rectangle	11.7%	0.10	30.0°

Test set: 94 images (image-wise 80/10/10 split, seed 7).

Usage

from model import load_model, predict_grasp

model = load_model("pytorch_model.bin")          # CPU or CUDA automatically
x, y, w, h, angle_deg = predict_grasp(model, "photo.jpg")
print(f"grasp at ({x:.0f}, {y:.0f}), size {w:.0f}x{h:.0f}, angle {angle_deg:.1f}°")

predict_grasp accepts a PIL image, a numpy array (HWC), or a filepath, and returns the rectangle in the input image's pixel coordinates. The network runs internally on a 128×96 resize.

Model details

• Architecture: 4 conv blocks (24→48→96→96 channels) + FC head, ~358,758 parameters. Input 128×96 RGB; output (xc, yc, w, h, sin2θ, cos2θ) with positions/sizes normalised to [0, 1] and the angle encoded continuously across the ±90° wrap.
• Training: plain MSE on the 6-vector, AdamW, cosine schedule, early stopping on validation loss. One random positive label per image per epoch + horizontal/vertical flip augmentation (CPU training, 60 min).

Limitations (read before deploying)

• Single rectangle output — the dataset labels several valid grasps per image; the model commits to one, which caps the rectangle metric.
• Lab-tabletop domain — 240 household objects photographed on a table from a single Kinect viewpoint; not cluttered bins, not novel lighting.
• RGB only — depth is available in the dataset and usually helps; omitted here to keep the model tiny and CPU-trainable.
• Image-wise split — the same object can appear in train and test in different poses; an object-wise split is the stricter generalisation test (see the project repo for those numbers).

Dataset

Cornell Grasping Dataset — Lenz, Lee & Saxena, "Deep Learning for Detecting Robotic Grasps", IJRR 2015; grasp labels from Jiang, Moseson & Saxena, "Efficient Grasping from RGBD Images", ICRA 2011. Official page http://pr.cs.cornell.edu/grasping/rect_data/data.php (currently offline; data fetched from the Wayback Machine archive of that page). Intended for research use; please cite the authors above.