Papers
arxiv:2611.18919

Correcting Persistent Failures in Point-Cloud Grasping Policies using Human Demonstrations in Virtual Reality

Published on Nov 13
· Submitted by
Chris von Csefalvay
on Nov 17
Authors:
,
,
,

Abstract

A human-in-the-loop failure-mining pipeline turns persistently unsolved grasps of a reinforcement-learning point-cloud policy into virtual-reality human demonstrations, improving hard-case success without reducing generalization to unseen objects.

Robots in homes, warehouses and farms must pick up objects they have never seen, usually without a 3D model or a category label. A grasping policy that works directly on the point cloud from a depth sensor needs only the geometry the robot observes, so it applies to any new object. Offline grasp generators such as Contact-GraspNet work this way, but they rank grasps by learned plausibility rather than by whether the grasp actually lifts the object. Reinforcement learning (RL) can optimize for that directly and make a point-cloud grasping policy much better on average, yet it leaves specific grasps persistently unsolved. We present a human-in-the-loop failure loop that targets those grasps: situations an RL policy keeps failing are mined automatically, sent to a virtual-reality application where a person picks the object up with their own hands, and returned to training as two grasp points. The person contributes only where to grasp; the robot chooses its own approach. In simulation, RL raises success on unseen objects from 55% with Contact-GraspNet to 83%. Adding 58 human demonstrations nearly doubles success on the hard cases RL kept failing, from 24% to 45%, and recovers performance that RL alone loses on similar cases that were never demonstrated, without reducing success on unseen objects.

Community

We found that RL makes a point-cloud grasping policy much better on average, but can leave particular object–pose combinations persistently unsolved. So we built a failure loop: mine those cases, show them to a person in VR, then use their two fingertip contacts to retrain the policy. The robot chooses its own approach, and each demonstration is checked with its gripper before training.

In simulation, RL improved lift success on unseen flat-lying objects from 55% with Contact-GraspNet to 83%. Adding 58 verified demonstrations collected in 32 minutes raised success on demonstrated hard cases from 24% to 45%, without a measurable loss on held-out objects. We also saw recovery on undemonstrated cases involving similar objects. The next step is to separate the effects of human supervision and targeted replay, then test on hardware.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.18919
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.18919 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.18919 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.18919 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.