[Bug Report] Robotiq Gripper Issue in Custom UR5 Environment

[jmSNU]

Describe the bug

Dear NVIDIA Support Team,

I am a researcher in the field of robot learning and would like to seek your advice on an issue I am currently facing. I have created a .usd file featuring a UR5 robotic arm with a Robotiq 2F-140 gripper attached, and I am using this model to train an agent through reinforcement learning. I have also implemented a custom environment using DirectRLEnv.

However, I have encountered a problem where the gripper on the robotic arm does not behave as expected and appears to be malfunctioning. This issue seems to interfere with the training process and is also affecting the convergence of the IK solver.

I would greatly appreciate any guidance or suggestions you could provide to help resolve this issue.

Thank you for your time and assistance.

Steps to reproduce

1. Import URDF of Robotiq 2F-140 Gripper on IsaacSim
2. Move the gripper joint to control gripper

A detailed video demonstration related to the issue is available at the following link

Please see my URDF and usd of UR5 with Robotiq 2F-140 gripper.

-->

System Info

Describe the characteristic of your environment:

• Commit: [e.g. 8f3b9ca]
• Isaac Sim Version: 4.1.0
• OS: Ubuntu 22.04 LTS
• GPU: RTX 4090
• CUDA: 12.6
• GPU Driver: 560.35.03

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Criteria 1
• Criteria 2

[AlanSunHR]

When you reset your robot, did you reset all the joints of the robot? Especially for the robotiq gripper, since it has a kinematic loop, you may have to specify the initial joint positions of all joints (including those that are not actuated by actuators).

We have been using Isaac Sim to simulate our robots with kinematic loops for years, and we found this very important but as far as I know, this has not been mentioned in any official documentation. Please check the following forum discussions for more details:

Closed Chain Robot In Isaac Sim?

Support Parallel Robots?

[jmSNU]

When you reset your robot, did you reset all the joints of the robot? Especially for the robotiq gripper, since it has a kinematic loop, you may have to specify the initial joint positions of all joints (including those that are not actuated by actuators).

We have been using Isaac Sim to simulate our robots with kinematic loops for years, and we found this very important but as far as I know, this has not been mentioned in any official documentation. Please check the following forum discussions for more details:

Closed Chain Robot In Isaac Sim?

Support Parallel Robots?

Thank you for your response, @AlanSunHR!

I have carefully reviewed your feedback, and upon further examination, it seems that the issue is not related to how the joints are being reset. As indicated in the '_reset_idx' function within my 'DirectRLEnv' class and the 'ArticulationCfg' configuration for my robot, the joint reset process appears to be functioning as intended.

Please let me know if there is any aspect of this that I might be misunderstanding. I truly appreciate your insights and guidance!

def _reset_idx(self, env_ids: Sequence[int] | None):
        if env_ids is None:
            env_ids = self.robot._ALL_INDICES

        self.robot.reset(env_ids)
        self.camera.reset(env_ids)
        self.sensor.reset(env_ids)
        self.table.reset(env_ids)
        self.target.reset(env_ids)
        self.diff_ik_controller.reset(env_ids)
        self.diff_ik_controller_position.reset(env_ids)
        super()._reset_idx(env_ids)

        arm_joints = self.joint_ids[:6]
        joint_pos = self.robot.data.default_joint_pos[env_ids]
        joint_pos[:, arm_joints] += sample_uniform(
            -0.125,
            0.125,
            (len(env_ids), 6),
            self.device,
        )
        joint_vel = torch.zeros_like(joint_pos)
        self.robot.set_joint_position_target(joint_pos, env_ids=env_ids)
        self.robot.write_joint_state_to_sim(joint_pos, joint_vel, env_ids=env_ids)

UR5_CFG = ArticulationCfg(
    spawn=sim_utils.UsdFileCfg(
        usd_path="usd/ur5_with_gripper.usd",
        rigid_props=sim_utils.RigidBodyPropertiesCfg(
            disable_gravity=False,
            max_depenetration_velocity=5.0,
        ),
        collision_props=sim_utils.CollisionPropertiesCfg(
            collision_enabled = True,
        ),
        activate_contact_sensors=True,
    ),
    init_state=ArticulationCfg.InitialStateCfg(
        joint_pos={
            "shoulder_pan_joint": 0.0,
            "shoulder_lift_joint": -1.65,
            "elbow_joint": 1.5,
            "wrist_1_joint": -1.712,
            "wrist_2_joint": -1.712,
            "wrist_3_joint": 0.0,
            "left_inner_knuckle_joint" : 0.00033,
            "right_inner_knuckle_joint" : -0.00031,
            "left_inner_finger_joint" : 0.00051,
            "right_inner_finger_joint" : -0.00015,
            "right_outer_knuckle_joint" : -0.00034,
            "finger_joint" : 0.0
        },
        pos = (-0.2,0.0,1.0)
    ),
    actuators={
        "arm": ImplicitActuatorCfg(
            joint_names_expr=["shoulder_pan_joint", "shoulder_lift_joint", "elbow_joint", "wrist_.*_joint"],
            velocity_limit=10.0,
            effort_limit=87.0,
            stiffness=800.0,
            damping=40.0,
        ),
        "finger": ImplicitActuatorCfg(
            joint_names_expr=["finger_joint", "right_outer_knuckle_joint"],
            velocity_limit=0.2,
            effort_limit=200.0,
            stiffness=2000.0,
            damping=2000.0,
        ),
    },
)

[RandomOakForest]

Thank you for posting this. Is it possible for you to try Isaac Sim 4.2 and the latest Isaac Lab 1.4.0?

[jmSNU]

Thank you for posting this. Is it possible for you to try Isaac Sim 4.2 and the latest Isaac Lab 1.4.0?

Thank you for your response @RandomOakForest !

Unfortunately, the issue persists even after updating to Isaac Sim 4.2.0 and Isaac Lab 1.3.0. (I was wondering if Isaac Lab 1.4.0 is now available for use.)

I have shared the implementation of my DirectRLEnv in my repository for your reference. I wonder whether there is fatal or faulty implementation in my DirectRLEnv.

Additionally, I have recorded a video showing the rollout of an SAC-trained agent, with the gripper both activated and disabled. Based on the video, I believe this issue is not related to the implementation of DirectRLEnv.

Moreover, when substituting the robot with a Franka Panda Arm (a 7-DoF robot equipped with a gripper), both the training and rollout proceed without any problems. A colleague also observed that disabling the mimic joint functionality for the Robotiq gripper does not resolve the issue, which further suggests the problem may be specific to the Robotiq gripper in Isaac Sim.

In light of these observations, it seems likely that the issue stems from the Robotiq gripper implementation within Isaac Sim.

Thank you for your time and assistance in addressing this matter.

gripper activated

gripper_abled.mp4

gripper disabled

gripper_disabled.mp4

[StrainFlow]

This looks like a physics rigging issue to me. Have you tried simulating the manipulator in Isaac Sim on its own? I would check these things first:

1. If there are any articulation loops, one joint needs to be explicitly excluded from articulation in its settings
2. Make sure the masses and inertias of each rigid body are set to reasonable values
3. Make sure that rigid bodies collision boundary boxes do not overlap, especially if they are in separate articulations

How are you attaching the end effector to the manipulator?

[SantiDiazC]

Hi @jmSNU ,
additionally to what @StrainFlow Suggested, I recommend you to try attaching the gripper in code using the robot assembler extension, in this way you can control and debug the robot and the gripper independently.

[Kenny-Vilella]

Hello @jmSNU ,

After investigated the USD file you provided, it seems to me that there is no kinematic loop closure.
I also noticed the presence of 5 mimic joints, while one mimic joint (finger_joint) is enough.
These two observations make me wonder, are you trying to emulate the kinematic loop closure with mimic joints?
If this is the case, then I first recommend to remove the 4 extra mimic joints and close the kinematic loop.
The best option is probably to close it at the inner knuckle joint.

I suspect that there are other issues with the USD file, but it will be easier to investigate after fixing this.

[RandomOakForest]

Thank you for following up. I will move this to a discussion for follow up. If the issues persist in Isaac Lab 2.0 and Isaac Sim 4.5, we can move this back to our issues as a bug report. Thank you.