IK Solver for Xlerobot in Isaac Sim

Introduction

Recently I was involved in a robotic hackathon, and we planned to use the Xlerobot (https://github.com/Vector-Wangel/XLeRobot) in the simulation to train some policies. This blog will introduce the IK (Inverse Kinematics) for controlling arms and discuss some challenges I faced and how to deal with them.

Full video introduction(Step by step build extension and standalone code):

Lula IK

Lula IK requires preparing the URDF and YAML config for the IK solver.

Attention 1: The YAML file can be generated by the Lula robot description editor, and it’s important to note that root_link is required for the c-space definition.

api_version: 1.0
cspace:
    - Rotation_2
    - Pitch_2
    - Elbow_2
    - Wrist_Pitch_2
    - Wrist_Roll_2
root_link: Base_2

Attention 2: Velocity in the URDF should all be over 0.

Challenges in Lula IK - Rare movement due to low tolerance in Lula IK

Lula IK is based on CCD (Cyclic Coordinate Descent) and BFGS (quasi-Newton optimization algorithm). The tolerance setting is too low to avoid any movement. When Lula IK fails, it does not output a guaranteed valid joint update, so if you don’t apply its “near-solution” anyway, the articulation state never changes—leaving the seed unchanged and the arm frozen.

My solution applies the near-solution from a failed IK attempt, keeping the seed close to the target so the solver can converge in the next step instead of restarting far away.

# Apply near-solution anyway so the seed can converge across steps
if self._apply_action_on_fail and action_r is not None:
    try:
        self._articulation.apply_action(action_r)
        if (self._step_count % self._debug_every_n) == 0:
            print(f"[XleRobot][Right] applying action despite IK failure")
    except Exception:
        if (self._step_count % self._debug_every_n) == 0:
            print("[XleRobot][Right] IK failed and action could not be applied")
if (self._step_count % self._debug_every_n) == 0:
    try:
        ee_pos_r_b, ee_rot_r_b = self._right_articulation_ik.compute_end_effector_pose()
        pos_err_b = float(np.linalg.norm(tgt_pos_r_b - np.array(ee_pos_r_b)))
        print(f"[XleRobot][Right] IK failed. world_pos_err={pos_err_r:.4f} base_pos_err={pos_err_b:.4f} tgt_b={tgt_pos_r_b.tolist()}")
    except Exception:
        print("[XleRobot][Right] IK failed target=", trg_pos)

Differential IK

Since my teammate created an edited USD file, I switched to differential IK without the URDF and YAML.

Challenge 1: This is a floating base robot, so the IK solver will have different PhysX data size from the fixed base. Without the offset, the DOF indices and Jacobian will gain the wrong columns. In the floating base, there will be a base_dof_offset added to the PhysX.

# Floating base PhysX DOFs is longer than the robot DOFs so we need to offset the joint indices
total_dofs = jacobians.shape[-1]
base_dof_offset = max(total_dofs - len(self.robot.data.joint_names), 0)
left_dof_indices = self._left_joint_ids + base_dof_offset if base_dof_offset else self._left_joint_ids
left_jacobian = jacobians[:, left_ee_jacobi_idx, :, left_dof_indices].clone()
left_joint_pos = self.robot.data.joint_pos[:, self._left_joint_ids]

Challenge 2: Differential IK is plain IK, which requires more strategies to optimize the movement and parameter adjustment.

Optimization 1: Adaptive damping avoids jitter at low lambda and latency at high values.

def _adaptive_lambda(self, J: torch.Tensor, pos_err_mag: torch.Tensor, *,
        base_lambda: float = 0.1,   # small, always-on damping
        lam_max: float = 0.8,       # cap (adjust after you see variation)
        k_manip: float = 0.10,       # manipulability sensitivity
        k_err: float = 0.20,         # error sensitivity
        sigma_ref: float = 0.20,     # "healthy" σ_min
        manip_slope: float = 0.08,   # softness of manipulability curve
        err_cap: float = 0.30,       # 50 cm → avoids instant saturation
        use_position_rows: bool = True,
        log_debug: bool = False
    ) -> torch.Tensor:
        """
        Adaptive DLS damping using a smooth manipulability term (via softplus) and a capped
        error term. Accepts J of shape [6,dof]/[B,6,dof] or [3,dof]/[B,3,dof].
        Returns scalar (unbatched) or [B] (batched).
        """
        # ---- normalize shapes ----
        batched = (J.dim() == 3)
        if not batched:
            J = J.unsqueeze(0)  # [1, r, dof]

        # ---- choose rows (prefer 3xDOF for 5-DOF arm) ----
        if use_position_rows and J.shape[1] >= 6:
            Jt = J[:, 0:3, :]
        else:
            Jt = J

        device, dtype = Jt.device, Jt.dtype

        # ---- singular values and σ_min ----
        svals = torch.linalg.svdvals(Jt)          # [B, min(r,dof)]
        sigma_min = svals.min(dim=1).values       # [B]

        # ---- constants as tensors (fixes the softplus float error) ----
        sigma_ref_t   = torch.as_tensor(sigma_ref,   device=device, dtype=dtype)
        manip_slope_t = torch.as_tensor(manip_slope, device=device, dtype=dtype)
        err_cap_t     = torch.as_tensor(err_cap,     device=device, dtype=dtype)
        base_lambda_t = torch.as_tensor(base_lambda, device=device, dtype=dtype)

        # ---- manipulability term (smooth, bounded, ~0 when σ>=sigma_ref) ----
        # x grows as sigma_min drops below sigma_ref; softplus keeps it gentle
        x = (sigma_ref_t - sigma_min) / manip_slope_t
        num   = torch.nn.functional.softplus(x)
        denom = torch.nn.functional.softplus(sigma_ref_t / manip_slope_t) + 1e-6
        manip_term = (num / denom).clamp(0.0, 1.0)    # [B]

        # ---- error term (0→1 over 0→err_cap, then capped) ----
        if pos_err_mag.dim() == 0:
            pos_err_mag = pos_err_mag.expand(Jt.shape[0])
        elif pos_err_mag.dim() > 1:
            pos_err_mag = pos_err_mag.reshape(-1)
        err_term = (pos_err_mag / err_cap_t).clamp(0.0, 1.0)  # [B]

        # ---- assemble λ and clamp ----
        lam = base_lambda_t + k_manip * manip_term + k_err * err_term
        lam = lam.clamp(max=lam_max)                           # [B]

        if log_debug:
            print(f"[AdaptiveLambdaDBG] "
                f"sigma_min_mean={sigma_min.mean().item():.4f} "
                f"manip_term_mean={manip_term.mean().item():.3f} "
                f"err_term_mean={err_term.mean().item():.3f} "
                f"lambda_mean={lam.mean().item():.3f}")

        return lam if batched else lam.squeeze(0)

Optimization 2: I also compared with the low pass filter, which did not yield good results, so I did not apply it. I applied error gating to avoid unwanted movement.

Optimization 3: When we move the base, the arm’s unwanted movement will affect the inertial momentum due to no sync with the target. So I applied sync strategies and the cool-down method to avoid the arm’s dragging back movement.

Controller integrated

I also integrate the game controller to control tha arms and base movement for simple demo.

Performance Comparison

IK Method	Avg. Time per Frame
Differential IK (2 arms)	~3.1 ms
Differential IK (2 arms, Adaptive Damping)	~5-6 ms
Lula IK (2 arms)	~12 ms

P.S.: More features are being added. GitHub repository coming soon!