Learning to leap

The problem

The real objective was simple, but the reward was sparse.

The goal was to jump to a commanded location and finish the episode standing stably inside the target area. That outcome is easy to state, but a reward given only at the end is difficult to discover. Most early rollouts never produce a useful jump or landing, so PPO has little signal that points toward successful behavior.

The challenge was therefore not only control. It was discovery: how to make successful behavior learnable without changing the definition of what success meant.

Technical stack

Isaac Lab simulation, PPO training, and experiment tooling.

I built the learning workflow in Python with Isaac Lab and Isaac Sim, using RL-Games for PPO training and Weights & Biases for sweep configuration and experiment logging. Training ran on a single RTX 3090 with 2,048 vectorized environments.

• Policy and control. A shared [256, 128, 64] actor-critic network mapped a 49-dimensional observation to 12 joint-position targets at 120 Hz.
• Physics and actuation. The physics, servo, and parallel-spring actuator models ran at 360 Hz. I added sensor noise and motion-capture latency to stress the policy against less-than-ideal observations.
• Evaluation. Command-bucketed metrics and episode diagnostics were used to evaluate 20,019 height-and-distance requests across the thesis range.

My approach

I decomposed the jump to create a denser learning signal.

I formulated the task as takeoff, flight, and landing phases. During takeoff, the landing command was continuously converted into the velocity vector required for a ballistic trajectory from the robot's current position. Tracking that vector supplied a dense learning signal while the outcome was still controllable.

During descent, the objective shifted to a level body attitude; after contact, it shifted again to low center-of-mass velocity and stability. A shared actor-critic network was trained with PPO in 2,048 parallel Isaac Lab environments. The policy produced desired positions for all 12 joints at 120 Hz, while analytical actuator models ran at the 360 Hz physics rate.

What made it hard

The policy kept finding technically valid ways to do the wrong thing.

• Phase timing became an exploit. Too much landing reward encouraged tiny jumps that reached the profitable phase quickly; too much flight reward encouraged unnecessarily vertical jumps that maximized airtime.
• Contact sensing was ambiguous. Isaac Lab reported aggregate contact forces without identifying the collision partner. I switched phase detection to center-of-mass acceleration so self-contact could not masquerade as ground contact.
• Landing found a local optimum. The robot learned to brake with a foot, then settle on its motor housings. It satisfied low-velocity and upright-body rewards, but it did not achieve a stable feet-only landing.

Results

Good command tracking across most of the training range, with a clear boundary.

The final policy was evaluated on 20,019 uniformly sampled height-and-distance commands. Most jumps stayed within a few centimeters of their targets and reached landing with a flat attitude. A representative 0.40 m-high, 0.30 m-long command overshot both targets by about 2 cm.

The weak region was also informative. Shallow, long jumps below roughly 0.35 m in height and above 0.45 m in distance were consistently harder and could fall short by up to 15 cm. Training took about 1 hour 19 minutes on a single RTX 3090. The policy was not deployed to the physical robot, so these are simulation results, not a sim-to-real claim.

What I took from it

Keep the objective simple. Solve discovery separately.

I split the task into takeoff, flight, and landing because the endpoint reward was too sparse for the policy to discover. Each phase received a heuristic objective. The center-of-mass velocity target during takeoff is the clearest example. This made the problem more gradual and granular, but it also encoded a strong inductive bias about how a jump should be performed.

The policy could optimize those proxies instead of the outcome I actually wanted. My reward formulation was doing two jobs at once: defining success and trying to make success discoverable. That opened the door to phase-timing exploits and other local optima.

I would now keep the objective much closer to the real task: finish the episode standing stably inside the target area. I would address sparse discovery separately, for example by initializing the robot in states where successful behavior is easier to find, then expanding the starting-state distribution. Obstacles could also make jumping emerge as a useful solution. The reward should describe the outcome, not prescribe the motion used to reach it.