YouDuck.ai
≡

The sim2real recipe

Last updated

A policy that walks beautifully in simulation and falls over the moment it reaches hardware is the standard failure mode for this kind of project. microduck_rl encodes the whole sim2real recipe into the repository itself rather than leaving it in one person’s head.

This page walks through what it gets right — understanding these is worth more than copying the parameters.

1. The actuator cannot be an ideal model

The Dynamixel XL330 is modelled with the M6 model from BAM, which covers:

  • the voltage control law
  • back-EMF
  • Coulomb, Stribeck and load-dependent friction

This layer is the foundation of sim2real. Treat a servo as an ideal torque source and the gait learned in simulation will almost certainly fail on hardware — the friction and voltage behaviour of a real servo eat exactly the response the policy was relying on.

2. What gets randomised is physics, not noise

Domain randomisation is applied per environment to four quantities:

Randomised Why this one
Battery voltage A full duck and a nearly empty one do not have the same torque available
Voltage sag under load Voltage collapses when several servos pull at once, and that directly shapes the gait
Command delay The control chain of a real system has latency, and the policy has to be robust to it
Friction magnitude Unit-to-unit variation and wear

The implementation is FrictionDRBamActuator under src/mjlab_microduck/actuator/.

Notice that all four are physical parameters, not Gaussian noise added to observations. These are two completely different kinds of randomisation — the first teaches the policy to cope with the distribution of parameters it will meet in the world, the second only makes it insensitive to observation noise.

3. The backlash has to be modelled on the correct side

This is the easiest part to get wrong, and the implementation detail in microduck_rl most worth learning from.

Every main task has a Backlash variant, with ±1° (2° total) of gear play in series with each of the 14 servo joints.

Everything turns on where the encoder sits:

On the real robot, the encoder is on the output side of the play.

So the simulation has to match that structure:

  1. Each servo gets a non-actuated passive_<joint>_backlash hinge;
  2. The firmware PD emulation (BacklashEncoderBamActuator) and the joint_pos / joint_vel observations all read through the backlash — that is, qpos[servo] + qpos[backlash].

Model the encoder on the input side instead and the policy believes it knows the joint position exactly, while on hardware what it reads is the value after the play. The discrepancy is invisible while things are static and arrives all at once on reversal.

There is also an engineering benefit: neither the observation nor the action dimension changes, so the ONNX export and the onboard runtime need no modification at all. A policy trained with backlash and one trained without are directly interchangeable.

4. Several policies share one observation contract

On deployment the runtime hot-swaps between the walk / recovery / trick policies, backed by a shared 61-dimensional observation contract.

The point of that design: any policy can take the robot over at any moment. The fall-recovery policy does not wait for the walking policy to hand over — it can step in whenever it needs to.

Rehearse this in simulation:

uv run scripts/infer_policy.py --walking walk.onnx --standing stand.onnx \
    --sitstand sitstand.onnx --roulade roulade.onnx --new-cmd-obs

This script rehearses the very switching logic of the onboard runtime. It supports --debug, --save-csv and --record, which exist specifically for sim2real comparison — run the same command sequence in simulation and on the robot, then compare the curves.

What to watch once it is on the robot

Once a policy is on hardware, robotctl monitor is the main window into it. It shows what the client asked for alongside what was actually executed, and names the reason whenever the two differ.

The most common situation during sim2real is a safety limit clamping the policy’s output — the stick is pushed all the way and the robot does not move. The reason is stated explicitly, for example:

deadman — no intent arrived recently, velocity zeroed

The bottom border shows which .onnx is currently loaded. That matters when you are swapping policies to debug: walk is a mode name, and two versions with completely different gaits both report as walk.

Export the full state for offline analysis:

robotctl monitor --json --hz 50 > run.jsonl

Together with the simulation data from infer_policy.py --save-csv, that gives you a joint-by-joint sim2real comparison.