The robot walked. Then it refused to turn on the spot. Finding out why led me to a second policy, Unitree’s own from unitree_rl_mjlab, and a small experiment.

All the code is in the exercises/ folder.

Setting up the G1

The G1 model file describes the robot’s body. It has no floor, no motors and no starting pose. I added these with MuJoCo’s MjSpec, which lets you edit a model in Python before you compile it:

spec = mujoco.MjSpec.from_file("robot/g1.xml")
apply_collision_settings(spec)            # which parts touch, how the feet grip
add_floor(spec)
actuator_names = add_position_actuators(spec)   # 29 motors, gains from motor specs
add_init_keyframe(spec, actuator_names)         # knees-bent starting pose, pelvis at 0.76 m
apply_solver_settings(spec)               # 5 ms timestep = 200 physics steps per second
model = spec.compile()

The full function is build_g1_model in g1_utils.py.

One surprise: with the policy’s gains, the robot cannot stand on its own. It slowly sinks and falls over. The gains are soft on purpose. The trained policy is what keeps the robot balanced. Without it, a humanoid is just an inverted pendulum.

Three parts of the setup took me some time to understand: which parts touch, how the feet grip, and where the motors’ “armature” comes from.

Which parts touch: collision bits

A robot model has two kinds of shapes:

• Visual meshes: detailed 3D models that make the robot look right.
• Collision geoms: simple capsules and spheres that stand in for the body when things touch.

Checking contacts with detailed meshes would be slow, so the G1 uses 33 simple shapes for collisions: 14 small capsules under the feet and 19 others on the legs, pelvis and arms.

MuJoCo uses two numbers per shape to decide what can touch what: contype and conaffinity. Two shapes A and B can collide if:

The & compares the two numbers bit by bit. Think of contype as “the groups I am in” and conaffinity as “the groups I collide with”. If either shape accepts the other, they collide.

The capsules are also 1 / 1 with each other, so the robot’s limbs can hit each other too. MuJoCo skips each body’s direct parent automatically, and a short <exclude> list in the model turns off a few more pairs.

The bits can do more than on and off. The training code has a “no self-collision” option that sets the robot’s capsules to contype = 0, conaffinity = 1:

• Robot and floor: floor contype 1 & capsule conaffinity 1 = 1. They collide.
• Robot and robot: 0 & 1 = 0 both ways. They don’t.

With one pair of numbers, the feet stay on the ground while the legs can pass through each other.

Friction: how the feet grip

Walking only works if the feet don’t slide. When a foot presses on the floor, the force has two parts: the normal force straight up, and the friction force along the floor. Friction can be anything up to a limit, and the limit grows with how hard the foot presses:

The G1’s feet use \(\mu = 0.6\), about rubber on a hard floor. If you draw the total force as one arrow, this rule says the arrow must stay inside a cone around the vertical, \(\arctan(0.6) \approx 31°\) wide on each side. Inside the cone the foot sticks. Outside, it slips.

For the 33 kg robot standing still, that is about 196 N of sideways grip. It also limits how fast the robot can speed up: at most \(\mu g \approx 5.9\) m/s². No controller can push harder than the floor allows.

condim sets how many directions a contact can push in:

The feet get condim = 3, enough to walk. The other capsules get condim = 1: if two knees bump, they just slide apart, and that is also faster to compute.

Whose friction wins. A foot touching the floor is two shapes, each with its own friction. By default MuJoCo uses the bigger one, so the floor’s 1.0 would beat the foot’s 0.6. So the feet get a higher priority (1, the floor has 0), and the higher-priority shape’s settings are used. Without this, the robot would walk on a grippier floor than the one it was trained on.

Pyramid or cone. A perfectly round cone is slow to solve, so by default MuJoCo uses a pyramid instead. With condim = 3 the pyramid has 4 sides. Seen from above (panel b), the cone is a circle and the pyramid is a diamond inside it. Along the axes they match, but on the diagonals the pyramid gives only 71 % of the real grip. MuJoCo also offers an exact cone, cone="elliptic", which is slower. The training framework (mjlab) uses the pyramid, and so do I: the policy learned to walk with it.

All of this is one small function, apply_collision_settings in g1_utils.py (slightly shortened here):

for geom in spec.geoms:
    if re.fullmatch(FOOT_PATTERN, geom.name):        # the 14 foot capsules
        geom.condim, geom.priority = 3, 1
        geom.friction[0] = 0.6
        geom.contype = geom.conaffinity = 1
    elif re.fullmatch(".*_collision", geom.name):    # other collision capsules
        geom.condim, geom.priority = 1, 0
        geom.contype = geom.conaffinity = 1
    else:                                            # visual meshes: never collide
        geom.contype = geom.conaffinity = 0

Motors and gearboxes

In Week 1, armature was just a number I added to a pendulum joint. On the G1 it comes from real hardware. Understanding it took me longer than anything else in the setup.

An electric motor spins fast but is weak. A gearbox trades speed for strength. With a gear ratio \(N\), the joint turns \(N\) times slower than the motor and gets \(N\) times the torque. The G1 uses planetary gearboxes, each with two stages.

One stage. A planetary stage has a sun gear in the middle (the input), planet gears around it, and a fixed outer ring. The planets sit on a carrier, which is the output. When the sun turns, the planets roll along the ring and drag the carrier round, more slowly. How much slower depends only on the teeth:

That explains the odd numbers in the motor specs, like \(1 + 48/22 = 3.18\).

Two stages. In the animation, the motor is the red sun of stage 1. Stage 1’s blue carrier turns 4.5 times slower and drives stage 2’s blue sun. Stage 2’s green carrier is the joint, another 3.18 times slower. The ratios multiply: \(4.5 \times 3.18 = 14.3\). That’s in the motor’s name: 7520_14.

Why the motor feels heavy at the joint. A spinning part stores energy \(\tfrac{1}{2} I \omega^2\). Say a part spins \(k\) times faster than the joint:

So from the joint’s point of view, that part weighs like \(I k^2\). The motor spins \(4.5 \times 3.18\) times faster than the joint, the middle stage 3.18 times, and the output just once. Adding all three gives the armature:

For the 7520_14 (the colored rows in the animation): 0.01003 + 0.000099 + 0.000053 = 0.0102 kg·m². The motor rotor is the lightest part, but it makes up 98 % of the total, because its factor is the biggest: \(14.3^2 \approx 205\).

In code it’s two short functions, in g1_utils.py:

def reflected_inertia(rotor_inertias, gears):
    r1, r2, r3 = rotor_inertias
    g1, g2, g3 = gears
    return r1 * (g2 * g3)**2 + r2 * g3**2 + r3

def motor_gains(armature, natural_freq_hz=10.0, damping_ratio=2.0):
    wn = 2 * np.pi * natural_freq_hz          # Hz -> rad/s
    return armature * wn**2, 2 * damping_ratio * armature * wn   # KP, KD

The G1 has four motor types:

The ankles and the waist pitch and roll each use two 5020 motors, so their numbers are doubled.

From armature to gains. The training setup doesn’t tune each joint by hand. It picks the behavior it wants, a 10 Hz natural frequency and a damping ratio of 2, and uses last week’s formulas to get the gains:

For the hip pitch: \(K_P = 40.2\), \(K_D = 2.56\). The policy file stores the gains it was trained with, and all 29 of mine matched.

What the policy sees

The policy never sees the simulator. Every 20 ms it gets 99 numbers, and that is all it knows about the robot. The order comes from the policy file itself:

In code, from observation_utils.py:

obs = np.concatenate([
    read_sensor(model, data, "imu_lin_vel"),   # velocity of the IMU, pelvis frame
    read_sensor(model, data, "imu_ang_vel"),   # angular velocity, pelvis frame
    projected_gravity(model, data),            # "down" in the pelvis frame
    data.qpos[7:] - default_pos,               # joint angles relative to the starting pose
    data.qvel[6:],                             # joint velocities
    last_action,                               # previous policy output
    command,                                   # [vx, vy, wz]
]).astype(np.float32)

A number in the wrong place, or in the wrong frame, does not cause an error. The policy just reacts to a robot that doesn’t exist. So I checked each part against something I could work out by hand.

Which frame is which

The robot’s base is a free joint, and its velocity qvel[0:6] mixes two frames (panel a):

• qvel[0:3], the linear velocity, uses world axes.
• qvel[3:6], the angular velocity, uses the pelvis’s own axes.

The policy was trained on sensor readings, and both IMU sensors use the pelvis frame: “forward” is always the pelvis’s x axis, whichever way the robot faces. To check I understood them, I rebuilt both sensor readings from qvel:

• The gyro equals qvel[3:6]. It’s already in the pelvis frame.
• The velocimeter needs two steps. First turn the world velocity into the pelvis frame (\(R^\top v\)). Then add \(\omega \times r\), because the IMU sits 9.5 cm from the pelvis centre, and on a turning body, points further out move faster.

Both matched exactly. Leaving out \(\omega \times r\) gave an error of 0.03 m/s, which matters when the robot is asked to walk at 0.5 m/s.

Projected gravity (panel b) is “down” from the robot’s point of view, \(R^\top [0, 0, -1]\). Standing upright it’s \([0, 0, -1]\). Leaning 30° forward it becomes \([0.5, 0, -0.87]\): down now points partly forward. That’s how the policy knows which way it is falling.

def projected_gravity(model, data):
    R = data.xmat[model.body("pelvis").id].reshape(3, 3)   # pelvis frame -> world
    return R.T @ np.array([0.0, 0.0, -1.0])                 # world -> pelvis frame

A 5 ms lag

One detail surprised me. mj_step computes the sensors first and then moves the simulation forward. So right after a step, the joint angles are new, but the IMU readings are still from before the step, 5 ms old. One call to mj_forward updates them.

During normal walking the difference is tiny, 0.004 rad/s. But when the robot hits the floor, its rotation changes sharply, and the old gyro reading was off by up to 11 rad/s. Diego’s code uses the old values. The training framework updates them first (at least in the version I could read). On flat ground it barely changes the walk: 3.00 m against 3.08 m in 6 s.

Inside the network

The trained policy is one file, walk.onnx. ONNX is a standard format for trained neural networks. It stores the steps of the computation and the learned weights, so you don’t need the framework that trained it. The policy was trained in PyTorch, but to run it I only need ONNX Runtime. Diego’s C++ ROS node uses the same file.

I wrote a small reader to see what’s inside:

• A normalizer comes first. The network learned on scaled inputs, \((x - \text{mean}) / \text{std}\), and the mean and std from training are inside the file. So I give it the raw 99 numbers. Scaling them myself would scale them twice.
• Then a simple MLP: \(99 \to 512 \to 256 \to 128 \to 29\). Each layer is a matrix multiply plus a bias (ONNX calls it Gemm), followed by ELU, which is like ReLU but smooth below zero.
• About 219,000 weights. At 4 bytes each that’s 877 KB, and the file is 881 KB, so it’s almost all weights.
• Only the actor. Training also used a second network, the critic, which judged how good each moment was. It’s not in the file, because running the policy doesn’t need it.

The file also stores metadata: joint names, starting pose, gains, action scales and the observation order. Loading the policy and reading it takes a few lines, from walk.py:

session = ort.InferenceSession("walk.onnx", providers=["CPUExecutionProvider"])
meta = session.get_modelmeta().custom_metadata_map        # name -> comma-separated values
stiffness = [float(v) for v in meta["joint_stiffness"].split(",")]

I compared every entry with my model. All of them matched. That was the strongest check that my simulated robot is the one the policy was trained on.

It walks

The policy runs every 20 ms, and the physics runs 4 times in between. Here is a simplified version of the controller in policy_utils.py, which is called before every physics step:

def control(model, data):
    if step % 4 == 0:                                      # every 4th physics step = 50 Hz
        obs = make_observation(model, data, default_pos, last_action, command)
        action = session.run(None, {"obs": obs[None, :]})[0][0]     # 99 numbers in, 29 out
        target = default_pos + action_scale * action                 # joint angles, joint order
        data.ctrl[:] = target[joint_index_of_actuator]               # reorder for the actuators
        last_action = action                                         # fed back next time
    step += 1

Two details mattered:

• The policy gives joint angles in joint order, but MuJoCo’s actuators are in a different order (grouped by motor type). So the targets must be reordered.
• My first version was off by one. MuJoCo counts the robot’s free-floating base as joint 0, so a joint’s id is one more than its place in the policy’s list. A simple test caught it: a zero action must give exactly the starting pose.

Asked to walk at 0.6 m/s, the robot walked 3.0 m in 6 s without falling, including the time it takes to get going.

The dead zone

Testing every command

Walking forward is just one test. I tried every kind of command: faster, slower, backward, sideways, turning. For each one I measured the steady speed, over the last 3 s of an 8 s run, once the robot had settled into its walk (validate.py):

Normal walking works well. But below about 0.3 m/s the robot just stands, and it won’t turn on the spot at all, only while walking. A dead zone.

Ruling out my own mistakes

My first thought was that I had done something wrong. So I checked the differences I knew about:

• Sensor timing: old or updated IMU readings (the 5 ms lag) give the same dead zone.
• Integrator: Euler (as in Diego’s code) or implicitfast (the training default) give the same dead zone.
• Command code: the training framework doesn’t cut off small commands.

The dead zone stayed every time. Diego’s blog doesn’t mention it. It seemed to come from the policy itself.

A second policy

Unitree publishes its own walking policy for the G1, trained with the same framework. It takes 98 numbers instead of 99. It leaves out the linear velocity, which a real robot can’t measure directly, and it adds two numbers that Diego’s policy doesn’t have: a gait phase. I gave it its own robot model and starting pose, checked its file the same way, and ran the same tests (unitree.py):

No dead zone. Unitree’s policy walks slowly, steps sideways and turns on the spot, almost exactly as asked.

The gait clock

The gait phase is a clock that goes round once per step cycle, every 0.6 s. A single number from 0 to 1 would jump back to 0 at the end of every cycle, so instead it’s given as a point on a circle:

When the command is almost zero, the clock is turned off and reads \([0, 0]\). That’s the policy’s signal to stand still. In code, from unitree_utils.py:

def gait_phase(t, command, period=0.6):
    phi = (t % period) / period
    phase = np.array([np.sin(2 * np.pi * phi), np.cos(2 * np.pi * phi)])
    if np.linalg.norm(command) < 0.1:      # standing still: clock off
        phase = np.zeros(2)
    return phase

This gave me an idea. A turn-on-the-spot command is \([0, 0, 0.8]\). Its size is 0.8, so Unitree’s clock keeps ticking, which tells the policy to keep stepping. Diego’s policy has no clock. It has to decide from the command alone whether to step, and maybe it only learned to step when the command is large.

The experiment

To test this idea, I changed only the clock and kept everything else the same: same robot, same policy, same commands. I tried three versions of Unitree’s policy:

def with_phase(value):
    def obs_fn(model, data, default_pos, last_action, command):
        obs = make_unitree_observation(model, data, default_pos, last_action, command)
        obs[9:11] = value                  # replace the gait clock
        return obs
    return obs_fn

# normal clock | clock off: with_phase([0, 0]) | clock frozen: with_phase([0, 1])

With its clock turned off, Unitree’s policy behaves like Diego’s: it stands still for slow commands, won’t turn on the spot, and still walks at 0.6 m/s. With the clock frozen, it loses its rhythm, goes too fast, and falls when it tries to turn. So the clock has to tick, not just be there.

What I can and can’t say

For Unitree’s policy the answer is clear: the ticking gait clock is what makes it step for small commands. For bigger commands, the command alone is enough.

For Diego’s policy the evidence is strong but indirect. It has no clock, and it behaves like Unitree’s policy without its clock. That’s my best explanation for its dead zone. But it isn’t proof: the two policies were trained with different settings, and Diego’s training settings aren’t available.

Next

• ROS 2 and Gazebo. Diego’s project runs the policy in Gazebo, a different simulator, through a ROS 2 node. I’ll set that up (in Docker on my Mac) and see how much of the behavior carries over.
• Which policy is better? “It walks” isn’t a measurement. I want proper numbers to compare the two policies: how well they follow commands, how much they drift, how much effort they use, and how they handle pushes and different floors.

I’m working on a research project with JdeRobot around humanoid locomotion: making it easier to give humanoid robots useful movement skills. Before building anything on top of a walking robot, I want to understand how a learned walking policy actually works. A neural network reads some sensor values, outputs some numbers, and a robot with almost 30 joints walks. What happens in between? The physics simulation, the robot state, the observations, the actions and the control.

I’m learning this by rebuilding an existing project piece by piece. My reference is Diego López’s thesis project from Robotics Lab URJC. It runs a walking policy for the Unitree G1 humanoid, trained with mjlab on the MuJoCo physics simulator. My first goal is to make the G1 walk in plain MuJoCo with that trained policy, and to understand every part of it.

This week was about MuJoCo itself. All the code is in the exercises/ folder of the project repository.

Opening up the G1

I started by loading the robot and looking at what’s inside. MuJoCo splits a simulation into two objects:

• MjModel holds everything that stays fixed: bodies, joints, masses, sensors, the timestep.
• MjData holds everything that changes: joint positions and velocities, sensor readings, time.

model = mujoco.MjModel.from_xml_path("robot/g1.xml")
data = mujoco.MjData(model)

joint = model.joint(1)                          # look up joints by index or name
print(joint.name, joint.qposadr, joint.dofadr)  # left_hip_pitch_joint [7] [6]

The full inspection script is inspect_model.py.

For the G1:

A few of these numbers needed explaining. nq and nv differ because the robot’s base is a free joint that can move and rotate in any direction. Its position takes 7 numbers (x, y, z and a 4-number quaternion), but its velocity only needs 6 (3 linear, 3 angular). So the first hinge joint lives at qpos[7] but at qvel[6], and the reference code has constants for exactly that offset. The quaternion is stored as (w, x, y, z), while SciPy expects (x, y, z, w); mixing them up gives no error, only wrong rotations. And nu = 0 means the file has no motors at all. Without actuators, the robot is a ragdoll.

Free fall and the integrator

The model has no floor, so stepping the simulation makes the robot fall. That makes a nice first test. After time \(t\), the height should be:

The test itself is only a few lines:

mujoco.mj_resetData(model, data)
z0 = data.qpos[2]                      # pelvis height
while data.time < 0.5:
    mujoco.mj_step(model, data)

t = data.time
z_expected = z0 + 0.5 * model.opt.gravity[2] * t**2   # gravity[2] is -9.81
error = abs(data.qpos[2] - z_expected)

After 0.5 s the formula gives −0.4333 m, and MuJoCo gave −0.4382 m. The simulated robot fell 4.9 mm further than physics says it should.

The difference comes from how MuJoCo moves through time. Its default integrator, semi-implicit Euler, first updates the velocity and then moves the position using that new velocity:

Since the robot is speeding up, the end-of-step velocity is the fastest one within that step, so every step moves slightly too far. Summing over all the steps gives the error exactly:

The error is proportional to \(\Delta t\), and halving the timestep did halve it, to 2.45 mm. It’s a small number, but it showed me that the simulator is an approximation with rules I can predict.

One joint at a time: the pendulum

Before giving 29 joints their motors, I wanted to understand a single joint completely. So I wrote a pendulum in MuJoCo’s XML format: one hinge joint, a massless rod of length \(L = 0.5\) m, and a 1 kg ball at the end. The timestep is 0.005 s, the same one the G1 policy uses.

<mujoco>
  <option timestep="0.005"/>
  <worldbody>
    <body name="pole">
      <joint name="hinge" type="hinge" axis="0 1 0"/>
      <geom type="capsule" fromto="0 0 0 0 0 -0.5" size="0.01" density="0"/>  <!-- massless rod -->
      <geom type="sphere" pos="0 0 -0.5" size="0.05" mass="1"/>              <!-- 1 kg ball -->
    </body>
  </worldbody>
  <actuator> <!-- motors go here --> </actuator>
</mujoco>

The model is generated by a small function in pendulum.py, so the later experiments can swap in different actuators.

How long is one swing?

For small swings, a pendulum’s period is:

Starting from 0.1 rad, I measured the period by timing when the pendulum crossed zero:

zero_crossings = []
while data.time < 5.0:
    q_prev = data.qpos[0]
    mujoco.mj_step(model, data)
    if q_prev > 0 and data.qpos[0] < 0:     # crossed zero going down: one per swing
        zero_crossings.append(data.time)

measured_period = np.diff(zero_crossings).mean()

It came out at 1.4217 s, about 0.22% longer. That gap can be predicted almost exactly: the ball’s size adds 0.20%, the finite swing adds 0.06%, and timing on a 5 ms grid adds up to ±0.12%. Together that’s 0.14% to 0.38%, and the measurement falls inside.

A PD controller that stops short

Next I gave the pendulum a motor and wrote my own PD controller:

where \(q^*\) is the target angle. In code, the controller runs before every simulation step and writes the torque into data.ctrl:

def make_pd(kp, kd, target):
    def control(model, data):
        q, q_dot = data.qpos[0], data.qvel[0]
        data.ctrl[0] = kp * (target - q) - kd * q_dot   # torque (N·m) for a <motor>
    return control

The easiest way to picture it is as a spring and a damper:

The spring (\(K_P\)) pulls toward the target, harder the further away the pendulum is. The damper (\(K_D\)) resists motion and stops the spring from bouncing forever.

With \(K_P = 20\), \(K_D = 2\) and a target of 0.5 rad, I expected the pendulum to settle at 0.5. It settled at 0.404 rad.

The reason becomes clear once you put the controller into Newton’s second law for rotation:

At rest, both the acceleration and the velocity are zero, which leaves:

The left side is the spring’s torque and the right side is gravity’s. If the error \(q^* - q\) were zero, the spring would push with zero torque while gravity still pulls down, and the pendulum couldn’t stay there. For the pendulum to hold still, there has to be an error, just enough stretch in the spring to carry the weight. Checking the numbers: \(20 \times 0.096 = 1.93\) N·m, which is gravity’s torque at 0.404 rad. This gap is called steady-state error.

Three ways to close the gap

Rearranging the balance shows the size of the error:

That points to three fixes.

1. A stiffer spring. The error shrinks as \(1/K_P\). Going from \(K_P = 20\) to 80 to 320 reduced it from 0.096 to 0.028 to 0.007 rad. It never reaches zero, though, and as the next sections show, higher gains bring more overshoot, more torque, and eventually instability.

2. An integral term. Add \(K_I \int (q^* - q)\,dt\) to the controller. As long as any error remains, the integral keeps growing and pushes harder, so the only place the pendulum can rest is zero error. The cost is a slower response and a tendency to overshoot when the motor saturates (called windup). I haven’t tried this one yet.

3. Gravity compensation. If we know the gravity torque, we can simply add it to the motor command:

In code, that’s one extra term:

data.ctrl[0] = kp * (target - q) - kd * q_dot + m * g * L * np.sin(q)

Put that into Newton’s law and gravity cancels out completely:

Now the rest condition is simply \(K_P\,(q^* - q) = 0\), so the error must be zero.

In simulation it settled at exactly 0.5000 rad. The catch is that it needs an accurate model of the robot: its mass, its geometry, where its centre of mass is. On a real humanoid, that model is never perfect.

Predicting the motion

Steady-state error describes where the pendulum ends up. To describe how it gets there, I linearized the system around its resting angle. It then behaves like a mass on a spring with a damper, with total stiffness:

Gravity adds a small extra stiffness of its own, \(m g L \cos q\); that’s where the cosine appears. From \(k\) and \(I\) come the two numbers that describe the whole response:

The natural frequency \(\omega_n\) sets how fast it responds. The damping ratio \(\zeta\) sets how much it overshoots: below 1 it overshoots and wobbles, at exactly 1 it reaches the target fastest without overshooting, and above 1 it creeps in slowly. From these, the overshoot and settling time follow:

I implemented these formulas and ran sweeps over the gains to compare them with the simulation. Each formula is a one-line function in pd_experiments.py:

def natural_frequency(k_eff, I):
    return np.sqrt(k_eff / I)

def damping_ratio(kd, k_eff, I):
    return kd / (2 * np.sqrt(k_eff * I))

def predicted_overshoot(zeta, step):
    if zeta < 1:
        return step * np.exp(-np.pi * zeta / np.sqrt(1 - zeta**2))
    return 0.0

Watching a few lines of maths predict the simulation this closely was the best part of the week. The sweeps also made some trade-offs concrete. More \(K_P\) means a smaller error but more overshoot and more torque. More \(K_D\) removes overshoot, but too much makes the response slow again; \(K_D = 20\) took 3 s to settle. Settling speed depends only on \(K_D\) and inertia, because \(\zeta\,\omega_n = K_D / 2I\) and \(K_P\) cancels out, which is why every \(K_D = 2\) run settles in about a second. And critical damping isn’t the fastest option: \(\zeta = 0.8\) settled in 0.38 s with barely any overshoot, quicker than \(\zeta = 1\).

When the simulation breaks

Those formulas assume time flows smoothly, but the simulator moves in steps of \(\Delta t\). If the gains are too high, a single step overcorrects by more than the original error, and the error grows from step to step. For the damping term alone, each step multiplies the velocity by a fixed factor:

As long as that factor stays between −1 and 1, the motion dies out. Below −1, the velocity flips sign and grows every step. The spring term has a similar limit, and on their own the two give:

These are limits for each term alone, not a combined guarantee. When stiffness and damping act together, a step updates position and velocity jointly, and for this integrator the condition becomes:

The individual limits are just the cases where one gain is close to zero. A large \(K_D\) leaves less room for \(K_P\), and the other way around. The simulation matched: \(K_D = 99\) was fine and 101 exploded; \(K_P = 38{,}000\) survived (badly) and 42,000 exploded. Plugging those four runs into the condition gives 3.95, 4.03, 3.87 and 4.26: below 4 for the stable runs and above 4 for the ones that exploded.

MuJoCo’s own actuators

The position actuator

MuJoCo has a built-in position actuator. Every MuJoCo actuator computes its force from one general formula:

Swapping my motor for a position actuator is a one-line change in the XML:

<position joint="hinge" kp="20" kv="2"/>

and the parameters MuJoCo built from it can be printed:

print(model.actuator_gainprm[0, :3])   # [ 20.   0.   0.]
print(model.actuator_biasprm[0, :3])   # [  0. -20.  -2.]

A position actuator sets the gain to \([K_P, 0, 0]\) and the bias to \([0, -K_P, -K_D]\). Substituting gives \(K_P(\text{ctrl} - q) - K_D\,\dot q\): my PD controller, with ctrl as the target angle. Running both versions side by side, the trajectories matched to \(10^{-17}\) rad, which is just floating-point rounding.

Implicit integration

The two versions only behave differently beyond the stability limit. With \(K_D = 102\), just past it, I switched MuJoCo to its implicitfast integrator, which again is one attribute:

<option timestep="0.005" integrator="implicitfast"/>

My Python PD still exploded, but the built-in actuator stayed stable. An implicit integrator applies damping using the end-of-step velocity, and that update can never overcorrect, whatever the gain:

To do this, MuJoCo needs to know that the force depends on velocity. For the built-in actuator it knows the formula. For my Python PD it only receives a number, so it can’t help. That’s a practical reason to let MuJoCo compute the PD inside its actuators, and a reminder to use the same integrator as in training.

Torque limits

Real motors have a maximum torque. In MuJoCo that’s two attributes on the actuator:

<position joint="hinge" kp="20" kv="2" forcelimited="true" forcerange="-1 1"/>

I limited the actuator to 1 N·m, although holding the pendulum near the target needs 1.93 N·m. I expected it to settle at a lower angle. Instead, it kept swinging between 0 and 0.414 rad and never stopped.

The PD controller asks for more than 1 N·m the whole time, so the motor delivers a constant 1 N·m. A constant torque doesn’t depend on velocity, so the damping term gets clamped away along with everything else. What remains is a pendulum with gravity, a constant push and no damping, oscillating around the angle where gravity equals 1 N·m (0.205 rad). This is actuator saturation: when the requested PD torque goes past the motor’s limit, it gets clipped, and for as long as that lasts the controller can’t deliver the damping it was designed for. Here the saturation never ended, so the damping never came back.

Armature

A motor drives its joint through a gearbox and spins much faster than the joint does. Seen from the joint, the rotor’s inertia gets multiplied by the gear ratio squared. MuJoCo calls this armature and simply adds it to the joint’s inertia:

<joint name="hinge" type="hinge" axis="0 1 0" armature="0.1"/>

After mj_forward, data.M[0] (the joint inertia MuJoCo uses) goes from 0.251 to 0.351. All the actuator experiments are in pendulum_pd.py.

More inertia makes the response slower and less damped, just as the formulas predict. The final angle stays the same, because at rest nothing accelerates and inertia plays no part.

From the pendulum to a walking humanoid

All of this matters for the G1 because of where the controller sits. A walking policy could output torques directly, putting control inside the policy. The G1 policy I’m reproducing outputs joint targets instead, and MuJoCo’s position actuators turn them into torques with exactly the PD law above. Control sits outside the policy, in the simulator. Following the usual convention in legged-robot learning, each action is a scaled offset from a default pose:

So each pendulum lesson carries over. Because of steady-state error, the policy has learned to aim past the angle it really wants, so that the PD error produces the right torque; change \(K_P\) and every learned action means a different torque. The G1’s gains are computed from each joint’s armature, choosing a 10 Hz natural frequency (\(\omega_n = 2\pi \times 10 \approx 62.8\) rad/s) and a damping ratio of 2, through \(K_P = I\,\omega_n^2\) and \(K_D = 2\zeta I \omega_n\), the same formulas as above. Every joint also has a torque limit. Brief saturation is usually fine, but while a joint is saturated its PD torque is clipped and it can’t guarantee the damping it was tuned for, so persistent saturation can lead to oscillation. And the timestep and integrator have to match training: the policy acts at 50 Hz on top of 200 Hz physics.

How I’m learning

I’m using Claude as a tutor for this project. It suggests the next exercise, explains concepts when I get stuck, and reviews my code, while I write most of it myself. Having something that walks me through a structured sequence and asks “why do you think that happened?” has made a big difference to how quickly I get from confusion to understanding.

Next week

Next week I’ll build the full G1 setup in plain MuJoCo: a floor, 29 position actuators with the right gains, the starting pose, the 99-number observation and the 50 Hz control loop. Then I’ll load the trained policy and, hopefully, watch it walk.