Custom visualization¶

Aside from the true simulation state, MuJoCo provides ways to draw additional 3D geometries (geoms) onto an existing 3D scene.

Note

Custom visualization requires the viewer (or viewer-ui), renderer, or cpp-viewer Cargo feature to be enabled.

In MuJoCo-rs, drawing is done through MjvScene. There are three things that accept custom visual-only geoms:

Drawing to a scene¶

To draw custom geoms to a scene inside a viewer or a renderer, applications typically call two methods when updating custom geoms:

MjvScene::clear_geom, which will clear all the existing geoms.
MjvScene::create_geom, which will add a new geom to the scene.

Note

create_geom panics when the user scene has no free geom slots. Use try_create_geom for the fallible variant that returns Result. Increase the configured user-geom capacity if you hit this limit.

Warning

create_geom and try_create_geom are unsafe; clear_geom is safe. The renderer reads the geom’s matid, texid and, on a flex or a skin geom, objid as unchecked indices, so the caller must keep each one below the count of the model that the rendering context was created for. matid and texid also accept -1 for none, which is what the two methods write.

This example shows how to draw a line between two balls.

In the example, we start drawing by first obtaining a mutable reference to the user scene and clearing its geoms, which are otherwise preserved between syncs:

viewer.with_state_lock(|mut state_lock| {
    let scene = state_lock.user_scene_mut();  // obtain a mutable reference to the user scene.
    scene.clear_geom();  // clear existing geoms
}).unwrap();

We then initialize a new geom. We make it a line (MjtGeom::mjGEOM_LINE) and give it a pure white color (Some([1.0, 1.0, 1.0, 1.0])). We leave other fields at None, as they are not needed at this stage.

viewer.with_state_lock(|mut state_lock| {
    let scene = state_lock.user_scene_mut();  // obtain a mutable reference to the user scene.
    scene.clear_geom();  // clear existing geoms

    // SAFETY: the geom keeps the ids that create_geom sets to none.
    let new_geom = unsafe {
        scene.create_geom(
            MjtGeom::mjGEOM_LINE,  // type of geom to draw.
            None,  // size, ignore here as we set it below.
            None,  // position: ignore here as we set it below.
            None,  // rotational matrix: ignore here as we set it below.
            Some([1.0, 1.0, 1.0, 1.0])  // color (rgba): pure white.
        )
    };
}).unwrap();

In the above snippet, defining the fields that we’ve set to None would work, making this the final step. However, we would need to know their correct values. We obtain the needed values using MjvGeom::connect, which calculates the values to result in the geom pointing from one point to another.

viewer.with_state_lock(|mut state_lock| {
    let scene = state_lock.user_scene_mut();  // obtain a mutable reference to the user scene.
    scene.clear_geom();  // clear existing geoms

    // SAFETY: the geom keeps the ids that create_geom sets to none.
    let new_geom = unsafe {
        scene.create_geom(
            MjtGeom::mjGEOM_LINE,  // type of geom to draw.
            None,  // size, ignore here as we set it below.
            None,  // position: ignore here as we set it below.
            None,  // rotational matrix: ignore here as we set it below.
            Some([1.0, 1.0, 1.0, 1.0])  // color (rgba): pure white.
        )
    };

    /* Read X, Y and Z coordinates of both balls. */
    let ball1_position = ball1_joint_info.view(&data).qpos[..3]
        .try_into().unwrap();
    let ball2_position = ball2_joint_info.view(&data).qpos[..3]
        .try_into().unwrap();

    /* Modify the visual geom's position, orientation and length, to connect the balls */
    new_geom.connect(
        0.0,            // width
        ball1_position, // from
        ball2_position  // to
    );
}).unwrap();

The following image shows the result of the above example.