MH3 Genesis Hackathon Starter

Google Developers Group Windsor

Build with AI - Smart Automation, Robotics, and Vision

This repository is a self-contained, Python-based starting point for building projects with the Mirsee Robotics MH3 in Genesis World. It includes the simulated MH3, a default world with a floor, a browser dashboard, and a Python API for controlling and observing the robot.

Setup and Run

Supported Platforms

• Linux x64 (glibc 2.28+)
• Apple Silicon macOS (macOS 14+)
• Windows 10+ x64

Intel Macs, ARM Linux/Windows, and 32-bit platforms are not supported.

Prerequisites

• Linux: curl or wget, plus graphics runtime libraries (libgl1, libegl1, libglib2.0-0 on Ubuntu/Debian). On WSL2, ensure WSLg and updated GPU drivers are installed.
• Working GPU drivers and OpenGL/EGL runtime (required for cameras and the viewer, even if physics runs on CPU). See Genesis installation requirements.

Installation

The setup scripts install uv and the pinned Python version automatically; no existing Python installation is required.

Note

Initial setup may take a few minutes while kernels compile and the environment is verified.

Linux / Apple Silicon macOS:

./setup.sh

Windows x64 (PowerShell):

.\setup.ps1

Important

If Windows blocks script execution, run: powershell -ExecutionPolicy Bypass -File .\setup.ps1

Common options:

• Force CPU physics: ./setup.sh --cpu (or .\setup.ps1 --cpu)
• Reset environment: ./setup.sh --reset (or .\setup.ps1 --reset)

Running Scripts

After setup, run simulation scripts (such as hello_mh3.py) with the run script for your platform:

Linux / Apple Silicon macOS:

./run.sh hello_mh3.py

Windows x64 (PowerShell):

.\run.ps1 hello_mh3.py

Both scripts forward all arguments to uv run --no-sync, including script parameters and uv options (for example, ./run.sh my_script.py --steps 300 or .\run.ps1 my_script.py --steps 300). They use the project's local uv installation when available and run from the project directory.

Setup installs the locked shared dependencies first, then selects a PyTorch wheel for your hardware. On a fresh environment, Torch is deferred until this selection, so the separate Torch installation is expected.

The run scripts preserve the selected wheel: plain uv run synchronizes against uv.lock and can replace the selected CUDA wheel with the locked CPU build on Windows. --frozen still synchronizes packages and does not prevent this. If this has already happened, rerun setup to restore and verify the hardware-selected wheel, then use the run script.

Upgrading

To update dependencies to the latest Genesis World release:

Note

In most cases you should only try upgrading if a newer release of Genesis World has a feature or bug fix you need. Before upgrading, back up your current project.

Linux / Apple Silicon macOS:

./upgrade.sh

Windows x64 (PowerShell):

.\upgrade.ps1

Use --dry-run to preview changes or --lock-only to update lockfiles without installing.

Browser Dashboard

The dashboard provides an interactive web interface for driving the base, posing joints, testing inverse kinematics (IK), and monitoring live camera feeds.

Controls & Feeds

• Base drive: Hold on-screen buttons or use W / A / S / D (or arrow keys) to drive the base. Speed sliders adjust maximum linear and angular rates. Releasing a control stops base motion automatically.
• Joint controls: Adjust sliders or enter numeric values to pose joints. Click Return all joints home to restore the default startup configuration.
• IK tester: Interactively solve inverse kinematics for chosen end effectors and reference frames with position and orientation targets.
• Camera feeds: Displays live views for head and base cameras (left RGB, right RGB, and depth). In depth views, close surfaces are white and far distances are black.

Using the Dashboard

You can easily embed the dashboard in your custom simulation scripts:

from mh3_genesis import MH3ControlPanel, MH3Simulation, SimulatorConfig

sim = MH3Simulation(SimulatorConfig.from_yaml("config/mh3.yaml")).build()
panel = MH3ControlPanel(sim)
panel.start()  # Use open_browser=False to print the local URL without opening a browser tab
sim.add_step_hook(panel.update)

try:
    sim.run()
finally:
    panel.close()

The dashboard listens locally on 127.0.0.1 on an automatically assigned port. Commands are queued and applied safely on the simulation thread during panel.update(). Press Ctrl+C in your terminal to stop the simulation.

Camera Configuration & Customization

Cameras and vision feeds can be customized for performance, visual quality, and bandwidth in config/mh3.yaml or programmatically in Python. Feeds are addressed directly by name:

• Head stereo rig: head_left (RGB), head_right (RGB), head_depth (metric depth).
• Base stereo rig: base_left (RGB), base_right (RGB), base_depth (metric depth).

Configurable Camera Feed Settings

Every camera feed shares the same clean configuration options:

Option	Type	Default	Description
enabled	boolean	true	When false, the feed is completely turned off. If both RGB and depth for a physical sensor are disabled, Genesis omits that camera entirely from the scene to save GPU resources.
format	string	jpeg	Encoding format: jpeg (fast and compact) or png (lossless).
quality	integer	100	Compression quality for JPEG encoding (1 to 100).
width	integer	960	Render image width in pixels. If changed, height is automatically updated to maintain the camera's native 16:10 aspect ratio.
height	integer	600	Render image height in pixels. If changed, width is automatically updated to maintain the camera's native 16:10 aspect ratio.
near_m / far_m	float	0.1 / 20.0	Near and far clipping planes in metres.

Note

Modifying either width or height automatically computes and updates the other dimension to preserve the native optical 16:10 aspect ratio ($1.6$) of the physical ZED X cameras (for instance, setting width: 1920 automatically sets height: 1200, and setting height: 400 sets width: 640).

Fixed Camera Hardware Parameters

The following parameters represent the true physical characteristics of the robot's Stereolabs ZED X (base) and ZED X Mini (head) 2.2mm optics and real robot perception pipeline.

Parameter	Type	Value	Description
vertical_fov_deg	float	77.95°	Vertical field of view computed from the physical 2.2mm lens sensor aperture ($1.80,\text{mm}$ aperture / $1.1124,\text{mm}$ focal length).
horizontal_fov_deg	float	104.63°	Horizontal field of view computed from the physical 2.2mm lens sensor aperture ($2.88,\text{mm}$ aperture / $1.1124,\text{mm}$ focal length; ~110° nominal with distortion).
fps	float	15.0 FPS	Real robot sensor capture and ROS 2 perception publish rate (pub_frame_rate: 15.0).

YAML Example (config/mh3.yaml)

# Disable right RGB feeds to save GPU compute.
cameras:
  head_left:
    enabled: true
    format: jpeg
    quality: 100
  head_right:
    enabled: false
    format: jpeg
    quality: 100
  head_depth:
    enabled: true
    format: jpeg
    quality: 100
  base_left:
    enabled: true
    format: jpeg
    quality: 100
  base_right:
    enabled: false
    format: jpeg
    quality: 100
  base_depth:
    enabled: true
    format: jpeg
    quality: 100

Dashboard Adaptation

The browser dashboard automatically detects configured cameras and adapts in real time:

• Inactive feeds and individual camera figures are hidden automatically.
• If all cameras for a group (head or base) are disabled, that entire section is hidden.
• If all cameras are disabled, the dashboard indicates "All cameras disabled in configuration" and suspends camera network polling.

Robot Joints & Links

Units

• Revolute joints (arms, head, hands): Radians (rad).
• Prismatic joints (slider): Metres (m).
• Mobile base velocities: Metres per second (m/s) for linear speed, radians per second (rad/s) for angular speed.

Joint Names & Limits

Inspect available joints and limits via sim.joint_names and sim.get_joint_limits():

• Torso lift: slider (travels from 0.0 to 0.3 m).
• Head: head_yaw (−1.40 to 3.10 rad), head_pitch (−0.40 to 1.00 rad).
• Arms (7 DOF each): left_1 through left_7 and right_1 through right_7. Wrist joints 6 and 7 use the vendor model bounds of −0.30 to 0.30 rad.
• Hands (6 controls per hand):
  • Fingers: left_index, left_middle, left_ring, left_little (and matching right_* joints). Zero radians represents fully open; joints close through 1.57 rad. Dependent finger joints follow these root controls automatically via mimic constraints.
  • Thumbs: left_thumb_1 (up to 1.6 rad) and left_thumb_2 (up to 0.45 rad), along with their right_* counterparts.

Link Names

Discover all link identifiers using sim.link_names before passing one to sim.get_link() or sim.solve_ik():

• Arm segments: left_arm_1 through left_arm_7 (and right_arm_1 through right_arm_7).
• Gripper bases: left_gripper_base, right_gripper_base.
• Fingertips: left_index_tip, left_middle_tip, left_thumb_tip, etc.
• Torso & head: torso_base, head_base, head_camera_mount.

Using the API

Running Custom Scripts

After running setup, execute your custom scripts with ./run.sh (or .\run.ps1 on Windows):

./run.sh my_script.py

API Usage Example - hello_mh3.py

A complete, self-contained starter script demonstrating scene hooks, joint commands, base driving, state queries, inverse kinematics, camera capture, and step hooks is provided in hello_mh3.py. Run it directly with:

./run.sh hello_mh3.py  # Windows: .\run.ps1 hello_mh3.py

Or use it as a reference for your own scripts:

import numpy as np
from mh3_genesis import MH3ControlPanel, MH3Simulation, SimulatorConfig

# 1. Load configuration and initialize simulation.
config = SimulatorConfig.from_yaml("config/mh3.yaml")
sim = MH3Simulation(config)


# 2. Scene hook: add custom entities before Genesis compiles the scene.
def add_custom_world(simulation: MH3Simulation) -> None:
    simulation.scene.add_entity(simulation.gs.morphs.Box(pos=(1.0, 0.0, 0.25), size=(0.3, 0.3, 0.5)))


sim.add_scene_hook(add_custom_world)

# 3. Build simulation, compile kernels, and spawn articulation.
sim.build()

# Inspect available joints, links, and cameras.
print(sim.compute_backend)
print(sim.joint_names)
print(sim.link_names)
print(sim.camera_names)

# 4. Control robot joints and drive mobile base.
sim.set_joint_positions({"head_yaw": 0.2, "head_pitch": -0.2, "slider": 0.1})
sim.set_joint_positions({"left_index": 0.6, "left_thumb_1": 0.5})
sim.drive_base(linear_mps=0.1, angular_radps=0.0)

# Advance simulation steps.
for _ in range(30):
    sim.step()
sim.stop_base()

# 5. Read joint states (positions, velocities, efforts).
joint_state = sim.get_joint_state(["head_yaw", "head_pitch", "slider"])

# 6. Inverse Kinematics (IK) for arm manipulation (world space by default, or frame="link_name").
# Supports target position, orientation (Euler or quaternion), or position=None to re-orient in place.
left_arm_joints = [f"left_{i}" for i in range(1, 8)]
solution = sim.solve_ik(
    end_effector="left_gripper_base",
    position=(0.45, 0.25, 0.85),
    joints=left_arm_joints,
    apply=True,
)

# 7. Sensor capture (RGB, depth, point cloud).
frame = sim.capture_camera("head_left", rgb=True, depth=True)
points, valid_depth = sim.capture_pointcloud("head_left")


# 8. Step hook: register callbacks to run after each physics step.
def behavior_controller(simulation: MH3Simulation) -> None:
    # Example timed motion: level head between 1.0 and 1.5 seconds.
    if 1.0 <= simulation.sim_time < 1.5:
        simulation.set_joint_positions({"head_pitch": 0.0})


sim.add_step_hook(behavior_controller)

# 9. Start browser dashboard and run.
panel = MH3ControlPanel(sim)
panel.start()
sim.add_step_hook(panel.update)

try:
    sim.run()
finally:
    panel.close()

Advanced projects can also directly access sim.scene, sim.robot, sim.floor, and sim.gs for any native Genesis World feature.

Resources

These are resources which you may consider using with this project.

• Genesis World Documentation - Core physics and rendering engine of this project.
• Google Gemma - Google's lightweight, on-device multi-modal AI model.
• Gemini Robotics ER-2 - Google's flagship Embodied Reasoning (ER) Vision-Language Model (VLM).
• LeRobot - Machine learning framework for robotics by HuggingFace.
• py_trees - A Python behaviour tree implementation.