RM65 Stack Lab

SAM3 Socket Bridge

This repo supports running Isaac Lab and SAM3 in separate conda environments:

• lab: Isaac Lab simulation and demo collection
• vision: SAM3 mask inference

The two environments communicate over a localhost TCP socket. This keeps the Isaac Lab environment stable while still allowing real-time SAM3 mask overlays in collect_diffusion_demos.py.

1. Start The SAM3 Server In vision

Open a terminal:

cd /home/rae/rm65_stack_lab
conda activate vision
python scripts/vision_mask_server.py \
  --host 127.0.0.1 \
  --port 5567

If SAM3 is not in the default location, pass the checkpoint explicitly:

cd /home/rae/rm65_stack_lab
conda activate vision
python scripts/vision_mask_server.py \
  --host 127.0.0.1 \
  --port 5567 \
  --sam3-checkpoint /path/to/sam3.pt

Useful optional flags:

--device cuda
--amp-dtype auto
--confidence-threshold 0.20

Notes:

• --amp-dtype float16 can help if bfloat16 causes issues on your GPU.
• The server loads SAM3 once and then waits for requests from the lab env.

2. Run Demo Collection In lab

Open a second terminal:

cd /home/rae/rm65_stack_lab
conda activate lab
python scripts/collect_diffusion_demos.py \
  --mask-visualization-backend sam3_socket \
  --mask-server-host 127.0.0.1 \
  --mask-server-port 5567

Useful mask-related tuning flags:

--mask-text-prompt box
--mask-primary-max-dist 0.35
--mask-table-z-min 0.40
--mask-table-z-max 0.70

What this currently does:

• sends the head RGB image, head depth, camera pose, camera intrinsics, and current gripper center from lab to vision
• runs SAM3 in the vision env
• returns primary and target masks
• overlays those masks live in the collect_diffusion_demos.py perception UI

Important:

• this bridge is currently for visualization only
• the collector control logic still uses the existing perception-backend path such as task_context
• if the mask server is offline, the collector should keep running and fall back to the existing overlay behavior

3. Expected Workflow

Terminal 1:

cd /home/rae/rm65_stack_lab
conda activate vision
python scripts/vision_mask_server.py --host 127.0.0.1 --port 5567

Terminal 2:

cd /home/rae/rm65_stack_lab
conda activate lab
python scripts/collect_diffusion_demos.py \
  --mask-visualization-backend sam3_socket \
  --mask-server-host 127.0.0.1 \
  --mask-server-port 5567

4. Troubleshooting

If the collector UI shows that the mask server is offline:

• make sure vision_mask_server.py is running
• make sure both sides use the same host and port
• make sure the SAM3 checkpoint is accessible in the vision env

If SAM3 starts but crashes on dtype issues:

• restart the server with:

python scripts/vision_mask_server.py --host 127.0.0.1 --port 5567 --amp-dtype float16

If SAM3 cannot find weights:

• pass --sam3-checkpoint /path/to/sam3.pt

5. Related Files

• scripts/vision_mask_server.py: SAM3 socket server in the vision env
• scripts/collect_diffusion_demos.py: Isaac Lab collector with socket client and live mask overlay
• src/rm65_stack_lab/perception/mask_rpc.py: shared socket message helpers