AI-to-USD: An LLM Agent That Self-Corrects Industrial Scene Generation for Isaac Sim

I built an end-to-end pipeline that turns natural language descriptions into validated USD industrial scenes and synthetic training data for computer vision.

The pipeline has two stages:

Stage 1: Spec-to-Sim Copilot: You describe a battery module assembly task (“assemble a 2x3 grid of LG E63 cells with a UR10e robot”). An LLM (Gemini 2.0 Flash with structured output) generates a Pydantic-validated ModuleTask JSON. Then, 5 industrial safety rules are checked against the spec: thermal spacing between cells, robot arm reachability, cell rotation alignment for busbar welding, module tray bounds, and cell count limits. If validation fails, the system feeds structured error reports back to the LLM for self-correction (up to 3 repair attempts). This is the interesting part: pure LLM generation fails silently for spatial tasks — cells overlap by 5-10mm, coordinates fall outside robot reach — so the validation loop catches what the LLM can’t reason about.

Stage 2: Industrial SDG Lab: The validated scene gets domain randomization across 4 axes (lighting intensity, material color shift, camera pose, object placement jitter) to produce variant USDA scenes with COCO-format annotations — ready for training object detection models. An Omniverse-style 3D viewport built with Three.js lets you interactively inspect the scene with PBR materials, bloom post-processing, and a property inspector panel.

Live demo: I’ll type a prompt, show the LLM generating a spec, trigger a validation failure, watch the AI self-correct, then generate randomized variants with the 3D viewport.