How to Set Up a Domain Randomization Pipeline

Domain randomization operates across three categories: visual randomization (what the camera sees), physics randomization (how objects interact), and dynamics randomization (how the robot behaves). For each category, identify which specific parameters affect your task's sim-to-real gap.

Visual randomization axes: (1) Lighting — intensity (0.3x-3x ambient), color temperature (3000-7000K), direction (hemisphere sampling), number of lights (1-5), shadows on/off. (2) Textures — table surface (uniform color, wood, metal, fabric, random procedural), object textures, wall/background textures. (3) Camera — position (+-3-5 cm per axis from nominal), orientation (+-3-5 degrees), field of view (+-5 degrees), focal length, white balance, noise level. (4) Distractors — random objects placed in the scene that are not task-relevant, varying in count (0-10) and appearance.

Physics randomization axes: (1) Friction — surface-surface friction coefficient (0.3-1.5), object-table friction, gripper-object friction. (2) Mass — object mass (0.5x-2x nominal), center of mass offset (+-1 cm). (3) Restitution — bounciness coefficient (0.0-0.5). (4) Contact stiffness and damping (affects how objects feel when grasped).

Dynamics randomization axes: (1) Actuator gains — P and D gains of joint controllers (0.8x-1.2x). (2) Motor delay — control latency (0-20 ms). (3) Joint friction — per-joint Coulomb friction (0.5x-2x). (4) Observation noise — Gaussian noise added to joint position readings (0-0.005 rad) and force readings (0-0.5 N).

Document all axes in a randomization specification with the planned range for each parameter and the sampling distribution (uniform is the default; log-uniform for parameters spanning orders of magnitude like friction).

Set up the simulation environment with your robot model, task objects, and the randomization framework. The two main simulator ecosystems are:

NVIDIA Isaac Sim + Replicator: Isaac Sim provides the rendering engine (RTX ray tracing for photorealistic images), robot model loading (URDF/USD), and physics (PhysX). Replicator is the built-in randomization API. Create a Replicator graph that specifies which parameters to randomize and their distributions. Example: rep.create.light(position=rep.distribution.uniform((-2,-2,1),(2,2,3)), intensity=rep.distribution.uniform(500,5000)). Replicator can randomize textures, object poses, camera parameters, and lighting in a single graph definition. Run the graph at the start of each episode to generate a fresh randomized scene.

MuJoCo + custom randomization: MuJoCo provides fast physics simulation (10,000+ steps/second on CPU, 1M+ with GPU parallelization via MuJoCo XLA). Randomization is applied by modifying the MJCF model XML before each episode: change friction, mass, joint damping, and actuator parameters. For visual randomization, MuJoCo's native renderer is basic (no ray tracing), so either: (a) use MuJoCo for physics and render with a separate engine (Blender, Isaac Sim) for perception training, or (b) apply visual randomization as image augmentation (color jitter, random crops, Gaussian noise) after rendering. The augmentation approach is simpler but less physically accurate than rendered randomization.

For either simulator, create a randomization configuration file (YAML or JSON) that lists all randomized parameters with their distribution type and range. This config file is versioned alongside the codebase so that randomization settings are reproducible.

With the randomization pipeline configured, generate a large synthetic dataset. For perception pretraining (object detection, segmentation), generate 50,000-500,000 randomized images. For policy pretraining (manipulation, navigation), generate 10,000-100,000 randomized episodes.

During generation, log the randomization parameters used for each image or episode alongside the data. This metadata enables post-hoc analysis: if the real-world transfer fails for shiny objects, you can check whether your texture randomization included sufficient metallic/specular materials. Store the data in a format compatible with your training pipeline — images as .png with JSON annotations for perception, HDF5 or RLDS episodes for policy training.

Validate the synthetic data before training: (1) Visual plausibility — render 100 random images and verify they look reasonable (no objects floating in mid-air, no cameras inside walls, no extreme lighting that makes everything black or white). (2) Physics plausibility — for policy data, verify that the physics randomization does not produce impossible scenarios (objects with negative mass, friction > 2.0 causing numerical instability). (3) Distribution coverage — histogram each randomized parameter and verify the distribution matches the specification (uniform parameters should be flat, not peaked). (4) Label correctness — for perception data, overlay the generated labels (bounding boxes, masks) on the images and verify alignment.

Common generation pitfalls: Objects spawning inside each other (add collision checking during scene generation), cameras placed inside objects (add camera frustum validation), and lighting so extreme that images are black (clamp minimum scene brightness). Fix these issues before large-scale generation.

Train your model on the domain-randomized synthetic data and evaluate transfer to the real world. The evaluation protocol must be carefully designed to isolate the effect of domain randomization from other variables.

For perception models (detectors, segmenters): train on 100% synthetic data with domain randomization, then evaluate on a real-world test set (100-500 real images labeled by hand). Compare against: (a) a model trained on the same number of real images (upper bound), (b) a model trained on synthetic data without domain randomization (lower bound). The randomized model should close 50-80% of the gap between (a) and (b).

For manipulation policies: pretrain on domain-randomized simulation data, then evaluate success rate on 50-100 real-world trials per task. If sim-to-real transfer is below 60% success, the randomization ranges are likely too narrow (real conditions fall outside the training distribution) or there is a systematic gap the randomization does not cover (e.g., sensor delay, robot dynamics mismatch). Diagnose by replaying the real-world observations through the policy and inspecting where it makes incorrect predictions — if the policy fails at the perception stage (wrong object detection), widen visual randomization; if it fails at the control stage (correct detection but wrong action), widen dynamics randomization.

For policies that need higher real-world performance, add a fine-tuning stage: take the sim-pretrained policy and fine-tune on 500-2,000 real-world demonstrations. This hybrid (sim pretrain + real fine-tune) consistently outperforms either approach alone. The sim pretraining provides a strong initialization that the real data refines. Track the fine-tuning data efficiency: how many real demonstrations are needed to reach 90% success? Compare to training from scratch on real data to quantify the benefit of sim pretraining.

Domain randomization is not a one-shot setup — it requires iterative tuning based on real-world deployment feedback. Analyze real-world failures to identify which randomization axes need adjustment.

Build a failure analysis pipeline: (1) Record all real-world evaluation trials with full sensor data. (2) For each failure, classify the failure mode (perception failure, planning failure, control failure). (3) For perception failures, compare the real observation to the distribution of synthetic observations — is the real image out-of-distribution in color, texture, lighting, or clutter? If so, widen the corresponding randomization axis. (4) For control failures, compare the real dynamics (measured forces, achieved velocities) to the range of dynamics in simulation — is the real friction/mass/delay outside the randomized range? If so, widen the dynamics randomization.

Common iteration patterns: (a) Lighting too narrow — real-world shadows and highlights are more extreme than initial randomization. Widen lighting intensity range and add hard shadows. (b) Object textures too synthetic — procedural textures do not capture real-world wear, printing, and material variation. Add photorealistic texture assets from 3D asset libraries (Polyhaven, AmbientCG). (c) Camera position too narrow — real cameras drift over time due to vibration and temperature changes. Widen camera pose randomization to +-5 cm and +-5 degrees. (d) Friction too narrow — real gripper-object friction varies dramatically with object material and surface condition. Widen friction range to 0.2-1.5.

After each iteration, retrain and re-evaluate. Track the sim-to-real gap (performance difference between simulation and real world) across iterations — it should decrease monotonically. If it plateaus, the remaining gap is likely due to factors that domain randomization cannot address (deformable object physics, fluid dynamics, thermal effects), and real-world data fine-tuning is needed for further improvement.

For production deployment, combine domain-randomized synthetic data with real-world data in a structured training pipeline. The integration follows a standard two-phase approach:

Phase 1 — Sim pretraining: Train the model on 50,000-500,000 domain-randomized synthetic episodes (or images for perception). Use aggressive augmentation and randomization in this phase since the goal is to build general representations. Train until validation loss on a held-out synthetic set plateaus.

Phase 2 — Real fine-tuning: Starting from the Phase 1 checkpoint, fine-tune on real-world data with a reduced learning rate (typically 0.1x-0.01x the pretraining rate). The real data set is smaller (500-5,000 episodes) but high-quality. Use light augmentation in this phase (color jitter, random crop) — heavy augmentation is unnecessary because the real data distribution is what we want to match. Fine-tune until validation performance on a held-out real-world test set peaks.

Data mixing: Some practitioners mix synthetic and real data during fine-tuning rather than training sequentially. Use a sampling ratio of 3:1 to 5:1 (synthetic:real) in each batch to prevent the small real dataset from being overwhelmed. Gradually increase the real data proportion over the course of fine-tuning (curriculum approach).

Deployment validation: Before deploying, run a final evaluation on 100+ real-world trials covering the full task diversity. Track not just success rate but also the distribution of failure modes — if the failure distribution shifts after fine-tuning (fewer perception failures, more dynamics failures), this indicates the real data improved perception but more dynamics randomization or dynamics-specific real data is needed. Document the full training recipe (sim dataset, randomization config, real dataset, hyperparameters) for reproducibility.