Indoor RGB Video Dataset
RGB video of indoor environments with semantic annotations, instance segmentation, and room layout annotations for training 3D scene understanding and indoor navigation. Capture is RGB video — Claru does not deploy LiDAR or laser scanners.
Dataset at a Glance
Why Indoor Point Cloud Data Matters for Robotics
Indoor robots -- home assistants, office delivery bots, cleaning robots, retail inventory systems, and facility management platforms -- must build and reason about 3D representations of their environments. Point clouds provide the richest 3D representation available: millions of measured 3D points that capture room geometry, furniture placement, object locations, and navigable space with centimeter-level precision. Unlike 2D images or even depth maps, point clouds represent the full 3D structure of a scene, enabling robots to plan paths around furniture, reach for objects on shelves, and understand spatial relationships that are ambiguous in 2D projections.
Training 3D perception models on point cloud data is fundamentally different from training on images. Point clouds are unordered, irregular, and variable in density -- a nearby wall produces dense points while a distant corner may have sparse coverage. Models must be invariant to point ordering (PointNet), capture local geometric patterns (PointNet++, KPConv), or operate on voxelized representations (MinkowskiNet, SparseConvNet). Each approach requires training data with per-point annotations that accurately reflect the 3D semantic structure of real indoor environments.
Existing indoor point cloud datasets like ScanNet and S3DIS have driven significant progress in 3D scene understanding research, but they are limited in scale and diversity. ScanNet contains 1,513 scans of primarily academic and residential rooms. S3DIS covers 6 areas of a single university building. Real-world indoor robots encounter far more diverse environments: homes with varied architectural styles, offices with different furniture systems, retail stores with diverse shelving, hospitals with specialized equipment, hotels, restaurants, gyms, libraries, and more. Claru captures RGB video of indoor environments, not point clouds: 3D structure is supplied only as a derived annotation layer, such as monocular depth estimates computed from the frames.
Research from CVPR 2024 and 3DV 2024 demonstrates that 3D scene understanding models trained on diverse indoor environments achieve 25-35% better generalization to novel room types compared to models trained on single-building datasets, with the improvement driven by exposure to varied furniture styles, room scales, and architectural configurations that build robust feature representations.
Capture and Collection Methodology
Capture is RGB video recorded as walkthroughs of real indoor spaces. Claru does not deploy LiDAR, terrestrial laser scanners, or depth sensors. Where a project needs 3D structure, it is supplied as a derived annotation layer -- for example monocular depth estimates computed from the RGB frames -- and is labelled as an estimate rather than measured geometry.
Walkthrough video follows the path a mobile robot would take, starting from a room entrance and progressively revealing the space, which makes it usable for visual navigation and exploration research without a scanning rig.
Campaigns can target indoor environment types such as residential living spaces (apartments, houses, studios), commercial offices (open plan, private offices, conference rooms), retail spaces (stores, showrooms), hospitality (hotel rooms, lobbies, restaurants), healthcare (patient rooms, waiting areas, clinics), educational (classrooms, labs, libraries), fitness (gyms, studios), and industrial (workshops, server rooms). Room counts and mix are scoped per project.
Environmental metadata per recording can include room type, approximate floor area, ceiling height, number of distinct furniture items, dominant materials (wood, carpet, tile, concrete), lighting type (natural, fluorescent, LED), and clutter level (minimal, moderate, dense). For residential spaces, architectural style (modern, traditional, industrial, Scandinavian) can be documented. This metadata enables researchers to study how 3D perception performance varies with environment characteristics and to build room-type-aware models.
Comparison with Public Datasets
How Claru's indoor RGB video compares to publicly available alternatives for 3D indoor scene understanding.
| Dataset | Clips | Hours | Modalities | Environments | Annotations |
|---|---|---|---|---|---|
| ScanNet (CVPR 2017) | 1,513 scans | N/A | RGB-D (Kinect) | ~700 rooms (academic/residential) | 20 semantic classes, instances |
| S3DIS (CVPR 2016) | ~270 scans | N/A | Point cloud (Matterport) | 1 university building | 13 semantic classes |
| Matterport3D (3DV 2017) | ~10K views | N/A | RGB-D, mesh | 90 buildings | 40 semantic classes, instances |
| Claru Indoor RGB Video | Scoped per project | Scoped per project | RGB (depth as derived layer) | Residential, commercial, retail, hospitality and more | Semantic and instance masks, layout, navigable regions, affordance |
Annotation Pipeline and Quality Assurance
Stage one automated processing runs on the RGB frames: initial semantic and instance segmentation with pre-trained 2D models, floor-region detection, and -- where scoped -- monocular depth estimation to provide a derived relative-depth layer. Derived depth is delivered as an estimate with its provenance recorded, never as sensor ground truth.
Stage two human annotation adds per-pixel semantic labels across 40+ indoor object categories following a taxonomy compatible with ScanNet but extended with additional classes for commercial and hospitality environments: furniture (tables, chairs, desks, beds, sofas, shelving), fixtures (sinks, toilets, light fixtures, HVAC vents), electronics (monitors, TVs, printers), architectural elements (walls, floors, ceilings, doors, windows, columns, stairs), and room-specific items (kitchen appliances, bathroom fixtures, retail displays, gym equipment). Instance segmentation separates individual objects within each class.
Stage three adds higher-level spatial annotations: room layout annotation (wall, floor, and ceiling boundaries), functional zone delineation (work area, circulation path, storage zone, social area), navigable space mapping with clearance annotations (wheelchair-accessible paths, narrow passages, step hazards), and object affordance labels (sittable, openable, graspable, pushable). These spatial annotations go beyond per-point semantics to capture the functional structure that indoor robots need for task planning.
Stage four QA combines automated consistency checks with human review. Labels are checked for temporal consistency across frames -- the same chair must keep the same label as the camera moves through the room. Instance boundaries are reviewed against object edges in the frame, and navigable-region annotations are reviewed by a second annotator.
Use Cases
Indoor Scene Understanding
Training segmentation models that recognize furniture, architectural elements, and objects across diverse room types from RGB video, with labels that can be lifted to 3D by teams running their own reconstruction. Point-cloud networks such as Mask3D, PointNet++, MinkowskiNet, and Point Transformer need a 3D source; Claru supplies RGB and derived depth, not scanned point clouds.
Indoor Robot Navigation
Training visual navigation systems that robots use for path planning and obstacle avoidance. Navigable-region and clearance annotations on walkthrough video help models reason about traversability -- critical for robots that must navigate around furniture, through doorways, and over thresholds.
Scene Reconstruction and Simulation
Using RGB walkthrough video of real rooms as input to NeRF, Gaussian Splatting, or other reconstruction pipelines run by the customer, and as training data for scene completion and furniture layout prediction. Useful for building simulation environments for robot training.
Key References
- [1]Dai et al.. “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.” CVPR 2017, 2017. Link
- [2]Armeni et al.. “3D Semantic Parsing of Large-Scale Indoor Spaces.” CVPR 2016, 2016. Link
- [3]Schult et al.. “Mask3D: Mask Transformer for 3D Instance Segmentation.” ICRA 2023, 2023. Link
- [4]Wu et al.. “Point Transformer V2: Grouped Vector Attention and Partition-based Pooling.” NeurIPS 2022, 2022. Link
How Claru Delivers This Data
Claru's collector network records RGB video across indoor environments targeted per campaign -- apartments and hotels, co-working spaces and medical clinics, retail stores and fitness centers. This diversity helps perception models trained on the data generalize across the full range of indoor environments that robots encounter in deployment, not just the academic and residential settings covered by existing public datasets.
Custom campaigns can target specific room types (residential only, or office focus), architectural styles, furniture density levels, or accessibility requirements (ADA-compliant space annotations). Timelines are scoped per campaign.
Data is delivered as RGB video with per-frame semantic and instance labels, room layout annotations, and navigable-region maps. Monocular depth estimates, where scoped, ship as a separate derived layer. Formats include MP4+JSON, HDF5, RLDS, WebDataset, or a custom schema.
Frequently Asked Questions
RGB video. Claru does not deploy LiDAR, terrestrial laser scanners, or depth sensors. Where 3D structure is needed, it is provided as a derived annotation layer -- for example monocular depth estimates computed from the RGB frames -- not as scanned point clouds.
40+ classes extending the ScanNet taxonomy with additional categories for commercial, hospitality, healthcare, and retail environments. Per-pixel semantic labels, instance segmentation, room layout, navigable regions, and object affordance annotations can be scoped.
Campaigns can target residential (apartments, houses), offices (open plan, private, conference), retail (stores, showrooms), hospitality (hotels, restaurants), healthcare (patient rooms, clinics), educational (classrooms, labs), fitness (gyms), and industrial (workshops, server rooms) spaces. Room counts are scoped per project.
Yes. Floor-level navigable-region annotations with obstacle clearance, wheelchair-accessible path identification, narrow passages, and step hazard locations, labelled on the RGB video. These support training navigation systems that reason about traversability for different robot footprints.
As input, yes. RGB walkthrough video can feed NeRF, Gaussian Splatting, or mesh reconstruction pipelines that you run. Claru does not deliver scanned point clouds or metric ground-truth meshes, so reconstructions inherit the scale ambiguity of RGB-only capture.
Request a Sample Pack
Get a curated sample of indoor RGB video with semantic annotations to evaluate for your 3D perception or indoor navigation project.