# MIMIC Dataset Validation Guide ## Overview This guide helps you validate that your MIMIC-converted dataset is correctly formatted. Validation checks: - ✅ JSON structure and schema compliance - ✅ File references and integrity - ✅ Transform shape and rotation-matrix validity - ✅ Sensor data completeness - ✅ Timestamp consistency - ✅ Data type correctness Validation results are split into **errors** and **warnings**. Errors indicate broken schema or data integrity and make validation fail. Warnings are advisory heuristics; they do not change the exit status. Axis-direction assumptions are profile-specific and must not be treated as universal coordinate-convention checks. --- ## Quick Validation Checklist Use this checklist before submitting your dataset: ### Coordinate Systems - [ ] Camera extrinsics are camera-to-vehicle (not vehicle-to-camera) - [ ] Confirm the source and target frame conventions from authoritative metadata - [ ] Confirm vehicle axes against the declared convention, not world-axis signs - [ ] All matrices are 4×4 homogeneous transformations ### Sensors - [ ] Sensors required by the target workflow are present - [ ] Input dataset contains at least one camera - [ ] Input dataset contains at least one LiDAR - [ ] All camera images exist and are readable - [ ] All LiDAR point clouds are valid PLY files - [ ] LiDAR intensity is normalized to [0, 1] or -1 - [ ] Each sensor data list aligns with its own trajectory timestamps and poses - [ ] Per-sensor frame counts may differ; no sensor was padded or resampled to another sensor's timestamps - [ ] Each sensor pose equals `ego_vehicle_pose @ sensor.extrinsics` at the matching timestamp - [ ] Camera intrinsics have positive focal lengths - [ ] LiDAR vertical resolution is properly specified ### Annotations - [ ] Actor types are valid (vehicle, pedestrian, cyclist, other) - [ ] Actor dimensions are positive and reasonable - [ ] Actor trajectories have valid timestamps - [ ] 2D bounding boxes don't exceed image dimensions - [ ] All actors have at least one annotation frame ### Files & Format - [ ] `scene.json` is valid JSON - [ ] All file paths in JSON are relative and exist - [ ] No missing sensor data frames - [ ] PLY files have correct 6-column format - [ ] File permissions allow reading ## Reconstruction Readiness Schema validation checks structure and field constraints; it cannot determine whether images are real captures or whether the sequence is suitable for a particular reconstruction workflow. Before reconstruction, verify that camera images are readable photographs, camera poses and intrinsics are usable, and the sequence is continuous. If the workflow also uses LiDAR, confirm camera frames have LiDAR coverage within that workflow's matching tolerance. The standalone program below includes the complete customer-usable validation flow: schema and file checks, sampled image readability, trajectory and camera checks, sampled LiDAR loading, transform validity, optional camera/LiDAR timing, and a warning for unusually uniform images. Required shape, timestamp alignment, file integrity, and invalid rigid transforms are hard errors. Motion jumps, unusual image content, and sensor-coverage gaps are advisory warnings. Choose `general` for aerial, robotics, or unknown world-frame conventions. The optional `ground-vehicle` profile adds an advisory Z-up heuristic. Use it only after confirming that the source world frame is aligned with the expected ground-vehicle convention; the heuristic still warns rather than failing validation. The script uses the public `mimic_data_api` package. The image score is a heuristic, not proof of image content; inspect flagged samples yourself. ```python import argparse import random import sys from pathlib import Path import numpy as np from PIL import Image from mimic_data_api.file_utils.lidar_ply_handler import LidarPointCloudIOHandler from mimic_data_api.scene import Scene def flat_colour_fraction(path: Path) -> float: """Estimate how much of an image is one quantized colour.""" with Image.open(path) as source: image = source.convert("RGB") image.thumbnail((320, 320)) pixels = np.asarray(image, dtype=np.uint8).reshape(-1, 3) // 8 codes = (pixels[:, 0] << 10) | (pixels[:, 1] << 5) | pixels[:, 2] return float(np.bincount(codes).max() / len(codes)) def validate_scene( scene_path: Path, coordinate_profile: str = "general", frame_convention_confirmed: bool = False, ) -> int: if coordinate_profile not in {"general", "ground-vehicle"}: raise ValueError("coordinate_profile must be 'general' or 'ground-vehicle'") if coordinate_profile == "ground-vehicle" and not frame_convention_confirmed: raise ValueError( "Confirm the source world-frame convention before using the ground-vehicle profile" ) if scene_path.is_dir(): scene_json = scene_path / "scene.json" scene_dir = scene_path else: scene_json = scene_path scene_dir = scene_path.parent errors = [] warnings = [] try: scene = Scene.model_validate_json(scene_json.read_text()) except Exception as exc: print(f"Schema validation failed: {exc}") return 1 sensors = scene.ego_vehicle.sensors cameras = sensors.cameras lidars = sensors.lidars or [] ego_trajectory = scene.ego_vehicle.trajectory ego_timestamps = np.asarray(ego_trajectory.timestamps, dtype=float) ego_poses = np.asarray(ego_trajectory.object2world, dtype=float) if not np.all(np.diff(ego_timestamps) >= 0): errors.append("Ego trajectory timestamps are not monotonic") if len(ego_poses) > 1: jumps = np.linalg.norm(np.diff(ego_poses[:, :3, 3], axis=0), axis=1) if float(jumps.max()) > 5.0: warnings.append( f"Ego trajectory has a {float(jumps.max()):.2f} m frame-to-frame jump" ) ego_rotations = ego_poses[:, :3, :3] rotation_determinants = np.linalg.det(ego_rotations) orthogonality = np.matmul(ego_rotations.swapaxes(-1, -2), ego_rotations) invalid_rotations = (np.abs(rotation_determinants - 1.0) > 0.01) | ( np.max(np.abs(orthogonality - np.eye(3)), axis=(1, 2)) > 0.01 ) if np.any(invalid_rotations): errors.append( f"{int(np.sum(invalid_rotations))} ego pose rotation(s) are not orthonormal rotations" ) if not cameras: errors.append("Scene has no cameras") for camera in cameras: intrinsics = camera.intrinsics if intrinsics.fx <= 0 or intrinsics.fy <= 0: errors.append(f"Camera {camera.name}: focal lengths must be positive") if not (0 < intrinsics.cx < intrinsics.width and 0 < intrinsics.cy < intrinsics.height): warnings.append(f"Camera {camera.name}: principal point is outside the image") rotation = np.asarray(camera.extrinsics, dtype=float)[:3, :3] rotation_orthogonality = rotation.T @ rotation if ( abs(float(np.linalg.det(rotation)) - 1.0) > 0.01 or not np.allclose(rotation_orthogonality, np.eye(3), atol=0.01) ): errors.append( f"Camera {camera.name}: extrinsic rotation is not orthonormal" ) if not camera.images_data: errors.append(f"Camera {camera.name}: no images") continue available_images = [] for image_data in camera.images_data: image_path = scene_dir / image_data.filepath if not image_path.is_file(): errors.append(f"Missing image: {image_data.filepath}") else: available_images.append((image_data, image_path)) sample = random.sample(available_images, min(5, len(available_images))) fractions = [] for image_data, image_path in sample: try: with Image.open(image_path) as image: image.verify() fractions.append(flat_colour_fraction(image_path)) except Exception as exc: errors.append(f"Unreadable image {image_data.filepath}: {exc}") if fractions: median_fraction = float(np.median(fractions)) if median_fraction >= 0.40: warnings.append( f"Camera {camera.name}: sampled images are unusually uniform " f"({median_fraction:.0%}); inspect for plots or renderings" ) lidar_handler = LidarPointCloudIOHandler() for lidar in lidars: if not isinstance(lidar.use_for_training, bool): errors.append(f"LiDAR {lidar.name}: use_for_training must be boolean") horizontal_resolution = lidar.intrinsics.horizontal_resolution if horizontal_resolution is not None and not (0 < horizontal_resolution < 0.1): errors.append( f"LiDAR {lidar.name}: horizontal_resolution should be radians per point" ) files = [] for point_cloud in lidar.lidar_pcds_data: point_cloud_path = scene_dir / point_cloud.filepath if not point_cloud_path.is_file(): errors.append(f"Missing point cloud: {point_cloud.filepath}") else: files.append(point_cloud_path) for point_cloud_path in random.sample(files, min(5, len(files))): try: point_cloud = lidar_handler.load(point_cloud_path) if len(point_cloud.points) == 0: errors.append( f"Point cloud has no usable points after SDK loading: {point_cloud_path}; " "check whether every point has intensity -1 (missing return)" ) except Exception as exc: errors.append(f"Cannot read point cloud {point_cloud_path}: {exc}") if lidars: lidar_timestamp_arrays = [ np.asarray(lidar.trajectory.timestamps, dtype=float) for lidar in lidars if len(lidar.trajectory.timestamps) ] if lidar_timestamp_arrays: lidar_timestamps = np.sort(np.concatenate(lidar_timestamp_arrays)) for camera in cameras: camera_timestamps = np.asarray(camera.trajectory.timestamps, dtype=float) if camera_timestamps.size < 2: continue tolerance = 2.0 * float(np.percentile(np.diff(camera_timestamps), 90)) right = np.searchsorted(lidar_timestamps, camera_timestamps) left = np.clip(right - 1, 0, len(lidar_timestamps) - 1) right = np.clip(right, 0, len(lidar_timestamps) - 1) nearest = np.minimum( np.abs(camera_timestamps - lidar_timestamps[left]), np.abs(camera_timestamps - lidar_timestamps[right]), ) if np.any(nearest > tolerance): warnings.append( f"Camera {camera.name}: {int(np.sum(nearest > tolerance))} " f"frame(s) have no LiDAR sweep within {tolerance:.4f}s; " "reconstruction preprocessing may fail for these frames" ) else: warnings.append("LiDAR temporal coverage could not be checked: no LiDAR poses") if coordinate_profile == "ground-vehicle" and len(ego_poses): middle_pose = ego_poses[len(ego_poses) // 2] if middle_pose[2, 2] < 0: warnings.append( "Ground-vehicle profile: vehicle Z axis points downward at the sampled pose; " "verify the confirmed source frame convention" ) for message in errors: print(f"ERROR: {message}") for message in warnings: print(f"WARNING: {message}") if errors: return 1 print(f"Validated {scene_json}: {len(warnings)} warning(s)") return 0 if __name__ == "__main__": parser = argparse.ArgumentParser( description="Validate a MIMIC scene directory or scene.json" ) parser.add_argument( "scene", type=Path, help="Scene directory or path to scene.json" ) parser.add_argument( "--coordinate-profile", choices=("general", "ground-vehicle"), default="general", help="Apply advisory axis-direction checks for a confirmed ground-vehicle frame", ) parser.add_argument( "--confirm-frame-convention", action="store_true", help="Confirm the source world frame before using the ground-vehicle profile", ) args = parser.parse_args() try: result = validate_scene( args.scene, coordinate_profile=args.coordinate_profile, frame_convention_confirmed=args.confirm_frame_convention, ) except ValueError as exc: parser.error(str(exc)) sys.exit(result) ``` Save the complete example as `validate_scene.py` and run it with either a scene directory or the `scene.json` file: ```bash python validate_scene.py /path/to/scene ``` For data with a confirmed ground-vehicle world frame, enable the advisory direction check explicitly: ```bash python validate_scene.py /path/to/scene \\ --coordinate-profile ground-vehicle --confirm-frame-convention ``` The image-uniformity warning is not a pass/fail test for photographic content. Lighting, blank walls, night scenes, and other valid images can also produce a high score. Always inspect the sampled images, and tune the optional timing check to the target workflow's actual sensor-matching policy. --- ## Validation Script ### Installation ```bash curl -LsSf https://astral.sh/uv/install.sh | sh uv venv .venv source .venv/bin/activate uv pip install jsonschema numpy pillow plyfile ``` ### Automated schema validation via mimic_data_api If `mimic_data_api` is installed, you can validate any `scene.json` against the full data model in one call: ```python from mimic_data_api.scene import Scene text = open("scene.json").read() scene = Scene.model_validate_json(text) # raises ValidationError on failure ``` `model_validate_json` runs every Pydantic and numpydantic validator defined in the data model. The complete set of automatic checks is: **WorldTrajectory** (ego vehicle, every camera, every lidar, every actor) | Check | Detail | |---|---| | Array shape — timestamps | 1-D float array | | Array shape — object2world | (N, 4, 4) float array | | Length consistency | `len(timestamps) == len(object2world)` | | Angular velocity (optional) | shape (N, 3); `len == len(timestamps)` | | Linear velocity (optional) | shape (N, 3); `len == len(timestamps)` | **Camera** | Check | Detail | |---|---| | `camera_model` | Must be one of: `simple_pinhole`, `pinhole`, `simple_radial`, `radial`, `opencv`, `opencv_fisheye`, `equirectangular`, `omnidirectionalstereo_l`, `omnidirectionalstereo_r`, `vr180_l`, `vr180_r`, `orthophoto`, `fisheye624`, `mei` | | Image timestamps | `timestamp >= 0` on every `ImageData` entry | | Image file extension | `filepath` must end with `.png`, `.jpg`, or `.jpeg` | | Timestamp alignment | List of image timestamps must exactly match `camera.trajectory.timestamps` | **LiDAR** | Check | Detail | |---|---| | `use_for_training` | Required and must be a boolean: `true` or `false`; `null` is invalid | | `VerticalResolution` | Exactly one of `vertical_resolution` or `elevation_channel_mapping` must be set (not both, not neither) | | `horizontal_resolution` / `vertical_resolution` | Radians per point (angular step between adjacent beams), e.g. `2*pi/num_points_per_rotation`. **Not** points per radian — a real dataset export mistakenly used points per radian (values like `572.96`), which downstream renderers silently mis-consumed as a near-zero angular step. | | Rotating LiDAR | When `lidar_type == "rotating"`, `rotation_type` must be `"clockwise"` or `"counter-clockwise"` | **Actor** | Check | Detail | |---|---| | `boxtype` | Must be one of: `vehicle`, `pedestrian`, `bicycle`, `sign`, `unknown` | **General (all models)** | Check | Detail | |---|---| | Required fields | Every required field must be present | | Field types | Values must match declared annotations | | Literal fields | Constrained to their declared set of values | These checks are enforced by `Scene.model_validate_json` automatically. ### Basic Validation ```python """ Basic MIMIC dataset validation script """ import json import numpy as np from pathlib import Path from typing import List, Tuple, Dict, Any import sys class MIMICValidator: def __init__(self, scene_path: Path): self.scene_path = Path(scene_path) self.scene_dir = self.scene_path.parent self.scene_data = None self.errors = [] self.warnings = [] def validate(self) -> bool: """Run full validation.""" print("=" * 70) print("MIMIC Dataset Validation") print("=" * 70) print(f"Scene: {self.scene_path}") print() # Step 1: Load JSON if not self._validate_json(): return False # Step 2: Check structure self._validate_structure() # Step 3: Check ego vehicle self._validate_ego_vehicle() # Step 4: Check sensors self._validate_sensors() # Step 5: Check actors self._validate_actors() # Step 6: Check file references self._validate_files() # Print results return self._print_results() def _validate_json(self) -> bool: """Validate JSON file exists and is readable.""" print("1. Checking JSON file...") if not self.scene_path.exists(): self.errors.append(f"File not found: {self.scene_path}") return False try: with open(self.scene_path) as f: self.scene_data = json.load(f) print(" ✓ JSON file valid") return True except json.JSONDecodeError as e: self.errors.append(f"Invalid JSON: {e}") return False def _validate_structure(self): """Validate basic scene structure.""" print("\n2. Checking scene structure...") required_fields = ["ego_vehicle", "actors"] for field in required_fields: if field not in self.scene_data: self.errors.append(f"Missing required field: {field}") else: print(f" ✓ Found {field}") # Check ego_vehicle structure ego = self.scene_data.get("ego_vehicle", {}) if not isinstance(ego, dict): self.errors.append("ego_vehicle must be object") return for field in ["trajectory", "sensors"]: if field not in ego: self.errors.append(f"Missing ego_vehicle.{field}") def _validate_ego_vehicle(self): """Validate ego vehicle data.""" print("\n3. Checking ego vehicle...") ego = self.scene_data.get("ego_vehicle", {}) traj = ego.get("trajectory", {}) if not traj: self.errors.append("Missing ego trajectory") return timestamps = traj.get("timestamps", []) poses = traj.get("object2world", []) if len(timestamps) == 0: self.errors.append("Empty ego trajectory") return if len(timestamps) != len(poses): self.errors.append( f"Timestamp/pose mismatch: {len(timestamps)} vs {len(poses)}" ) return print(f" ✓ Ego trajectory: {len(timestamps)} frames") # Check timestamps are sorted ts_array = np.array(timestamps) if not np.all(ts_array[:-1] <= ts_array[1:]): self.warnings.append("Timestamps not monotonically increasing") # Check poses are 4x4 for i, pose in enumerate(poses): if not isinstance(pose, list) or len(pose) != 4: self.errors.append(f"Pose {i}: not 4x4 matrix") continue for row in pose: if not isinstance(row, list) or len(row) != 4: self.errors.append(f"Pose {i}: not 4x4 matrix") # Check coordinate system self._check_coordinate_system(poses[0], "Ego vehicle") def _check_coordinate_system(self, pose: List[List[float]], name: str): """Check rotation-matrix validity without assuming axis directions.""" pose_array = np.array(pose) R = pose_array[:3, :3] det = np.linalg.det(R) if not np.isclose(det, 1.0, atol=0.01) or not np.allclose( R.T @ R, np.eye(3), atol=0.01 ): self.errors.append(f"{name}: rotation matrix is not orthonormal") def _validate_sensors(self): """Validate sensor data.""" print("\n4. Checking sensors...") sensors = self.scene_data.get("ego_vehicle", {}).get("sensors", {}) cameras = sensors.get("cameras", []) lidars = sensors.get("lidars", []) radars = sensors.get("radars", []) print(f" ✓ Cameras: {len(cameras)}") print(f" ✓ LiDARs: {len(lidars)}") print(f" ✓ Radars: {len(radars)}") if len(cameras) == 0: self.errors.append("No cameras found (at least one required)") if len(lidars) == 0: self.errors.append("No LiDARs found (at least one required)") # Check cameras for i, camera in enumerate(cameras): self._validate_camera(camera, i) # Check LiDARs for i, lidar in enumerate(lidars): self._validate_lidar(lidar, i) def _validate_camera(self, camera: Dict, idx: int): """Validate camera data.""" required = ["id", "name", "extrinsics", "intrinsics", "images_data"] for field in required: if field not in camera: self.errors.append(f"Camera {idx}: missing {field}") intrinsics = camera.get("intrinsics", {}) # Check focal lengths are positive fx = intrinsics.get("fx", 0) fy = intrinsics.get("fy", 0) if fx <= 0 or fy <= 0: self.errors.append(f"Camera {idx}: invalid focal length") # Check resolution width = intrinsics.get("width", 0) height = intrinsics.get("height", 0) if width <= 0 or height <= 0: self.errors.append(f"Camera {idx}: invalid resolution") # Check principal point cx = intrinsics.get("cx", 0) cy = intrinsics.get("cy", 0) if not (0 <= cx <= width and 0 <= cy <= height): self.warnings.append( f"Camera {idx}: principal point outside image bounds" ) def _validate_lidar(self, lidar: Dict, idx: int): """Validate LiDAR data.""" required = [ "id", "name", "extrinsics", "intrinsics", "lidar_pcds_data", "use_for_training", ] for field in required: if field not in lidar: self.errors.append(f"LiDAR {idx}: missing {field}") use_for_training = lidar.get("use_for_training") if not isinstance(use_for_training, bool): self.errors.append( f"LiDAR {idx}: use_for_training must be a boolean (true or false)" ) intrinsics = lidar.get("intrinsics", {}) # Check LiDAR type lidar_type = intrinsics.get("lidar_type") if lidar_type not in ["rotating", "solid-state"]: self.errors.append(f"LiDAR {idx}: invalid lidar_type") # Check rotation type if rotating if lidar_type == "rotating": rotation_type = intrinsics.get("rotation_type") if rotation_type not in ["clockwise", "counter-clockwise"]: self.errors.append(f"LiDAR {idx}: invalid rotation_type") # Check FOV v_fov = intrinsics.get("vertical_fov", 0) h_fov = intrinsics.get("horizontal_fov", 0) if v_fov <= 0 or h_fov <= 0: self.errors.append(f"LiDAR {idx}: invalid FOV") # Check range max_range = intrinsics.get("max_range", 0) min_range = intrinsics.get("min_range", 0) if max_range <= 0 or min_range < 0 or min_range >= max_range: self.errors.append(f"LiDAR {idx}: invalid range") def _validate_actors(self): """Validate actor annotations.""" print("\n5. Checking actors...") actors = self.scene_data.get("actors", []) if len(actors) == 0: self.warnings.append("No actors in scene") return print(f" ✓ Actors: {len(actors)}") for i, actor in enumerate(actors): self._validate_actor(actor, i) def _validate_actor(self, actor: Dict, idx: int): """Validate single actor.""" required = ["id", "type", "dimensions", "trajectory"] for field in required: if field not in actor: self.errors.append(f"Actor {idx}: missing {field}") # Check type actor_type = actor.get("type") valid_types = ["vehicle", "pedestrian", "cyclist", "other"] if actor_type not in valid_types: self.errors.append(f"Actor {idx}: invalid type '{actor_type}'") # Check dimensions dims = actor.get("dimensions", {}) for field in ["length", "width", "height"]: value = dims.get(field, 0) if value <= 0: self.errors.append(f"Actor {idx}: invalid dimension {field}") # Check trajectory traj = actor.get("trajectory", {}) traj_ts = traj.get("timestamps", []) if len(traj_ts) == 0: self.errors.append(f"Actor {idx}: empty trajectory") def _validate_files(self): """Validate all file references exist.""" print("\n6. Checking file references...") missing_files = [] # Check camera images for camera in self.scene_data.get("ego_vehicle", {}).get("sensors", {}).get("cameras", []): for img_data in camera.get("images_data", []): filepath = img_data.get("filepath") if filepath: full_path = self.scene_dir / filepath if not full_path.exists(): missing_files.append(filepath) # Check LiDAR point clouds for lidar in self.scene_data.get("ego_vehicle", {}).get("sensors", {}).get("lidars", []): for pcd_data in lidar.get("lidar_pcds_data", []): filepath = pcd_data.get("filepath") if filepath: full_path = self.scene_dir / filepath if not full_path.exists(): missing_files.append(filepath) if missing_files: self.warnings.append(f"Missing {len(missing_files)} files") for f in missing_files[:5]: # Show first 5 self.warnings.append(f" - {f}") if len(missing_files) > 5: self.warnings.append(f" ... and {len(missing_files) - 5} more") else: print(" ✓ All file references found") def _print_results(self) -> bool: """Print validation results.""" print("\n" + "=" * 70) print("VALIDATION RESULTS") print("=" * 70) if self.errors: print(f"\n❌ ERRORS ({len(self.errors)}):") for error in self.errors: print(f" - {error}") if self.warnings: print(f"\n⚠️ WARNINGS ({len(self.warnings)}):") for warning in self.warnings: print(f" - {warning}") if not self.errors: print("\n✓ VALIDATION PASSED") print(f" Errors: 0") print(f" Warnings: {len(self.warnings)}") return True else: print(f"\n✗ VALIDATION FAILED") print(f" Errors: {len(self.errors)}") print(f" Warnings: {len(self.warnings)}") return False def main(): """Run validation on command-line argument.""" if len(sys.argv) < 2: print("Usage: uv run python validate_mimic_dataset.py ") sys.exit(1) scene_path = Path(sys.argv[1]) validator = MIMICValidator(scene_path) success = validator.validate() sys.exit(0 if success else 1) if __name__ == "__main__": main() ``` ### Usage ```bash # Save as validate_mimic_dataset.py uv run python validate_mimic_dataset.py /path/to/scene.json ``` Example output: ``` ====================================================================== MIMIC Dataset Validation ====================================================================== Scene: /path/to/scene.json 1. Checking JSON file... ✓ JSON file valid 2. Checking scene structure... ✓ Found ego_vehicle ✓ Found actors 3. Checking ego vehicle... ✓ Ego trajectory: 100 frames 4. Checking sensors... ✓ Cameras: 5 ✓ LiDARs: 1 ✓ Radars: 0 5. Checking actors... ✓ Actors: 42 6. Checking file references... ✓ All file references found ====================================================================== VALIDATION RESULTS ====================================================================== ✓ VALIDATION PASSED Errors: 0 Warnings: 0 ``` --- ## Deep Validation: LiDAR Intensity ### Check Intensity Normalization In a LiDAR PLY, `i = -1` marks a ray with no valid return. The SDK point-cloud loader removes those rows by default, so it is not a placeholder for intensity that the source did not measure. If the source has no intensity channel, use a documented value in `[0, 1]` (such as `0.0`) instead. The standalone validator above samples PLY files through the SDK loader and reports a sampled cloud that becomes empty. ```python import numpy as np from plyfile import PlyData def validate_lidar_intensity(ply_path: str): """Check if LiDAR intensity is properly normalized.""" ply_data = PlyData.read(ply_path) intensities = ply_data['vertex']['i'] # Check statistics valid_mask = intensities >= 0 # -1 indicates missing valid_intensities = intensities[valid_mask] print(f"Point cloud: {len(intensities)} points") print(f" Valid points: {np.sum(valid_mask)}") print(f" Missing points: {np.sum(~valid_mask)}") if len(valid_intensities) > 0: print(f" Intensity range: [{valid_intensities.min():.3f}, {valid_intensities.max():.3f}]") print(f" Mean intensity: {valid_intensities.mean():.3f}") print(f" Std intensity: {valid_intensities.std():.3f}") # Check if in [0, 1] range if valid_intensities.min() >= 0.0 and valid_intensities.max() <= 1.0: print("✓ Intensity is normalized") else: print("✗ Intensity NOT normalized (should be in [0, 1])") # Check for missing indicator (-1) if np.any(intensities == -1): print(f"✓ Found {np.sum(intensities == -1)} missing point indicators (-1)") # Usage validate_lidar_intensity("lidar/top_lidar/0000.ply") ``` ### Check Point Cloud Format ```python import numpy as np from plyfile import PlyData def validate_ply_format(ply_path: str): """Verify PLY has correct 6-column format.""" ply_data = PlyData.read(ply_path) required_fields = ['x', 'y', 'z', 'i', 't', 'c'] vertex = ply_data['vertex'] print(f"PLY file: {ply_path}") print(f" Points: {len(vertex)}") print(f" Fields: {vertex.dtype.names}") # Check all required fields all_present = True for field in required_fields: if field in vertex.dtype.names: print(f" ✓ {field}") else: print(f" ✗ MISSING: {field}") all_present = False if all_present: print("✓ PLY format valid") else: print("✗ PLY format invalid") return all_present # Usage validate_ply_format("lidar/top_lidar/0000.ply") ``` --- ## Validation Best Practices 1. **Validate early**: Check after each dataset section 2. **Use type hints**: Catch errors before processing 3. **Log issues**: Keep detailed logs of validation results 4. **Test on sample**: Validate one scene before converting full dataset 5. **Automate checks**: Run validation in CI/CD pipeline --- ## Common Issues and Fixes | Issue | Cause | Fix | |-------|-------|-----| | "Invalid intensity values" | Not normalized | Apply log-space normalization (see guide) | | "Coordinate system error" | Wrong transform | Verify axes point in expected directions | | "File not found" | Relative path issue | Use paths relative to scene.json location | | "PLY format error" | Missing columns | Ensure 6-column format: x, y, z, i, t, c | | "Timestamp mismatch" | Sensor data/pose mismatch | Ensure each sensor's timestamp count and order match its own trajectory poses | --- ## See Also - [conversion-guide.md](conversion-guide.md) - Full conversion guide - [quick-reference.md](quick-reference.md) - Code snippets - [example_converter.py](example_converter.py) - Example converter