MIMIC Dataset Validation Guide¶
Overview¶
This guide helps you validate that your MIMIC-converted dataset is correctly formatted.
Validation checks:
✅ JSON structure and schema compliance
✅ File references and integrity
✅ Transform shape and rotation-matrix validity
✅ Sensor data completeness
✅ Timestamp consistency
✅ Data type correctness
Validation results are split into errors and warnings. Errors indicate broken schema or data integrity and make validation fail. Warnings are advisory heuristics; they do not change the exit status. Axis-direction assumptions are profile-specific and must not be treated as universal coordinate-convention checks.
Quick Validation Checklist¶
Use this checklist before submitting your dataset:
Coordinate Systems¶
[ ] Camera extrinsics are camera-to-vehicle (not vehicle-to-camera)
[ ] Confirm the source and target frame conventions from authoritative metadata
[ ] Confirm vehicle axes against the declared convention, not world-axis signs
[ ] All matrices are 4×4 homogeneous transformations
Sensors¶
[ ] Sensors required by the target workflow are present
[ ] Input dataset contains at least one camera
[ ] Input dataset contains at least one LiDAR
[ ] All camera images exist and are readable
[ ] All LiDAR point clouds are valid PLY files
[ ] LiDAR intensity is normalized to [0, 1] or -1
[ ] Each sensor data list aligns with its own trajectory timestamps and poses
[ ] Per-sensor frame counts may differ; no sensor was padded or resampled to another sensor’s timestamps
[ ] Each sensor pose equals
ego_vehicle_pose @ sensor.extrinsicsat the matching timestamp[ ] Camera intrinsics have positive focal lengths
[ ] LiDAR vertical resolution is properly specified
Annotations¶
[ ] Actor types are valid (vehicle, pedestrian, cyclist, other)
[ ] Actor dimensions are positive and reasonable
[ ] Actor trajectories have valid timestamps
[ ] 2D bounding boxes don’t exceed image dimensions
[ ] All actors have at least one annotation frame
Files & Format¶
[ ]
scene.jsonis valid JSON[ ] All file paths in JSON are relative and exist
[ ] No missing sensor data frames
[ ] PLY files have correct 6-column format
[ ] File permissions allow reading
Reconstruction Readiness¶
Schema validation checks structure and field constraints; it cannot determine whether images are real captures or whether the sequence is suitable for a particular reconstruction workflow. Before reconstruction, verify that camera images are readable photographs, camera poses and intrinsics are usable, and the sequence is continuous. If the workflow also uses LiDAR, confirm camera frames have LiDAR coverage within that workflow’s matching tolerance.
The standalone program below includes the complete customer-usable validation flow: schema and file checks, sampled image readability, trajectory and camera checks, sampled LiDAR loading, transform validity, optional camera/LiDAR timing, and a warning for unusually uniform images. Required shape, timestamp alignment, file integrity, and invalid rigid transforms are hard errors. Motion jumps, unusual image content, and sensor-coverage gaps are advisory warnings.
Choose general for aerial, robotics, or unknown world-frame conventions. The
optional ground-vehicle profile adds an advisory Z-up heuristic. Use it only
after confirming that the source world frame is aligned with the expected
ground-vehicle convention; the heuristic still warns rather than failing
validation. The script uses the public
mimic_data_api package. The image score is a heuristic, not proof of image
content; inspect flagged samples yourself.
import argparse
import random
import sys
from pathlib import Path
import numpy as np
from PIL import Image
from mimic_data_api.file_utils.lidar_ply_handler import LidarPointCloudIOHandler
from mimic_data_api.scene import Scene
def flat_colour_fraction(path: Path) -> float:
"""Estimate how much of an image is one quantized colour."""
with Image.open(path) as source:
image = source.convert("RGB")
image.thumbnail((320, 320))
pixels = np.asarray(image, dtype=np.uint8).reshape(-1, 3) // 8
codes = (pixels[:, 0] << 10) | (pixels[:, 1] << 5) | pixels[:, 2]
return float(np.bincount(codes).max() / len(codes))
def validate_scene(
scene_path: Path,
coordinate_profile: str = "general",
frame_convention_confirmed: bool = False,
) -> int:
if coordinate_profile not in {"general", "ground-vehicle"}:
raise ValueError("coordinate_profile must be 'general' or 'ground-vehicle'")
if coordinate_profile == "ground-vehicle" and not frame_convention_confirmed:
raise ValueError(
"Confirm the source world-frame convention before using the ground-vehicle profile"
)
if scene_path.is_dir():
scene_json = scene_path / "scene.json"
scene_dir = scene_path
else:
scene_json = scene_path
scene_dir = scene_path.parent
errors = []
warnings = []
try:
scene = Scene.model_validate_json(scene_json.read_text())
except Exception as exc:
print(f"Schema validation failed: {exc}")
return 1
sensors = scene.ego_vehicle.sensors
cameras = sensors.cameras
lidars = sensors.lidars or []
ego_trajectory = scene.ego_vehicle.trajectory
ego_timestamps = np.asarray(ego_trajectory.timestamps, dtype=float)
ego_poses = np.asarray(ego_trajectory.object2world, dtype=float)
if not np.all(np.diff(ego_timestamps) >= 0):
errors.append("Ego trajectory timestamps are not monotonic")
if len(ego_poses) > 1:
jumps = np.linalg.norm(np.diff(ego_poses[:, :3, 3], axis=0), axis=1)
if float(jumps.max()) > 5.0:
warnings.append(
f"Ego trajectory has a {float(jumps.max()):.2f} m frame-to-frame jump"
)
ego_rotations = ego_poses[:, :3, :3]
rotation_determinants = np.linalg.det(ego_rotations)
orthogonality = np.matmul(ego_rotations.swapaxes(-1, -2), ego_rotations)
invalid_rotations = (np.abs(rotation_determinants - 1.0) > 0.01) | (
np.max(np.abs(orthogonality - np.eye(3)), axis=(1, 2)) > 0.01
)
if np.any(invalid_rotations):
errors.append(
f"{int(np.sum(invalid_rotations))} ego pose rotation(s) are not orthonormal rotations"
)
if not cameras:
errors.append("Scene has no cameras")
for camera in cameras:
intrinsics = camera.intrinsics
if intrinsics.fx <= 0 or intrinsics.fy <= 0:
errors.append(f"Camera {camera.name}: focal lengths must be positive")
if not (0 < intrinsics.cx < intrinsics.width and 0 < intrinsics.cy < intrinsics.height):
warnings.append(f"Camera {camera.name}: principal point is outside the image")
rotation = np.asarray(camera.extrinsics, dtype=float)[:3, :3]
rotation_orthogonality = rotation.T @ rotation
if (
abs(float(np.linalg.det(rotation)) - 1.0) > 0.01
or not np.allclose(rotation_orthogonality, np.eye(3), atol=0.01)
):
errors.append(
f"Camera {camera.name}: extrinsic rotation is not orthonormal"
)
if not camera.images_data:
errors.append(f"Camera {camera.name}: no images")
continue
available_images = []
for image_data in camera.images_data:
image_path = scene_dir / image_data.filepath
if not image_path.is_file():
errors.append(f"Missing image: {image_data.filepath}")
else:
available_images.append((image_data, image_path))
sample = random.sample(available_images, min(5, len(available_images)))
fractions = []
for image_data, image_path in sample:
try:
with Image.open(image_path) as image:
image.verify()
fractions.append(flat_colour_fraction(image_path))
except Exception as exc:
errors.append(f"Unreadable image {image_data.filepath}: {exc}")
if fractions:
median_fraction = float(np.median(fractions))
if median_fraction >= 0.40:
warnings.append(
f"Camera {camera.name}: sampled images are unusually uniform "
f"({median_fraction:.0%}); inspect for plots or renderings"
)
lidar_handler = LidarPointCloudIOHandler()
for lidar in lidars:
if not isinstance(lidar.use_for_training, bool):
errors.append(f"LiDAR {lidar.name}: use_for_training must be boolean")
horizontal_resolution = lidar.intrinsics.horizontal_resolution
if horizontal_resolution is not None and not (0 < horizontal_resolution < 0.1):
errors.append(
f"LiDAR {lidar.name}: horizontal_resolution should be radians per point"
)
files = []
for point_cloud in lidar.lidar_pcds_data:
point_cloud_path = scene_dir / point_cloud.filepath
if not point_cloud_path.is_file():
errors.append(f"Missing point cloud: {point_cloud.filepath}")
else:
files.append(point_cloud_path)
for point_cloud_path in random.sample(files, min(5, len(files))):
try:
point_cloud = lidar_handler.load(point_cloud_path)
if len(point_cloud.points) == 0:
errors.append(
f"Point cloud has no usable points after SDK loading: {point_cloud_path}; "
"check whether every point has intensity -1 (missing return)"
)
except Exception as exc:
errors.append(f"Cannot read point cloud {point_cloud_path}: {exc}")
if lidars:
lidar_timestamp_arrays = [
np.asarray(lidar.trajectory.timestamps, dtype=float)
for lidar in lidars
if len(lidar.trajectory.timestamps)
]
if lidar_timestamp_arrays:
lidar_timestamps = np.sort(np.concatenate(lidar_timestamp_arrays))
for camera in cameras:
camera_timestamps = np.asarray(camera.trajectory.timestamps, dtype=float)
if camera_timestamps.size < 2:
continue
tolerance = 2.0 * float(np.percentile(np.diff(camera_timestamps), 90))
right = np.searchsorted(lidar_timestamps, camera_timestamps)
left = np.clip(right - 1, 0, len(lidar_timestamps) - 1)
right = np.clip(right, 0, len(lidar_timestamps) - 1)
nearest = np.minimum(
np.abs(camera_timestamps - lidar_timestamps[left]),
np.abs(camera_timestamps - lidar_timestamps[right]),
)
if np.any(nearest > tolerance):
warnings.append(
f"Camera {camera.name}: {int(np.sum(nearest > tolerance))} "
f"frame(s) have no LiDAR sweep within {tolerance:.4f}s; "
"reconstruction preprocessing may fail for these frames"
)
else:
warnings.append("LiDAR temporal coverage could not be checked: no LiDAR poses")
if coordinate_profile == "ground-vehicle" and len(ego_poses):
middle_pose = ego_poses[len(ego_poses) // 2]
if middle_pose[2, 2] < 0:
warnings.append(
"Ground-vehicle profile: vehicle Z axis points downward at the sampled pose; "
"verify the confirmed source frame convention"
)
for message in errors:
print(f"ERROR: {message}")
for message in warnings:
print(f"WARNING: {message}")
if errors:
return 1
print(f"Validated {scene_json}: {len(warnings)} warning(s)")
return 0
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Validate a MIMIC scene directory or scene.json"
)
parser.add_argument(
"scene", type=Path, help="Scene directory or path to scene.json"
)
parser.add_argument(
"--coordinate-profile",
choices=("general", "ground-vehicle"),
default="general",
help="Apply advisory axis-direction checks for a confirmed ground-vehicle frame",
)
parser.add_argument(
"--confirm-frame-convention",
action="store_true",
help="Confirm the source world frame before using the ground-vehicle profile",
)
args = parser.parse_args()
try:
result = validate_scene(
args.scene,
coordinate_profile=args.coordinate_profile,
frame_convention_confirmed=args.confirm_frame_convention,
)
except ValueError as exc:
parser.error(str(exc))
sys.exit(result)
Save the complete example as validate_scene.py and run it with either a scene
directory or the scene.json file:
python validate_scene.py /path/to/scene
For data with a confirmed ground-vehicle world frame, enable the advisory direction check explicitly:
python validate_scene.py /path/to/scene \\
--coordinate-profile ground-vehicle --confirm-frame-convention
The image-uniformity warning is not a pass/fail test for photographic content. Lighting, blank walls, night scenes, and other valid images can also produce a high score. Always inspect the sampled images, and tune the optional timing check to the target workflow’s actual sensor-matching policy.
Validation Script¶
Installation¶
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv .venv
source .venv/bin/activate
uv pip install jsonschema numpy pillow plyfile
Automated schema validation via mimic_data_api¶
If mimic_data_api is installed, you can validate any scene.json against the full
data model in one call:
from mimic_data_api.scene import Scene
text = open("scene.json").read()
scene = Scene.model_validate_json(text) # raises ValidationError on failure
model_validate_json runs every Pydantic and numpydantic validator defined in
the data model. The complete set of automatic checks is:
WorldTrajectory (ego vehicle, every camera, every lidar, every actor)
Check |
Detail |
|---|---|
Array shape — timestamps |
1-D float array |
Array shape — object2world |
(N, 4, 4) float array |
Length consistency |
|
Angular velocity (optional) |
shape (N, 3); |
Linear velocity (optional) |
shape (N, 3); |
Camera
Check |
Detail |
|---|---|
|
Must be one of: |
Image timestamps |
|
Image file extension |
|
Timestamp alignment |
List of image timestamps must exactly match |
LiDAR
Check |
Detail |
|---|---|
|
Required and must be a boolean: |
|
Exactly one of |
|
Radians per point (angular step between adjacent beams), e.g. |
Rotating LiDAR |
When |
Actor
Check |
Detail |
|---|---|
|
Must be one of: |
General (all models)
Check |
Detail |
|---|---|
Required fields |
Every required field must be present |
Field types |
Values must match declared annotations |
Literal fields |
Constrained to their declared set of values |
These checks are enforced by Scene.model_validate_json automatically.
Basic Validation¶
"""
Basic MIMIC dataset validation script
"""
import json
import numpy as np
from pathlib import Path
from typing import List, Tuple, Dict, Any
import sys
class MIMICValidator:
def __init__(self, scene_path: Path):
self.scene_path = Path(scene_path)
self.scene_dir = self.scene_path.parent
self.scene_data = None
self.errors = []
self.warnings = []
def validate(self) -> bool:
"""Run full validation."""
print("=" * 70)
print("MIMIC Dataset Validation")
print("=" * 70)
print(f"Scene: {self.scene_path}")
print()
# Step 1: Load JSON
if not self._validate_json():
return False
# Step 2: Check structure
self._validate_structure()
# Step 3: Check ego vehicle
self._validate_ego_vehicle()
# Step 4: Check sensors
self._validate_sensors()
# Step 5: Check actors
self._validate_actors()
# Step 6: Check file references
self._validate_files()
# Print results
return self._print_results()
def _validate_json(self) -> bool:
"""Validate JSON file exists and is readable."""
print("1. Checking JSON file...")
if not self.scene_path.exists():
self.errors.append(f"File not found: {self.scene_path}")
return False
try:
with open(self.scene_path) as f:
self.scene_data = json.load(f)
print(" ✓ JSON file valid")
return True
except json.JSONDecodeError as e:
self.errors.append(f"Invalid JSON: {e}")
return False
def _validate_structure(self):
"""Validate basic scene structure."""
print("\n2. Checking scene structure...")
required_fields = ["ego_vehicle", "actors"]
for field in required_fields:
if field not in self.scene_data:
self.errors.append(f"Missing required field: {field}")
else:
print(f" ✓ Found {field}")
# Check ego_vehicle structure
ego = self.scene_data.get("ego_vehicle", {})
if not isinstance(ego, dict):
self.errors.append("ego_vehicle must be object")
return
for field in ["trajectory", "sensors"]:
if field not in ego:
self.errors.append(f"Missing ego_vehicle.{field}")
def _validate_ego_vehicle(self):
"""Validate ego vehicle data."""
print("\n3. Checking ego vehicle...")
ego = self.scene_data.get("ego_vehicle", {})
traj = ego.get("trajectory", {})
if not traj:
self.errors.append("Missing ego trajectory")
return
timestamps = traj.get("timestamps", [])
poses = traj.get("object2world", [])
if len(timestamps) == 0:
self.errors.append("Empty ego trajectory")
return
if len(timestamps) != len(poses):
self.errors.append(
f"Timestamp/pose mismatch: {len(timestamps)} vs {len(poses)}"
)
return
print(f" ✓ Ego trajectory: {len(timestamps)} frames")
# Check timestamps are sorted
ts_array = np.array(timestamps)
if not np.all(ts_array[:-1] <= ts_array[1:]):
self.warnings.append("Timestamps not monotonically increasing")
# Check poses are 4x4
for i, pose in enumerate(poses):
if not isinstance(pose, list) or len(pose) != 4:
self.errors.append(f"Pose {i}: not 4x4 matrix")
continue
for row in pose:
if not isinstance(row, list) or len(row) != 4:
self.errors.append(f"Pose {i}: not 4x4 matrix")
# Check coordinate system
self._check_coordinate_system(poses[0], "Ego vehicle")
def _check_coordinate_system(self, pose: List[List[float]], name: str):
"""Check rotation-matrix validity without assuming axis directions."""
pose_array = np.array(pose)
R = pose_array[:3, :3]
det = np.linalg.det(R)
if not np.isclose(det, 1.0, atol=0.01) or not np.allclose(
R.T @ R, np.eye(3), atol=0.01
):
self.errors.append(f"{name}: rotation matrix is not orthonormal")
def _validate_sensors(self):
"""Validate sensor data."""
print("\n4. Checking sensors...")
sensors = self.scene_data.get("ego_vehicle", {}).get("sensors", {})
cameras = sensors.get("cameras", [])
lidars = sensors.get("lidars", [])
radars = sensors.get("radars", [])
print(f" ✓ Cameras: {len(cameras)}")
print(f" ✓ LiDARs: {len(lidars)}")
print(f" ✓ Radars: {len(radars)}")
if len(cameras) == 0:
self.errors.append("No cameras found (at least one required)")
if len(lidars) == 0:
self.errors.append("No LiDARs found (at least one required)")
# Check cameras
for i, camera in enumerate(cameras):
self._validate_camera(camera, i)
# Check LiDARs
for i, lidar in enumerate(lidars):
self._validate_lidar(lidar, i)
def _validate_camera(self, camera: Dict, idx: int):
"""Validate camera data."""
required = ["id", "name", "extrinsics", "intrinsics", "images_data"]
for field in required:
if field not in camera:
self.errors.append(f"Camera {idx}: missing {field}")
intrinsics = camera.get("intrinsics", {})
# Check focal lengths are positive
fx = intrinsics.get("fx", 0)
fy = intrinsics.get("fy", 0)
if fx <= 0 or fy <= 0:
self.errors.append(f"Camera {idx}: invalid focal length")
# Check resolution
width = intrinsics.get("width", 0)
height = intrinsics.get("height", 0)
if width <= 0 or height <= 0:
self.errors.append(f"Camera {idx}: invalid resolution")
# Check principal point
cx = intrinsics.get("cx", 0)
cy = intrinsics.get("cy", 0)
if not (0 <= cx <= width and 0 <= cy <= height):
self.warnings.append(
f"Camera {idx}: principal point outside image bounds"
)
def _validate_lidar(self, lidar: Dict, idx: int):
"""Validate LiDAR data."""
required = [
"id",
"name",
"extrinsics",
"intrinsics",
"lidar_pcds_data",
"use_for_training",
]
for field in required:
if field not in lidar:
self.errors.append(f"LiDAR {idx}: missing {field}")
use_for_training = lidar.get("use_for_training")
if not isinstance(use_for_training, bool):
self.errors.append(
f"LiDAR {idx}: use_for_training must be a boolean (true or false)"
)
intrinsics = lidar.get("intrinsics", {})
# Check LiDAR type
lidar_type = intrinsics.get("lidar_type")
if lidar_type not in ["rotating", "solid-state"]:
self.errors.append(f"LiDAR {idx}: invalid lidar_type")
# Check rotation type if rotating
if lidar_type == "rotating":
rotation_type = intrinsics.get("rotation_type")
if rotation_type not in ["clockwise", "counter-clockwise"]:
self.errors.append(f"LiDAR {idx}: invalid rotation_type")
# Check FOV
v_fov = intrinsics.get("vertical_fov", 0)
h_fov = intrinsics.get("horizontal_fov", 0)
if v_fov <= 0 or h_fov <= 0:
self.errors.append(f"LiDAR {idx}: invalid FOV")
# Check range
max_range = intrinsics.get("max_range", 0)
min_range = intrinsics.get("min_range", 0)
if max_range <= 0 or min_range < 0 or min_range >= max_range:
self.errors.append(f"LiDAR {idx}: invalid range")
def _validate_actors(self):
"""Validate actor annotations."""
print("\n5. Checking actors...")
actors = self.scene_data.get("actors", [])
if len(actors) == 0:
self.warnings.append("No actors in scene")
return
print(f" ✓ Actors: {len(actors)}")
for i, actor in enumerate(actors):
self._validate_actor(actor, i)
def _validate_actor(self, actor: Dict, idx: int):
"""Validate single actor."""
required = ["id", "type", "dimensions", "trajectory"]
for field in required:
if field not in actor:
self.errors.append(f"Actor {idx}: missing {field}")
# Check type
actor_type = actor.get("type")
valid_types = ["vehicle", "pedestrian", "cyclist", "other"]
if actor_type not in valid_types:
self.errors.append(f"Actor {idx}: invalid type '{actor_type}'")
# Check dimensions
dims = actor.get("dimensions", {})
for field in ["length", "width", "height"]:
value = dims.get(field, 0)
if value <= 0:
self.errors.append(f"Actor {idx}: invalid dimension {field}")
# Check trajectory
traj = actor.get("trajectory", {})
traj_ts = traj.get("timestamps", [])
if len(traj_ts) == 0:
self.errors.append(f"Actor {idx}: empty trajectory")
def _validate_files(self):
"""Validate all file references exist."""
print("\n6. Checking file references...")
missing_files = []
# Check camera images
for camera in self.scene_data.get("ego_vehicle", {}).get("sensors", {}).get("cameras", []):
for img_data in camera.get("images_data", []):
filepath = img_data.get("filepath")
if filepath:
full_path = self.scene_dir / filepath
if not full_path.exists():
missing_files.append(filepath)
# Check LiDAR point clouds
for lidar in self.scene_data.get("ego_vehicle", {}).get("sensors", {}).get("lidars", []):
for pcd_data in lidar.get("lidar_pcds_data", []):
filepath = pcd_data.get("filepath")
if filepath:
full_path = self.scene_dir / filepath
if not full_path.exists():
missing_files.append(filepath)
if missing_files:
self.warnings.append(f"Missing {len(missing_files)} files")
for f in missing_files[:5]: # Show first 5
self.warnings.append(f" - {f}")
if len(missing_files) > 5:
self.warnings.append(f" ... and {len(missing_files) - 5} more")
else:
print(" ✓ All file references found")
def _print_results(self) -> bool:
"""Print validation results."""
print("\n" + "=" * 70)
print("VALIDATION RESULTS")
print("=" * 70)
if self.errors:
print(f"\n❌ ERRORS ({len(self.errors)}):")
for error in self.errors:
print(f" - {error}")
if self.warnings:
print(f"\n⚠️ WARNINGS ({len(self.warnings)}):")
for warning in self.warnings:
print(f" - {warning}")
if not self.errors:
print("\n✓ VALIDATION PASSED")
print(f" Errors: 0")
print(f" Warnings: {len(self.warnings)}")
return True
else:
print(f"\n✗ VALIDATION FAILED")
print(f" Errors: {len(self.errors)}")
print(f" Warnings: {len(self.warnings)}")
return False
def main():
"""Run validation on command-line argument."""
if len(sys.argv) < 2:
print("Usage: uv run python validate_mimic_dataset.py <path_to_scene.json>")
sys.exit(1)
scene_path = Path(sys.argv[1])
validator = MIMICValidator(scene_path)
success = validator.validate()
sys.exit(0 if success else 1)
if __name__ == "__main__":
main()
Usage¶
# Save as validate_mimic_dataset.py
uv run python validate_mimic_dataset.py /path/to/scene.json
Example output:
======================================================================
MIMIC Dataset Validation
======================================================================
Scene: /path/to/scene.json
1. Checking JSON file...
✓ JSON file valid
2. Checking scene structure...
✓ Found ego_vehicle
✓ Found actors
3. Checking ego vehicle...
✓ Ego trajectory: 100 frames
4. Checking sensors...
✓ Cameras: 5
✓ LiDARs: 1
✓ Radars: 0
5. Checking actors...
✓ Actors: 42
6. Checking file references...
✓ All file references found
======================================================================
VALIDATION RESULTS
======================================================================
✓ VALIDATION PASSED
Errors: 0
Warnings: 0
Deep Validation: LiDAR Intensity¶
Check Intensity Normalization¶
In a LiDAR PLY, i = -1 marks a ray with no valid return. The SDK point-cloud
loader removes those rows by default, so it is not a placeholder for intensity
that the source did not measure. If the source has no intensity channel, use a
documented value in [0, 1] (such as 0.0) instead. The standalone validator
above samples PLY files through the SDK loader and reports a sampled cloud that
becomes empty.
import numpy as np
from plyfile import PlyData
def validate_lidar_intensity(ply_path: str):
"""Check if LiDAR intensity is properly normalized."""
ply_data = PlyData.read(ply_path)
intensities = ply_data['vertex']['i']
# Check statistics
valid_mask = intensities >= 0 # -1 indicates missing
valid_intensities = intensities[valid_mask]
print(f"Point cloud: {len(intensities)} points")
print(f" Valid points: {np.sum(valid_mask)}")
print(f" Missing points: {np.sum(~valid_mask)}")
if len(valid_intensities) > 0:
print(f" Intensity range: [{valid_intensities.min():.3f}, {valid_intensities.max():.3f}]")
print(f" Mean intensity: {valid_intensities.mean():.3f}")
print(f" Std intensity: {valid_intensities.std():.3f}")
# Check if in [0, 1] range
if valid_intensities.min() >= 0.0 and valid_intensities.max() <= 1.0:
print("✓ Intensity is normalized")
else:
print("✗ Intensity NOT normalized (should be in [0, 1])")
# Check for missing indicator (-1)
if np.any(intensities == -1):
print(f"✓ Found {np.sum(intensities == -1)} missing point indicators (-1)")
# Usage
validate_lidar_intensity("lidar/top_lidar/0000.ply")
Check Point Cloud Format¶
import numpy as np
from plyfile import PlyData
def validate_ply_format(ply_path: str):
"""Verify PLY has correct 6-column format."""
ply_data = PlyData.read(ply_path)
required_fields = ['x', 'y', 'z', 'i', 't', 'c']
vertex = ply_data['vertex']
print(f"PLY file: {ply_path}")
print(f" Points: {len(vertex)}")
print(f" Fields: {vertex.dtype.names}")
# Check all required fields
all_present = True
for field in required_fields:
if field in vertex.dtype.names:
print(f" ✓ {field}")
else:
print(f" ✗ MISSING: {field}")
all_present = False
if all_present:
print("✓ PLY format valid")
else:
print("✗ PLY format invalid")
return all_present
# Usage
validate_ply_format("lidar/top_lidar/0000.ply")
Validation Best Practices¶
Validate early: Check after each dataset section
Use type hints: Catch errors before processing
Log issues: Keep detailed logs of validation results
Test on sample: Validate one scene before converting full dataset
Automate checks: Run validation in CI/CD pipeline
Common Issues and Fixes¶
Issue |
Cause |
Fix |
|---|---|---|
“Invalid intensity values” |
Not normalized |
Apply log-space normalization (see guide) |
“Coordinate system error” |
Wrong transform |
Verify axes point in expected directions |
“File not found” |
Relative path issue |
Use paths relative to scene.json location |
“PLY format error” |
Missing columns |
Ensure 6-column format: x, y, z, i, t, c |
“Timestamp mismatch” |
Sensor data/pose mismatch |
Ensure each sensor’s timestamp count and order match its own trajectory poses |
See Also¶
conversion-guide.md - Full conversion guide
quick-reference.md - Code snippets
example_converter.py - Example converter