# MIMIC Data API - Quick Reference ## Installation ```bash curl -LsSf https://astral.sh/uv/install.sh | sh uv venv --python 3.10 .venv source .venv/bin/activate uv pip install qvyon-mimic-data-api ``` For the TestPyPI release, use a clean environment and public PyPI for third-party dependencies: ```bash uv pip install \ --index-url https://test.pypi.org/simple/ \ --extra-index-url https://pypi.org/simple/ \ qvyon-mimic-filesystem qvyon-mimic-data-api ``` Use a fresh Python 3.10 ``.venv`` for each SDK release. Install the converter packages from TestPyPI with the same index settings shown above. For WOD, add the Waymo runtime with ``--no-deps`` followed by ``dacite==1.8.1`` and ``typeguard==4.4.4`` because its metadata pins conflict with the shared SDK dependencies. - Verify converter commands after install: ```bash command -v kitti360-convert command -v osdar23-convert command -v nuscene-convert command -v wod-convert command -v pandaset-convert ``` ## Core Concepts ### Scene Hierarchy ``` Scene ├── ego_vehicle: EgoVehicle │ ├── trajectory: WorldTrajectory │ └── sensors: Sensors │ ├── cameras: list[Camera] │ ├── lidars: list[Lidar] │ └── radars: list[Radar] └── actors: list[Actor] ``` ### Coordinate Systems **Camera (OpenCV)**: X-right, Y-down, Z-forward **Vehicle (ISO)**: X-forward, Y-left, Z-up ## Common Tasks ### 1. Create a Scene ```python import numpy as np from mimic_data_api.scene import Scene, EgoVehicle, Sensors from mimic_data_api.camera import Camera, CameraIntrinsics, DistortionParameters, ImageData from mimic_data_api.lidar import Lidar, LidarIntrinsics, VerticalResolution, LidarPointCloudData from mimic_data_api.trajectory_numpy import WorldTrajectory # Trajectory timestamps = np.array([0.0, 0.1, 0.2]) poses = np.eye(4).reshape(1, 4, 4) poses = np.repeat(poses, len(timestamps), axis=0) ego_traj = WorldTrajectory(timestamps=timestamps, object2world=poses) # A sensor trajectory is sensor-to-world, not the raw ego trajectory. camera_extrinsics = np.eye(4) camera_poses = np.stack([ ego_pose @ camera_extrinsics for ego_pose in ego_traj.object2world ]) camera_traj = WorldTrajectory( timestamps=timestamps, object2world=camera_poses, ) # Camera camera = Camera( id=0, name="front_camera", camera_model="pinhole", extrinsics=camera_extrinsics, distortion_parameters=DistortionParameters(), intrinsics=CameraIntrinsics( fx=1000.0, fy=1000.0, cx=960.0, cy=540.0, skew=0.0, width=1920.0, height=1080.0, ), trajectory=camera_traj, images_data=[ ImageData(filepath="images/front/0000.png", timestamp=float(timestamps[0]), undistorded=True), ImageData(filepath="images/front/0001.png", timestamp=float(timestamps[1]), undistorded=True), ImageData(filepath="images/front/0002.png", timestamp=float(timestamps[2]), undistorded=True), ], ) # LiDAR uses the same sensor-to-world composition rule. lidar_extrinsics = np.eye(4) lidar_poses = np.stack([ ego_pose @ lidar_extrinsics for ego_pose in ego_traj.object2world ]) lidar_timestamps = timestamps[:1] lidar = Lidar( id=0, name="top_lidar", extrinsics=lidar_extrinsics, intrinsics=LidarIntrinsics( lidar_type="rotating", rotation_type="clockwise", vertical_fov=0.7, horizontal_fov=2 * np.pi, horizontal_resolution=0.002, vertical_resolution=VerticalResolution(vertical_resolution=0.035), max_range=120.0, min_range=0.5, ), trajectory=WorldTrajectory( timestamps=lidar_timestamps, object2world=lidar_poses[:1], ), lidar_pcds_data=[ LidarPointCloudData( filepath="lidar/top_lidar/0000.ply", timestamp=float(lidar_timestamps[0]), egomotion_compensated=True, has_missing_points=False, ) ], min_cut_off_range=0.5, use_for_training=True, ) # Scene scene = Scene( ego_vehicle=EgoVehicle( trajectory=ego_traj, sensors=Sensors(cameras=[camera], lidars=[lidar]) ), actors=[] ) ``` ### 2. Load Scene from JSON ```python import json from mimic_data_api.scene import Scene with open('scene.json') as f: scene_data = json.load(f) scene = Scene.model_validate(scene_data) print(f"Scene has {len(scene.actors)} actors") ``` ### 3. Normalize LiDAR Intensity ```python import numpy as np def normalize_lidar_intensity(intensities: np.ndarray) -> np.ndarray: x = intensities.astype(np.float32, copy=True) finite_mask = np.isfinite(x) valid_raw = x[finite_mask & (x > 0)] if valid_raw.size == 0: x[~finite_mask | (x <= 0)] = -1.0 return x p99 = float(np.quantile(valid_raw, 0.999)) fit_vals = valid_raw[valid_raw < p99] if fit_vals.size == 0: fit_vals = valid_raw log_fit_vals = np.log(np.clip(fit_vals, 1e-12, None)) i_min = float(np.min(log_fit_vals)) i_max = float(np.max(log_fit_vals)) denom = max(i_max - i_min, 1e-12) nonreturn = (~finite_mask) | (x <= 0) outlier = (~nonreturn) & (x >= p99) valid = (~nonreturn) & (~outlier) if np.any(valid): x[valid] = (np.log(np.clip(x[valid], 1e-12, None)) - i_min) / denom x[nonreturn] = -1.0 x[outlier] = 1.0 return np.clip(x, -1.0, 1.0) # Usage raw_intensity = np.array([0.1, 0.5, 1.0, 5.0, 10.0]) normalized = normalize_lidar_intensity(raw_intensity) ``` Use `-1` only for a known missing return; the SDK loader removes those points by default. This helper treats non-positive measured values as missing returns. If the source has no intensity channel, bypass the helper and write a documented non-sentinel placeholder such as `0.0`; set `has_missing_points` according to actual missing returns, not to the absence of an intensity measurement. ### 4. Create LiDAR ```python import numpy as np from mimic_data_api.lidar import Lidar, LidarIntrinsics, VerticalResolution, LidarPointCloudData from mimic_data_api.trajectory_numpy import WorldTrajectory ego_traj = WorldTrajectory( timestamps=np.array([0.0]), object2world=np.array([np.eye(4)]), ) lidar_extrinsics = np.eye(4) lidar_pose = ego_traj.object2world[0] @ lidar_extrinsics lidar_trajectory = WorldTrajectory( timestamps=np.array([0.0]), object2world=np.array([lidar_pose]), ) lidar = Lidar( id=0, name="top_lidar", extrinsics=lidar_extrinsics, intrinsics=LidarIntrinsics( lidar_type="rotating", rotation_type="clockwise", vertical_fov=0.7, horizontal_fov=2 * np.pi, horizontal_resolution=0.002, vertical_resolution=VerticalResolution( elevation_channel_mapping={ 0: -0.26, 1: -0.09, 2: 0.09, 3: 0.26 } ), max_range=120.0, min_range=0.5, ), trajectory=lidar_trajectory, lidar_pcds_data=[ LidarPointCloudData( filepath="lidar/top_lidar/0000.ply", timestamp=0.0, egomotion_compensated=True, has_missing_points=False, ) ], min_cut_off_range=0.5, use_for_training=True, ) ``` `Lidar.use_for_training` is a required boolean training selector. It defaults to `True`; use `False` to disable a LiDAR stream. Do not write `null`. ### 5. Access Scene Data ```python # Ego position ego_pos = scene.ego_vehicle.trajectory.object2world[0, :3, 3] print(f"Ego at: {ego_pos}") # Cameras for camera in scene.ego_vehicle.sensors.cameras: for img in camera.images_data: print(f"{camera.name}: {img.filepath} @ {img.timestamp}s") # Actors for actor in scene.actors: print(f"{actor.id}: {actor.type}") ``` ### 6. Write Scene to File ```python import json from pathlib import Path scene_dict = scene.model_dump(mode='json') output_path = Path("output/scene.json") output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, 'w') as f: json.dump(scene_dict, f, indent=2) ``` ### 7. Validate Coordinate System ```python import numpy as np pose = np.array([...]) # 4x4 matrix R = pose[:3, :3] # Check orthonormality x_axis = R[:, 0] y_axis = R[:, 1] z_axis = R[:, 2] assert np.isclose(np.dot(x_axis, y_axis), 0.0), "X and Y not orthogonal" assert np.isclose(np.dot(y_axis, z_axis), 0.0), "Y and Z not orthogonal" ``` ## API Reference ### Scene ```python Scene( ego_vehicle: EgoVehicle, actors: list[Actor] = [] ) ``` ### EgoVehicle ```python EgoVehicle( trajectory: WorldTrajectory, sensors: Sensors ) ``` ### Sensors ```python Sensors( cameras: list[Camera], lidars: list[Lidar] | None = None, radars: list[Radar] | None = None ) ``` ### Camera ```python Camera( id: int, name: str, extrinsics: ndarray, # 4x4 intrinsics: CameraIntrinsics, trajectory: WorldTrajectory | None = None, images_data: list[ImageData] = [] ) ``` ### CameraIntrinsics ```python CameraIntrinsics( resolution: list[int], # [width, height] focal_length: list[float], # [fx, fy] principal_point: list[float], # [cx, cy] distortion: DistortionParameters | None = None, rolling_shutter: RollingShutterParameters | None = None ) ``` ### Lidar ```python Lidar( id: int, name: str, extrinsics: ndarray, # 4x4 intrinsics: LidarIntrinsics, trajectory: WorldTrajectory | None = None, lidar_pcds_data: list[LidarPointCloudData] = [] ) ``` ### LidarIntrinsics ```python LidarIntrinsics( lidar_type: Literal["rotating", "solid-state"], rotation_type: Literal["clockwise", "counter-clockwise"] | None, vertical_fov: float, horizontal_fov: float, horizontal_resolution: float, vertical_resolution: VerticalResolution, max_range: float, min_range: float ) ``` ### Actor ```python Actor( id: str, type: Literal["vehicle", "pedestrian", "cyclist", "other"], dimensions: ActorDimensions, trajectory: WorldTrajectory, annotations: list[ActorAnnotation] | None = None ) ``` ## Dataset Metadata Set these metadata flags correctly when building your converter: ### Camera Metadata ```python from mimic_data_api.camera import ImageData, RollingShutterParameters, DistortionParameters # Per-image metadata ImageData( filepath="images/cam_0/0000.jpg", timestamp=0.0, undistorded=False, # True if image is already rectified/undistorted rolling_shutter=RollingShutterParameters( readout_direction="top-to-bottom", # or "left-to-right" readout_time=0.009 # sensor readout duration in seconds; 0.0 for global shutter ), ) # Sensor-level: camera_model must be set Camera( id=0, name="cam_0", camera_model="opencv", # or "mei" for fisheye/omnidirectional distortion_parameters=DistortionParameters( k1=0.1, k2=-0.05, k3=0.0, p1=0.0, p2=0.0, xi=None # set for MEI fisheye model only ), intrinsics=CameraIntrinsics(...), ... ) ``` **Key Fields**: - `camera_model`: `"opencv"` for perspective cameras; `"mei"` for fisheye/omnidirectional - `undistorded`: `True` if each image is already rectified; `False` if raw distortion is present - `rolling_shutter`: Supply when the sensor does not have a global shutter - `distortion_parameters`: Coefficient values from your sensor calibration file ### LiDAR Metadata ```python from mimic_data_api.lidar import LidarPointCloudData LidarPointCloudData( filepath="lidar/top_lidar/0000.ply", timestamp=0.0, egomotion_compensated=False, # True if points are already motion-compensated has_missing_points=True # True if any points have intensity=-1 ) ``` **Key Fields**: - `egomotion_compensated`: `False` for raw rolling captures; `True` if the provider already unrolled the cloud - `has_missing_points`: `True` if the scan contains actual no-return points marked with intensity = -1; the SDK loader removes those points by default ## Error Handling ```python from pydantic import ValidationError try: scene = Scene.model_validate(scene_data) except ValidationError as e: for error in e.errors(): print(f"Error in {error['loc']}: {error['msg']}") ``` ## Debugging ```python import json # Pretty print scene print(json.dumps(scene.model_dump(mode='json'), indent=2)) # Check specific field print(scene.ego_vehicle.trajectory.timestamps) # Verify types print(type(scene.actors[0].dimensions)) ``` ## See Also - [Conversion Guide](conversion-guide.md) - [Validation Guide](validation-guide.md)