# Converting Your Dataset to MIMIC Data Format ## Overview The MIMIC data format is a standardized schema for autonomous driving datasets. This guide walks you through converting your dataset step-by-step. ### Minimum Input Dataset Requirements The MIMIC scene schema requires at least one camera and one LiDAR. Requirements also depend on the target workflow and converter. For reconstruction, real captured RGB imagery, usable camera poses, and intrinsics are essential; LiDAR does not replace camera imagery. Individual converters may impose additional source-data requirements. ### What is MIMIC Format? MIMIC format is a JSON-based schema that describes: - **Ego vehicle** trajectory and motion data - **Sensor data** (cameras, LiDAR, radar) - **Annotations** (3D bounding boxes, actor tracks) - **Coordinate system** definitions and transformations - **File references** to actual sensor data (images, point clouds) --- ## Step 0: Source Suitability and Inspection Before writing or running a converter, check that the source is suitable for the target reconstruction workflow. A scene can be valid MIMIC data without being suitable for reconstruction: it depends on visual information in posed RGB images, so point clouds, labels, or rendered visualizations cannot replace real camera imagery. - [ ] **Real RGB captures:** Open representative images from the source. Do not trust file extensions or dataset descriptions alone; detection datasets often store point-cloud plots, depth maps, or annotation overlays as image files. - [ ] **Camera poses:** Confirm camera-to-world poses are known or can be recovered consistently across the sequence. A separate ego trajectory is optional if camera poses are supplied directly. - [ ] **Camera intrinsics:** Confirm calibration is available or can be recovered from the images and capture setup. - [ ] **Sequence continuity:** Confirm the selected frames belong to one continuous recording. Inspect timestamp gaps, pose jumps, and visual transitions; split concatenated or discontinuous runs into separate scenes. LiDAR, radar, annotations, and a separate ego trajectory can provide useful data for other workflows, but they are not substitutes for posed RGB images. ### Structural Inspection Run a structural inspection before writing any conversion code. File counts, directory layouts, representative filenames, and small text samples reveal most of what a converter needs, without loading raw sensor data. `mimic-data-api` ships this as the `mimic-inspect-dataset` command. Install the published package from TestPyPI if it is not already installed: ```bash uv pip install --index-url https://test.pypi.org/simple/ \ --extra-index-url https://pypi.org/simple/ qvyon-mimic-data-api mimic-inspect-dataset /path/to/dataset ``` The report contains file counts and total size per extension, a directory tree, first lines of small text files such as CSV headers, PLY vertex property names and types, and representative filenames per type. This exposes naming conventions, frame index formats, column names, units, and schema issues before a full conversion run. To also verify point cloud units, pass the sensor's maximum range in metres: ```bash mimic-inspect-dataset /path/to/dataset --max-range 120 ``` The same report is available programmatically: ```python from pathlib import Path from mimic_data_api.dataset_inspect import inspect_dataset report = inspect_dataset(Path("/path/to/dataset"), max_range=120.0) print(report.render()) ``` ### Finding Calibration Metadata Calibration may be stored in ZIP or TAR sidecars rather than beside the scene files. Inspect archive listings before concluding that calibration is missing: ```bash unzip -l /path/to/source.zip tar -tf /path/to/source.tar.gz ``` Look for calibration manifests, camera intrinsics, sensor extrinsics, frame definitions, and camera-to-stream mappings. Filenames are clues, not proof; inspect candidate files and confirm they describe the exact sensor streams and image variant being converted. Prefer authoritative source-provided calibration over guessed defaults. If sources conflict or calibration is incomplete, record the selected source and its provenance, and use an explicit reviewed override rather than silently inventing values. Inspect listings first and extract only the needed files from large archives. ### Verifying Point Cloud Units MIMIC expects PLY coordinates in **metres**. Some source pipelines export millimetres, and `Scene.model_validate_json` cannot detect this because coordinates are just floats. Verify units empirically: compare the largest absolute coordinate against the sensor's maximum range. The `--max-range` check above reports this ratio and classifies it: - **Within range** - consistent with metres. - **Ratio near 1000** - the source is in millimetres; divide coordinates by 1000 before writing the MIMIC PLY. - **Modest overrun (a few times the range)** - usually still metres. Real sensors return points beyond their nominal range, and `max_range` in the scene may be understated. Do not rescale this data; confirm `max_range` and inspect outlier returns instead. --- ## Step 1: Understanding Coordinate Systems ### Critical: Coordinate System Conventions MIMIC uses two different coordinate systems. **Getting these wrong is the most common source of errors.** #### 1. **Camera Coordinate System** (OpenCV Convention) Used for camera intrinsics and image operations. ``` Z (forward) ↑ | X (right) | / | / +--→ Y (down) ``` - **X-axis**: points right in image - **Y-axis**: points down in image - **Z-axis**: points forward (into scene) #### 2. **Vehicle Coordinate System** (ISO Standard) Used for ego vehicle, LiDAR, radar, and 3D annotations. ``` Z (up) ↑ | Y ← + → X (forward) (left) ``` - **X-axis**: forward direction of vehicle - **Y-axis**: left direction (perpendicular to forward) - **Z-axis**: up direction ### ROS Frame Conventions The standard ROS `base_link` convention (FLU: X forward, Y left, Z up) matches the MIMIC vehicle frame. ROS camera optical frames (X right, Y down, Z forward) match the MIMIC camera frame. When the source actually uses these conventions, no axis rotation is needed for those frames. Verify the frame IDs and transform direction rather than trusting names alone. In a `TransformStamped`, `header.frame_id` is the parent and `child_frame_id` is the child; a `/tf_static` transform gives the child frame's pose in the parent. Thus `base_link` as parent and a sensor frame as child provide the sensor-to-vehicle extrinsic expected by MIMIC. Confirm the values against the physical mounting position and orientation. Not every ROS stack uses FLU for its vehicle frame. Aerial systems commonly use FRD or NED, and vendor VIO APIs may use a camera-start or other world frame. Determine the convention independently for every source frame before composing poses. For example, converting vector components from an aligned camera-like IMAGE basis (X right, Y down, Z forward) to vehicle FLU is: ```python IMAGE_TO_FLU = np.array([ [0, 0, 1], # forward = image Z [-1, 0, 0], # left = -image X [0, -1, 0], # up = -image Y ]) vehicle_vector = IMAGE_TO_FLU @ image_vector ``` This is a basis change for vector components, not a replacement for the camera's measured mounting extrinsics. For a transform `T_A_B` from frame B to frame A, if `C_A` and `C_B` convert the parent and child vector components respectively, the re-expressed transform is `C_A @ T_A_B @ inverse(C_B)`. Do not conjugate a pose with one matrix unless both frames use the same source and target basis conversion. ### Converting Axis-Angle Extrinsics Some calibration files store a rotation axis and angle with a translation instead of a 4×4 matrix. Confirm the source's angle units, translation units, coordinate basis, and transform direction first. The following builds a camera-to-vehicle transform from an axis-angle rotation in degrees and a translation in metres: ```python import numpy as np from scipy.spatial.transform import Rotation axis = np.asarray([axis_x, axis_y, axis_z], dtype=float) axis_norm = np.linalg.norm(axis) angle_degrees = source_angle_degrees translation_m = np.asarray([tx_m, ty_m, tz_m], dtype=float) if axis_norm == 0.0: if angle_degrees != 0.0: raise ValueError("A nonzero angle requires a nonzero rotation axis") rotation_matrix = np.eye(3) else: rotation_vector = axis / axis_norm * np.deg2rad(angle_degrees) rotation_matrix = Rotation.from_rotvec(rotation_vector).as_matrix() camera_to_vehicle = np.eye(4) camera_to_vehicle[:3, :3] = rotation_matrix camera_to_vehicle[:3, 3] = translation_m ``` If the source provides the inverse transform, invert it rather than relabeling it. Verify the resulting translation and orientation against the measured sensor mounting and the conventions in this guide. ### Sensor Trajectory and Extrinsics Contract The ego vehicle trajectory and each sensor trajectory use different frames. The ego vehicle trajectory stores vehicle-to-world poses. A sensor's ``extrinsics`` stores the static sensor-to-vehicle calibration. The sensor trajectory must store sensor-to-world poses with that calibration already composed: ```python sensor_to_world[i] = vehicle_to_world[i] @ sensor_to_vehicle ``` This rule applies to cameras, LiDARs, and radars. Do not assign the raw ego trajectory directly to a sensor. Sensor data files remain in the sensor-local coordinate frame; the sensor trajectory is what places those points or poses in the world frame. The sensor trajectory must also contain one pose for each entry in that sensor's data list, in the same timestamp order. If a sensor is sampled at a different rate from the ego trajectory, first obtain ego poses at the sensor timestamps, then compose each pose with the sensor's extrinsics. For cameras, use that camera's image timestamps; different cameras may have different timestamps. If the source provides ego poses at each image timestamp, use those poses directly. If it provides sensor-to-world poses, convert their coordinate convention as needed and do not apply the sensor extrinsics a second time. Otherwise, interpolate the ego trajectory to each camera's image timestamps: use linear interpolation for translation and spherical linear interpolation (SLERP) for rotation, then compose the camera-to-vehicle extrinsics as shown above. Interpolate only within the source trajectory's timestamp coverage; do not silently extrapolate. The shared [`example_converter.py`](example_converter.py) shows this pattern using `scipy.spatial.transform` (install `scipy` if it is not already available). ### Verifying Your Coordinate System ```python import numpy as np # Check a pose/transform matrix pose = np.array([...]) # Your 4x4 matrix # Extract axes R = pose[:3, :3] # Rotation matrix position = pose[:3, 3] # Position # Check axes point in right directions forward = R[:, 0] # X-axis should point forward left = R[:, 1] # Y-axis should point left up = R[:, 2] # Z-axis should point up # Verify they're orthonormal print("X·Y =", np.dot(forward, left), "should be ≈0") print("Y·Z =", np.dot(left, up), "should be ≈0") print("Z·X =", np.dot(up, forward), "should be ≈0") ``` ### Transformation Example If your dataset uses different conventions: ```python import numpy as np # Example: Converting from Waymo (X=forward, Y=right, Z=down) to MIMIC WAYMO_TO_MIMIC = np.array([ [0, 0, 1, 0], # X_mimic = Z_waymo (forward) [-1, 0, 0, 0], # Y_mimic = -X_waymo (left) [0, -1, 0, 0], # Z_mimic = -Y_waymo (up) [0, 0, 0, 1], ]) mimic_pose = WAYMO_TO_MIMIC @ waymo_pose ``` --- ## Step 2: Preparing Sensor Data ### Image Data #### Requirements - **Format**: PNG, JPG, or JPEG - **Color space**: RGB (not BGR) - **Size**: 1280×720 to 1920×1080 typical - **Distortion**: Can be raw (distorted) or undistorted - **Naming**: Store in `images//.png` #### Per-image mask fields The schema defines several optional mask paths, but they are not interchangeable: | Field | Current processing behavior | |---|---| | `dynamic_mask_filepath` | Read as a dynamic-object mask by the current processing path. | | `vignetting_mask_filepath` | Read by vignetting-specific camera-mask handling. | | `mask_filepath` | Accepted by the schema but not read by the current processing path; setting it alone does not apply a mask. | Do not populate `mask_filepath` expecting it to exclude pixels. Use a mask field only for the behavior it currently supports, and confirm file paths and image dimensions match the corresponding camera frames. To exclude moving objects from reconstruction, generate one single-channel PNG mask per image and set that image's `dynamic_mask_filepath` to the path relative to the scene root. Match each mask to the same camera and frame as its image, and make its dimensions match the image. In the current loader, **black (0) marks pixels to suppress** and **white (255) keeps pixels**; this is the opposite of some segmentation-mask conventions. Frames without masks may leave the field unset. The reconstruction loader reads this field by default. Prefer ground-truth instance/semantic masks or projected dynamic actor boxes when available. Background subtraction is appropriate only for a fixed or near-static camera; ego motion changes the background and makes that approach unreliable. Before running reconstruction, verify that every non-null mask path exists, opens as a single-channel image, and matches its corresponding image's dimensions. #### Supported Camera Models MIMIC currently supports these `camera_model` values: - `simple_pinhole`: `fx`, `cx`, `cy`; use `fy=fx` and zero distortion. - `pinhole`: `fx`, `fy`, `cx`, `cy`; use zero distortion for rectified images. - `simple_radial`: `fx`, `cx`, `cy`, `k1`. - `radial`: `fx`, `cx`, `cy`, `k1`, `k2`. - `opencv`: `fx`, `fy`, `cx`, `cy`, `k1`, `k2`, `p1`, `p2`; `k3` is also available. - `opencv_fisheye`: `fx`, `fy`, `cx`, `cy`, `k1`, `k2`, `k3`, `k4`. - `equirectangular`, `omnidirectionalstereo_l`, `omnidirectionalstereo_r`, `vr180_l`, `vr180_r`, and `orthophoto` for their corresponding projection formats. - `fisheye624` and `mei` for supported extended fisheye/omnidirectional calibrations; provide model-specific coefficients such as `xi` for `mei`. All camera models require `fx`, `fy`, `cx`, `cy`, `skew`, `width`, and `height` in `CameraIntrinsics`, plus a `DistortionParameters` object. If the source camera model is not supported, convert the images and calibration to one of these MIMIC-supported models before creating the scene. `fx` and `fy` are independent values. They do not need to be equal; wide-angle, fisheye, or anamorphic cameras may legitimately have non-square pixels. Preserve the calibrated values instead of forcing `fy=fx` unless the source calibration actually uses square pixels. #### Mapping Camera Identifiers to Image Streams Use this recovery workflow only when the source's ID-to-stream mapping is missing or unreliable. Existing converters use explicit format mappings: WOD camera enums and Parquet calibration, nuScenes `CAM_*` channels and calibrated- sensor tokens, KITTI-360 fixed camera IDs and calibration files, and OSDaR23 Raillabel sensor IDs with per-frame sensor references. When a mapping does need recovery, do not assume a summary/config mapping is complete or that its IDs are unique; some datasets use repeated placeholders. Build the mapping from per-frame metadata that contains both a camera-parameter identifier and an image path: 1. Group records by the camera-parameter identifier and infer the image stream from each record's path or explicit stream field. 2. Use the most common stream for each identifier as the candidate mapping. 3. Inspect disagreements and reject ties or mappings without a clear majority; do not resolve conflicts by folder sort order. 4. Confirm the final mapping is one-to-one for the streams being converted, then join it to calibration blocks by camera-parameter identifier. Check representative images, dimensions, and calibration for every mapped stream before creating `Camera` objects. Keep the resolved mapping with the converter inputs so the assignment is reproducible. #### Metadata Requirements For each camera: ```python from mimic_data_api.camera import Camera, CameraIntrinsics, DistortionParameters camera = Camera( id=0, name="front_camera", extrinsics=camera_to_vehicle_transform, # 4x4 matrix intrinsics=CameraIntrinsics( fx=1000.0, fy=1000.0, cx=960.0, cy=540.0, skew=0.0, width=1920.0, height=1080.0, ), camera_model="opencv", distortion_parameters=DistortionParameters(k1=0.1, k2=-0.05), trajectory=camera_trajectory, images_data=[...] ) ``` ### LiDAR Data #### Requirements - **Format**: PLY (Polygon File Format) - **Coordinate frame**: Sensor-local (not world/vehicle frame) - **Point structure**: X, Y, Z, `i`, `t`, `c` - **Column order**: MUST be exactly in this order - **Intensity**: NORMALIZED to [0, 1] range (see below) - **Missing points**: Marked with intensity = -1 - **Naming**: Store in `lidar//.ply` **Sentinel semantics:** `i = -1` means the ray produced no valid return; it does not mean that intensity was unavailable. `LidarPointCloudIOHandler.load()` removes these points by default, so writing `-1` for every point can make a non-empty PLY load as an empty point cloud. If the source has no intensity measurement channel, write a documented non-sentinel placeholder such as `0.0` for its points; do not send an all-zero placeholder array through the normalization helper below, which interprets non-positive inputs as no returns. Set `has_missing_points=True` only when the cloud contains actual missing-return sentinels. For ROS1/ROS2 bags, including `PointCloud2` field layout discovery and the mapping to MIMIC PLY properties, see the [ROS bag conversion guide](ros-bag-conversion.md). #### Angular Resolution Units `LidarIntrinsics.horizontal_resolution` and `VerticalResolution.vertical_resolution` are **radians per point** — the angular step between adjacent beams — not points per radian. Compute it as `2 * np.pi / num_points_per_rotation`, or `np.deg2rad(degrees_between_beams)` if you know the angular spacing directly. Typical values are small (roughly `0.001`–`0.03` radians); a value in the hundreds almost certainly means points-per-radian was used by mistake. Every first-party converter (`mimic-wod`, `mimic-nuscene`, `mimic-kitti360`, `mimic-osdar23`) follows this convention. #### Point Cloud Format Each PLY file must have exactly 6 columns: ``` ply format binary_little_endian element vertex 123456 property float x property float y property float z property float i ← Intensity; MUST be [0,1] or -1 property float t ← Time offset from scan timestamp property ushort c ← Beam/channel ID (`uint16`) end_header [binary point data] ``` Use `LidarPointCloudIOHandler().load(path)` to verify generated PLY files. A successful `Scene.model_validate_json` call alone does not validate PLY vertex property names because the schema does not open the referenced files. #### Intensity Normalization (CRITICAL!) **All LiDAR intensity MUST be normalized to [0, 1] range.** ##### Algorithm ```python import numpy as np def normalize_lidar_intensity(intensities: np.ndarray) -> np.ndarray: """Normalize measured intensities to [0, 1]; encode known no-returns as -1.""" x = intensities.astype(np.float32, copy=True) finite_mask = np.isfinite(x) valid_raw = x[finite_mask & (x > 0)] if valid_raw.size == 0: x[~finite_mask | (x <= 0)] = -1.0 return x # Find 99.9th percentile (handle outliers) p99 = float(np.quantile(valid_raw, 0.999)) fit_vals = valid_raw[valid_raw < p99] if fit_vals.size == 0: fit_vals = valid_raw # Log-transform for normalization log_fit_vals = np.log(np.clip(fit_vals, 1e-12, None)) i_min = float(np.min(log_fit_vals)) i_max = float(np.max(log_fit_vals)) denom = max(i_max - i_min, 1e-12) # Apply transformations nonreturn = (~finite_mask) | (x <= 0) outlier = (~nonreturn) & (x >= p99) valid = (~nonreturn) & (~outlier) if np.any(valid): x[valid] = (np.log(np.clip(x[valid], 1e-12, None)) - i_min) / denom x[nonreturn] = -1.0 x[outlier] = 1.0 return np.clip(x, -1.0, 1.0) # Usage raw_intensity = np.array([...]) normalized = normalize_lidar_intensity(raw_intensity) ``` ##### Why Log-Space Normalization? - Raw LiDAR intensity is exponentially distributed - Log-transform makes distribution more uniform - Improves machine learning model performance - Handles outliers gracefully - Standard in computer vision #### LiDAR Point Cloud Metadata When creating `LidarPointCloudData` entries, you must provide metadata flags that affect how the pipeline processes point clouds: **egomotion_compensated** (bool) - Indicates whether individual points have been temporally compensated for sensor motion during scan acquisition - **True**: Cloud is "unrolled" — all points are in the same reference frame regardless of when they were captured - **False**: Cloud is "rolled" — points reflect the sensor's position/orientation at their individual capture times (raw sensor output) - **Examples from this repository's converters**: - WOD sets the flag according to whether its range-image conversion applies per-pixel pose compensation; its selected test frames skip that compensation. - nuScenes reads raw sweep coordinates and estimates point-time offsets from point order and channel count, so its exported points are marked False for downstream compensation. - KITTI-360 and OSDaR23 export raw scans and mark them False. **has_missing_points** (bool) - Indicates whether this scan contains invalid/missing points (e.g. rain occlusion, sensor dropouts) - **True**: Scan has points marked as invalid (intensity = -1) - **False**: All points are valid Choose these values from the actual source data; the converter author is responsible for setting them correctly: - Set `egomotion_compensated=True` when all points have already been expressed in one common sensor frame for the scan (deskewed/unrolled). - Set `egomotion_compensated=False` when points retain their individual capture timing and represent a raw rolling scan. Preserve their `t` timestamp offsets. - Some sources emit separate point clouds at each IMU tick. If you combine them, do not set `egomotion_compensated=True` merely because the per-point times were discarded: preprocessing interprets `True` as already motion-compensated and skips compensation. Either transform all points into one common sensor frame before dropping their times and set `True`, or preserve the point-time offsets in `t` and set `False`. If neither is possible, keep the clouds separate or otherwise retain enough timing information to process them correctly. - Set `has_missing_points=True` when any output point uses intensity `-1` as the missing-return marker; otherwise set it to `False`. Do not infer these flags from the dataset name alone. Inspect the source point cloud representation and the transformations applied by your converter. **Example**: ```python from mimic_data_api.lidar import LidarPointCloudData lidar_pcds_data.append( LidarPointCloudData( filepath="lidar/top/0042.ply", timestamp=1234567890.123, egomotion_compensated=False, # raw rolling capture has_missing_points=True, # contains invalid points at intensity=-1 ) ) --- ## Step 3: Creating Ego Vehicle Trajectory ### Post-conversion Slicing Choose the slice window in time, then filter each sensor independently. Do not use one camera's timestamps as a frame-by-frame template for other sensors: cameras, LiDARs, and radars may have different rates, dropped frames, and different counts. Keep each stream's original samples and poses; filtering is not interpolation or resampling. For a camera-led slice, take the lower and upper time references from the chosen front-camera frames, then keep every other sensor sample and actor trajectory sample whose own timestamp falls in that interval. This assumes all timestamps use a common time base; do not match each sample to a front-camera frame or to an ego-trajectory index. Use the half-open boundary convention below for adjacent chunks. For a sensor, first verify that its data timestamps exactly match its trajectory timestamps in both length and order. Apply the same time mask to both arrays and to any per-pose velocity arrays. Camera timestamps must remain exactly aligned with their image entries. Actor trajectories are filtered independently to the same time window. ```python import numpy as np def filter_sensor_window(data_entries, trajectory, start_s, end_s): """Filter one timestamp-aligned data list and trajectory to [start_s, end_s).""" data_timestamps = np.asarray( [entry["timestamp"] for entry in data_entries], dtype=float ) trajectory_timestamps = np.asarray(trajectory["timestamps"], dtype=float) if not start_s < end_s: raise ValueError("Slice end must be later than slice start") if data_timestamps.shape != trajectory_timestamps.shape or not np.array_equal( data_timestamps, trajectory_timestamps ): raise ValueError("Sensor data and trajectory timestamps must match before slicing") if np.any(np.diff(trajectory_timestamps) <= 0): raise ValueError("Sensor timestamps must be strictly increasing") keep = (trajectory_timestamps >= start_s) & (trajectory_timestamps < end_s) filtered_entries = [entry for entry, selected in zip(data_entries, keep) if selected] filtered_trajectory = dict(trajectory) for key in ("timestamps", "object2world", "linear_velocity", "angular_velocity"): values = trajectory.get(key) if values is None: continue if len(values) != len(trajectory_timestamps): raise ValueError(f"Trajectory field {key} has a mismatched length") filtered_trajectory[key] = np.asarray(values)[keep].tolist() return filtered_entries, filtered_trajectory ``` Use half-open windows (`start <= timestamp < end`) for adjacent chunks so a boundary sample is not duplicated; choose the upper reference accordingly, or include the final endpoint explicitly for the last chunk if required. Preserve the original pose values rather than re-interpolating them. Copy or hard-link referenced image, mask, and point-cloud files into the output while preserving their relative paths; fall back to copying when hard links are unavailable. Drop or report a stream with no samples in the window according to the target workflow instead of manufacturing frames. After slicing, validate each sensor's data-to-trajectory timestamp equality, all referenced paths, and the complete scene with `Scene.model_validate_json`. Different sensors are not expected to have equal frame counts, and that divergence is schema-valid: alignment is required within each sensor, not across sensors. For reconstruction, also check cross-sensor temporal coverage: a camera frame can be individually valid and still lack a LiDAR sweep within the pipeline's matching tolerance. Current preprocessing selects the nearest sweep within `2 * camera.sampling_time`, where `camera.sampling_time` is the 90th percentile of that camera's timestamp intervals. If no LiDAR has a sweep in that window, the frame has no LiDAR depth support and preprocessing may fail later while handling the empty accumulation. This can happen at a slice boundary even when the source recording is valid. Before reconstruction, trim uncovered camera frames and apply the same selection to `images_data`, camera trajectory poses, and any per-pose velocity arrays. Keep the camera frames only if the target workflow can process them without LiDAR support. The ego vehicle trajectory is a sequence of poses over time. ```python import numpy as np from mimic_data_api.trajectory_numpy import WorldTrajectory # Create timestamps timestamps = np.linspace(0.0, 10.0, 100) # 0 to 10 seconds # Create 4x4 transformation matrices num_frames = len(timestamps) poses = np.zeros((num_frames, 4, 4)) for i in range(num_frames): # Each pose: 4x4 matrix from vehicle coords to world coords poses[i] = create_pose_at_time(timestamps[i]) trajectory = WorldTrajectory( timestamps=timestamps, object2world=poses # vehicle-to-world transformations ) ``` ### Selecting the Ego Pose Source If a dataset provides multiple pose sources, choose the source that best fits your use case and preserves a consistent world frame across the full scene. Examples include raw GPS, local odometry, map-frame fused odometry, and INS. The SDK does not prescribe one source for every dataset; the converter author must determine which source is reliable and appropriate for the data. For a typical vehicle dataset, use this default priority: 1. A globally consistent map-frame fused trajectory, such as an EKF estimate combining GNSS, INS, and odometry. 2. A calibrated INS trajectory with adequate drift and timestamp quality for the scene. 3. Raw GPS as a fallback for coarse global position; it often does not provide reliable orientation on its own. Local odometry is relative and can drift. Use it only when it is anchored to a consistent map frame or when a relative trajectory is explicitly appropriate. Dataset-specific evidence can change this priority; verify the frame, coverage, and quality instead of relying on the source label alone. Avoid mixing poses from incompatible frames or using an odometry frame that drifts when a globally referenced source is required. Record the chosen source and any frame conversion in the converter documentation. ### LiDAR Training Selection Every `Lidar` must have a boolean `use_for_training` flag. It accepts only `true` or `false` and defaults to `true`; `null` is not a valid value because it can break downstream training configuration. Set it to `false` only when the LiDAR stream should be excluded from training. ```python lidar = Lidar( # ... other required fields ... min_cut_off_range=minimum_cutoff_from_sensor_spec, use_for_training=True, ) ``` `min_cut_off_range` is required and is measured in metres. Set it to the sensor's blind zone or the minimum range used to remove points belonging to the ego vehicle. The converter author must choose the value from the sensor blind-zone specification and preprocessing requirements. --- ## Step 4: Annotating Actors (Objects) Actors represent all objects in the scene (vehicles, pedestrians, etc.). ```python from mimic_data_api.actor import Actor, ActorDimensions import numpy as np # Define actor trajectory actor_timestamps = np.array([0.0, 0.1, 0.2, 0.3, 0.4]) actor_poses = np.zeros((len(actor_timestamps), 4, 4)) # ... populate actor poses ... actor = Actor( id="vehicle_123", type="vehicle", # One of: vehicle, pedestrian, cyclist, other dimensions=ActorDimensions( length=4.5, # Forward direction (meters) width=2.0, # Left direction (meters) height=1.5 # Up direction (meters) ), trajectory=WorldTrajectory( timestamps=actor_timestamps, object2world=actor_poses ), ) ``` ### Actor Types - **"vehicle"**: Cars, trucks, buses - **"pedestrian"**: People on foot - **"cyclist"**: People on bicycles - **"other"**: Everything else --- ## Step 5: Assembling the Scene ```python from mimic_data_api.scene import Scene, EgoVehicle, Sensors scene = Scene( ego_vehicle=EgoVehicle( trajectory=ego_trajectory, sensors=Sensors( cameras=[front_camera, rear_camera, ...], lidars=[top_lidar], radars=[front_radar] ) ), actors=[actor_1, actor_2, ...] ) ``` --- ## Step 6: Serialization and Storage ### Writing Scene to JSON ```python import json # Serialize scene scene_dict = scene.model_dump(mode='json') # Write to file with open('scene.json', 'w') as f: json.dump(scene_dict, f, indent=2) ``` ### Directory Structure ``` my_dataset/ └── scenes/ ├── scene_001/ │ ├── scene.json # Main metadata │ ├── images/ │ │ ├── front_camera/0000.png │ │ ├── rear_camera/0000.png │ │ └── ... │ ├── lidar/ │ │ ├── top_lidar/0000.ply │ │ └── ... │ └── radar/ │ └── front_radar/0000.ply └── scene_002/ └── ... ``` --- ## Step 7: Validation Checklist Before submitting your dataset, verify: ### Coordinate Systems - [ ] Camera extrinsics are camera-to-vehicle transformations - [ ] Vehicle Z-axis points upward - [ ] Vehicle X-axis points forward - [ ] Vehicle Y-axis points left - [ ] All transformations use 4x4 homogeneous matrices ### Sensors - [ ] All camera images exist and are readable - [ ] All LiDAR point clouds are valid PLY files - [ ] LiDAR intensity normalized to [0, 1] or -1 - [ ] Each sensor's timestamps match its own trajectory poses - [ ] Camera intrinsics have positive focal lengths ### Annotations - [ ] Actor types are valid (vehicle, pedestrian, cyclist, other) - [ ] Actor dimensions are positive and reasonable - [ ] Actor trajectories have valid timestamps ### Files - [ ] scene.json is valid JSON - [ ] All file paths in JSON exist on disk - [ ] No missing sensor data frames - [ ] PLY files have correct 6-column format --- ## Troubleshooting ### Issue: "LiDAR intensity out of range" **Solution**: Ensure intensity is normalized. ```python from mimic_data_api.unstructured.pointclouds import LidarPointCloud # After loading, normalize column 3 (intensity) points[:, 3] = normalize_lidar_intensity(points[:, 3]) ``` ### Issue: "Camera distortion not supported" **Supported models**: - Radial-tangential (Brown-Conrady) - Fisheye (Kannala-Brandes) - Equidistant ### Issue: "Mismatched image and trajectory timestamps" **Solution**: A camera or LiDAR may use timestamps that differ from the ego trajectory. This is valid when the sensor has its own clock or sampling rate. The required invariant is per sensor: the number and order of timestamps in the sensor data list must exactly match the number and order of poses in that sensor's `trajectory.object2world` array. Sensor timestamps do not need to match the ego timestamps. ### Issue: "Invalid coordinate transformation" **Debugging**: See Step 1 verification code above. --- ## Best Practices 1. **Start Small**: Convert one scene first 2. **Validate Early**: Check coordinate systems before full conversion 3. **Test Normalization**: Verify LiDAR intensity on sample 4. **Use Reference Implementations**: Adapt existing converters 5. **Document Assumptions**: Record any dataset-specific transformations 6. **Automated Checks**: Run validation tools regularly --- ## Next Steps Once converted: 1. Run validation tools 2. Submit your dataset 3. Report any issues For API details, see [Quick Reference](quick-reference.md).