Converting Your Dataset to MIMIC Data Format

Overview

The MIMIC data format is a standardized schema for autonomous driving datasets. This guide walks you through converting your dataset step-by-step.

Minimum Input Dataset Requirements

The MIMIC scene schema requires at least one camera and one LiDAR. Requirements also depend on the target workflow and converter. For reconstruction, real captured RGB imagery, usable camera poses, and intrinsics are essential; LiDAR does not replace camera imagery. Individual converters may impose additional source-data requirements.

What is MIMIC Format?

MIMIC format is a JSON-based schema that describes:

  • Ego vehicle trajectory and motion data

  • Sensor data (cameras, LiDAR, radar)

  • Annotations (3D bounding boxes, actor tracks)

  • Coordinate system definitions and transformations

  • File references to actual sensor data (images, point clouds)

Step 0: Source Suitability and Inspection

Before writing or running a converter, check that the source is suitable for the target reconstruction workflow. A scene can be valid MIMIC data without being suitable for reconstruction: it depends on visual information in posed RGB images, so point clouds, labels, or rendered visualizations cannot replace real camera imagery.

  • [ ] Real RGB captures: Open representative images from the source. Do not trust file extensions or dataset descriptions alone; detection datasets often store point-cloud plots, depth maps, or annotation overlays as image files.

  • [ ] Camera poses: Confirm camera-to-world poses are known or can be recovered consistently across the sequence. A separate ego trajectory is optional if camera poses are supplied directly.

  • [ ] Camera intrinsics: Confirm calibration is available or can be recovered from the images and capture setup.

  • [ ] Sequence continuity: Confirm the selected frames belong to one continuous recording. Inspect timestamp gaps, pose jumps, and visual transitions; split concatenated or discontinuous runs into separate scenes.

LiDAR, radar, annotations, and a separate ego trajectory can provide useful data for other workflows, but they are not substitutes for posed RGB images.

Structural Inspection

Run a structural inspection before writing any conversion code. File counts, directory layouts, representative filenames, and small text samples reveal most of what a converter needs, without loading raw sensor data.

mimic-data-api ships this as the mimic-inspect-dataset command. Install the published package from TestPyPI if it is not already installed:

uv pip install --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ qvyon-mimic-data-api

mimic-inspect-dataset /path/to/dataset

The report contains file counts and total size per extension, a directory tree, first lines of small text files such as CSV headers, PLY vertex property names and types, and representative filenames per type. This exposes naming conventions, frame index formats, column names, units, and schema issues before a full conversion run.

To also verify point cloud units, pass the sensor’s maximum range in metres:

mimic-inspect-dataset /path/to/dataset --max-range 120

The same report is available programmatically:

from pathlib import Path
from mimic_data_api.dataset_inspect import inspect_dataset

report = inspect_dataset(Path("/path/to/dataset"), max_range=120.0)
print(report.render())

Finding Calibration Metadata

Calibration may be stored in ZIP or TAR sidecars rather than beside the scene files. Inspect archive listings before concluding that calibration is missing:

unzip -l /path/to/source.zip
tar -tf /path/to/source.tar.gz

Look for calibration manifests, camera intrinsics, sensor extrinsics, frame definitions, and camera-to-stream mappings. Filenames are clues, not proof; inspect candidate files and confirm they describe the exact sensor streams and image variant being converted. Prefer authoritative source-provided calibration over guessed defaults. If sources conflict or calibration is incomplete, record the selected source and its provenance, and use an explicit reviewed override rather than silently inventing values. Inspect listings first and extract only the needed files from large archives.

Verifying Point Cloud Units

MIMIC expects PLY coordinates in metres. Some source pipelines export millimetres, and Scene.model_validate_json cannot detect this because coordinates are just floats.

Verify units empirically: compare the largest absolute coordinate against the sensor’s maximum range. The --max-range check above reports this ratio and classifies it:

  • Within range - consistent with metres.

  • Ratio near 1000 - the source is in millimetres; divide coordinates by 1000 before writing the MIMIC PLY.

  • Modest overrun (a few times the range) - usually still metres. Real sensors return points beyond their nominal range, and max_range in the scene may be understated. Do not rescale this data; confirm max_range and inspect outlier returns instead.

Step 1: Understanding Coordinate Systems

Critical: Coordinate System Conventions

MIMIC uses two different coordinate systems. Getting these wrong is the most common source of errors.

1. Camera Coordinate System (OpenCV Convention)

Used for camera intrinsics and image operations.

    Z (forward)
    ↑
    |     X (right)
    |    /
    |   /
    +--→ Y (down)

  • X-axis: points right in image

  • Y-axis: points down in image

  • Z-axis: points forward (into scene)

2. Vehicle Coordinate System (ISO Standard)

Used for ego vehicle, LiDAR, radar, and 3D annotations.

        Z (up)
        ↑
        |
    Y ← + → X (forward)
   (left)

  • X-axis: forward direction of vehicle

  • Y-axis: left direction (perpendicular to forward)

  • Z-axis: up direction

ROS Frame Conventions

The standard ROS base_link convention (FLU: X forward, Y left, Z up) matches the MIMIC vehicle frame. ROS camera optical frames (X right, Y down, Z forward) match the MIMIC camera frame. When the source actually uses these conventions, no axis rotation is needed for those frames.

Verify the frame IDs and transform direction rather than trusting names alone. In a TransformStamped, header.frame_id is the parent and child_frame_id is the child; a /tf_static transform gives the child frame’s pose in the parent. Thus base_link as parent and a sensor frame as child provide the sensor-to-vehicle extrinsic expected by MIMIC. Confirm the values against the physical mounting position and orientation.

Not every ROS stack uses FLU for its vehicle frame. Aerial systems commonly use FRD or NED, and vendor VIO APIs may use a camera-start or other world frame. Determine the convention independently for every source frame before composing poses. For example, converting vector components from an aligned camera-like IMAGE basis (X right, Y down, Z forward) to vehicle FLU is:

IMAGE_TO_FLU = np.array([
    [0,  0, 1],  # forward = image Z
    [-1, 0, 0],  # left = -image X
    [0, -1, 0],  # up = -image Y
])
vehicle_vector = IMAGE_TO_FLU @ image_vector

This is a basis change for vector components, not a replacement for the camera’s measured mounting extrinsics. For a transform T_A_B from frame B to frame A, if C_A and C_B convert the parent and child vector components respectively, the re-expressed transform is C_A @ T_A_B @ inverse(C_B). Do not conjugate a pose with one matrix unless both frames use the same source and target basis conversion.

Converting Axis-Angle Extrinsics

Some calibration files store a rotation axis and angle with a translation instead of a 4×4 matrix. Confirm the source’s angle units, translation units, coordinate basis, and transform direction first. The following builds a camera-to-vehicle transform from an axis-angle rotation in degrees and a translation in metres:

import numpy as np
from scipy.spatial.transform import Rotation

axis = np.asarray([axis_x, axis_y, axis_z], dtype=float)
axis_norm = np.linalg.norm(axis)
angle_degrees = source_angle_degrees
translation_m = np.asarray([tx_m, ty_m, tz_m], dtype=float)

if axis_norm == 0.0:
    if angle_degrees != 0.0:
        raise ValueError("A nonzero angle requires a nonzero rotation axis")
    rotation_matrix = np.eye(3)
else:
    rotation_vector = axis / axis_norm * np.deg2rad(angle_degrees)
    rotation_matrix = Rotation.from_rotvec(rotation_vector).as_matrix()

camera_to_vehicle = np.eye(4)
camera_to_vehicle[:3, :3] = rotation_matrix
camera_to_vehicle[:3, 3] = translation_m

If the source provides the inverse transform, invert it rather than relabeling it. Verify the resulting translation and orientation against the measured sensor mounting and the conventions in this guide.

Sensor Trajectory and Extrinsics Contract

The ego vehicle trajectory and each sensor trajectory use different frames. The ego vehicle trajectory stores vehicle-to-world poses. A sensor’s extrinsics stores the static sensor-to-vehicle calibration. The sensor trajectory must store sensor-to-world poses with that calibration already composed:

sensor_to_world[i] = vehicle_to_world[i] @ sensor_to_vehicle

This rule applies to cameras, LiDARs, and radars. Do not assign the raw ego trajectory directly to a sensor. Sensor data files remain in the sensor-local coordinate frame; the sensor trajectory is what places those points or poses in the world frame.

The sensor trajectory must also contain one pose for each entry in that sensor’s data list, in the same timestamp order. If a sensor is sampled at a different rate from the ego trajectory, first obtain ego poses at the sensor timestamps, then compose each pose with the sensor’s extrinsics. For cameras, use that camera’s image timestamps; different cameras may have different timestamps.

If the source provides ego poses at each image timestamp, use those poses directly. If it provides sensor-to-world poses, convert their coordinate convention as needed and do not apply the sensor extrinsics a second time. Otherwise, interpolate the ego trajectory to each camera’s image timestamps: use linear interpolation for translation and spherical linear interpolation (SLERP) for rotation, then compose the camera-to-vehicle extrinsics as shown above. Interpolate only within the source trajectory’s timestamp coverage; do not silently extrapolate. The shared example_converter.py shows this pattern using scipy.spatial.transform (install scipy if it is not already available).

Verifying Your Coordinate System

import numpy as np

# Check a pose/transform matrix
pose = np.array([...])  # Your 4x4 matrix

# Extract axes
R = pose[:3, :3]  # Rotation matrix
position = pose[:3, 3]  # Position

# Check axes point in right directions
forward = R[:, 0]  # X-axis should point forward
left = R[:, 1]     # Y-axis should point left
up = R[:, 2]       # Z-axis should point up

# Verify they're orthonormal
print("X·Y =", np.dot(forward, left), "should be ≈0")
print("Y·Z =", np.dot(left, up), "should be ≈0")
print("Z·X =", np.dot(up, forward), "should be ≈0")

Transformation Example

If your dataset uses different conventions:

import numpy as np

# Example: Converting from Waymo (X=forward, Y=right, Z=down) to MIMIC
WAYMO_TO_MIMIC = np.array([
    [0, 0, 1, 0],    # X_mimic = Z_waymo (forward)
    [-1, 0, 0, 0],   # Y_mimic = -X_waymo (left)
    [0, -1, 0, 0],   # Z_mimic = -Y_waymo (up)
    [0, 0, 0, 1],
])

mimic_pose = WAYMO_TO_MIMIC @ waymo_pose

Step 2: Preparing Sensor Data

Image Data

Requirements

  • Format: PNG, JPG, or JPEG

  • Color space: RGB (not BGR)

  • Size: 1280×720 to 1920×1080 typical

  • Distortion: Can be raw (distorted) or undistorted

  • Naming: Store in images/<camera_name>/<frame_id>.png

Per-image mask fields

The schema defines several optional mask paths, but they are not interchangeable:

Field

Current processing behavior

dynamic_mask_filepath

Read as a dynamic-object mask by the current processing path.

vignetting_mask_filepath

Read by vignetting-specific camera-mask handling.

mask_filepath

Accepted by the schema but not read by the current processing path; setting it alone does not apply a mask.

Do not populate mask_filepath expecting it to exclude pixels. Use a mask field only for the behavior it currently supports, and confirm file paths and image dimensions match the corresponding camera frames.

To exclude moving objects from reconstruction, generate one single-channel PNG mask per image and set that image’s dynamic_mask_filepath to the path relative to the scene root. Match each mask to the same camera and frame as its image, and make its dimensions match the image. In the current loader, black (0) marks pixels to suppress and white (255) keeps pixels; this is the opposite of some segmentation-mask conventions. Frames without masks may leave the field unset. The reconstruction loader reads this field by default.

Prefer ground-truth instance/semantic masks or projected dynamic actor boxes when available. Background subtraction is appropriate only for a fixed or near-static camera; ego motion changes the background and makes that approach unreliable. Before running reconstruction, verify that every non-null mask path exists, opens as a single-channel image, and matches its corresponding image’s dimensions.

Supported Camera Models

MIMIC currently supports these camera_model values:

  • simple_pinhole: fx, cx, cy; use fy=fx and zero distortion.

  • pinhole: fx, fy, cx, cy; use zero distortion for rectified images.

  • simple_radial: fx, cx, cy, k1.

  • radial: fx, cx, cy, k1, k2.

  • opencv: fx, fy, cx, cy, k1, k2, p1, p2; k3 is also available.

  • opencv_fisheye: fx, fy, cx, cy, k1, k2, k3, k4.

  • equirectangular, omnidirectionalstereo_l, omnidirectionalstereo_r, vr180_l, vr180_r, and orthophoto for their corresponding projection formats.

  • fisheye624 and mei for supported extended fisheye/omnidirectional calibrations; provide model-specific coefficients such as xi for mei.

All camera models require fx, fy, cx, cy, skew, width, and height in CameraIntrinsics, plus a DistortionParameters object. If the source camera model is not supported, convert the images and calibration to one of these MIMIC-supported models before creating the scene.

fx and fy are independent values. They do not need to be equal; wide-angle, fisheye, or anamorphic cameras may legitimately have non-square pixels. Preserve the calibrated values instead of forcing fy=fx unless the source calibration actually uses square pixels.

Mapping Camera Identifiers to Image Streams

Use this recovery workflow only when the source’s ID-to-stream mapping is missing or unreliable. Existing converters use explicit format mappings: WOD camera enums and Parquet calibration, nuScenes CAM_* channels and calibrated- sensor tokens, KITTI-360 fixed camera IDs and calibration files, and OSDaR23 Raillabel sensor IDs with per-frame sensor references.

When a mapping does need recovery, do not assume a summary/config mapping is complete or that its IDs are unique; some datasets use repeated placeholders. Build the mapping from per-frame metadata that contains both a camera-parameter identifier and an image path:

  1. Group records by the camera-parameter identifier and infer the image stream from each record’s path or explicit stream field.

  2. Use the most common stream for each identifier as the candidate mapping.

  3. Inspect disagreements and reject ties or mappings without a clear majority; do not resolve conflicts by folder sort order.

  4. Confirm the final mapping is one-to-one for the streams being converted, then join it to calibration blocks by camera-parameter identifier.

Check representative images, dimensions, and calibration for every mapped stream before creating Camera objects. Keep the resolved mapping with the converter inputs so the assignment is reproducible.

Metadata Requirements

For each camera:

from mimic_data_api.camera import Camera, CameraIntrinsics, DistortionParameters

camera = Camera(
    id=0,
    name="front_camera",
    extrinsics=camera_to_vehicle_transform,  # 4x4 matrix
    intrinsics=CameraIntrinsics(
        fx=1000.0,
        fy=1000.0,
        cx=960.0,
        cy=540.0,
        skew=0.0,
        width=1920.0,
        height=1080.0,
    ),
    camera_model="opencv",
    distortion_parameters=DistortionParameters(k1=0.1, k2=-0.05),
    trajectory=camera_trajectory,
    images_data=[...]
)

LiDAR Data

Requirements

  • Format: PLY (Polygon File Format)

  • Coordinate frame: Sensor-local (not world/vehicle frame)

  • Point structure: X, Y, Z, i, t, c

  • Column order: MUST be exactly in this order

  • Intensity: NORMALIZED to [0, 1] range (see below)

  • Missing points: Marked with intensity = -1

  • Naming: Store in lidar/<sensor_name>/<frame_id>.ply

Sentinel semantics: i = -1 means the ray produced no valid return; it does not mean that intensity was unavailable. LidarPointCloudIOHandler.load() removes these points by default, so writing -1 for every point can make a non-empty PLY load as an empty point cloud. If the source has no intensity measurement channel, write a documented non-sentinel placeholder such as 0.0 for its points; do not send an all-zero placeholder array through the normalization helper below, which interprets non-positive inputs as no returns. Set has_missing_points=True only when the cloud contains actual missing-return sentinels.

For ROS1/ROS2 bags, including PointCloud2 field layout discovery and the mapping to MIMIC PLY properties, see the ROS bag conversion guide.

Angular Resolution Units

LidarIntrinsics.horizontal_resolution and VerticalResolution.vertical_resolution are radians per point — the angular step between adjacent beams — not points per radian. Compute it as 2 * np.pi / num_points_per_rotation, or np.deg2rad(degrees_between_beams) if you know the angular spacing directly. Typical values are small (roughly 0.001–0.03 radians); a value in the hundreds almost certainly means points-per-radian was used by mistake. Every first-party converter (mimic-wod, mimic-nuscene, mimic-kitti360, mimic-osdar23) follows this convention.

Point Cloud Format

Each PLY file must have exactly 6 columns:

ply
format binary_little_endian
element vertex 123456
property float x
property float y
property float z
property float i               ← Intensity; MUST be [0,1] or -1
property float t               ← Time offset from scan timestamp
property ushort c              ← Beam/channel ID (`uint16`)
end_header
[binary point data]

Use LidarPointCloudIOHandler().load(path) to verify generated PLY files. A successful Scene.model_validate_json call alone does not validate PLY vertex property names because the schema does not open the referenced files.

Intensity Normalization (CRITICAL!)

All LiDAR intensity MUST be normalized to [0, 1] range.

Algorithm
import numpy as np

def normalize_lidar_intensity(intensities: np.ndarray) -> np.ndarray:
    """Normalize measured intensities to [0, 1]; encode known no-returns as -1."""
    x = intensities.astype(np.float32, copy=True)
    finite_mask = np.isfinite(x)
    valid_raw = x[finite_mask & (x > 0)]
    
    if valid_raw.size == 0:
        x[~finite_mask | (x <= 0)] = -1.0
        return x
    
    # Find 99.9th percentile (handle outliers)
    p99 = float(np.quantile(valid_raw, 0.999))
    fit_vals = valid_raw[valid_raw < p99]
    if fit_vals.size == 0:
        fit_vals = valid_raw
    
    # Log-transform for normalization
    log_fit_vals = np.log(np.clip(fit_vals, 1e-12, None))
    i_min = float(np.min(log_fit_vals))
    i_max = float(np.max(log_fit_vals))
    denom = max(i_max - i_min, 1e-12)
    
    # Apply transformations
    nonreturn = (~finite_mask) | (x <= 0)
    outlier = (~nonreturn) & (x >= p99)
    valid = (~nonreturn) & (~outlier)
    
    if np.any(valid):
        x[valid] = (np.log(np.clip(x[valid], 1e-12, None)) - i_min) / denom
    
    x[nonreturn] = -1.0
    x[outlier] = 1.0
    
    return np.clip(x, -1.0, 1.0)

# Usage
raw_intensity = np.array([...])
normalized = normalize_lidar_intensity(raw_intensity)
Why Log-Space Normalization?

  • Raw LiDAR intensity is exponentially distributed

  • Log-transform makes distribution more uniform

  • Improves machine learning model performance

  • Handles outliers gracefully

  • Standard in computer vision

LiDAR Point Cloud Metadata

When creating LidarPointCloudData entries, you must provide metadata flags that affect how the pipeline processes point clouds:

egomotion_compensated (bool)

  • Indicates whether individual points have been temporally compensated for sensor motion during scan acquisition

  • True: Cloud is “unrolled” — all points are in the same reference frame regardless of when they were captured

  • False: Cloud is “rolled” — points reflect the sensor’s position/orientation at their individual capture times (raw sensor output)

  • Examples from this repository’s converters:

    • WOD sets the flag according to whether its range-image conversion applies per-pixel pose compensation; its selected test frames skip that compensation.

    • nuScenes reads raw sweep coordinates and estimates point-time offsets from point order and channel count, so its exported points are marked False for downstream compensation.

    • KITTI-360 and OSDaR23 export raw scans and mark them False.

has_missing_points (bool)

  • Indicates whether this scan contains invalid/missing points (e.g. rain occlusion, sensor dropouts)

  • True: Scan has points marked as invalid (intensity = -1)

  • False: All points are valid

Choose these values from the actual source data; the converter author is responsible for setting them correctly:

  • Set egomotion_compensated=True when all points have already been expressed in one common sensor frame for the scan (deskewed/unrolled).

  • Set egomotion_compensated=False when points retain their individual capture timing and represent a raw rolling scan. Preserve their t timestamp offsets.

  • Some sources emit separate point clouds at each IMU tick. If you combine them, do not set egomotion_compensated=True merely because the per-point times were discarded: preprocessing interprets True as already motion-compensated and skips compensation. Either transform all points into one common sensor frame before dropping their times and set True, or preserve the point-time offsets in t and set False. If neither is possible, keep the clouds separate or otherwise retain enough timing information to process them correctly.

  • Set has_missing_points=True when any output point uses intensity -1 as the missing-return marker; otherwise set it to False.

Do not infer these flags from the dataset name alone. Inspect the source point cloud representation and the transformations applied by your converter.

Example:

from mimic_data_api.lidar import LidarPointCloudData

lidar_pcds_data.append(
    LidarPointCloudData(
        filepath="lidar/top/0042.ply",
        timestamp=1234567890.123,
        egomotion_compensated=False,  # raw rolling capture
        has_missing_points=True,      # contains invalid points at intensity=-1
    )
)

---

## Step 3: Creating Ego Vehicle Trajectory

### Post-conversion Slicing

Choose the slice window in time, then filter each sensor independently. Do not
use one camera's timestamps as a frame-by-frame template for other sensors:
cameras, LiDARs, and radars may have different rates, dropped frames, and
different counts. Keep each stream's original samples and poses; filtering is
not interpolation or resampling.

For a camera-led slice, take the lower and upper time references from the chosen
front-camera frames, then keep every other sensor sample and actor trajectory
sample whose own timestamp falls in that interval. This assumes all timestamps
use a common time base; do not match each sample to a front-camera frame or to an
ego-trajectory index. Use the half-open boundary convention below for adjacent
chunks.

For a sensor, first verify that its data timestamps exactly match its trajectory
timestamps in both length and order. Apply the same time mask to both arrays and
to any per-pose velocity arrays. Camera timestamps must remain exactly aligned
with their image entries. Actor trajectories are filtered independently to the
same time window.

```python
import numpy as np


def filter_sensor_window(data_entries, trajectory, start_s, end_s):
    """Filter one timestamp-aligned data list and trajectory to [start_s, end_s)."""
    data_timestamps = np.asarray(
        [entry["timestamp"] for entry in data_entries], dtype=float
    )
    trajectory_timestamps = np.asarray(trajectory["timestamps"], dtype=float)

    if not start_s < end_s:
        raise ValueError("Slice end must be later than slice start")
    if data_timestamps.shape != trajectory_timestamps.shape or not np.array_equal(
        data_timestamps, trajectory_timestamps
    ):
        raise ValueError("Sensor data and trajectory timestamps must match before slicing")
    if np.any(np.diff(trajectory_timestamps) <= 0):
        raise ValueError("Sensor timestamps must be strictly increasing")

    keep = (trajectory_timestamps >= start_s) & (trajectory_timestamps < end_s)
    filtered_entries = [entry for entry, selected in zip(data_entries, keep) if selected]
    filtered_trajectory = dict(trajectory)
    for key in ("timestamps", "object2world", "linear_velocity", "angular_velocity"):
        values = trajectory.get(key)
        if values is None:
            continue
        if len(values) != len(trajectory_timestamps):
            raise ValueError(f"Trajectory field {key} has a mismatched length")
        filtered_trajectory[key] = np.asarray(values)[keep].tolist()

    return filtered_entries, filtered_trajectory

Use half-open windows (start <= timestamp < end) for adjacent chunks so a boundary sample is not duplicated; choose the upper reference accordingly, or include the final endpoint explicitly for the last chunk if required. Preserve the original pose values rather than re-interpolating them. Copy or hard-link referenced image, mask, and point-cloud files into the output while preserving their relative paths; fall back to copying when hard links are unavailable. Drop or report a stream with no samples in the window according to the target workflow instead of manufacturing frames.

After slicing, validate each sensor’s data-to-trajectory timestamp equality, all referenced paths, and the complete scene with Scene.model_validate_json. Different sensors are not expected to have equal frame counts, and that divergence is schema-valid: alignment is required within each sensor, not across sensors. For reconstruction, also check cross-sensor temporal coverage: a camera frame can be individually valid and still lack a LiDAR sweep within the pipeline’s matching tolerance. Current preprocessing selects the nearest sweep within 2 * camera.sampling_time, where camera.sampling_time is the 90th percentile of that camera’s timestamp intervals. If no LiDAR has a sweep in that window, the frame has no LiDAR depth support and preprocessing may fail later while handling the empty accumulation. This can happen at a slice boundary even when the source recording is valid. Before reconstruction, trim uncovered camera frames and apply the same selection to images_data, camera trajectory poses, and any per-pose velocity arrays. Keep the camera frames only if the target workflow can process them without LiDAR support.

The ego vehicle trajectory is a sequence of poses over time.

import numpy as np
from mimic_data_api.trajectory_numpy import WorldTrajectory

# Create timestamps
timestamps = np.linspace(0.0, 10.0, 100)  # 0 to 10 seconds

# Create 4x4 transformation matrices
num_frames = len(timestamps)
poses = np.zeros((num_frames, 4, 4))

for i in range(num_frames):
    # Each pose: 4x4 matrix from vehicle coords to world coords
    poses[i] = create_pose_at_time(timestamps[i])

trajectory = WorldTrajectory(
    timestamps=timestamps,
    object2world=poses  # vehicle-to-world transformations
)

Selecting the Ego Pose Source

If a dataset provides multiple pose sources, choose the source that best fits your use case and preserves a consistent world frame across the full scene. Examples include raw GPS, local odometry, map-frame fused odometry, and INS. The SDK does not prescribe one source for every dataset; the converter author must determine which source is reliable and appropriate for the data.

For a typical vehicle dataset, use this default priority:

  1. A globally consistent map-frame fused trajectory, such as an EKF estimate combining GNSS, INS, and odometry.

  2. A calibrated INS trajectory with adequate drift and timestamp quality for the scene.

  3. Raw GPS as a fallback for coarse global position; it often does not provide reliable orientation on its own.

Local odometry is relative and can drift. Use it only when it is anchored to a consistent map frame or when a relative trajectory is explicitly appropriate. Dataset-specific evidence can change this priority; verify the frame, coverage, and quality instead of relying on the source label alone.

Avoid mixing poses from incompatible frames or using an odometry frame that drifts when a globally referenced source is required. Record the chosen source and any frame conversion in the converter documentation.

LiDAR Training Selection

Every Lidar must have a boolean use_for_training flag. It accepts only true or false and defaults to true; null is not a valid value because it can break downstream training configuration. Set it to false only when the LiDAR stream should be excluded from training.

lidar = Lidar(
    # ... other required fields ...
    min_cut_off_range=minimum_cutoff_from_sensor_spec,
    use_for_training=True,
)

min_cut_off_range is required and is measured in metres. Set it to the sensor’s blind zone or the minimum range used to remove points belonging to the ego vehicle. The converter author must choose the value from the sensor blind-zone specification and preprocessing requirements.

Step 4: Annotating Actors (Objects)

Actors represent all objects in the scene (vehicles, pedestrians, etc.).

from mimic_data_api.actor import Actor, ActorDimensions
import numpy as np

# Define actor trajectory
actor_timestamps = np.array([0.0, 0.1, 0.2, 0.3, 0.4])
actor_poses = np.zeros((len(actor_timestamps), 4, 4))
# ... populate actor poses ...

actor = Actor(
    id="vehicle_123",
    type="vehicle",  # One of: vehicle, pedestrian, cyclist, other
    dimensions=ActorDimensions(
        length=4.5,   # Forward direction (meters)
        width=2.0,    # Left direction (meters)
        height=1.5    # Up direction (meters)
    ),
    trajectory=WorldTrajectory(
        timestamps=actor_timestamps,
        object2world=actor_poses
    ),
)

Actor Types

  • “vehicle”: Cars, trucks, buses

  • “pedestrian”: People on foot

  • “cyclist”: People on bicycles

  • “other”: Everything else

Step 5: Assembling the Scene

from mimic_data_api.scene import Scene, EgoVehicle, Sensors

scene = Scene(
    ego_vehicle=EgoVehicle(
        trajectory=ego_trajectory,
        sensors=Sensors(
            cameras=[front_camera, rear_camera, ...],
            lidars=[top_lidar],
            radars=[front_radar]
        )
    ),
    actors=[actor_1, actor_2, ...]
)

Step 6: Serialization and Storage

Writing Scene to JSON

import json

# Serialize scene
scene_dict = scene.model_dump(mode='json')

# Write to file
with open('scene.json', 'w') as f:
    json.dump(scene_dict, f, indent=2)

Directory Structure

my_dataset/
└── scenes/
    ├── scene_001/
    │   ├── scene.json                    # Main metadata
    │   ├── images/
    │   │   ├── front_camera/0000.png
    │   │   ├── rear_camera/0000.png
    │   │   └── ...
    │   ├── lidar/
    │   │   ├── top_lidar/0000.ply
    │   │   └── ...
    │   └── radar/
    │       └── front_radar/0000.ply
    └── scene_002/
        └── ...

Step 7: Validation Checklist

Before submitting your dataset, verify:

Coordinate Systems

  • [ ] Camera extrinsics are camera-to-vehicle transformations

  • [ ] Vehicle Z-axis points upward

  • [ ] Vehicle X-axis points forward

  • [ ] Vehicle Y-axis points left

  • [ ] All transformations use 4x4 homogeneous matrices

Sensors

  • [ ] All camera images exist and are readable

  • [ ] All LiDAR point clouds are valid PLY files

  • [ ] LiDAR intensity normalized to [0, 1] or -1

  • [ ] Each sensor’s timestamps match its own trajectory poses

  • [ ] Camera intrinsics have positive focal lengths

Annotations

  • [ ] Actor types are valid (vehicle, pedestrian, cyclist, other)

  • [ ] Actor dimensions are positive and reasonable

  • [ ] Actor trajectories have valid timestamps

Files

  • [ ] scene.json is valid JSON

  • [ ] All file paths in JSON exist on disk

  • [ ] No missing sensor data frames

  • [ ] PLY files have correct 6-column format

Troubleshooting

Issue: “LiDAR intensity out of range”

Solution: Ensure intensity is normalized.

from mimic_data_api.unstructured.pointclouds import LidarPointCloud

# After loading, normalize column 3 (intensity)
points[:, 3] = normalize_lidar_intensity(points[:, 3])

Issue: “Camera distortion not supported”

Supported models:

  • Radial-tangential (Brown-Conrady)

  • Fisheye (Kannala-Brandes)

  • Equidistant

Issue: “Mismatched image and trajectory timestamps”

Solution: A camera or LiDAR may use timestamps that differ from the ego trajectory. This is valid when the sensor has its own clock or sampling rate. The required invariant is per sensor: the number and order of timestamps in the sensor data list must exactly match the number and order of poses in that sensor’s trajectory.object2world array. Sensor timestamps do not need to match the ego timestamps.

Issue: “Invalid coordinate transformation”

Debugging: See Step 1 verification code above.

Best Practices

  1. Start Small: Convert one scene first

  2. Validate Early: Check coordinate systems before full conversion

  3. Test Normalization: Verify LiDAR intensity on sample

  4. Use Reference Implementations: Adapt existing converters

  5. Document Assumptions: Record any dataset-specific transformations

  6. Automated Checks: Run validation tools regularly

Next Steps

Once converted:

  1. Run validation tools

  2. Submit your dataset

  3. Report any issues

For API details, see Quick Reference.