API Brief Description

This guide will help you get started with the MIMIC Data API.

Dependencies

The Data API requires:

  • Python 3.10+

  • NumPy

  • Pydantic v2

  • numpydantic

Basic Concepts

Pydantic Models

All data structures in the MIMIC Data API are built on Pydantic models, providing:

  • Validation: Automatic type checking and validation on construction

  • Serialization: Easy JSON import/export with model_dump_json() and model_validate_json()

  • Type Hints: Full IDE support and autocomplete

Example:

from mimic_data_api import Actor, ActorDimensions, WorldTrajectory
import numpy as np

# Create with validation
actor = Actor(
    id=1,
    name="Vehicle_001",
    dimensions=ActorDimensions(length=4.5, width=1.8, height=1.5),
    boxtype="vehicle",
    dynamic=True,
    trajectory=WorldTrajectory(
        timestamps=np.array([0.0, 0.1]),
        object2world=np.stack([np.eye(4), np.eye(4)])
    )
)

# Serialize to JSON
json_str = actor.model_dump_json(indent=2)

# Deserialize from JSON
actor_copy = Actor.model_validate_json(json_str)

Coordinate Frames

The API uses right-handed coordinate frames that vary by sensor type:

Vehicle/LiDAR/Radar Frame (FLU - Forward, Left, Up):

  • X-axis: Forward (direction of travel)

  • Y-axis: Left (perpendicular to travel direction)

  • Z-axis: Up (vertical)

Camera Frame (OpenCV Convention):

  • X-axis: Right (horizontal, perpendicular to optical axis)

  • Y-axis: Down (vertical, perpendicular to optical axis)

  • Z-axis: Forward (optical axis, pointing into the scene)

World Frame:

  • Fixed global coordinates

  • Typically ENU (East-North-Up) or a local tangent plane

  • Scene-dependent orientation

Transformation Matrices

Poses are represented as 4x4 homogeneous transformation matrices:

\[\begin{split}T = \begin{bmatrix} R_{3 \times 3} & t_{3 \times 1} \\ 0_{1 \times 3} & 1 \end{bmatrix}\end{split}\]

Where:

  • \(R\) is a 3x3 rotation matrix

  • \(t\) is a 3x1 translation vector

To transform a point from local to world coordinates:

import numpy as np

# Point in local coordinates (homogeneous)
p_local = np.array([1.0, 0.0, 0.0, 1.0])

# Transformation matrix
T = trajectory.object2world[0]  # First pose

# Transform to world coordinates
p_world = T @ p_local

WorldTrajectory

The WorldTrajectory class is the fundamental building block for representing time-series pose data:

from mimic_data_api import WorldTrajectory
import numpy as np

# Create a 100-frame trajectory at 10 Hz
n_frames = 100
trajectory = WorldTrajectory(
    timestamps=np.linspace(0, 10, n_frames),  # 0 to 10 seconds
    object2world=np.stack([np.eye(4)] * n_frames),  # Identity poses
    linear_velocity=np.zeros((n_frames, 3)),   # Optional velocities
    angular_velocity=np.zeros((n_frames, 3))   # Optional angular velocities
)

# Access specific time
t5_pose = trajectory.object2world[50]  # Pose at t=5s
position = t5_pose[:3, 3]  # Extract translation
rotation = t5_pose[:3, :3]  # Extract rotation matrix

Working with Sensors

Cameras

Camera sensors include intrinsic calibration, distortion parameters, and image data:

from mimic_data_api import (
    Camera, CameraIntrinsics, DistortionParameters, ImageData
)
import numpy as np

camera = Camera(
    id=1,
    name="front_camera",
    camera_model="opencv",  # Supports various models
    intrinsics=CameraIntrinsics(
        fx=1000.0, fy=1000.0,  # Focal lengths
        cx=640.0, cy=360.0,    # Principal point
        skew=0.0,
        width=1280, height=720
    ),
    distortion_parameters=DistortionParameters(
        k1=-0.1, k2=0.01,  # Radial distortion
        p1=0.001, p2=-0.001  # Tangential distortion
    ),
    extrinsics=np.eye(4),  # Sensor to vehicle transform
    trajectory=ego_trajectory,
    images_data=[
        ImageData(timestamp=0.0, filepath="img_000.png", undistorded=False),
        ImageData(timestamp=0.1, filepath="img_001.png", undistorded=False),
    ]
)

# Get intrinsic matrix
K = camera.intrinsics.as_K()  # Returns 3x3 numpy array

LiDAR Sensors

LiDAR sensors support both rotating and solid-state types:

from mimic_data_api import (
    Lidar, LidarIntrinsics, VerticalResolution, LidarPointCloudData
)
import numpy as np

lidar = Lidar(
    id=1,
    name="top_lidar",
    extrinsics=np.eye(4),
    intrinsics=LidarIntrinsics(
        lidar_type="rotating",
        rotation_type="clockwise",
        vertical_fov=0.7,  # ~40 degrees in radians
        horizontal_fov=2 * np.pi,  # 360 degrees
        horizontal_resolution=0.002,  # radians/point (angular step), not points/radian
        vertical_resolution=VerticalResolution(vertical_resolution=0.035),
        max_range=120.0,
        min_range=0.5
    ),
    trajectory=ego_trajectory,
    lidar_pcds_data=[
        LidarPointCloudData(
            filepath="lidar_000.ply",
            timestamp=0.0,
            egomotion_compensated=True,
            has_missing_points=False
        )
    ],
    min_cut_off_range=2.0  # Filter ego vehicle points
)

Radar Sensors

Radar sensors include velocity measurement capabilities:

from mimic_data_api import Radar, RadarIntrinsics, RadarPointCloudData
import numpy as np

radar = Radar(
    id=1,
    name="front_radar",
    extrinsics=np.eye(4),
    intrinsics=RadarIntrinsics(
        range=200.0,  # Max range in meters
        azimuth_fov=2.0,  # Horizontal FOV in radians
        elevation_fov=0.35,  # Vertical FOV in radians
        azimuth_resolution=[(0.0, 0.017), (1.0, 0.035)],  # Angle-dependent
        elevation_resolution=None,  # Non-imaging radar
        min_speed_resolution=-70.0,  # m/s
        max_speed_resolution=70.0    # m/s
    ),
    trajectory=ego_trajectory,
    radar_pcds_data=[
        RadarPointCloudData(filepath="radar_000.ply", timestamp=0.0)
    ]
)

Working with Actors

Actors represent other objects in the scene:

from mimic_data_api import Actor, ActorDimensions, WorldTrajectory
import numpy as np

# Create a moving vehicle
vehicle = Actor(
    id=42,
    name="sedan_001",
    dimensions=ActorDimensions(length=4.5, width=1.8, height=1.5),
    boxtype="vehicle",
    original_boxtype="sedan",  # Preserve original label
    dynamic=True,
    trajectory=WorldTrajectory(
        timestamps=np.array([0.0, 0.1, 0.2]),
        object2world=vehicle_poses  # (3, 4, 4) array
    ),
    description="Blue sedan in adjacent lane"
)

# Create a static sign
sign = Actor(
    id=100,
    name="stop_sign",
    dimensions=ActorDimensions(length=0.1, width=0.6, height=2.0),
    boxtype="sign",
    dynamic=False,
    trajectory=WorldTrajectory(
        timestamps=np.array([0.0]),
        object2world=np.eye(4).reshape(1, 4, 4)
    )
)

Creating Complete Scenes

Combine all elements into a Scene:

from mimic_data_api import Scene, EgoVehicle, Sensors

scene = Scene(
    ego_vehicle=EgoVehicle(
        trajectory=ego_trajectory,
        sensors=Sensors(
            cameras=[front_camera, rear_camera],
            lidars=[top_lidar],
            radars=[front_radar, rear_radar]
        )
    ),
    actors=[vehicle, pedestrian, cyclist, sign]
)

# Serialize to JSON
with open("scene.json", "w") as f:
    f.write(scene.model_dump_json(indent=2))

# Load from JSON
with open("scene.json", "r") as f:
    loaded_scene = Scene.model_validate_json(f.read())

JSON Serialization

The Data API uses Pydantic’s JSON serialization, which handles NumPy arrays automatically:

# Export
json_str = scene.model_dump_json(indent=2)

# Import
scene = Scene.model_validate_json(json_str)

# Export to dict (for custom processing)
data_dict = scene.model_dump()

For working with files:

from pathlib import Path

# Save scene
Path("scene.json").write_text(scene.model_dump_json(indent=2))

# Load scene
scene = Scene.model_validate_json(Path("scene.json").read_text())

Next Steps