Camera Module¶

The camera module provides data structures for representing camera sensors, including intrinsic parameters, distortion models, and captured image data.

Camera¶

class mimic_data_api.camera.Camera(*, id, name, camera_model, distortion_parameters, extrinsics, trajectory, intrinsics, images_data)[source]¶

Bases: BaseModel

Complete camera sensor representation.

This class encapsulates all information about a camera sensor including its calibration (intrinsic and extrinsic), projection model, trajectory, and captured image data.

id¶

Unique identifier for the camera within the sensor suite.

Type:: int

name¶

Human-readable name for the camera (e.g., “front_left”, “rear”).

Type:: str

camera_model¶

Projection model type (e.g., “pinhole”, “opencv”, “opencv_fisheye”).

Type:: CameraModels

distortion_parameters¶

Lens distortion coefficients for the camera model.

Type:: DistortionParameters

extrinsics¶

4x4 transformation matrix from sensor frame to vehicle frame.

Type:: SensorExtrinsic

trajectory¶

Time-series of camera poses in world coordinates.

Type:: WorldTrajectory

intrinsics¶

Internal camera parameters (focal length, principal point, image size).

Type:: CameraIntrinsics

images_data¶

List of captured images with their metadata.

Type:: list of ImageData

Raises:: ValueError – If image timestamps don’t match trajectory timestamps.

Note

The images_data timestamps must exactly match the trajectory.timestamps array. This ensures consistent indexing between poses and images.

Example

>>> camera = Camera(
...     id=0,
...     name="front_camera",
...     camera_model="pinhole",
...     distortion_parameters=DistortionParameters(),
...     extrinsics=np.eye(4),
...     trajectory=trajectory,
...     intrinsics=CameraIntrinsics(
...         fx=500, fy=500, cx=320, cy=240,
...         skew=0, width=640, height=480
...     ),
...     images_data=[ImageData(timestamp=0.0, filepath="img.png", undistorded=True)]
... )

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

id: int¶

name: str¶

camera_model: CameraModels¶

distortion_parameters: DistortionParameters¶

extrinsics: SensorExtrinsic¶

trajectory: WorldTrajectory¶

intrinsics: CameraIntrinsics¶

images_data: list[ImageData]¶

validate_images_trajectory_timestamps()[source]¶

Validate that image timestamps match trajectory timestamps.

Returns:: The validated camera instance.
Return type:: Camera
Raises:: ValueError – If image timestamps don’t match trajectory timestamps.

__eq__(other)[source]¶

Check equality with another Camera.

Performs deep comparison of all camera attributes including calibration parameters, trajectory, and image data.

Parameters:: other (object) – The object to compare against.
Returns:: True if cameras are equal, False otherwise.
Return type:: bool

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Note

Camera.trajectory.object2world contains camera-to-world poses. It is formed by composing the matching vehicle-to-world pose with Camera.extrinsics (camera-to-vehicle); it is not the raw ego trajectory.

CameraIntrinsics¶

class mimic_data_api.camera.CameraIntrinsics(*, fx, fy, cx, cy, skew, height, width)[source]¶

Bases: BaseModel

Internal camera calibration parameters.

This class encapsulates the intrinsic camera matrix parameters that define the projection from 3D camera coordinates to 2D image coordinates.

fx¶

Focal length in pixels along the x-axis.

Type:: float

fy¶

Focal length in pixels along the y-axis.

Type:: float

cx¶

Principal point x-coordinate in pixels (typically image center).

Type:: float

cy¶

Principal point y-coordinate in pixels (typically image center).

Type:: float

skew¶

Skew coefficient (typically 0 for modern cameras).

Type:: float

height¶

Image height in pixels.

Type:: float

width¶

Image width in pixels.

Type:: float

Note

The intrinsic matrix K is structured as:

\[\begin{split}K = \begin{bmatrix} f_x & skew & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}\end{split}\]

Example

>>> intrinsics = CameraIntrinsics(
...     fx=1000.0, fy=1000.0,
...     cx=640.0, cy=360.0,
...     skew=0.0,
...     width=1280.0, height=720.0
... )
>>> K = intrinsics.as_K()
>>> print(K.shape)
(3, 3)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Intrinsic Matrix

The intrinsic matrix K is structured as:

\[\begin{split}K = \begin{bmatrix} f_x & skew & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}\end{split}\]

fx: float¶

fy: float¶

cx: float¶

cy: float¶

skew: float¶

height: float¶

width: float¶

as_K(*, dtype=<class 'numpy.float32'>)[source]¶

Return the 3x3 intrinsic matrix for pinhole-like models.

Constructs the camera calibration matrix K that projects 3D points in camera coordinates to 2D homogeneous image coordinates.

Parameters:: dtype (numpy.dtype, optional) – Data type for the output array. Default: np.float32
Returns:: A 3x3 intrinsic calibration matrix K.
Return type:: numpy.ndarray

Example

>>> intrinsics = CameraIntrinsics(
...     fx=500.0, fy=500.0, cx=320.0, cy=240.0,
...     skew=0.0, width=640.0, height=480.0
... )
>>> K = intrinsics.as_K()
>>> # Project a 3D point at (1, 0, 5) in camera coordinates
>>> point_3d = np.array([1.0, 0.0, 5.0])
>>> point_2d_homogeneous = K @ point_3d
>>> point_2d = point_2d_homogeneous[:2] / point_2d_homogeneous[2]

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

DistortionParameters¶

class mimic_data_api.camera.DistortionParameters(*, p1=0.0, p2=0.0, k1=0.0, k2=0.0, k3=0.0, k4=0.0, k5=None, k6=None, k0=None, p0=None, s0=None, s1=None, s2=None, s3=None, sx1=None, sy1=None, xi=None)[source]¶

Bases: BaseModel

Lens distortion coefficients for camera calibration.

This class supports various distortion models including standard OpenCV radial-tangential distortion, fisheye models, and extended models with thin prism distortion.

The distortion is typically applied to normalized image coordinates before projection through the intrinsic matrix.

p1¶

First tangential distortion coefficient. Default: 0.0

Type:: float

p2¶

Second tangential distortion coefficient. Default: 0.0

Type:: float

k1¶

First radial distortion coefficient. Default: 0.0

Type:: float

k2¶

Second radial distortion coefficient. Default: 0.0

Type:: float

k3¶

Third radial distortion coefficient. Default: 0.0

Type:: float

k4¶

Fourth radial distortion coefficient (fisheye models). Default: 0.0

Type:: float

k5¶

Fifth radial distortion coefficient (FullOpenCV model). Optional.

Type:: float or None

k6¶

Sixth radial distortion coefficient (FullOpenCV model). Optional.

Type:: float or None

k0¶

Zeroth radial coefficient (RadTanThinPrismFisheye model). Optional.

Type:: float or None

p0¶

Zeroth tangential coefficient (RadTanThinPrismFisheye model). Optional.

Type:: float or None

s0¶

Zeroth thin prism distortion coefficient. Optional.

Type:: float or None

s1¶

First thin prism distortion coefficient. Optional.

Type:: float or None

s2¶

Second thin prism distortion coefficient. Optional.

Type:: float or None

s3¶

Third thin prism distortion coefficient. Optional.

Type:: float or None

sx1¶

First thin prism x-coefficient (ThinPrismFisheye model). Optional.

Type:: float or None

sy1¶

First thin prism y-coefficient (ThinPrismFisheye model). Optional.

Type:: float or None

xi¶

Projection parameter (Mei / Omnidirectional models). Optional.

Type:: float or None

Note

The standard OpenCV distortion model uses:

\[ \begin{align}\begin{aligned}x_{distorted} = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2p_1 xy + p_2(r^2 + 2x^2)\\y_{distorted} = y(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + p_1(r^2 + 2y^2) + 2p_2 xy\end{aligned}\end{align} \]

where \(r^2 = x^2 + y^2\).

Example

>>> distortion = DistortionParameters(k1=-0.1, k2=0.01, p1=0.001, p2=-0.001)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Distortion Model

The standard OpenCV distortion model:

\[x_{distorted} = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2p_1 xy + p_2(r^2 + 2x^2)\]

\[y_{distorted} = y(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + p_1(r^2 + 2y^2) + 2p_2 xy\]

where \(r^2 = x^2 + y^2\).

p1: float¶

p2: float¶

k1: float¶

k2: float¶

k3: float¶

k4: float¶

k5: float | None¶

k6: float | None¶

k0: float | None¶

p0: float | None¶

s0: float | None¶

s1: float | None¶

s2: float | None¶

s3: float | None¶

sx1: float | None¶

sy1: float | None¶

xi: float | None¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

RollingShutterParameters¶

class mimic_data_api.camera.RollingShutterParameters(*, readout_time=0.0, readout_direction='global')[source]¶

Bases: BaseModel

Rolling shutter timing and direction parameters.

Defines the rolling shutter behavior for cameras that don’t capture all pixels simultaneously. Essential for accurate motion compensation in dynamic scenes.

readout_time¶

Total time to read out the entire sensor in seconds. A value of 0.0 indicates a global shutter. Default: 0.0

Type:: float

readout_direction¶

Direction in which sensor rows/columns are read out. "global" indicates simultaneous readout (global shutter). Default: "global"

Type:: {“up_to_down”, “down_to_up”, “left_to_right”, “right_to_left”, “global”}

Note

For rolling shutter cameras, different rows are captured at slightly different times. The time offset for row r is:

\[t_{row} = t_{frame} + \frac{r}{H} \cdot t_{readout}\]

where H is the image height and the direction is top-to-bottom.

Example

>>> rs = RollingShutterParameters(
...     readout_time=0.033,  # 33ms readout
...     readout_direction="up_to_down"
... )

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

readout_time: float¶

readout_direction: Literal['up_to_down', 'down_to_up', 'left_to_right', 'right_to_left', 'global']¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ImageData¶

class mimic_data_api.camera.ImageData(*, filepath, timestamp, undistorded, rolling_shutter=RollingShutterParameters(readout_time=0.0, readout_direction='global'), image=None, semantic_segmentation_filepath=None, semantic_segmentation_image=None, instance_segmentation_filepath=None, instance_segmentation_image=None, lidar_depth_filepath=None, lidar_depth_image=None, monodepth_filepath=None, monodepth_image=None, normal_filepath=None, normal_segmentation_image=None, wb_params=None, brightness_control=None, dynamic_mask_filepath=None, dynamic_mask_image=None, mask_filepath=None, mask_image=None, vignetting_mask_filepath=None, vignetting_mask_image=None)[source]¶

Bases: BaseModel

Metadata and optional pixel data for a captured camera image.

This class stores information about individual images captured by a camera, including the file location, timestamp, and processing state.

filepath¶

Path to the image file. Must be PNG or JPEG format.

Type:: str

timestamp¶

Capture timestamp in seconds (must be >= 0).

Type:: float

undistorded¶

True if the image has been undistorted (lens distortion removed), False if it contains original lens distortion.

Type:: bool

rolling_shutter¶

Rolling shutter parameters for this specific image. Default: global shutter (no rolling shutter effect).

Type:: RollingShutterParameters

image¶

Optional in-memory image data. Not serialized to JSON.

Type:: RGBImage or None

Raises:: ValueError – If filepath doesn’t end with .png, .jpg, or .jpeg.

Example

>>> img_data = ImageData(
...     filepath="data/images/frame_000.png",
...     timestamp=0.0,
...     undistorded=False
... )

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

filepath: str¶

timestamp: float¶

undistorded: bool¶

rolling_shutter: RollingShutterParameters¶

image: RGBImage | None¶

semantic_segmentation_filepath: str | None¶

semantic_segmentation_image: SegmentationImage | None¶

instance_segmentation_filepath: str | None¶

instance_segmentation_image: SegmentationImage | None¶

lidar_depth_filepath: str | None¶

lidar_depth_image: DepthImage | None¶

monodepth_filepath: str | None¶

monodepth_image: DepthImage | None¶

normal_filepath: str | None¶

normal_segmentation_image: NormalImage | None¶

wb_params: list[float] | None¶

brightness_control: float | None¶

dynamic_mask_filepath: str | None¶

dynamic_mask_image: SegmentationImage | None¶

mask_filepath: str | None¶

mask_image: SegmentationImage | None¶

vignetting_mask_filepath: str | None¶

vignetting_mask_image: SegmentationImage | None¶

classmethod validate_filepath(v)[source]¶

Validate that the filepath has a supported image extension.

Parameters:: v (str) – The filepath to validate.
Returns:: The validated filepath.
Return type:: str
Raises:: ValueError – If the filepath doesn’t end with a supported extension.

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Camera Models¶

mimic_data_api.camera.CameraModels¶

Supported camera projection models (aligned with COLMAP):

simple_pinhole: Single focal length (fx = fy), no distortion
pinhole: Separate fx, fy, no distortion
simple_radial: Single focal length with k1 distortion
radial: Single focal length with k1, k2 distortion
opencv: Full OpenCV model (fx, fy, cx, cy, k1, k2, p1, p2)
opencv_fisheye: OpenCV fisheye model
equirectangular: 360° equirectangular projection
omnidirectionalstereo_l: ODS left eye
omnidirectionalstereo_r: ODS right eye
vr180_l: VR180 left eye
vr180_r: VR180 right eye
orthophoto: Orthographic projection
fisheye624: 6+2+4 parameter fisheye model

See COLMAP Camera Models for detailed documentation.