MIMIC Data API - Customer Documentation

Welcome! This folder contains everything you need to convert your autonomous driving dataset to the MIMIC data format.

Minimum Input Dataset Requirements

MIMIC supports input datasets with at least one camera and one LiDAR. Radar sensors are optional. Additional cameras, LiDARs, and radar sensors are supported when the converter can provide their calibration, trajectories, and sensor data.

Setup (TestPyPI Installation)

Mandatory requirement: use a fresh Python 3.10 virtual environment for each SDK release.

Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

Open a new terminal after installation so the uv command is available on your PATH.

Install the SDK and Converters

uv venv --python 3.10 .venv
source .venv/bin/activate
uv pip install \
    --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    qvyon-mimic-filesystem qvyon-mimic-data-api qvyon-mimic-raillabel \
    qvyon-mimic-wod qvyon-mimic-nuscene qvyon-mimic-kitti360 \
    qvyon-mimic-osdar23 qvyon-mimic-pandaset
uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4

Verify the Environment

uv pip list | grep -E "qvyon-mimic-filesystem|qvyon-mimic-data-api"

When you need to run Python commands, either activate .venv first or use uv run python. The converter examples below use uv run python so they work without a manual shell activation step.

Verify the Environment

uv pip list | grep -E "qvyon-mimic-(filesystem|data-api|raillabel|wod|nuscene|kitti360|osdar23|pandaset)"
command -v kitti360-convert
command -v osdar23-convert
command -v nuscene-convert
command -v wod-convert
command -v pandaset-convert

mimic_wod requires Python 3.10 because its TensorFlow runtime does not provide compatible wheels for newer Python versions. Install the Waymo package with --no-deps: its published metadata pins older NumPy/Pandas versions that conflict with the shared SDK environment. Install dacite==1.8.1 and typeguard==4.4.4 separately.

For the published converter wheels, use Python 3.10, TestPyPI for MIMIC packages, and public PyPI for third-party dependencies:

uv pip install \
    --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    qvyon-mimic-wod qvyon-mimic-nuscene qvyon-mimic-kitti360 \
    qvyon-mimic-osdar23 qvyon-mimic-pandaset
uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4

If any command is missing, recreate the clean .venv and repeat the TestPyPI installation.

Wheel Package Guide

Core SDK Packages

  • qvyon-mimic-filesystem: Filesystem abstraction used by SDK and converters; import it as mimic_filesystem.

  • qvyon-mimic-data-api: Canonical MIMIC schema classes (Scene, sensors, trajectories, actors), plus validation/serialization model definitions; import it as mimic_data_api.

Example Converter Packages

  • qvyon-mimic-wod: Example converter for Waymo Open Dataset; import it as mimic_wod.

  • qvyon-mimic-nuscene: Example converter for nuScenes Dataset; import it as mimic_nuscene.

  • qvyon-mimic-kitti360: Example converter for KITTI-360; import it as mimic_kitti360.

  • qvyon-mimic-osdar23: Example converter for OSDaR23; import it as mimic_osdar23.

  • qvyon-mimic-pandaset: Example converter for PandaSet, including calibrated Hesai Pandar64 LiDAR export; import it as mimic_pandaset.

These examples show how to build your own converter using mimic-data-api + mimic-filesystem.

Example Converter Sensor Profiles

  • Waymo (mimic-wod): 5 cameras, 1 lidar.

  • nuScenes (mimic-nuscene): 6 cameras, 1 lidar, 5 radars.

  • KITTI-360 (mimic-kitti360): 1 pinhole camera + 2 fisheye cameras + 1 lidar (common profile; converter is configurable up to 4 cameras).

  • OSDaR23 (mimic-osdar23): 3 cameras + 1 merged lidar (default 3-cam preset). Radar data exists in dataset but converter currently skips radar output.

  • PandaSet (mimic-pandaset): supports 6 camera streams + 1 Hesai Pandar64 LiDAR; the default and example conversion uses 3 front cameras. It does not default to a single camera.

Example Converter Setup Commands

All examples below assume a single shared .venv where all TestPyPI packages are already installed.

source .venv/bin/activate

For mimic_wod, add the Waymo runtime packages to the same .venv after the MIMIC packages:

uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4

Command patterns for wod-convert, nuscene-convert, and osdar23-convert are aligned with mimic-data-pipeline-bash task invocations.

Waymo Example (5 cameras, 1 lidar)

Main conversion command behavior:

  • Reads a Waymo context/sequence from your local Waymo dataset mirror.

  • Converts selected camera/lidar frames into one MIMIC scene.json plus sensor files.

  • Pulls data from --env.input_path (typically a directory synced from gs://waymo_open_dataset_v_2_0_0/).

  • Exports camera images with native distortion parameters; it does not create an undistorted-images folder.

wod-convert --env.input_path /data/datasets/wod/ --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/wod/wod_1050_3cam_augment --env.output_file_name scene.json --op.context_name=10500357041547037089_1474_800_1494_800 --op.start_frame=0 --op.end_frame=200 --op.cameras FRONT FRONT_LEFT FRONT_RIGHT

Mandatory post-step for WOD:

  • Run mimic-scene-preprocess if your workflow requires undistorted camera images.

  • wod-convert alone will not produce an undistorted-images directory.

TensorFlow CUDA/TensorRT warning lines are expected on CPU-only machines and do not indicate conversion failure when wod-convert finishes successfully.

nuScenes Example (6 cameras, 1 lidar, 5 radars)

Main conversion command behavior:

  • Reads one nuScenes scene from your local nuScenes dataset root.

  • Converts all requested camera/lidar/radar streams into MIMIC scene output.

  • Pulls data from --env.input_path (dataset root containing v1.0-* metadata and sensor folders).

nuscene-convert --env.input_path /data/datasets/nuscenes --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/nuscene/nuscenes_0655_3cam_no_augment --env.output_file_name scene.json --op.sequence=scene-0655 --op.version=v1.0-trainval --op.cameras FRONT FRONT_LEFT FRONT_RIGHT --op.lidars LIDAR_TOP --op.radars all --op.train_split_fraction 1.0

KITTI-360 Example (1 pinhole, 2 fisheye, 1 lidar)

Main conversion command behavior:

  • Reads one KITTI-360 drive sequence from your local KITTI-360 root.

  • Converts chosen camera set and Velodyne lidar into MIMIC scene output.

  • Pulls data from --env.kitti360_root (root with data_2d_raw, data_3d_raw, calibration, data_poses).

kitti360-convert --env.kitti360_root /data/datasets/kitti-360 --env.output_path /data/kitti_00_900 --env.output_file_name scene.json --op.sequence_name 2013_05_28_drive_0000_sync --op.start_frame 900 --op.end_frame 1100 --op.cameras cam_00 cam_02 cam_03 --op.rectified True --op.process_lidar True --op.process_sick_lidar True --op.process_actors True --op.unroll_lidar True --op.fisheye_mask_skip 100 --op.fisheye_mask_threshold 25.0 --op.mask_sanity True --op.debug_export_world_pcl True

OSDaR23 Example (3 cameras, 1 lidar)

Main conversion command behavior:

  • Reads one OSDaR23 sequence folder and its OpenLABEL annotations.

  • Converts selected camera preset and merged lidar into MIMIC scene output.

  • Pulls data from --env.input_path (OSDaR23 dataset root) and --op.sequence.

osdar23-convert --env.input_path /data/datasets/osdar23_data/ --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/osdar23/osdar_6_2_1cam_augment_ba --env.output_file_name scene.json --op.sequence 6_station_klein_flottbek_6.2 --op.cameras 1cam_high_resolution --op.start_frame 60 --op.end_frame 160 --op.process_lidar True --op.process_actors True

PandaSet Example (3 cameras, 1 LiDAR)

Main conversion command behavior:

  • Reads one PandaSet sequence from your local PandaSet root.

  • Converts the selected cameras, calibrated Pandar64 LiDAR, ego trajectory, and cuboid tracks into a MIMIC scene.

  • Uses --input_path and --output_path; --end_frame is exclusive.

pandaset-convert --input_path /data/datasets/pandaset --output_path /data/datasets/pandaset/mimic/001 --sequence 001 --cameras front_left_camera front_camera front_right_camera

Use pandaset-convert --help for all six camera names and frame-range options. See the package README for PandaSet download layout, calibration details, and pipeline configuration.

For OSDaR23, frame IDs are sequence-specific. Ensure --op.end_frame >= --op.start_frame; otherwise the selected range becomes empty.

📚 Quick Start

For Dataset Conversion

  1. Start here: Dataset Conversion Guide

  2. ROS bags: ROS Bag Conversion

  3. Code examples: Quick Reference

  4. Check your work: Validation Guide

  5. Use as template: Converter Template

For Developers

🎯 What is MIMIC Format?

MIMIC is a standardized JSON-based format for autonomous driving datasets. It describes:

  • Ego vehicle: Position, orientation, sensor positions over time

  • Sensors: Camera intrinsics/extrinsics, LiDAR specs, radar parameters

  • Actors: Other objects in the scene with positions and dimensions

  • Data files: References to actual images and point clouds

🚀 Common Tasks

Convert a Dataset

# 1. Prepare your data
# 2. Edit the converter template
cp example_converter.py my_converter.py
# [Edit my_converter.py for your dataset format]

# 3. Run conversion
uv run python my_converter.py /path/to/input --output-path /path/to/output

# 4. Validate result
uv run python validate_dataset.py /path/to/output/scene.json

Validate Your Dataset

uv run python validate_dataset.py /path/to/scene.json

Load Scene Example

import json
from pathlib import Path

# Load scene
with open('scene.json') as f:
    scene_data = json.load(f)

# Access ego vehicle
ego = scene_data['ego_vehicle']
print(f"Ego trajectory: {len(ego['trajectory']['timestamps'])} frames")

# Access actors
actors = scene_data['actors']
print(f"Scene contains {len(actors)} actors")

⚠️ Critical Requirements

1. Coordinate Systems

MIMIC uses specific coordinate conventions - getting this wrong will break everything.

Camera (OpenCV convention):

  • X-axis: right in image

  • Y-axis: down in image

  • Z-axis: forward (into scene)

Vehicle (ISO standard):

  • X-axis: forward direction

  • Y-axis: left direction (perpendicular to forward)

  • Z-axis: up direction

See Coordinate Systems for verification steps.

2. LiDAR Intensity Normalization

Normalize measured LiDAR intensity to [0, 1]; use -1 only for known no-return points.

The SDK loader removes points with intensity -1 by default. If the source has no intensity channel, use a documented non-sentinel placeholder such as 0.0 instead of marking every point as a missing return.

See Intensity Normalization for the full algorithm.

3. File Organization

dataset/
└── scenes/
    └── scene_001/
        ├── scene.json              ← Metadata
        ├── images/camera_0/        ← Camera images (PNG/JPG)
        ├── lidar/top_lidar/        ← Point clouds (PLY, 6-column)
        └── radar/front_radar/      ← Radar point clouds

📖 Documentation Files

File

Purpose

Time

conversion-guide.md

Complete step-by-step guide

30 min

ros-bag-conversion.md

ROS1/ROS2 bags and PointCloud2 conversion

20 min

quick-reference.md

API cheat sheet and examples

15 min

validation-guide.md

Validate your converted dataset

20 min

example_converter.py

Template to customize for your format

-

🔗 Reference Datasets

We provide example converters for common datasets:

  • Waymo Open Dataset: 5-camera + 1-lidar example

  • KITTI-360: 1 pinhole + 2 fisheye + 1 lidar common example

  • nuScenes: 6-camera + 1-lidar + 5-radar example

  • OSDAR23: 3-camera + 1-lidar example (radar currently skipped in exporter)

These show how to handle different sensor layouts and coordinate systems.

❓ FAQ

Q: Which coordinate system should I use? A: Vehicle uses ISO standard (X-forward, Y-left, Z-up). Cameras use OpenCV (X-right, Y-down, Z-forward).

Q: Do I need to normalize LiDAR intensity? A: Yes, absolutely required. Use log-space normalization algorithm.

Q: What if my dataset has fisheye cameras? A: Use FisheyeDistortion parameter in CameraIntrinsics.

Q: How do I handle multiple cameras? A: Create separate Camera objects with unique IDs and names.

📞 Support

🔍 What’s Inside

This package includes:

✅ Complete Conversion Guide: Step-by-step instructions with examples ✅ API Reference: All classes and methods documented ✅ Validation Tools: Automated checks for your dataset ✅ Code Examples: Real-world converter template ✅ Troubleshooting: Solutions for common problems ✅ Reference Implementations: Examples from public datasets

📋 Next Steps

  1. Read Dataset Conversion Guide

  2. Review Quick Reference for code examples

  3. Adapt Converter Template for your dataset

  4. Validate with Validation Guide

  5. Submit your converted dataset

Version: 1.0 | Last Updated: 2026-07-28

For more information, visit the complete API documentation or contact support.