# MIMIC Data API - Customer Documentation

Welcome! This folder contains everything you need to convert your autonomous driving dataset to the MIMIC data format.

## Minimum Input Dataset Requirements

MIMIC supports input datasets with at least **one camera** and **one LiDAR**.
Radar sensors are optional. Additional cameras, LiDARs, and radar sensors are
supported when the converter can provide their calibration, trajectories, and
sensor data.

## Setup (TestPyPI Installation)

> **Mandatory requirement:** use a fresh Python 3.10 virtual environment for each SDK release.

### Install uv

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Open a new terminal after installation so the ``uv`` command is available on your ``PATH``.

### Install the SDK and Converters

```bash
uv venv --python 3.10 .venv
source .venv/bin/activate
uv pip install \
    --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    qvyon-mimic-filesystem qvyon-mimic-data-api qvyon-mimic-raillabel \
    qvyon-mimic-wod qvyon-mimic-nuscene qvyon-mimic-kitti360 \
    qvyon-mimic-osdar23 qvyon-mimic-pandaset
uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4
```

### Verify the Environment

```bash
uv pip list | grep -E "qvyon-mimic-filesystem|qvyon-mimic-data-api"
```

When you need to run Python commands, either activate ``.venv`` first or use ``uv run python``. The converter examples below use ``uv run python`` so they work without a manual shell activation step.

### Verify the Environment

```bash
uv pip list | grep -E "qvyon-mimic-(filesystem|data-api|raillabel|wod|nuscene|kitti360|osdar23|pandaset)"
command -v kitti360-convert
command -v osdar23-convert
command -v nuscene-convert
command -v wod-convert
command -v pandaset-convert
```

`mimic_wod` requires Python 3.10 because its TensorFlow runtime does not
provide compatible wheels for newer Python versions. Install the Waymo package
with `--no-deps`: its published metadata pins older NumPy/Pandas versions that
conflict with the shared SDK environment. Install `dacite==1.8.1` and
`typeguard==4.4.4` separately.

For the published converter wheels, use Python 3.10, TestPyPI for MIMIC
packages, and public PyPI for third-party dependencies:

```bash
uv pip install \
    --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    qvyon-mimic-wod qvyon-mimic-nuscene qvyon-mimic-kitti360 \
    qvyon-mimic-osdar23 qvyon-mimic-pandaset
uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4
```

If any command is missing, recreate the clean `.venv` and repeat the TestPyPI installation.

## Wheel Package Guide

### Core SDK Packages

- **qvyon-mimic-filesystem**: Filesystem abstraction used by SDK and converters; import it as `mimic_filesystem`.
- **qvyon-mimic-data-api**: Canonical MIMIC schema classes (Scene, sensors, trajectories, actors), plus validation/serialization model definitions; import it as `mimic_data_api`.

### Example Converter Packages

- **qvyon-mimic-wod**: Example converter for Waymo Open Dataset; import it as `mimic_wod`.
- **qvyon-mimic-nuscene**: Example converter for nuScenes Dataset; import it as `mimic_nuscene`.
- **qvyon-mimic-kitti360**: Example converter for KITTI-360; import it as `mimic_kitti360`.
- **qvyon-mimic-osdar23**: Example converter for OSDaR23; import it as `mimic_osdar23`.
- **qvyon-mimic-pandaset**: Example converter for PandaSet, including calibrated Hesai Pandar64 LiDAR export; import it as `mimic_pandaset`.

These examples show how to build your own converter using `mimic-data-api` + `mimic-filesystem`.

## Example Converter Sensor Profiles

- **Waymo (`mimic-wod`)**: 5 cameras, 1 lidar.
- **nuScenes (`mimic-nuscene`)**: 6 cameras, 1 lidar, 5 radars.
- **KITTI-360 (`mimic-kitti360`)**: 1 pinhole camera + 2 fisheye cameras + 1 lidar (common profile; converter is configurable up to 4 cameras).
- **OSDaR23 (`mimic-osdar23`)**: 3 cameras + 1 merged lidar (default 3-cam preset). Radar data exists in dataset but converter currently skips radar output.
- **PandaSet (`mimic-pandaset`)**: supports 6 camera streams + 1 Hesai Pandar64 LiDAR; the default and example conversion uses 3 front cameras. It does not default to a single camera.

## Example Converter Setup Commands

All examples below assume a single shared `.venv` where all TestPyPI packages are already installed.

```bash
source .venv/bin/activate
```

For `mimic_wod`, add the Waymo runtime packages to the same `.venv` after the
MIMIC packages:

```bash
uv pip install --no-deps waymo-open-dataset-tf-2-11-0==1.6.1
uv pip install dacite==1.8.1 typeguard==4.4.4
```

Command patterns for `wod-convert`, `nuscene-convert`, and `osdar23-convert` are aligned with `mimic-data-pipeline-bash` task invocations.

### Waymo Example (5 cameras, 1 lidar)

Main conversion command behavior:
- Reads a Waymo context/sequence from your local Waymo dataset mirror.
- Converts selected camera/lidar frames into one MIMIC `scene.json` plus sensor files.
- Pulls data from `--env.input_path` (typically a directory synced from `gs://waymo_open_dataset_v_2_0_0/`).
- Exports camera images with native distortion parameters; it does not create an undistorted-images folder.

```bash
wod-convert --env.input_path /data/datasets/wod/ --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/wod/wod_1050_3cam_augment --env.output_file_name scene.json --op.context_name=10500357041547037089_1474_800_1494_800 --op.start_frame=0 --op.end_frame=200 --op.cameras FRONT FRONT_LEFT FRONT_RIGHT
```

Mandatory post-step for WOD:
- Run `mimic-scene-preprocess` if your workflow requires undistorted camera images.
- `wod-convert` alone will not produce an undistorted-images directory.

TensorFlow CUDA/TensorRT warning lines are expected on CPU-only machines and do not indicate conversion failure when `wod-convert` finishes successfully.

### nuScenes Example (6 cameras, 1 lidar, 5 radars)

Main conversion command behavior:
- Reads one nuScenes scene from your local nuScenes dataset root.
- Converts all requested camera/lidar/radar streams into MIMIC scene output.
- Pulls data from `--env.input_path` (dataset root containing `v1.0-*` metadata and sensor folders).

```bash
nuscene-convert --env.input_path /data/datasets/nuscenes --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/nuscene/nuscenes_0655_3cam_no_augment --env.output_file_name scene.json --op.sequence=scene-0655 --op.version=v1.0-trainval --op.cameras FRONT FRONT_LEFT FRONT_RIGHT --op.lidars LIDAR_TOP --op.radars all --op.train_split_fraction 1.0
```

### KITTI-360 Example (1 pinhole, 2 fisheye, 1 lidar)

Main conversion command behavior:
- Reads one KITTI-360 drive sequence from your local KITTI-360 root.
- Converts chosen camera set and Velodyne lidar into MIMIC scene output.
- Pulls data from `--env.kitti360_root` (root with `data_2d_raw`, `data_3d_raw`, `calibration`, `data_poses`).

```bash
kitti360-convert --env.kitti360_root /data/datasets/kitti-360 --env.output_path /data/kitti_00_900 --env.output_file_name scene.json --op.sequence_name 2013_05_28_drive_0000_sync --op.start_frame 900 --op.end_frame 1100 --op.cameras cam_00 cam_02 cam_03 --op.rectified True --op.process_lidar True --op.process_sick_lidar True --op.process_actors True --op.unroll_lidar True --op.fisheye_mask_skip 100 --op.fisheye_mask_threshold 25.0 --op.mask_sanity True --op.debug_export_world_pcl True
```

### OSDaR23 Example (3 cameras, 1 lidar)

Main conversion command behavior:
- Reads one OSDaR23 sequence folder and its OpenLABEL annotations.
- Converts selected camera preset and merged lidar into MIMIC scene output.
- Pulls data from `--env.input_path` (OSDaR23 dataset root) and `--op.sequence`.

```bash
osdar23-convert --env.input_path /data/datasets/osdar23_data/ --env.output_path /data/development/jayam.patel/executor-v0.7.0-gpu2-test-data-only/scenes/osdar23/osdar_6_2_1cam_augment_ba --env.output_file_name scene.json --op.sequence 6_station_klein_flottbek_6.2 --op.cameras 1cam_high_resolution --op.start_frame 60 --op.end_frame 160 --op.process_lidar True --op.process_actors True
```

### PandaSet Example (3 cameras, 1 LiDAR)

Main conversion command behavior:
- Reads one PandaSet sequence from your local PandaSet root.
- Converts the selected cameras, calibrated Pandar64 LiDAR, ego trajectory, and cuboid tracks into a MIMIC scene.
- Uses `--input_path` and `--output_path`; `--end_frame` is exclusive.

```bash
pandaset-convert --input_path /data/datasets/pandaset --output_path /data/datasets/pandaset/mimic/001 --sequence 001 --cameras front_left_camera front_camera front_right_camera
```

Use `pandaset-convert --help` for all six camera names and frame-range options. See the package README for PandaSet download layout, calibration details, and pipeline configuration.

For OSDaR23, frame IDs are sequence-specific. Ensure `--op.end_frame >= --op.start_frame`; otherwise the selected range becomes empty.

## 📚 Quick Start

### For Dataset Conversion
1. **Start here**: [Dataset Conversion Guide](conversion-guide.md)
2. **ROS bags**: [ROS Bag Conversion](ros-bag-conversion.md)
3. **Code examples**: [Quick Reference](quick-reference.md)
4. **Check your work**: [Validation Guide](validation-guide.md)
5. **Use as template**: [Converter Template](example_converter.py)

### For Developers
- Read: [Dataset Conversion Guide](conversion-guide.md)
- Reference: [API Reference](quick-reference.md)
- Example: [Converter Template](example_converter.py)

## 🎯 What is MIMIC Format?

MIMIC is a standardized JSON-based format for autonomous driving datasets. It describes:

- **Ego vehicle**: Position, orientation, sensor positions over time
- **Sensors**: Camera intrinsics/extrinsics, LiDAR specs, radar parameters
- **Actors**: Other objects in the scene with positions and dimensions
- **Data files**: References to actual images and point clouds

## 🚀 Common Tasks

### Convert a Dataset
```bash
# 1. Prepare your data
# 2. Edit the converter template
cp example_converter.py my_converter.py
# [Edit my_converter.py for your dataset format]

# 3. Run conversion
uv run python my_converter.py /path/to/input --output-path /path/to/output

# 4. Validate result
uv run python validate_dataset.py /path/to/output/scene.json
```

### Validate Your Dataset
```bash
uv run python validate_dataset.py /path/to/scene.json
```

### Load Scene Example
```python
import json
from pathlib import Path

# Load scene
with open('scene.json') as f:
    scene_data = json.load(f)

# Access ego vehicle
ego = scene_data['ego_vehicle']
print(f"Ego trajectory: {len(ego['trajectory']['timestamps'])} frames")

# Access actors
actors = scene_data['actors']
print(f"Scene contains {len(actors)} actors")
```

## ⚠️ Critical Requirements

### 1. Coordinate Systems
MIMIC uses specific coordinate conventions - **getting this wrong will break everything**.

**Camera (OpenCV convention)**:
- X-axis: right in image
- Y-axis: down in image
- Z-axis: forward (into scene)

**Vehicle (ISO standard)**:
- X-axis: forward direction
- Y-axis: left direction (perpendicular to forward)
- Z-axis: up direction

See [Coordinate Systems](conversion-guide.md) for verification steps.

### 2. LiDAR Intensity Normalization
**Normalize measured LiDAR intensity to [0, 1]; use -1 only for known no-return points.**

The SDK loader removes points with intensity `-1` by default. If the source has
no intensity channel, use a documented non-sentinel placeholder such as `0.0`
instead of marking every point as a missing return.

See [Intensity Normalization](conversion-guide.md) for the full algorithm.

### 3. File Organization
```
dataset/
└── scenes/
    └── scene_001/
        ├── scene.json              ← Metadata
        ├── images/camera_0/        ← Camera images (PNG/JPG)
        ├── lidar/top_lidar/        ← Point clouds (PLY, 6-column)
        └── radar/front_radar/      ← Radar point clouds
```

## 📖 Documentation Files

| File | Purpose | Time |
|------|---------|------|
| [conversion-guide.md](conversion-guide.md) | Complete step-by-step guide | 30 min |
| [ros-bag-conversion.md](ros-bag-conversion.md) | ROS1/ROS2 bags and PointCloud2 conversion | 20 min |
| [quick-reference.md](quick-reference.md) | API cheat sheet and examples | 15 min |
| [validation-guide.md](validation-guide.md) | Validate your converted dataset | 20 min |
| [example_converter.py](example_converter.py) | Template to customize for your format | - |

## 🔗 Reference Datasets

We provide example converters for common datasets:

- **Waymo Open Dataset**: 5-camera + 1-lidar example
- **KITTI-360**: 1 pinhole + 2 fisheye + 1 lidar common example
- **nuScenes**: 6-camera + 1-lidar + 5-radar example
- **OSDAR23**: 3-camera + 1-lidar example (radar currently skipped in exporter)

These show how to handle different sensor layouts and coordinate systems.

## ❓ FAQ

**Q: Which coordinate system should I use?**
A: Vehicle uses ISO standard (X-forward, Y-left, Z-up). Cameras use OpenCV (X-right, Y-down, Z-forward).

**Q: Do I need to normalize LiDAR intensity?**
A: Yes, absolutely required. Use log-space normalization algorithm.

**Q: What if my dataset has fisheye cameras?**
A: Use FisheyeDistortion parameter in CameraIntrinsics.

**Q: How do I handle multiple cameras?**
A: Create separate Camera objects with unique IDs and names.

## 📞 Support

- **API questions?** → See [Quick Reference](quick-reference.md)
- **Conversion issues?** → Check [Conversion Guide](conversion-guide.md)
- **Validation errors?** → Read [Validation Guide](validation-guide.md)
- **Code examples?** → See [Quick Reference](quick-reference.md)

## 🔍 What's Inside

This package includes:

✅ **Complete Conversion Guide**: Step-by-step instructions with examples
✅ **API Reference**: All classes and methods documented
✅ **Validation Tools**: Automated checks for your dataset
✅ **Code Examples**: Real-world converter template
✅ **Troubleshooting**: Solutions for common problems
✅ **Reference Implementations**: Examples from public datasets

## 📋 Next Steps

1. Read [Dataset Conversion Guide](conversion-guide.md)
2. Review [Quick Reference](quick-reference.md) for code examples
3. Adapt [Converter Template](example_converter.py) for your dataset
4. Validate with [Validation Guide](validation-guide.md)
5. Submit your converted dataset

---

**Version**: 1.0 | **Last Updated**: 2026-07-28

For more information, visit the complete API documentation or contact support.