OpenDriveLab/DriveAGI

[ NEW❗️] 2024/09/08 We released a mini version of OpenDV-YouTube, containing 25 hours of driving videos. Feel free to try the mini subset by following instructions at OpenDV-mini!

2024/05/28 We released our latest research, Vista, a generalizable driving world model. It's capable of predicting high-fidelity and long-horizon futures, executing multi-modal actions, and serving as a generalizable reward function to assess driving behaviors.

2024/03/24 OpenDV-YouTube Update: Full suite of toolkits for OpenDV-YouTube is now available, including data downloading and processing scripts, as well as language annotations. Please refer to OpenDV-YouTube.

2024/03/15 We released the complete video list of OpenDV-YouTube, a large-scale driving video dataset, for GenAD project. Data downloading and processing script, as well as language annotations, will be released next week. Stay tuned.

2024/01/24 We are excited to announce some update to our survey and would like to thank John Lambert, Klemens Esterle from the public community for their advice to improve the manuscript.

🌏 A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024)

Quick facts:

• Introducing the world's first generalizable driving world model.
• Task: High-fidelity, action-conditioned, and long-horizon future prediction for driving scenes in the wild.
• Dataset: OpenDV-YouTube, nuScenes
• Code and model: https://github.com/OpenDriveLab/Vista
• Video Demo: https://vista-demo.github.io
• Related work: Vista, GenAD

@inproceedings{gao2024vista,
 title={Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability}, 
 author={Shenyuan Gao and Jiazhi Yang and Li Chen and Kashyap Chitta and Yihang Qiu and Andreas Geiger and Jun Zhang and Hongyang Li},
 booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
 year={2024}
}

@inproceedings{yang2024genad,
  title={{Generalized Predictive Model for Autonomous Driving}},
  author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

Quick facts:

• Task: given the language prompts as input, predict the trajectory in the scene
• Origin dataset: nuScenes, CARLA (To be released)
• Repo: https://github.com/OpenDriveLab/DriveLM, https://github.com/OpenDriveLab/ELM
• Related work: DriveLM, ELM
• Related challenge: Driving with Language AGC Challenge 2024

Abstract

With the continuous maturation and application of autonomous driving technology, a systematic examination of open-source autonomous driving datasets becomes instrumental in fostering the robust evolution of the industry ecosystem. In this survey, we provide a comprehensive analysis of more than 70 papers on the timeline, impact, challenges, and future trends in autonomous driving dataset.

Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future

• English Version
• Chinese Version Accepted at SCIENTIA SINICA Informationis (中文版)

@article{li2024_driving_dataset_survey,
 title = {Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future},
 author = {Hongyang Li and Yang Li and Huijie Wang and Jia Zeng and Huilin Xu and Pinlong Cai and Li Chen and Junchi Yan and Feng Xu and Lu Xiong and Jingdong Wang and Futang Zhu and Chunjing Xu and Tiancai Wang and Fei Xia and Beipeng Mu and Zhihui Peng and Dahua Lin and Yu Qiao},
 journal = {SCIENTIA SINICA Informationis},
 year = {2024},
 doi = {10.1360/SSI-2023-0313}
}

Current autonomous driving datasets can broadly be categorized into two generations since the 2010s. We define the Impact (y-axis) of a dataset based on sensor configuration, input modality, task category, data scale, ecosystem, etc.

Related Work Collection

We present comprehensive paper collections, leaderboards, and challenges.(Click to expand)

The Largest up-to-date 3D Occupancy Forecasting dataset for visual pre-training.

Quick facts:

• Task: given the large amount of data, predict the 3D occupancy in the environment.
• Origin dataset: nuPlan
• Repo: https://github.com/OpenDriveLab/OpenScene
• Related work: OccNet
• Related challenge: 3D Occupancy Prediction Challenge 2023, Occupancy and Flow AGC Challenge 2024, Predictive World Model AGC Challenge 2024

Flourishing OpenLane-V2 with Standard Definition (SD) Map and Map Elements.

Quick facts:

• Task: given multi-view images and SD-map (also known as ADAS map) as input, build the driving scene on the fly without the aid of HD-map.
• Repo: https://github.com/OpenDriveLab/OpenLane-V2
• Related work: OpenLane-V2, TopoNet, LaneSegNet
• Related challenge: Lane Topology Challenge 2023, Mapless Driving AGC Challenge 2024