OpenGVLab/Ask-Anything

🦜 VideoChat Family: Ask-Anything

| | | |
[VideoChat-7B-8Bit] End2End ChatBOT for video and image. [InternVideo2-Chat-8B-HD]

中文 README 及 中文交流群 | Paper

⭐️: We are also working on a updated version, stay tuned!

:fire: Updates

• 2026/07/17: 🚀🚀 We release VideoChat3, a fully open, efficient 4B Video MLLM for general, long-form, and streaming video understanding. VideoChat3 improves 18/19 offline and 10/11 streaming metrics over Qwen3-VL-4B. We release the model weights, code, training recipes, and complete datasets. Check out our paper and homepage!

• 2025/01/18: We release videochat-flash and videochat-tpo to extend MLLMs' capabilities on both long and accurate video understanding. videochat-flash sets new records in mutiple video benchmarks (for both short and long videos), improving code usability by leveaging LLaVA and others. videochat-tpo exploits classical vision task annotations (e.g. tracking) to optimize MLLMs in a DPO manner, enhancing MLLMs' performance and enabling capabilities in tracking, segmentation, and more.

• 2024/06/25: We release the branch of videochat2 using vllm, speed up the inference of videochat2.

• 2024/06/19: 🎉🎉 Our VideoChat2 achieves the best performances among the open-sourced VideoLLMs on MLVU, a multi-task long video understanding benchmark.

• 2024/06/13: Fix some bug and give testing scripts/

  • :warning: We replace some repeated (~30) QAs in MVBench, which may only affect the results by 0.5%.
  • :loudspeaker: We give the scripts for testing EgoSchema and Video-MME, please check the demo_mistral.ipynb and demo_mistral_hd.ipynb.

• 2024/06/07: :fire::fire::fire: We release VideoChat2_HD, which is fine-tuned with high-resolution data and is capable of handling more diverse tasks. It showcases better performance on different benchmarks, especially for detailed captioning. Furthermore, it achieves 54.8% on Video-MME, the best score among 7B MLLMs. Have a try! 🏃🏻‍♀️🏃🏻

• 2024/06/06: We release VideoChat2_phi3, a faster model with robust performaces.

• 2024/05/22: We release VideoChat2_mistral, which shows better capacity on diverse tasks (60.4% on MVBench, 78.6% on NExT-QA, 63.8% on STAR, 46.4% on TVQA, 54.4% on EgoSchema-full and 80.5% on IntentQA). More details have been updated in the paper.

• 2024/04/05 MVBench is selected as Poster (Highlight)!

• 2024/2/27 MVBench is accepted by CVPR2024.

• 2023/11/29 VideoChat2 and MVBench are released.

  • VideoChat2 is a robust baseline built on UMT and Vicuna-v0.
  • 2M diverse instruction data are released for effective tuning.
  • MVBench is a comprehensive benchmark for video understanding.

• 2023/05/11 End-to-end VideoChat and its technical report.

  • VideoChat1: Instruction tuning for video chatting (also supports image one).
  • Paper: We present how we craft VideoChat with two versions (via text and embed) along with some discussions on its background, applications, and more.

• 2023/04/25 Watch videos longer than one minute with chatGPT

  • VideoChat LongVideo: Incorporating langchain and whisper into VideoChat.

• 2023/04/21 Chat with MOSS

  • VideoChat with MOSS: Explicit communication with MOSS.

• 2023/04/20: Chat with StableLM

  • VideoChat with StableLM: Explicit communication with StableLM.

• 2023/04/19: Code release & Online Demo

  • VideoChat with ChatGPT: Explicit communication with ChatGPT. Sensitive with time.
  • MiniGPT-4 for video: Implicit communication with Vicuna. Not sensitive with time. (Simple extension of MiniGPT-4, which will be improved in the future.)

🔨 Getting Started

Build video chat with:

• End2End
• ChatGPT
• StableLM
• MOSS
• MiniGPT-4

:clapper: [End2End ChatBot]

https://github.com/OpenGVLab/Ask-Anything/assets/24236723/a8667e87-49dd-4fc8-a620-3e408c058e26

Your browser does not support the video tag.

:movie_camera: [Communication with ChatGPT]

https://user-images.githubusercontent.com/24236723/233630363-b20304ab-763b-40e5-b526-e2a6b9e9cae2.mp4

Your browser does not support the video tag.

:page_facing_up: Citation

If you find this project useful in your research, please consider cite:

@article{2023videochat,
  title={VideoChat: Chat-Centric Video Understanding},
  author={KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao},
  journal={arXiv preprint arXiv:2305.06355},
  year={2023}
}

@inproceedings{li2024mvbench,
  title={Mvbench: A comprehensive multi-modal video understanding benchmark},
  author={Li, Kunchang and Wang, Yali and He, Yinan and Li, Yizhuo and Wang, Yi and Liu, Yi and Wang, Zun and Xu, Jilan and Chen, Guo and Luo, Ping and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={22195--22206},
  year={2024}
}

🌤️ Discussion Group

If you have any questions during the trial, running or deployment, feel free to join our WeChat group discussion! If you have any ideas or suggestions for the project, you are also welcome to join our WeChat group discussion!

We are hiring researchers, engineers and interns in General Vision Group, Shanghai AI Lab. If you are interested in working with us, please contact Yi Wang ([email protected]).