A look at NousResearch/Obsidian: 206 stars on GitHub, written primarily in Python. Maybe the new state of the art vision model? we'll see š¤·āāļø
Snapshot summary built from the project's own GitHub metadata ā there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then ā see the READ ME tab for the original README in full.
You just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen. You may notice that there is no model in the model list. Do not worry, as we have not launched any model worker yet. It will be automatically updated when you launch a model worker.
Launch a model worker
This is the actual worker that performs the inference on the GPU. Each worker is responsible for a single model specified in --model-path.
Wait until the process finishes loading the model and you see "Uvicorn running on ...". Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.
Training
1. Pretraining
Please download the 558K subset of the LAION-CC-SBU dataset with BLIP captions here.
Pretrain takes around 2.5 hours for Obsidian-3B-V0.5 on 4x A100 (80G), at 336px for the vision module.
Training script with DeepSpeed ZeRO-2: pretrain.sh.
--mm_projector_type mlp2x_gelu: the two-layer MLP vision-language connector.
Please download the annotation of the final mixture our instruction tuning data llava_v1_5_mix665k.json, and download the images from constituting datasets:
COCO: train2017
GQA: images
OCR-VQA: download script, we save all files as .jpg
TextVQA: train_val_images
VisualGenome: part1, part2
After downloading all of them, organize the data as follows in ./playground/data,
Our GPT-assisted evaluation pipeline for multimodal modeling is provided for a comprehensive understanding of the capabilities of vision-language models. Please see our paper for more details.
Evaluate the generated responses. In our case, answer-file-ref.jsonl is the response generated by text-only GPT-4 (0314), with the context captions/boxes provided.
Does NousResearch/Obsidian have a project website?
No homepage URL was recorded for NousResearch/Obsidian in TopGit's last sync. The README tab above frequently contains screenshots and demo links, or check the repository description on GitHub.
How many stars does NousResearch/Obsidian have?
NousResearch/Obsidian has 206 GitHub stars ā refresh the page for the live number, or check github.com/NousResearch/Obsidian. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is NousResearch/Obsidian open source?
Yes ā NousResearch/Obsidian ships under the Apache-2.0 license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/NousResearch/Obsidian.
What is NousResearch/Obsidian?
NousResearch/Obsidian (NousResearch/Obsidian) is a Python project on GitHub. From the project's own README: Maybe the new state of the art vision model? we'll see š¤·āāļø
Where do I read more about NousResearch/Obsidian?
This TopGit page is a snapshot ā the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/NousResearch/Obsidian is the definitive source.
Read full README in the tab above.
Want a second opinion on Obsidian?
Ask an AI that can read this page ā one click and you get its take on Obsidian.