LingBot-Map: Streaming 3D Reconstruction Model

Understanding LingBot-Map

LingBot-Map is a feed-forward 3D foundation model from the Robbyant Team that reconstructs 3D scenes from streaming video or image sequences. Instead of iteratively optimizing a scene like classic SLAM or bundle adjustment, its Geometric Context Transformer combines anchor context, a pose-reference window, and trajectory memory to predict geometry and camera pose in a single forward pass, at roughly 20 FPS on 518×378 images.

Core Capabilities

• ✓Geometric Context Transformer unifies coordinate grounding, dense geometric cues, and drift correction through anchor context, a pose-reference window, and trajectory memory — all in one streaming architecture.
• ✓Feed-forward inference with paged KV cache attention holds ~20 FPS at 518×378 resolution on sequences beyond 10,000 frames, per the README.
• ✓Keyframe interval and windowed inference modes (--mode windowed, --window_size, --overlap_keyframes) extend the model past its ~320-frame training range for very long recordings.
• ✓Three checkpoints ship on Hugging Face and ModelScope: lingbot-map-long for large scenes, lingbot-map as the balanced default used in the paper, and lingbot-map-stage1, which can load into VGGT for bidirectional inference.
• ✓Offline rendering pipeline (demo_render/batch_demo.py) turns a video or image folder into a headless point-cloud flythrough MP4 without the interactive viewer.
• ✓Sky masking uses an auto-downloaded ONNX segmentation model to strip sky points from outdoor reconstructions, with masks cached to disk between runs.
• ✓Evaluation scripts cover benchmarks including KITTI, Oxford Spires, VBR, Droid-W, TUM-D, 7-scenes, ETH3D, Tanks and Temples, and NRGBD.

Setting Up LingBot-Map

Installation is documented step by step. Create a conda environment with Python 3.10, then install PyTorch 2.8.0 with torchvision 0.23.0 against CUDA 12.8 (`pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128`) — this version is recommended because NVIDIA Kaolin, needed for the batch renderer, ships prebuilt wheels only for torch-2.8.0_cu128. Install the package itself with `pip install -e .`, then add FlashInfer (`pip install --index-url https://pypi.org/simple flashinfer-python`) for paged KV cache attention; without it the model falls back to PyTorch's native SDPA attention via `--use_sdpa`. Visualization extras are optional: `pip install -e ".[vis]"`. The offline rendering pipeline needs more: `pip install -e ".[vis,render]"` for open3d and pyyaml, onnxruntime-gpu for batched sky masking, Kaolin built against the same torch/CUDA combo, ffmpeg, and a local build step (`cd demo_render/render_cuda_ext && python setup.py build_ext --inplace`) for its CUDA extensions.

Other 3D Reconstruction Approaches

Frequently Asked Questions

Best use cases

• •Prototyping streaming 3D reconstruction from a phone or camera video without building a SLAM pipeline from scratch.
• •Batch-rendering long indoor or outdoor walkthroughs (the README's example runs to roughly 25,000 frames) into a point-cloud flythrough MP4 via the offline pipeline.
• •Evaluating streaming reconstruction methods against public benchmarks such as KITTI and Oxford Spires using the released evaluation scripts.
• •Reconstructing scenes from video generated by LingBot-World (the README's companion world model) to inspect its output geometry.
• •Loading the stage-1 checkpoint into VGGT for bidirectional, non-streaming inference when you don't need the streaming property.

How to use

Download one of three checkpoints from Hugging Face or ModelScope (robbyant/lingbot-map) — lingbot-map-long, lingbot-map, or lingbot-map-stage1 — then point demo.py at it: `python demo.py --model_path /path/to/lingbot-map.pt --image_folder example/courthouse --mask_sky`. That launches an interactive viser viewer at http://localhost:8080 using the three bundled example scenes (courthouse, university, loop). For sequences past roughly 320 frames — the model's video RoPE training range — set `--keyframe_interval` to thin the KV cache, or switch to `--mode windowed --window_size 128` for anything past 3,000 frames. Sequences too long for the browser viewer, like the README's ~25,000-frame, 13-minute indoor walkthrough, go through `demo_render/batch_demo.py` instead, which renders a headless point-cloud flythrough MP4 directly from a video file or image folder using the same checkpoint.

Who should try it — and who should skip

Try LingBot-Map if you're prototyping streaming 3D reconstruction from video or photo sequences on an NVIDIA GPU and want a single feed-forward pass instead of assembling a SLAM or SfM pipeline from scratch — the three example scenes and viser viewer get you to a first result fast. Skip it if you need CPU-only inference, published accuracy numbers before you commit GPU time, or a mature production deployment story: this reads as a research release with an active TODO list and a 'coming soon' checkpoint, not a finished product.