lingbot-map / README.md
Flor's picture
card: update from fastforks 1cbf1671
7a6de40 verified
|
Raw History Blame Contribute Delete
5.62 kB
---
library_name: "fastforks"
pipeline_tag: "image-to-3d"
license: "apache-2.0"
tags:
- "multiview"
- "multi-view-stereo"
- "3d-reconstruction"
- "camera-pose-estimation"
- "cuda"
- "pytorch"
- "fastforks"
- "fastforks-card"
base_model:
- "robbyant/lingbot-map"
---
# LingBot-Map (fastforks)
LingBot-Map (Robbyant, 2026): streaming 3D reconstruction. A VGGT-style aggregator (DINOv2 ViT-L patch embed, 24 frame / global attention blocks, 3D RoPE) runs causally over the views in order with a KV cache (scale frames + a sliding window), then a camera head and a DPT depth head give cameras and depth of every view.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/Robbyant/lingbot-map> at `b9dd1b1`.
Author weight repos: <https://huggingface.co/robbyant/lingbot-map>.
Task: multiview. Tier: experimental. Outputs: `depth`, `confidence`, `intrinsics`, `cam_to_world`. Backends: `portable`, `kernels`.
Input: several images of one scene (posed and calibrated for multi-view stereo, unposed otherwise). Output: depth maps and, where the model predicts them, cameras and 3-D points.
## Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/multiview/models/lingbot_map/card.toml).
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
| `lingbot-map@1` | [lingbot-map.pt](https://huggingface.co/robbyant/lingbot-map/resolve/e8984048a940fbdfa19909f8cc0c0ad834c1ee5a/lingbot-map.pt) | `ee665103348e` | Apache-2.0 | yes |
## Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Licence files: `LICENSE`.
Training data: Per the README: a stage-1 bidirectional (VGGT-style) checkpoint, then streaming training; datasets are not listed.
Licence evidence: https://github.com/Robbyant/lingbot-map/blob/b9dd1b1/LICENSE.txt and https://huggingface.co/robbyant/lingbot-map (no licence field)
## Input size
any H x W >= 1, resized or padded inside the model
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `lingbot-map` 3000x4000 (5.5 GiB peak allocated, 6.8 reserved)
## Speed and memory
### Multiview, synthetic
| model | input | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `lingbot-map` | 480x640 | – | – | 730 | – | – | – | – | 4512 / 4770 | – |
### Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- |
| `lingbot-map` | – | 6.58 (480x640) | – | – | 1.08 (480x640) | – |
### torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `lingbot-map` | 480x640 | 730 ms | planned for 0.2.1 | deferred (build >5 min) | – | – | – | 4512 / 4770 | – | – | – |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
## Accuracy
### Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.02 and p99 <= 0.25 (absolute error of the output `depth`).
## Limitations
- Views are a sequence (a video): each view only sees the first `scale_frames` views and the views before it (older than `window` frames: their 6 special tokens only). At most 1024 views.
## Usage
```python
from fastforks import multiview
model = multiview.load("lingbot-map") # mode="auto": the preferred backend from recorded measurements
result = model(["view0.jpg", "view1.jpg", "view2.jpg"]) # the views of one scene, no cameras needed
result.depth # [V, H, W] float32 tensor; result.kind says what the maps mean
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
```
Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `lingbot-map`.
## Kernels backend
`multiview.load("lingbot-map", mode="kernels")` runs our CUDA kernels (`fastbb`, `cudnn_hacc`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
<!-- staged from fastforks 1cbf1671 (2026-11-15) -->