cvg/glue-factory

To evaluate the pre-trained SuperPoint+LightGlue model on HPatches, run:

python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite

You should expect the following results

{'H_error_dlt@1px': 0.3515,
 'H_error_dlt@3px': 0.6723,
 'H_error_dlt@5px': 0.7756,
 'H_error_ransac@1px': 0.3428,
 'H_error_ransac@3px': 0.5763,
 'H_error_ransac@5px': 0.6943,
 'mnum_keypoints': 1024.0,
 'mnum_matches': 560.756,
 'mprec@1px': 0.337,
 'mprec@3px': 0.89,
 'mransac_inl': 130.081,
 'mransac_inl%': 0.217,
 'ransac_mAA': 0.5378}

The default robust estimator is opencv, but we strongly recommend to use poselib instead:

python -m gluefactory.eval.hpatches --conf superpoint+lightglue-official --overwrite \
    eval.estimator=poselib eval.ransac_th=-1

Setting eval.ransac_th=-1 auto-tunes the RANSAC inlier threshold by running the evaluation with a range of thresholds and reports results for the optimal value. Here are the results as Area Under the Curve (AUC) of the homography error at 1/3/5 pixels:

To evaluate GlueStick on HPatches, run:

python -m gluefactory.eval.hpatches --conf gluefactory/configs/superpoint+lsd+gluestick.yaml --overwrite

You should expect the following results

{"mprec@1px": 0.245,
 "mprec@3px": 0.838,
 "mnum_matches": 1290.5,
 "mnum_keypoints": 2287.5,
 "mH_error_dlt": null,
 "H_error_dlt@1px": 0.3355,
 "H_error_dlt@3px": 0.6637,
 "H_error_dlt@5px": 0.7713,
 "H_error_ransac@1px": 0.3915,
 "H_error_ransac@3px": 0.6972,
 "H_error_ransac@5px": 0.7955,
 "H_error_ransac_mAA": 0.62806,
 "mH_error_ransac": null}

Since we use points and lines to solve for the homography, we use a different robust estimator here: Hest. Here are the results as Area Under the Curve (AUC) of the homography error at 1/3/5 pixels:

To evaluate the pre-trained SuperPoint+LightGlue model on MegaDepth-1500, run:

python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official
# or the adaptive variant
python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official \
    model.matcher.{depth_confidence=0.95,width_confidence=0.95}

The first command should print the following results

{'mepi_prec@1e-3': 0.795,
 'mepi_prec@1e-4': 0.15,
 'mepi_prec@5e-4': 0.567,
 'mnum_keypoints': 2048.0,
 'mnum_matches': 613.287,
 'mransac_inl': 280.518,
 'mransac_inl%': 0.442,
 'rel_pose_error@10°': 0.681,
 'rel_pose_error@20°': 0.8065,
 'rel_pose_error@5°': 0.5102,
 'ransac_mAA': 0.6659}

To use the PoseLib estimator:

python -m gluefactory.eval.megadepth1500 --conf superpoint+lightglue-official \
    eval.estimator=poselib eval.ransac_th=2.0

To evaluate the pre-trained SuperPoint+GlueStick model on MegaDepth-1500, run:

python -m gluefactory.eval.megadepth1500 --conf gluefactory/configs/superpoint+lsd+gluestick.yaml

Here are the results as Area Under the Curve (AUC) of the pose error at 5/10/20 degrees:

To evaluate the pre-trained SuperPoint+LightGlue model on Scannet-1500, run:

python -m gluefactory.eval.scannet1500 --conf superpoint+lightglue-official
# or the adaptive variant
python -m gluefactory.eval.scannet1500 --conf superpoint+lightglue-official \
    model.matcher.{depth_confidence=0.95,width_confidence=0.95}

Here are the results as Area Under the Curve (AUC) of the pose error at 5/10/20 degrees:

To evaluate GlueStick on ETH3D, run:

python -m gluefactory.eval.eth3d --conf gluefactory/configs/superpoint+lsd+gluestick.yaml

You should expect the following results

AP: 77.92
AP_lines: 69.22

python -m gluefactory.eval.inspect hpatches superpoint+lightglue-official

Click on a point to visualize matches on this pair.

To compare multiple methods on a dataset:

python -m gluefactory.eval.inspect hpatches superpoint+lightglue-official superpoint+superglue-official

All current benchmarks are supported by the viewer.

We show how to train LightGlue with SuperPoint. We first pre-train LightGlue on the homography dataset:

python -m gluefactory.train sp+lg_homography \  # experiment name
    --conf gluefactory/configs/superpoint+lightglue_homography.yaml

Feel free to use any other experiment name. By default the checkpoints are written to outputs/training/. The default batch size of 128 corresponds to the results reported in the paper and requires 2x 3090 GPUs with 24GB of VRAM each as well as PyTorch >= 2.0 (FlashAttention). Configurations are managed by OmegaConf so any entry can be overridden from the command line. If you have PyTorch < 2.0 or weaker GPUs, you may thus need to reduce the batch size via:

python -m gluefactory.train sp+lg_homography \
    --conf gluefactory/configs/superpoint+lightglue_homography.yaml  \
    data.batch_size=32  # for 1x 1080 GPU

Be aware that this can impact the overall performance. You might need to adjust the learning rate accordingly.

We then fine-tune the model on the MegaDepth dataset:

python -m gluefactory.train sp+lg_megadepth \
    --conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
    train.load_experiment=sp+lg_homography

Here the default batch size is 32. To speed up training on MegaDepth, we suggest to cache the local features before training (requires around 150 GB of disk space):

# extract features
python -m gluefactory.scripts.export_megadepth --method sp --num_workers 8
# run training with cached features
python -m gluefactory.train sp+lg_megadepth \
    --conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
    train.load_experiment=sp+lg_homography \
    data.load_features.do=True

The model can then be evaluated using its experiment name:

python -m gluefactory.eval.megadepth1500 --checkpoint sp+lg_megadepth

You can also run all benchmarks after each training epoch with the option --run_benchmarks.

We first pre-train GlueStick on the homography dataset:

python -m gluefactory.train gluestick_H --conf gluefactory/configs/superpoint+lsd+gluestick-homography.yaml --distributed

Feel free to use any other experiment name. Configurations are managed by OmegaConf so any entry can be overridden from the command line.

We then fine-tune the model on the MegaDepth dataset:

python -m gluefactory.train gluestick_MD --conf gluefactory/configs/superpoint+lsd+gluestick-megadepth.yaml --distributed

Note that we used the training splits train_scenes.txt and valid_scenes.txt to train the original model, which contains some overlap with the IMC challenge. The new default splits are now train_scenes_clean.txt and valid_scenes_clean.txt, without this overlap.