MatAnyone2/docs/EVAL.md

Evaluation Documentation

This document shows the way to reproduce the quantitative results in our paper on two synthetic benchmarks (YouTubeMatte and VideoMatte) and one real benchmark (CRGNN).

YouTubeMatte

📦 We provide the inference results with MatAnyone 2 on the YouTubeMatte benchmark here.

Preparation

• YouTubeMatte.zip (6.24G)
• YouTubeMatte_first_frame_seg_mask.zip (310K)

To run the inference scripts, your files should be arranged as:

data
   |- YouTubeMatte_first_frame_seg_mask   # for inference only
   |- YouTubeMatte
        |- youtubematte_512x288
        |- youtubematte_1920x1080

Batch Inference

Empirically, for low-resolution (youtubematte_512x288) and high-resolution (youtubematte_1920x1080) data, we set different hyperparameter values for --warmup, --erode_kernel, and --dilate_kernel.

# lr: youtubematte_512x288
bash evaluation/infer_batch_lr_yt.sh

# hr: youtubematte_1920x1080
bash evaluation/infer_batch_hr_yt.sh

Evaluation

To run the evaluation scripts, your files should be arranged as:

data
   |- YouTubeMatte
        |- youtubematte_512x288
        |- youtubematte_1920x1080

   |- results
        |- youtubematte_512x288
        |- youtubematte_1920x1080

# lr: youtubematte_512x288
python evaluation/eval_lr.py \
    --pred-dir ./data/results/youtubematte_512x288 \
    --true-dir ./data/YouTubeMatte/youtubematte_512x288 

# hr: youtubematte_1920x1080
python evaluation/eval_hr.py \
    --pred-dir ./data/results/youtubematte_1920x1080 \
    --true-dir ./data/YouTubeMatte/youtubematte_1920x1080 

VideoMatte

📦 We provide the inference results with MatAnyone 2 on the VideoMatte benchmark here.

Preparation

• videomatte_512x512.tar (1.8G)
• videomatte_1920x1080.tar (2.2G)
• VideoMatte_first_frame_seg_mask.zip (416K)

To run the inference scripts, your files should be arranged as:

data
   |- VideoMatte_first_frame_seg_mask   # for inference only
   |- VideoMatte
        |- videomatte_512x288
        |- videomatte_1920x1080

Batch Inference

Empirically, for low-resolution (videomatte_512x288) and high-resolution (videomatte_1920x1080) data, we set different hyperparameter values for --warmup, --erode_kernel, and --dilate_kernel.

# lr: videomatte_512x288
bash evaluation/infer_batch_lr_vm.sh

# hr: videomatte_1920x1080
bash evaluation/infer_batch_hr_vm.sh

Evaluation

To run the evaluation scripts, your files should be arranged as:

data
   |- VideoMatte
        |- videomatte_512x288
        |- videomatte_1920x1080

   |- results
        |- videomatte_512x288
        |- videomatte_512x288

# lr: videomatte_512x288
python evaluation/eval_lr.py \
    --pred-dir ./data/results/videomatte_512x288 \
    --true-dir ./data/VideoMatte/videomatte_512x288 

# hr: videomatte_1920x1080
python evaluation/eval_hr.py \
    --pred-dir ./data/results/videomatte_1920x1080 \
    --true-dir ./data/VideoMatte/videomatte_1920x1080 

CRGNN

📦 We provide the inference results with MatAnyone 2 on the CRGNN benchmark here.

Preparation

• real_human_data
• CRGNN_first_frame_seg_mask.zip (151K)

To run the inference scripts, your files should be arranged as:

data
   |- crgnn   
     |- alpha
     |- image_allframe
     |- mask              # first frame seg mask

Batch Inference

bash evaluation/infer_batch_crgnn.sh

Evaluation

To run the evaluation scripts, your files should be arranged as:

data
   |- crgnn
        |- alpha

   |- results
        |- crgnn

python evaluation/eval_crgnn.py \
    --pred-dir ./data/results/crgnn \
    --true-dir ./data/crgnn/alpha