New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test mmbenchv1.1 #45

Open

ZhaoyangLi-nju opened this issue Dec 23, 2024 · 1 comment

ZhaoyangLi-nju commented Dec 23, 2024

how to test mmbenchv1.1?

Member

kennymckormick commented Dec 26, 2024

Hi, @ZhaoyangLi-nju ,

We recommend you to adopt VLMEvalKit: https://github.com/open-compass/VLMEvalKit to evaluate mmbench v1.1. You need to first clone the repo and install it locally.

Usage: https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md

# Multi-GPU
torchrun --nproc-per-node={N_PROC} run.py --data {DATASET_NAME} --model {MODEL_NAME}
# Single-GPU
python run.py --data {DATASET_NAME} --model {MODEL_NAME}

For MMBench v1.1, the corresponding dataset names are:

MMBench_DEV_EN_V11
MMBench_DEV_CN_V11
MMBench_TEST_EN_V11
MMBench_TEST_CN_V11

For test splits, you need to submit the prediction files to our evaluation service: https://mmbench.opencompass.org.cn/mmbench-submission to get the final scores.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment