VBench Evaluation Format

[DaLawnChair]

Hello,

I was wondering how VBench is set up for evaluating the generated videos.

The original VBench format are simplistic prompts. Self Forcing and Causal Forcing notes that they use LLM extended prompts. Are the LLM extended prompts used for both generation and evaluation, ie. do you make modifications to the vbench_full_info.json file to use the LLM-extended prompts?

There had been a mention of it in this thread, but it had not been answered:
#18 (comment)

If this is the case, then do you modify VBench itself in order to get results for it's non-supported operations, like ['object_class', 'multiple_objects', 'scene', 'appearance_style', 'color', 'spatial_relationship']? These are required for cumulative scoring, but VBench doesn't support custom prompts on these. I also notice the same issue with human_action as is made to parse non-extended prompts.

I'm trying to run evaluation myself, and am conflicted on what to evaluate between.

[DaLawnChair]

Another question, VBench suggests a miniumum of 5 seeds per prompt, with 25 seeds for temporal flickering. Does Causal Forcing follow this?

[zhuhz22]

Hi @DaLawnChair ,

• Yes both generation and evaluation used the extended prompts in Self Forcing, as mentioned in Question about Vbench scores #18.
• We indeed need to modify the VBench evaluation, but it's simple as you only need to modify the character strings. You can ask LLM agent to modify. This step is indispensable, but it has also been the customary practice since Self Foricng.
• We uniformly use 5 seeds. In fact, changing the seed has little impact on the generation results.

[DaLawnChair]

We indeed need to modify the VBench evaluation, but it's simple as you only need to modify the character strings. You can ask LLM agent to modify. This step is indispensable, but it has also been the customary practice since Self Foricng.

When you mean "character strings", are you referring to the prompts or the parsing?

In my current setup, for the non-supported dimension and human_action, the videos are being generated with the LLM extended prompts, but are evaluated using the base prompts to avoid parsing issues.

IE. For dimensions like human_action, they fit action directly to the prompt.
prompt: "person is riding a bike"
Action: "riding a bike"
Evaluation (unmodified) : checking if Action matches directly to its topK list of actions from the evaluation model.

[zhuhz22]

You can refer to the following code for human_action:

def human_action(umt_path, video_list, device):
    state_dict = torch.load(umt_path, map_location='cpu')
    model = create_model(
        "vit_large_patch16_224",
        pretrained=False,
        num_classes=400,
        all_frames=16,
        tubelet_size=1,
        use_learnable_pos_emb=False,
        fc_drop_rate=0.,
        drop_rate=0.,
        drop_path_rate=0.2,
        attn_drop_rate=0.,
        drop_block_rate=None,
        use_checkpoint=False,
        checkpoint_num=16,
        use_mean_pooling=True,
        init_scale=0.001,
    )
    data_transform = Compose([
        Resize(256, interpolation='bilinear'),
        CenterCrop(size=(224, 224)),
        ClipToTensor(),
        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])
    model = model.to(device)
    model.load_state_dict(state_dict, strict=False)
    model.eval()
    cat_dict = build_dict()
    cnt= 0
    cor_num = 0
    video_results = []
    for video_path in tqdm(video_list, disable=get_rank() > 0):
        cor_num_per_video = 0
        video_label_ls = video_path
        from pathlib import Path
        ext_path = "<path to prompts/vbench/all_dimension_extend.txt>"
        with open(ext_path, "r", encoding="utf-8") as f:
            trunc2full = {}
            for ln in f:
                s = ln.rstrip("\n")
                k = s[:100] # We saved the videos with names of prompt[:100] when generation
                if k not in trunc2full: 
                    trunc2full[k] = s

        v = video_label_ls[0] if isinstance(video_label_ls, list) else video_label_ls
        stem = Path(v).stem
        video_label_ls=trunc2full[stem]

        cnt += 1
        images = load_video(video_path, data_transform, num_frames=16)
        images = images.unsqueeze(0)
        images = images.to(device)
        with torch.no_grad():
            logits = torch.sigmoid(model(images))
            results, indices = torch.topk(logits, 5, dim=1)
        indices = indices.squeeze().tolist()
        results = results.squeeze().tolist()
        results = [round(f, 4) for f in results]
        cat_ls = []
        for i in range(5):
            if results[i] >= 0.85:
                cat_ls.append(cat_dict[str(indices[i])])
        flag = False
        for cat in cat_ls:
            import re
            def words(s):
                return set(re.findall(r"\b[\w']+\b", s.lower()))
            cat_w = words(cat)
            if isinstance(video_label_ls, str):
                video_label_ls = [video_label_ls]
            has_intersection = any(cat_w & words(v) for v in video_label_ls)
            if has_intersection:
                cor_num += 1
                cor_num_per_video += 1
                flag = True
                break
        if flag is False:
            pass
        video_results.append({
            'video_path': video_path, 
            'video_results': flag,
            'cor_num_per_video': cor_num_per_video,})
    acc = cor_num / cnt
    return acc, video_results

Here, we only require that a match exists to count as a correct human action prediction, because otherwise (e.g., in your implementation) the predicted category may not be a substring of the LLM-extended prompts — they are not in a containment relationship. For example, prompt: "A person is riding a bike", extended prompt: "A person is riding a luxurious bike", predicted action: "riding a bike" — this would fail a strict string match. Moreover, we found that this relaxed criterion does not introduce frequent false positives.