Seed Questions in the Dataset

[joshinh]

@danyaljj Firstly thanks for releasing the dataset!

If I understood correctly, every perturbed example in this dataset has been created from some seed example from the original BOOLQ dataset. But there seem to be a few clusters (eg. id=25894) which have no seed questions. Is this behaviour expected or am I missing something?

[danyaljj]

Hey there 👋

You're right, some clusters in the data don't have instances that belong to BoolQ.
Here is why: after perturbing the BoolQ [subset] instances, we have a verification step where humans check if a given question is well-formed questions (i.e., humans agree on its answer). Few of the BoolQ [subset] instances were deleted at this stage since our human annotators didn't like them.

Does that answer your question?

[joshinh]

Thanks, that makes sense! I was under the impression that the verification step is only performed for perturbed instances. I suppose that is also the reason why some clusters have size = 1.