Evidentiality-guided Generator - Retrieval model

[patrickvonplaten]

🌟 New model addition

Model description

In this paper, we introduce Evidentiality-guided GGenerator, which incorporates evidentiality of passages---whether a passage contains correct evidence to support the output---into training the generator via multi-task learning of answer generation and evidentiality prediction for retrieval-augmented generation. Experimental results show large improvements across three knowledge intensive tasks: open question answering, fact verification and knowledge-enhanced dialogue.

Open source status

• the model implementation is available: https://github.com/AkariAsai/evidentiality_qa/tree/main/evi_gen
• the model weights are available: https://github.com/AkariAsai/evidentiality_qa#fine-tuned-models
• who are the authors: @AkariAsai

Happy to guide anyone who is interested through adding this model! Seems like it gives some nice improvements over RAG!

@qqaatw - maybe interesting for you ;-)

[patrickvonplaten]

cc @patil-suraj

[qqaatw]

I think we could firstly add Fusion-in-Decoder (FiD), paper and code, since Evidentiality-guided Generator is based on it.

[bhavitvyamalik]

Hi everyone, Can I work on adding Fusion-in-Decoder (FiD) to Transformers?

[patrickvonplaten]

Yeah that makes sense - @patil-suraj actually mentioned the same!

[patrickvonplaten]

@bhavitvyamalik - definitely! I think @qqaatw and I would be happy to guide you. Do you want to open a PR for it?

[bhavitvyamalik]

I was just going through the paper and will open a PR for it! Since this will be my first model contribution to Transformers, I wanted to know if there any guidlines that I should follow for the same?

[qqaatw]

@patrickvonplaten - absolutely! After Fusion-in-Decoder (FiD) is merged, I can open a follow-up addition for Evidentiality-guided Generator.

@bhavitvyamalik - I think having a look at contributing guidelines would be helpful in the first step :-)

[bhavitvyamalik]

Update: Preparing the original train/val/test data took some time as they were loading the full wikipedia dump (~13GB) in memory which was not possible from my side. I tried forward pass with available pretrained weights after removing functions that were not required and it worked well! I had a doubt related to FiD:

Integrating the retrieval in FiD is part of their future work, which means the users will have to input the context by themselves. So in what format should I get input from user? Current implementation is taking input like this:

[
    {
        "question":"who got the first nobel prize in physics",
        "answers":[
            "Wilhelm Conrad Röntgen"
        ],
        "ctxs":[
            {
                "title":"Nobel Prize in Physics",
                "text":"Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was awarded to physicist Wilhelm Röntgen in recognition of the extraordinary services he"
            },
            {
                "title":"Nobel Prize",
                "text":"His son, George Paget Thomson, received the same prize in 1937 for showing that they also have the properties of waves. William Henry Bragg and his son, William Lawrence Bragg, shared the Physics Prize in 1915 for inventing the X-ray spectrometer. Niels Bohr was awarded the Physics prize in 1922, as was his son, Aage Bohr, in 1975. Manne Siegbahn, who received the Physics Prize in 1924, was the father of Kai Siegbahn, who received the Physics Prize in 1981. Hans von Euler-Chelpin, who received the Chemistry Prize in 1929, was the father of Ulf von Euler, who was awarded"
            },
            {
                "title":"Nobel Prize in Physics",
                "text":"receive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for mankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901; the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. The first Nobel Prize in Physics was"
            }
        ]
    },
]

[patrickvonplaten]

Great question @bhavitvyamalik ! We've discussed this a bit with @patil-suraj as well. @patil-suraj - would you be interested in guiding @bhavitvyamalik here a bit regarding the design?

[patrickvonplaten]

Happy to help otherwise as well if you're too busy :-)