How can I extract embeddings?

[isaacOnline]

Hi! Would someone be able to provide an example showing how to extract embeddings?

[isaacOnline]

I created a fork to extract embeddings here. The embeddings get stored as a part of the transcription result that's returned when you call model.transcribe() and are done at the segment level.

[haiderasad]

What is the shape of the embeddings? Why do you want them in the first place?

I’m extracting embeddings for each segment that the models splits the audio into, and for each segment I’m extracting one set of embeddings for the encoder and one set for the decoder. Each of those sets will have four dimensions:

• One corresponding to the number of samples you're processing
• One corresponding to the number of transformer layers in the Whisper model that you’re using
• One corresponding to the the length of the segment
• One corresponding to the width of the Whisper model you’re using

I'm using them for adapting Whisper to another task.

can you tell what the decoder embeddings represent overall? the finalized text in the specified language or is it related to speech embeddings

[isaacOnline]

Unfortunately I've mostly been using the encoder embeddings myself.

[tusharagarwal25]

hi @haiderasad, were you able to work with the embeddings?

[tusharagarwal25]

hi @isaacOnline, getting this error: UnboundLocalError: local variable 'encoder_embeddings' referenced before assignment

Can you please help here?

[isaacOnline]

The stuff I wrote is in a separate "extract-embeddings" branch that you'll need to checkout. I think I've gotten the error you show when I've forgotten to switch to the right branch. You may also need to do the pip install:

git checkout extract-embeddings
pip install --editable ./

[KMC07]

Did you ever try using the embeddings for speaker diarization, and if not what were you using the embedding for?

[saeidmokaram]

Thanks @isaacOnline, This is very helpful. I was wondering what sort of information we should expect to be captured by the encoder_embeddings and what information by the decoder_embeddings?

[alxmares]

You can extract embeddings from the Whisper model using the Transformers library as follows:

processor = WhisperProcessor.from_pretrained("openai/whisper-base")
model = WhisperModel.from_pretrained("openai/whisper-base")

inputs = processor(speech_array, sampling_rate=16000, return_tensors="pt")

with torch.no_grad():
    encoder_outputs = model.encoder(inputs.input_features)

embeddings = encoder_outputs.last_hidden_state