Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add self attention encoder #139

Open
Adamits opened this issue Sep 28, 2023 · 1 comment
Open

Add self attention encoder #139

Adamits opened this issue Sep 28, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@Adamits
Copy link
Collaborator

Adamits commented Sep 28, 2023

With the decoupling of encoders and decoders, we have added a Linear encoder, which seems to just embed the inputs and pass them along. We should also add a SelfAttention encoder, which encodes the embeddings with a self attention layer (and no positional encoding).

This contextualizes the embeddings by representing each as a linear combination of itself wrt all other embeddings.

@Adamits Adamits self-assigned this Sep 28, 2023
@kylebgorman
Copy link
Contributor

+1. Makes sense.

@kylebgorman kylebgorman added the enhancement New feature or request label Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants