Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better SVD computation for convolution #89

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Better SVD computation for convolution #89

wants to merge 3 commits into from

Conversation

cofri
Copy link
Collaborator

@cofri cofri commented Feb 21, 2024

  • The creation of the equivalent convolution matrix requires to remove bias and activation in the convolutional layer. This wasn't done before. This is now fixed.
  • The equivalent matrix is now sparse and the SVD computation is also based on sparse matrices. Note that the smallest singular value is very expensive to compute with scipy.sparse

@cofri
fix(compute_layer_sv): remove bias and activation in convolution SVD
0e161a9 
When building the equivalent matrix of a convolution layer, the bias and
activation must be removed to only keep the convolutional operation of the
layer.
By cloning the layer and setting use_bias and activation to False and None
respectively, only the convolution is performed.
Note that cloning the model already built the new model with the correct
input shape.
@cofri cofri force-pushed the fix/bias_and_activations_in_conv_SVD branch from 82b21d8 to 161519e Compare February 22, 2024 12:53
@cofri cofri force-pushed the fix/bias_and_activations_in_conv_SVD branch from 161519e to 4c1d441 Compare February 22, 2024 14:49
@cofri
test(compute_layer_sv): add unit test with bias and activation
d2c6542 
SpectralConv2D SV computation must remove bias and activation before
computing the equivalent matrix. A unit test of a SpectralConv2D with
bias and ReLU is added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@cofri