Skip to content

Added more activation function support#2

Open
grantbruer wants to merge 1 commit into
gbruer/input-1dfrom
gbruer/more-activation-functions
Open

Added more activation function support#2
grantbruer wants to merge 1 commit into
gbruer/input-1dfrom
gbruer/more-activation-functions

Conversation

@grantbruer

Copy link
Copy Markdown
Collaborator

This PR lets users use more activation functions.

It has three main parts:

  1. Activation functions are no longer restricted to be invertible or non-invertible for certain networks. The previous behavior would silently give incorrect gradients if an invertible function was given where a non-invertible function was expected and vice versa.
    • I implemented this by making a common interface apply_backward() for backpropagating through invertible and non-invertible functions.
  2. I implemented softplus and hyperbolic activation functions.
  3. I added tests for these changes.

I based it off slimgroup#119, which I expect to be accepted first.

@grantbruer

Copy link
Copy Markdown
Collaborator Author

Probably apply_backward() should be renamed to backward(). I was initially worried about some name clashing with that name, but it is consistent with all the other backward functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant