an example of a word-level language model using FluxML #357
Conversation
| grad_x = clamp!(gradient[x], -args.clip, args.clip) | ||
| # backprop | ||
| x .-= lr .* grad_x |
There was a problem hiding this comment.
In the spirit of promoting best practices, we have https://fluxml.ai/Flux.jl/stable/training/optimisers/#Flux.Optimise.ClipValue and the rest of https://fluxml.ai/Flux.jl/stable/training/optimisers/ for this. I imagine you'd want to use something more sophisticated than plain SGD anyhow.
There was a problem hiding this comment.
@ToucheSir I opted out of the library methods as this explicit calculation provided better performance. However, for the promotion of best practices I agree and I can switch to using the library methods.
|
|
||
| # logit cross entropy loss function | ||
| function loss(x, y) | ||
| Flux.reset!(model) |
There was a problem hiding this comment.
Would highly recommend moving this outside of the loss function (i.e. into the training loop).
There was a problem hiding this comment.
@ToucheSir Can you elaborate on this? My understanding is that the loss function is called for each batch in a single training loop, thus for a sequential language model such as this one we want to reset the hidden state after each batch.
There was a problem hiding this comment.
I presume the suggestion is to keep it inside the for batch in data_loader loop, but outside the gradient call.
| hold_out = zip(x_train[end-5:end], y_train[end-5:end]) | ||
|
|
||
| # used for updating hyperparameters | ||
| best_val_loss = nothing |
There was a problem hiding this comment.
| best_val_loss = nothing | |
| local best_val_loss |
Co-authored-by: Brian Chen <[email protected]>
No description provided.