Proper way of registering gluon V2.0 custom parameter?

[feevos]

Dear all,

I was wondering as to what is the proper way to add a custom parameter in mxnet 2.0? It is my understanding that in the latest version of mxnet (2.++), we can use definition of forward with a custom HybridBlock, and the operation can still be hybridized. For example, this is working in the sense that I get a x2-x3 speed up after hybridizing (in my tests):

class V2Test(HybridBlock):
    def __init__(self, **kwards):
        super().__init__(**kwards)
        
        self.conv = gluon.nn.HybridSequential()
        for _ in range(5):
            self. conv. add(gluon.nn.Conv2D(channels=32,kernel_size=3,padding=1))
        
    def forward(self, input):
        return self.conv(input)

net = V2Test()
net.initialize()
net.hybridize()
.... 

Now, adding a custom parameter, with the "old" 1.X this works fine:

import mxnet as mx
from mxnet import gluon
from mxnet.gluon import HybridBlock

from mxnet import nd 

class SigmoidCrisp(HybridBlock):
    def __init__(self, smooth=1.e-2,**kwards):
        super().__init__(**kwards)


        self.smooth = smooth
        self.gamma  = gluon.Parameter('gamma', shape=(1,), init=mx.init.One())


    def hybrid_forward(self, F, input, gamma):
            out = self.smooth + F.sigmoid(gamma)
            out = F.reciprocal(out)

            out = input*out
            out = F.sigmoid(out)
            return out

net = SigmoidCrisp() 
net.initialize() 

xx = nd.random.uniform(shape=[3,32,128,128])
net.summary(xx)
--------------------------------------------------------------------------------
        Layer (type)                                Output Shape         Param #
================================================================================
               Input                           (3, 32, 128, 128)               0
      SigmoidCrisp-1                           (3, 32, 128, 128)               1
================================================================================
Parameters in forward computation graph, duplicate included
   Total params: 1
   Trainable params: 1
   Non-trainable params: 0
Shared params in forward computation graph: 0
Unique parameters in model: 1
--------------------------------------------------------------------------------

But when I try this with the new mxnet V2.0 format to add a parameter, like this:

import mxnet as mx
from mxnet import gluon
from mxnet.gluon import HybridBlock

from mxnet import np as F
from mxnet import npx as Fx

class SigmoidCrisp(HybridBlock):
    def __init__(self, smooth=1.e-2,**kwards):
        super().__init__(**kwards)


        self.smooth = smooth
        self.gamma  = gluon.Parameter('gamma', shape=(1,), init=mx.init.One())


    def forward(self, input, gamma):
    #def hybrid_forward(self, F, input, gamma):
            out = self.smooth + Fx.sigmoid(gamma)
            out = F.reciprocal(out)

            out = input*out
            out = Fx.sigmoid(out)
            return out



Fx.set_np()
net = SigmoidCrisp()
net.initialize()

# numpy like random 
xx = np.random.rand(3,32,128,128)

net.summary(xx)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-4cea76f26e76> in <module>
----> 1 net.summary(xx)

/usr/local/lib/python3.7/site-packages/mxnet/gluon/block.py in summary(self, *inputs)
    829         try:
    830             self.apply(_register_summary_hook)
--> 831             self(*inputs)
    832 
    833             line_format = '{:>20}  {:>42} {:>15}'

/usr/local/lib/python3.7/site-packages/mxnet/gluon/block.py in __call__(self, x, *args)
   1406             if not self._active:
   1407                 # Normal imperative computation of forward()
-> 1408                 return super().__call__(x, *args)
   1409 
   1410             if dc.is_deferred_compute():

/usr/local/lib/python3.7/site-packages/mxnet/gluon/block.py in __call__(self, *args)
    709             hook(self, args)
    710 
--> 711         out = self.forward(*args)
    712 
    713         for hook in self._forward_hooks.values():

*TypeError: forward() missing 1 required positional argument: 'gamma'*

If I use the hybrid_forward(self, F, input, gamma) definition, it recognizes gamma, and can accept mx.np input array, but I have to be cautious not to use F for any operations (because they are treated as ndarray or symbol). I am aware of the workaround ( F --> F.npx or F--> F.np, depending on the case), but I was wondering if there is a better way to register a parameter with mxnet/gluon 2.0 and still use simple def forward(self, input)?

Thank you very much for your time and this awesome library,
Foivos

[leezu]

To access parameters in HybridBlock.forward API you should follow the same approach as in Block.forward. Thus, you can access your custom parameter via self.gamma.data() inside the forward method.

[feevos]

thank you very much!