MXNET Gluon: Error when calling two consecutive backward - Check failed: type_ != nullptr

[Arhamna]

I need to call backward function for two different models, separately.

Loss_1



Losses_1 = []
with autograd.record():
    
    
   for batch_per_gpu in batch
es:        
       pred = model_1(batch_per_gpu)
        
       loss = loss_func(pred, truth)
        
       losses_1.append(loss)
   mx.autograd.backward(losses_1)


model_1_trainer.allreduce_grads()
model_1_trainer.update()




Loss_2

Losses_2 = []
with autograd.record():
    
    
   for batch_per_gpu in batches:
        
       pred = model_2(batch_per_gpu)
        
       loss = loss_func(pred, truth)

       losses_2.append(loss)
   mx.autograd.backward(losses_2)

model_2_trainer.allreduce_grads()

model_2_trainer.update()

I get this error on the second call of the backward function:

Check failed: type_ != nullptr: The any container is empty

Does anyone know how to fix this?

My MXNET version is 1.6.0

My issue is quite similar to this discussion #9712, except that I need to treat both losses separately.

[szha]

Hi @Arhamna. What's likely happening is that some of the variables/arrays are shared between these two models, while each NDArray only supports one entry for autograd tracing. As a result, the gradient tape in the first backward is overwritten by the second. Sorry that the error message is unclear and it would be really helpful if you could create a bug report for it.

A workaround I'd recommend is adding the losses together and do backward together. Since derivative is a linear function you can get the same result as doing two backward steps separately.