Ternarization of Models

[ajay-vikram]

I have a ternarized model where the weights are in { -1. 0. 1 }. Model weights require only 2 bits, and we have no MACs but rather only ACs since all multiplications can be essentially converted to additions and subtractions. However, according to the neurobench code, this will not be manifested. It still considers the operations as MACs and not ACs. Also, the memory footprint is calculated using the data type size of the tensor elements in bytes. In our case, we only need 2 bits for the ternarized weights. So we don't need an entire 32 bits to store 1 weight.
What is to be done? Is there a way to incorporate this for my model or do I need to mention stuff in the comments of the submission file? Also, what is the deadline time for submission in Indian Standard Time?

[jasonlyik]

Hi Ajay, that's great, I have suggestions for your cases:

1. For footprint, it should be simple to extend the existing footprint metric (function is here), where you calculate based on 2 bits per parameter/element if the weights are ternary. You can extend the existing metric or define a new metric in a local cloned repo, and use poetry to run the harness from the local repo instead of the pip package (instructions here).
2. For the synops we currently account for ACs only when activations are binary/ternary, and not when weights are binary/ternary. You could take a similar approach as above where you extend the existing metric, but if all weights are ternary then you can simply count all reported MACs as ACs, and outline your calculations in your submission report.

If you do extend the harness for your metrics calculations, it would be great if you could issue a pull request to add the feature as well.

[jasonlyik]

Tagging @vinniesun We are starting to get questions on the Grand Challenge submission, can you answer? It may also be good to send a reminder to participants that the deadline is approaching.

[vinniesun]

@jasonlyik @ajay-vikram I've sent out a couple of reminder emails with regards to how the submission process should be proceeded to each team leader. Below is the emailed content:

_- Submission Format:
For your self-reported metrics, please include them in your submitted test script like the examples in Neurobench:

They will be updated daily to a Google Sheet (https://docs.google.com/spreadsheets/d/100-8LmTZ8_qWtA8eF0giFMVqH1FEVCwMI8KszWDDeXQ/edit?usp=sharing), and the verified result will be updated daily on the scoreboard page of the Grand Challenge (http://1.117.17.41/neural-decoding-grand-challenge/).

• Deadline

Just a gentle reminder email that the deadline for this challenge is on the 2nd of August. Please remember to submit your code to the Form Submit page at http://1.117.17.41/neural-decoding-grand-challenge/

_

If you have multiple files that we need to run to verify your result, please zip them all together when submitting them. The submission date is midnight of the 2nd of August, Hong Kong Time.

[ajay-vikram]

Okay thanks a lot!

[ajay-vikram]

Hey @jasonlyik Would it be possible to get a deadline extension. We are working on incorporating ternarization into the metrics calculations. The ternarized training takes a lot of time and it would be hard to get all 6 models ternarized and the metrics code changed in the remaining time. Can something be done?

[jasonlyik]

@vinniesun Are there any plans to allow a deadline extension?

[vinniesun]

@jasonlyik @ajay-vikram Sorry guys, unfortunately there's no plan for a deadline extension, as we need to align our Grand Challenge's schedule with the rest of BioCAS.