You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As discussed in the meeting with Kevin. For most skills (practically, all but IoTs) device IDs don't need to be correct at parse time because they can be added automatically as postprocessing.
We still want to include them as "exact match accuracy" because they are part of the target program. We want pre-normalization token-by-token exact match accuracy to be the target metric due to how seq2seq works.
To account for this, and remove some less-relevant errors from error analysis of devices, we should introduce a new partial accuracy metric, "ok_without_device_id". This would be implemented in SentenceEvaluator and in the associated cmdline code.
We can implement this using some token manipulation (recognizing the sequence of tokens "id = GENERIC_ENTITY_*" and removing it), or with a proper NodeVisitor that visits all DeviceSelectors and sets the id to null.
The text was updated successfully, but these errors were encountered:
As discussed in the meeting with Kevin. For most skills (practically, all but IoTs) device IDs don't need to be correct at parse time because they can be added automatically as postprocessing.
We still want to include them as "exact match accuracy" because they are part of the target program. We want pre-normalization token-by-token exact match accuracy to be the target metric due to how seq2seq works.
To account for this, and remove some less-relevant errors from error analysis of devices, we should introduce a new partial accuracy metric, "ok_without_device_id". This would be implemented in SentenceEvaluator and in the associated cmdline code.
We can implement this using some token manipulation (recognizing the sequence of tokens "id = GENERIC_ENTITY_*" and removing it), or with a proper NodeVisitor that visits all DeviceSelectors and sets the
id
tonull
.The text was updated successfully, but these errors were encountered: