Why do we select only the 1st char in the decoded string in get_next_state() ?

[ekagra-ranjan]

Hi! I faced an issue where in the FSM in CFG Guide is getting self.generation differently from what the model is outputting. Debugging it lead to this line

Shouldn;t it be just self.generation += self.tokenizer.decode([token_id]) instead of self.generation += self.tokenizer.decode([token_id])[0] since the decode function in HF returns a str?

@rlouf

[ekagra-ranjan]

gentle reminder: @brandonwillard @rlouf

[brandonwillard]

@benlipkin and/or @lapp0 might know, but I don't off the top of my head.

[lapp0]

This appears to be a serious bug. Perhaps it's the root cause of #796

It looks like the signature of decode was confused with the signature of batch_decode.

Will investigate as part of #796

[lapp0]

@ekagra-ranjan I was mistaken in my initial review. This is an outlines.models.tokenizer.Tokenizer which returns List[str] for decode()

Could you please share a full reproduction script?