pub fn encode_for_translation(sequence: &str, direction: Direction) -> Vec<u32>Expand description
Token ids for a ProstT5 translation input: the direction token, the
sequence, then </s>.
The direction prefix is not decoration — ProstT5 is trained with it, and
omitting it or using the wrong one produces a fluent, wrong translation
rather than an error. The prefix is the only thing direction controls:
the characters themselves are mapped by vocab_id, which reads case as
meaning. Whitespace is skipped, as everywhere else here.