Skip to main content

vocab_id

Function vocab_id 

Source
pub fn vocab_id(c: char) -> u32
Expand description

The token id for any character in the shared vocabulary.

Case is meaning here, not formatting. ProstT5 puts amino acids and 3Di states in one vocabulary and distinguishes them by case: M is methionine (19) and m is a structural state (144). So this maps uppercase through residue_id and lowercase through three_di_id, exactly as HuggingFace’s tokenizer does — and deliberately does not case-fold.

Folding case would be an unforced deviation from the reference that changes the ids: lowercasing an amino-acid sequence would silently reinterpret it as a structural one, and the model would translate it fluently and wrongly.