pub fn pad_token_batch(
rows: &[Vec<u32>],
pad_id: u32,
device: &Device,
) -> Result<TokenBatch>Expand description
Right-pad token rows to a common length and build the matching mask.
pad_id must be the tokenizer’s actual pad token, not an arbitrary unused
id. It is not cosmetic: ESMC::forward
reconstructs its own key-padding mask as tokens.ne(pad_id) when none is
supplied, so padding with a different id would leave those positions
looking like real residues to the one model that infers its own mask.