Skip to main content

pad_token_batch

Function pad_token_batch 

Source
pub fn pad_token_batch(
    rows: &[Vec<u32>],
    pad_id: u32,
    device: &Device,
) -> Result<TokenBatch>
Expand description

Right-pad token rows to a common length and build the matching mask.

pad_id must be the tokenizer’s actual pad token, not an arbitrary unused id. It is not cosmetic: ESMC::forward reconstructs its own key-padding mask as tokens.ne(pad_id) when none is supplied, so padding with a different id would leave those positions looking like real residues to the one model that infers its own mask.