Skip to main content

encode

Function encode 

Source
pub fn encode(sequence: &str) -> Vec<u32>
Expand description

Token ids for sequence, with the trailing </s> and no BOS.

Whitespace is skipped, so "MKT" and "M K T" are the same input.