pub fn encode(sequence: &str) -> Vec<u32>
Token ids for sequence, with the trailing </s> and no BOS.
sequence
</s>
Whitespace is skipped, so "MKT" and "M K T" are the same input.
"MKT"
"M K T"