pub fn tokenize_structure(codes: &[u32], add_special_tokens: bool) -> Vec<u32>Expand description
Wrap VQ-VAE codebook indices with optional BOS/EOS special tokens.
codes should contain raw VQ-VAE indices in [0, 4095]. The function
does not validate range — that is left to the VQ-VAE encoder output.