Expand description
ESM3 structure tokenization.
Structure tokens are VQ-VAE codebook indices (0..4095). This module handles wrapping them with BOS/EOS special tokens for model input.
Functionsยง
- tokenize_
structure - Wrap VQ-VAE codebook indices with optional BOS/EOS special tokens.