Module saprot_tokenizer 

SaProt’s structure-aware tokenizer (ferritin-goh.3).

SaProt is the ESM-2 architecture with a different alphabet. Each residue is an (amino acid, 3Di structural state) pair over a 20×20 product alphabet — "Ma", "Ld", "Kp" — so a sequence is read two characters at a time and "MdAaLp" is three residues, not six.

Why not tokenizers::Tokenizer

SaProt’s repo ships no tokenizer.json. It has a bare vocab.txt: 446 lines, one token per line, the line number being the id. Reconstructing an HF tokenizer from that would mean assembling a WordLevel model plus a two-character Split pre-tokenizer to reproduce behaviour that is, written directly, a lookup table and chunks(2).

Vocabulary layout

0  <cls>
1  <pad>
2  <eos>
3  <unk>
4  <mask>
5+ Ap, Ay, An, ...   (amino acid, 3Di state) pairs

Those ids match config.json’s pad_token_id: 1 and mask_token_id: 4, which is the cross-check that the file is being read as intended.

Structs

SaProtTokenizer

A tokenizer built from SaProt’s vocab.txt.

Constants

CHARS_PER_RESIDUE

Characters per residue in SaProt’s product alphabet.

UNKNOWN_STATE

SaProt’s placeholder for an unknown amino acid or unknown structural state.

Functions

interleave

Weave an amino-acid sequence together with a 3Di structural string into SaProt’s product alphabet (ferritin-goh.13).