Struct SaProtTokenizer 

pub struct SaProtTokenizer { /* private fields */ }

A tokenizer built from SaProt’s vocab.txt.

Implementations

impl SaProtTokenizer

pub fn from_vocab_txt(contents: &str) -> Result<Self>

Parse a vocab.txt: one token per line, line number is the id.

Rejects a file whose special tokens are absent or misplaced, since the ids are load-bearing — <cls> and <eos> wrap every sequence, and a silently-wrong <unk> would turn unknown residues into a real amino acid rather than an unknown.

pub fn len(&self) -> usize

Number of tokens in the vocabulary.

pub fn is_empty(&self) -> bool

Whether the vocabulary is empty. Never true for a valid vocab.

pub fn token_to_id(&self, token: &str) -> Option<u32>

Look up a token’s id.

pub fn residue_count(&self, sequence: &str) -> usize

How many residues sequence encodes.

A trailing odd character is its own (malformed) residue rather than being dropped, so encode can report it as <unk> instead of silently shortening the sequence.

pub fn encode(&self, sequence: &str) -> Vec<u32>

Encode a SaProt sequence to token ids, without special tokens.

Unknown pairs map to <unk> rather than erroring: a residue whose 3Di state could not be determined is written with a # placeholder in practice, and refusing the whole sequence for one such residue would be unhelpful.

pub fn decode(&self, ids: &[u32]) -> String

Decode ids back to a SaProt sequence, skipping special tokens.

Trait Implementations

impl Clone for SaProtTokenizer

fn clone(&self) -> SaProtTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SaProtTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.