Enum TokenizerSpec 

pub enum TokenizerSpec {
    HfJson,
    Embedded(&'static str),
    BuiltinVocab(&'static str),
    HfVocabTxt(VocabAlphabet),
    None,
}

Where a model’s tokenizer comes from.

Not incidental detail: the four ported families genuinely differ here, and that knowledge used to be buried in six separate runners.

Variants

HfJson

A tokenizer.json downloaded from the model’s own HF repo.

Embedded(&'static str)

A tokenizer.json compiled into the binary with include_bytes!, named by its path under src/.

BuiltinVocab(&'static str)

A hand-written vocabulary table in Rust, named by its module path.

HfVocabTxt(VocabAlphabet)

A bare vocab.txt from the model’s repo — one token per line, the line number being the id. SaProt ships this instead of a tokenizer.json, and so do the BERT-style protein models, over a completely different alphabet — so the file format alone does not say how to read a sequence. The VocabAlphabet does.

None

No tokenizer: the model consumes structure, not sequence.

Trait Implementations

impl Clone for TokenizerSpec

fn clone(&self) -> TokenizerSpec

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenizerSpec

impl Debug for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerSpec

impl PartialEq for TokenizerSpec

fn eq(&self, other: &TokenizerSpec) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenizerSpec