Struct SpecialTokenLayout 

pub struct SpecialTokenLayout {
    pub leading: usize,
    pub trailing: usize,
}

How many special tokens a runner’s tokenizer wraps a sequence in.

Named counts rather than an enum of model names, so porting a new model means declaring its shape rather than editing a match somewhere else.

Fields

leading: usize

Tokens prepended before the first residue (e.g. BOS / <cls>).

trailing: usize

Tokens appended after the last residue (e.g. EOS).

Implementations

impl SpecialTokenLayout

pub const NONE: Self

No special tokens: model output rows map 1:1 onto residues.

pub const BOS_EOS: Self

One leading BOS and one trailing EOS — what every currently ported model does.

pub const EOS_ONLY: Self

One trailing EOS and no BOS — T5’s layout, which ProtT5 uses.

Distinct from BOS_EOS by one leading row, which is exactly enough to misalign every residue if assumed rather than read.

pub const fn total(&self) -> usize

Total number of non-residue rows.

Trait Implementations

impl Clone for SpecialTokenLayout

fn clone(&self) -> SpecialTokenLayout

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for SpecialTokenLayout

impl Debug for SpecialTokenLayout

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for SpecialTokenLayout

impl PartialEq for SpecialTokenLayout

fn eq(&self, other: &SpecialTokenLayout) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for SpecialTokenLayout