Struct T5Runner 

pub struct T5Runner { /* private fields */ }

A T5 encoder wrapped for PlmRunner.

Why the model sits behind a Mutex

T5EncoderModel::forward takes &mut self, because the same T5Attention code serves the decoder, where it maintains a KV cache. PlmRunner::embed takes &self, and widening the trait to &mut self would make every runner harder to share across threads for the sake of one model.

The encoder does not actually cache. T5Attention::load sets use_cache: cfg.use_cache && decoder, and T5EncoderModel::load builds its stack with decoder = false, so the cache branch is dead on this path whatever config.json says. load_config also forces use_cache: false so the guarantee does not depend on a detail of candle’s internals, and test_repeated_embed_is_deterministic fails if a future candle version starts carrying state across calls.

So the Mutex buys the &mut the signature demands while keeping the runner Send + Sync, and it is uncontended in the sense that matters: there is no state to protect. It does serialise concurrent embed calls on one runner, which is a real cost — clone the runner per thread if that bites.

Implementations

impl T5Runner

pub fn from_pretrained(model: T5Models, device: Device) -> Result<Self>

Load from the hub at the checkpoint’s own published precision.

See T5Models::published_dtype: F16 for ProtT5, F32 for Ankh. Use from_pretrained_with to override.

pub fn from_pretrained_with(model: T5Models, opts: &LoadOptions) -> Result<Self>

Load with an explicit device and dtype.

Trait Implementations

impl PlmRunner for T5Runner

fn embed(&self, sequence: &str) -> Result<Tensor>

Run the T5 encoder and return per-token hidden states.

Shape: (1, L + 1, d_model) — the trailing </s> row is included, and there is no BOS row.

fn special_tokens(&self) -> SpecialTokenLayout

ProtT5 appends </s> and prepends nothing: T5’s build_inputs_with_special_tokens is ids + [eos], and the vocabulary has no BOS piece at all.

Every other runner in the crate is BOS_EOS, which is exactly why the layout is declared rather than assumed — reading a ProtT5 embedding as if row 0 were a BOS token shifts every residue by one.

fn residue_count(&self, sequence: &str) -> usize

Whitespace is not a residue: "M K T" is three residues, not five.

fn model_name(&self) -> &str

Model name / identifier string (e.g. “esm2”, “amplify”, “esmc”).

fn metadata(&self) -> ModelMetadata

Architecture dimensions, for sizing downstream layers.

fn device(&self) -> &Device

Device the model’s weights live on.

fn embed_residues(&self, sequence: &str) -> Result<Tensor>

Per-residue embeddings with special-token rows removed.

fn logits(&self, _sequence: &str) -> Result<Tensor>

Masked-LM logits over the token vocabulary.

fn embed_batch(&self, sequences: &[&str]) -> Result<Tensor>

Embed several sequences in one forward pass.

fn embed_residues_batch(&self, sequences: &[&str]) -> Result<Tensor>

Per-residue embeddings for a batch, special-token rows removed.