Struct SwiGLU 

pub struct SwiGLU { /* private fields */ }

Implementations

impl SwiGLU

pub fn load(vb: VarBuilder<'_>, config: &ESM3Config) -> Result<Self>

Load the FFN, taking the bias terms from the checkpoint rather than assuming they are absent (ferritin-100.27).

The two checkpoints that share this block disagree, and hardcoding either answer is silently wrong for the other:

checkpoint	ffn.1.bias / ffn.3.bias
esm3_sm_open_v1.pth (the sequence model)	absent
esm3_structure_encoder_v0.pth (the VQ-VAE encoder)	present

This used to be linear_no_bias unconditionally, which is correct for the sequence model and drops two real tensors per block for the structure encoder. Nothing caught it: the loader only ever asked for the weights, so “every tensor resolves” stayed true while the biases sat unread in the file. The cost was ~4% drift in the encoder’s latent — invisible to any self-consistency check, but enough to move 11 of 93 residues onto a different codebook entry.

Probed rather than made a config flag because the checkpoint is the authority here, and the same probing idiom already resolves ESM-C’s wrapper prefix in loader::optional_prefix.

Trait Implementations

impl Module for SwiGLU

fn forward(&self, x: &Tensor) -> Result<Tensor>