Skip to main content

Module registry

Module registry 

Source
Expand description

One table describing every supported model.

Before this, the same facts were spread across six enums that each encoded a different subset in a different shape: AmplifyModels returned a WeightSource, ESM2Models a source plus a config, ESMCModels a source plus a filename plus a config wrapped in Result, and so on. Which tokenizer a model needs lived in its runner; how many special tokens it wraps lived in its PlmRunner impl; whether it had ever been checked against a Python reference lived only in the test suite.

REGISTRY is where that belongs. A model is a data row.

§Why a const table

The variation between models is data — strings, dimensions, token counts — so it stays data. The moment it becomes dyn, the compiler stops checking that every model is fully specified, which is exactly the property worth having: adding a row that omits a field will not compile.

Family is a closed enum for the same reason. Adding a backbone should be a deliberate, reviewed act rather than something that falls out of a string.

let card = lookup("esm2-t6-8m").expect("registered");
assert_eq!(card.metadata.d_model, 320);
assert_eq!(card.source.repo_id, "facebook/esm2_t6_8M_UR50D");

Structs§

ModelCard
Everything needed to identify, fetch, and load one model.

Enums§

Family
Architecture family a model belongs to.
ParityStatus
Whether this model’s numerics have been checked against a Python reference.
TokenizerSpec
Where a model’s tokenizer comes from.
VocabAlphabet
How to read a sequence against a bare vocab.txt.

Constants§

REGISTRY
Every model the public API exposes.

Functions§

by_family
Every card in a family.
loadable
Every card that can actually be loaded today.
lookup
Find a model by its registry id.
support_matrix_markdown
Render REGISTRY as a markdown support matrix.