Expand description
One table describing every supported model.
Before this, the same facts were spread across six enums that each encoded a
different subset in a different shape: AmplifyModels returned a
WeightSource, ESM2Models a source plus a config, ESMCModels a source
plus a filename plus a config wrapped in Result, and so on. Which
tokenizer a model needs lived in its runner; how many special tokens it
wraps lived in its PlmRunner impl; whether it had ever been checked
against a Python reference lived only in the test suite.
REGISTRY is where that belongs. A model is a data row.
§Why a const table
The variation between models is data — strings, dimensions, token counts —
so it stays data. The moment it becomes dyn, the compiler stops checking
that every model is fully specified, which is exactly the property worth
having: adding a row that omits a field will not compile.
Family is a closed enum for the same reason. Adding a backbone should be
a deliberate, reviewed act rather than something that falls out of a string.
let card = lookup("esm2-t6-8m").expect("registered");
assert_eq!(card.metadata.d_model, 320);
assert_eq!(card.source.repo_id, "facebook/esm2_t6_8M_UR50D");Structs§
- Model
Card - Everything needed to identify, fetch, and load one model.
Enums§
- Family
- Architecture family a model belongs to.
- Parity
Status - Whether this model’s numerics have been checked against a Python reference.
- Tokenizer
Spec - Where a model’s tokenizer comes from.
- Vocab
Alphabet - How to read a sequence against a bare
vocab.txt.
Constants§
- REGISTRY
- Every model the public API exposes.
Functions§
- by_
family - Every card in a family.
- loadable
- Every card that can actually be loaded today.
- lookup
- Find a model by its registry id.
- support_
matrix_ markdown - Render
REGISTRYas a markdown support matrix.