Annotation Vocabulary: Teaching Protein Models the Language of Function

Protein sequences are abundant. Clean function labels are not. Far less than one percent of documented protein sequences have ever been synthesized, and fewer still have been carefully annotated.

Most protein models learn from amino acid strings because strings are what exist at scale. But researchers care about a richer question: what does this protein do, where does it act, which domains does it contain, what reaction might it catalyze.

Natural language can express all of that, and it expresses it badly for machine consumption. Two curators describe the same protein differently. Some entries are long and redundant, others sparse. Boilerplate abounds. A model learning function from free text spends much of its capacity learning how biologists write.

Annotations as a language

The Annotation Vocabulary takes the structure that already exists and treats it as a token vocabulary. Six ontologies are each assigned a contiguous integer range. Every term gets a unique integer. A protein becomes a sequence of those integers, sorted ascending.

The sorting is not incidental. It gives the sequence a fixed grammar, which is what makes semi-supervised denoising work: mask 15 percent of a protein's annotation tokens and train the model to recover them from the rest. That model is the Annotation Transformer, and it never sees an amino acid.

The contrastive loss inside CAMP contains the least obvious design choice. The usual approach would pull each protein's embedding toward its annotation's embedding. The reasoning runs against this: there is no reason to assume the best representation for a protein must sit near the representation of its description. So instead of matching points, it matches relationships. The loss is the mean squared error between the pairwise cosine similarity matrix of the protein embeddings and that of the annotation embeddings, regularized by a term that prevents the trivial solution of collapsing everything to a point.

Does it produce better embeddings?

The test is a frozen-embedding probe: freeze the model, extract vectors, train a small head, and see how much functional information the representation already carried.

CAMP-EXP produces the only average F1 above 0.6, scoring 2.6 percent above ESM3 and 9.5 percent above ProteinVec, the closest methodological comparison. The best non-CAMP model is ANKH-base at 0.589.

The three-dollar figure is the central claim, more than the leaderboard position. The claim is not that this beats frontier models in general. It is that curating a latent space with structured supervision is a wildly better return on compute than adding parameters, when the task is annotation.

An unexpected secondary result: CAMP embeddings do well on protein-protein interaction tasks they were never trained for, outperforming other models by a clear margin on both yeast and human interaction datasets.

Where the vocabulary runs out

The same table read one column over gives the counterweight. On metal binding, CAMP-EXP scores below its own base model ESM2-650M, and ANKH models beat everything.

The reason is mechanical rather than mysterious. The EXP and RED datasets carry sparse cofactor annotation, and RED carries none. A vocabulary can only teach what it encodes. Curating a latent space toward the properties in your ontology inevitably curates it away from properties that are not.

The Annotation Vocabulary is a lens. Lenses focus.

Annotation as mask filling

If annotations are a language, then predicting a missing annotation is just mask filling, and this is the capability that became Translator. Hide one entire aspect of a protein's annotation, such as its enzyme commission number, and ask the model to recover it from the remaining aspects.

AT-EXP recovers enzyme commission numbers at 0.716 F1 without ever seeing the amino acid sequence, which is a striking demonstration that structured annotations constrain each other heavily. The two AT variants specialize in different directions: EXP is much stronger on EC, Pfam, and Gene3D, while RED is stronger on the gene ontology aspects, reflecting what each training corpus contained.

The genuinely surprising result is the one that goes the wrong way. ASM models, which get the full sequence in addition to the remaining annotations, perform worse than the annotation-only AT models on five of six aspects. More information should not hurt. Three candidate explanations are possible: ASM was undertrained, starting from a pretrained ESM2 checkpoint may be a poor initialization for a dual-vocabulary model, or some annotations are simply wrong. Each of those is testable.

Generating sequences from a description

The last capability is the most speculative. GSM takes an annotation prompt and a fully masked sequence and generates amino acids.

Evaluating generated proteins is its own problem, since perplexity says little about whether a sequence is protein-like. The evaluation uses an alignment-based score, calibrated against known conditions: randomly paired natural proteins score around 0.15, and above 0.5 implies extremely high similarity.

Below 50 percent masking, ordinary ESM2-150M fills sequences better than GSM does. At 70 percent masking and above, GSM's annotation prompts start paying off and it becomes clearly better. That crossover is exactly where you would want it, since it is only at high corruption that the prompt has to do real work.

Many GSM generations that look like hallucinations return statistically significant BLAST hits, and gene ontology enrichment on those hits matches the annotation prompt, including for prompts whose ground-truth protein has low sequence identity to anything in the training set. The model appears to be producing plausible novel realizations of a functional description rather than retrieving memorized sequences.

Why this line of work continues

Three ideas from this paper carried forward into everything since.

Structured vocabularies are a better supervision signal than free text for anything ontology-shaped. That result is reused directly in Translator and again in our embedding-translation work, where sequence-to-annotation turned out to be a far stronger cross-modal bridge than sequence-to-description.

Latent space curation buys more than scale, at least for annotation tasks. Three dollars of contrastive training on frozen encoders beat models with a thousand times the training budget on the metrics that matter here.

And evaluation protocol determines conclusions. These results do not follow a clean size-to-performance ordering. Probing embeddings measures how directly a representation correlates with a task, not how well the model could be finetuned for it. Those are different questions, and confusing them is how the field ends up with contradictory rankings.