cdsBERT: Why Codons Still Matter for Protein AI

Most protein models read proteins as amino acid strings. That is a reasonable simplification: amino acids are the direct building blocks, and they determine most of structure and function.

Biology does not start there. It starts with coding sequence. Three DNA letters form a codon, and up to six different codons encode the same amino acid. Mutations that swap one synonymous codon for another are conventionally called silent.

Silent is a claim about the protein product, not about the cell. Codon usage affects translation speed, organism-specific expression, mRNA secondary structure, and in some documented cases the folded structure of the protein itself. Two genes can encode an identical amino acid sequence while carrying quite different regulatory and evolutionary information.

cdsBERT asks a narrow version of that question. If you give a protein language model access to codons instead of amino acids, does it learn anything it did not have before?

Rebuilding a model's vocabulary

You cannot simply retrain from scratch on codons; the compute is prohibitive and the data is far scarcer than amino acid data. So the approach uses a transfer procedure called MELD, in three stages.

The seeding step is the interesting one. Because all six serine codons begin at exactly the same point in embedding space, any distance between them afterwards was learned from data. The initialization doubles as a control.

A practical aside with a pleasing solution: storing codons as three-character strings triples the dataset size, so a single-character code is defined for all 64 codons. Human-optimal codons keep their uppercase amino acid letter, second-most-frequent synonyms get the lowercase version, and stop codons map to ), }, and ], which has the side effect of making every open reading frame visible as matching brackets.

Does the codon vocabulary help?

The evaluation is enzyme commission number prediction from frozen embeddings, using both a k-nearest-neighbour classifier and a support vector machine over ten-fold cross-validation.

The trajectory matters more than any single bar. Straight after masked language modeling, cdsBERT is worse than the model it was seeded from: 62.4 percent against ProtBERT's 74.7 percent. The likely explanation is that the model is undertrained on its new vocabulary, which is credible given that the coding sequence corpus was large enough that a single epoch was never completed.

Knowledge distillation closes the gap and then some. cdsBERT+ reaches 78.7 percent, a relative gain of 16.3 percent, overtaking the original ProtBERT. ProtBERT given the same distillation treatment improves only to 78.1 percent.

That 0.6 point margin is the entire codon effect under a controlled comparison. The advantage holds across all k-nearest-neighbour metrics and support vector machine accuracy, while ProtBERT+ actually wins on support vector machine precision, recall, and F1. Ankh-base beats every variant.

Further distillation against either Ankh-base or Ankh-large produced no additional gain, suggesting both models had hit a ceiling set by architecture, size, or the modest 2,629-sequence enzyme dataset.

What moved, and why

The more interesting analysis is not the benchmark but the embeddings themselves.

Measuring mean squared error between weight matrices before and after each stage shows a clean division of labor. During masked language modeling, the token embedding matrix changed most, meaning the model was learning what its new tokens mean. During distillation, the intermediate linear layers and attention changed most, meaning the model was learning how to use them in context.

Because synonymous codons started identical, any separation is learned. Projecting the 64 codon embeddings into two dimensions shows them grouped by the physicochemical properties of their amino acids, with visible separation among synonyms. A pairwise distance map shows 21 distinct blocks along the diagonal, one per amino acid family, confirming synonyms stay related while no longer being identical.

The central result is a correlation. Comparing how far each codon's embedding moved against roughly 13,000 measurements of average codon usage across broad phylogeny gives a statistically significant medium negative relationship: codons used more frequently moved less.

The other correlations examined, between embedding movement, taxonomic prediction influence, and DNA-type prediction influence, were intuitive in direction but mostly negligible in size. Only usage against movement, and usage against DNA-type ranking, reached significance.

What it set up

Two things carried forward from this work.

The vocabulary-extension procedure itself. Seeding new tokens from semantically related existing ones, then recovering capability through distillation, is the same move later used to graft an annotation vocabulary onto a sequence model in the Annotation Vocabulary work. The trick generalizes past codons.

And the data bottleneck, identified as the real conclusion. High-quality coding sequence mappings for large protein repositories like UniProt largely do not exist. A codon foundation model is not blocked on architecture. It is blocked on someone assembling the corpus.