The Problem: Bidirectional Models Need to Know Order

Bidirectional transformers are the workhorses of many protein and language models. They let every token look at every other token, which is valuable when context can come from either side.

For proteins, that full context is especially important. A residue near the start of a sequence may interact with a residue far downstream once the protein folds.

But bidirectional attention has a vulnerability. Without positional information, it cannot inherently know the order of tokens. It sees relationships, but not the sequence direction that gives those relationships meaning.

Dual Triangle Attention was designed to address that problem.

The Core Idea

Causal language models, the left-to-right kind, get some positional structure from their attention mask. Each token can only see earlier tokens, so direction is built into the computation.

Bidirectional models do not have that same directional cue because every token can see every other token.

Dual Triangle Attention brings directional structure back into bidirectional modeling. It lets a model use information from both past and future positions while preserving useful order-aware behavior.

For nontechnical readers, the point is simple: the model keeps the benefits of full-context reading while becoming less dependent on explicit positional scaffolding.

What The Paper Showed

The paper tested Dual Triangle Attention on synthetic position tasks, natural-language masked modeling, and protein sequence modeling.

The key finding was that Dual Triangle Attention can learn positional information in settings where standard bidirectional attention struggles without explicit positional embeddings. It also performed strongly when positional embeddings were present, especially in longer-context evaluations.

That makes the result relevant beyond one benchmark. It suggests a general way to make bidirectional transformers more robust for sequences where order matters.

Why This Matters for Protein AI

Protein models depend on sequence order, but they also need global context. A good protein encoder must know both that residue A comes before residue B and that distant residues can become neighbors in three-dimensional space.

Dual Triangle Attention is useful because it targets that exact tension. It supports bidirectional context while giving the model a stronger inductive bias for position.

That could matter for long proteins, multi-domain proteins, protein complexes, genomic sequences, and other biological data where context is not purely local.

Synthyra's View

Synthyra builds products on top of open research, but production capabilities are not just paper replicas. Architecture research like Dual Triangle Attention gives us a path to improve future foundation models, especially where efficiency, context length, and biological sequence order matter.

This is foundational work. It is not a standalone wet-lab assay or user-facing prediction by itself. Its impact comes from making the next generation of models better suited to the biological data they read.

Limitations

Dual Triangle Attention is an architecture result, not a direct biological discovery. The practical gains will depend on model scale, training data, task, and integration into larger systems.

The paper shows that the mechanism is promising. The next step is using it inside larger biological foundation models and measuring where it changes real workflows.

---

This blog post summarizes work in the following paper:

Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings
Logan Hallee, Jason P. Gleghorn
Manuscript draft, 2026