7 min read

Dual Triangle Attention: Position Sense for Bidirectional Models

Dual Triangle Attention gives each attention head two complementary triangular masks. The model keeps full bidirectional context while inheriting the causal mask's implicit sense of order, and it holds up on both natural language and protein masked language modeling.

Logan Hallee

April 9, 2026

Attention mask geometry for bidirectional, causal, and the two Dual Triangle halves — Filled cells are positions a query may attend to. The two Dual Triangle halves are complementary: together they cover the full bidirectional field, but each half is ordered.

Bidirectional transformers are useful because every token can read every other token. That is exactly what you want for protein encoders, retrieval models, and any annotation task where the decisive evidence might sit on either side of the position you care about.

The cost is a strange blind spot. Standard bidirectional attention has no notion of order built into it. Every query attends to every key, so permuting the input permutes the output and nothing else changes. Positional embeddings are bolted on to fix this, and they work, but the mechanism itself remains order-blind underneath.

Causal models never had this problem. Their mask is triangular, so each token sees only the past, and direction is baked into the computation. That is why some autoregressive models can drop positional embeddings entirely and still work.

Dual Triangle Attention asks whether a bidirectional model can get the same structural cue without giving up full context.

The mechanism

Take one attention head. Split its query and key tensors in half along the feature dimension. Route the first half through a lower-triangular mask, so it attends to past and self. Route the second half through an upper-triangular mask, so it attends to future and self. The diagonal belongs to both halves, which keeps every attention row non-empty and the softmax well behaved. Each half normalizes independently.

Four attention mask matrices: bidirectional, causal, and the two complementary Dual Triangle halves — Figure 1. The masks, drawn from their definitions. Bidirectional permits everything and so encodes no order. Causal encodes order by forbidding the future. Dual Triangle splits one head into two ordered channels whose union is the full bidirectional field.

In plain terms

The model still reads the whole sequence. It just reads it through two one-way channels instead of one symmetric surface, and the fact that a token arrived through the "past" channel rather than the "future" channel is itself positional information.

What this costs

Nothing in parameters. There are no new weights beyond ordinary multi-head attention. The implementation reshapes the tensors from b × n × l × d to b × 2n × l × d/2, applies a head-routed block mask, and issues a single compiled flex_attention call. Because each direction uses half the query-key subspace, the attention operation itself needs roughly half the floating-point operations of standard bidirectional attention. No wall-clock or memory measurements are reported, so this is an analytic count for the attention operation, not an end-to-end speedup.

A test that isolates the question

Before real data, a synthetic probe with exactly one answer isolates the question. The model sees a sequence of random token IDs and must predict the position of the largest one. Vocabulary size 64, sequence length 64, so chance is about 1.6 percent.

This cannot be solved by recognizing tokens. The same token value appears at different positions across examples, so the model has to bind value to position. If an architecture has no access to positional information, it cannot do better than chance.

The result is clean. With positional embeddings, all three attention types solve it. Without them, causal attention still solves it, Dual Triangle Attention still solves it, and standard bidirectional attention fails completely. A control run with randomized labels puts every configuration back at chance, confirming the main result reflects learning rather than an artifact.

That is the mechanism claim, demonstrated in the smallest setting where it can be demonstrated at all.

Does it survive real data?

The probe proves the masks carry order. It says nothing about whether that helps on a task anyone cares about. So the evaluation trains 12-layer, 768-hidden transformers from scratch on one billion tokens, twice: English from FineWeb-Edu, and protein sequences from OMG-Prot50. Three seeds per configuration, identical architecture and optimizer across attention types.

Masked-token accuracy for three attention types with and without rotary positional embeddings, on English and protein data — Figure 2. Masked language modeling accuracy at the 256-token training context, mean and standard deviation over three seeds. With rotary embeddings, Dual Triangle Attention matches standard bidirectional attention on both domains. Without them, the gap is the entire result.

Given rotary embeddings, Dual Triangle Attention and standard bidirectional attention are statistically indistinguishable: 0.709 against 0.702 on English (p = 0.68), 0.248 against 0.252 on proteins (p = 0.21). Causal attention trails badly on both, which is expected, since one-directional context is a real handicap for a masked objective.

That parity held despite the split. Dual Triangle splits each head's query-key subspace between two directions, which could plausibly have reduced per-direction expressiveness. It did not measurably do so.

Without positional embeddings the picture changes completely. On English, standard bidirectional attention collapses to 0.130 accuracy while Dual Triangle Attention holds at 0.700, barely below its own rotary-embedding result. On proteins the effect is smaller but still clear: 0.237 against 0.203.

Longer than it was trained on

The more practically interesting result is context extension. Every model was trained at 256 tokens and evaluated at 1,024.

Masked-token accuracy at 1024 tokens after training at 256, for Dual Triangle and bidirectional attention — Figure 3. English masked language modeling evaluated at four times the training context. Dual Triangle Attention with rotary embeddings extends best. Without positional embeddings it retains meaningful performance where standard bidirectional attention is non-functional.

The best extended-context loss and accuracy in the study belong to Dual Triangle Attention with rotary embeddings, at 1.53 loss and 0.677 accuracy, against bidirectional attention's 1.81 and 0.636. The protein results mirror this.

A reasonable interpretation is that the triangular structure supplies a positional prior that does not degrade with distance the way a learned or rotary signal can when pushed past its training range.

What did not work

The evaluation also covers DroPE-style position dropping: train with rotary embeddings for the first 70 percent of the token budget, then remove them and continue. In autoregressive settings this works surprisingly well, because the causal mask keeps carrying position after the explicit signal is gone.

Test loss before and after dropping positional embeddings at 70 percent of training — Figure 4. Test loss with rotary embeddings kept throughout, against the same run with them dropped at 70 percent of the token budget. Every attention type gets worse, in both domains. Dual Triangle degrades least on English but is not spared.

It did not transfer. On English, Dual Triangle's test loss went from 1.53 to 4.05 and standard bidirectional attention's from 1.81 to 5.93. On proteins the degradation was smaller, and no variant recovered to its pre-drop language-modeling loss.

The likely explanation is that masked language modeling asks more of the recalibration than next-token prediction does. A causal model reconstructs forward context under an intact triangular mask. A masked model has to integrate evidence from both directions simultaneously after the positional signal is removed. The partial resilience seen here fits that story, since the dual triangles give the model some structural fallback where standard bidirectional attention has none. Surviving better is not the same as succeeding.

Positional tricks that work for causal models should not be assumed to carry over to bidirectional objectives.

Terms used here

Bidirectional attention. Every token attends to every other token in both directions. Standard for encoders and masked language models.

Causal attention. Each token attends only to itself and earlier tokens. Standard for text generation.

RoPE. Rotary positional embeddings. Position is injected by rotating query and key vectors by an angle that depends on their index.

Context extension. Evaluating a model on sequences longer than any it was trained on. Positional schemes tend to be where this breaks.

MLM. Masked language modeling. Hide a fraction of tokens and train the model to reconstruct them from surrounding context.

Why this matters for protein models

Protein models need global context because residues far apart in sequence can be neighbors in the folded structure. They also need order, because domains, motifs, and local grammar are all positional.

This architecture targets that tension directly, keeping the bidirectional field while giving the mechanism a native sense of direction. The open question is where the difference actually pays: longer proteins, complexes, genomic sequence, retrieval encoders, or hybrid models that need both local order and global comparison.

The experiments use a fixed 12-layer, 768-hidden architecture on a one-billion-token budget, which is modest. Three seeds is a small sample for the Welch tests, so effect sizes should be read as indicative. The training context of 256 tokens is short, so the four-times extension result would be more convincing at a larger base.

What the experiments establish: bidirectional attention does not have to be position-blind. When explicit positional embeddings are weak, removed, or pushed out of distribution, the shape of the mask can carry order on its own.

Source

Preparing for journal submission.