Attention Alone Improves Machine Translation and Training Efficiency

Before this paper, high-performing sequence transduction systems usually combined an encoder-decoder architecture with recurrence, convolution, and attention. That design made long-range dependencies expensive to model and limited parallelism during training, because recurrent computation had to step through tokens in order. The authors introduce the Transformer as a cleaner alternative: an encoder-decoder model built entirely from attention and position-wise feed-forward layers, with positional encodings supplying order information.

Core Contribution

The central claim is architectural rather than a single trick: recurrence is not required for competitive machine translation. The paper shows that self-attention can serve as the main sequence modeling operation in both the encoder and decoder, while cross-attention connects target-side decoding to source-side representations. This changes the training profile. Instead of sequential recurrence over length $n$, the model processes all positions in parallel, with the main attention cost scaling as $O(n^2 d)$ and a constant number of sequential operations.

That trade is favorable for the sentence lengths used in WMT translation. The Transformer pays quadratic attention over tokens, but it removes the long sequential path length of recurrent networks and gives every token direct access to every other token in a layer. The paper’s results support the practical version of that claim: better BLEU with less training time than prior recurrent and convolutional systems on the evaluated translation tasks.

Technical Approach

The Transformer uses stacked encoder and decoder blocks. Each encoder layer has multi-head self-attention followed by a position-wise feed-forward network; each decoder layer adds masked self-attention and encoder-decoder attention. Residual connections and layer normalization wrap each sublayer. Figure 1 captures the main design: attention is not an auxiliary module attached to a recurrent backbone, but the backbone itself.

The attention operation is scaled dot-product attention:

Multi-head attention runs this operation in parallel over learned projections, allowing different heads to attend to different relations. In the base model, the authors use 6 encoder and 6 decoder layers, model dimension 512, feed-forward dimension 2048, and 8 attention heads. The larger model keeps the same layer count but increases the model dimension to 1024, the feed-forward dimension to 4096, and the number of heads to 16. Figure 2 shows the decomposition from scaled dot-product attention to the multi-head version used throughout the network.

The model also uses sinusoidal positional encodings, label smoothing, residual dropout, and Adam with a custom learning-rate schedule. These details matter because the paper is not only proposing self-attention in isolation; it is presenting a trainable recipe that made the architecture competitive at the time.

Results and Analysis

On WMT 2014 English-to-German, the Transformer big model reports 28.4 BLEU, more than 2 BLEU above the prior best results cited by the authors, including ensembles. The base model reaches 27.3 BLEU while training for about 12 hours on 8 P100 GPUs. On WMT 2014 English-to-French, the big model reaches 41.8 BLEU after 3.5 days on 8 P100 GPUs, which the authors report as a new best single-model result with far lower training cost than earlier systems.

The comparison is convincing for the paper’s main setting. The baselines include recurrent and convolutional sequence models, and the gains are large enough that they are not just tuning noise. The efficiency claim is also supported in the regime tested: shorter training time and better BLEU are reported together, rather than as separate compromises. The ablation table strengthens the case by showing sensitivity to attention heads, key/value dimensions, model size, and dropout, though it is still an architecture-and-training recipe rather than a proof that attention alone is always preferable.

The attention visualizations on the final pages add qualitative support, showing heads that appear to track long-distance dependencies, anaphora, and sentence structure. Those plots are useful for interpretation, but they should not be treated as the main evidence. The measured translation scores carry the paper.

Limitations

The evaluation is concentrated on machine translation, with English constituency parsing used as a smaller generalization check. The paper does not test long-context settings where quadratic attention in sequence length becomes the main bottleneck. It also predates later concerns around scaling laws, data mixtures, retrieval, instruction tuning, and deployment behavior, so its evidence is strongest for supervised sequence transduction under the reported WMT training setup. Within that scope, the paper’s conclusion is well supported: self-attention can replace recurrent and convolutional sequence modeling while improving both accuracy and training efficiency.