Question 1. The paper "Attention Is All You Need" introduced the transformer architecture. What did the transformer replace in earlier sequence-to-sequence models?
- Option A: Attention mechanisms, which it replaced with recurrence
- Option B: Tokenisation, which it replaced with character-level input
- Option C: Recurrent and convolutional layers, which it replaced with attention
- Option D: Backpropagation, which it replaced with reinforcement learning
Show answer and explanation
Correct answer: C. Recurrent and convolutional layers, which it replaced with attention
Because self-attention relates every token to every other token directly, transformers process a sequence in parallel instead of step by step. This made much larger training runs practical and is the basis of today's LLMs.
- A (incorrect)It is the other way round: the transformer keeps attention and removes recurrence.
- B (incorrect)The transformer still works on tokens; the paper is about the network architecture, not the input encoding.
- C (correct)Correct. The transformer relies solely on attention mechanisms and dispenses with recurrence and convolutions entirely.
- D (incorrect)Transformers are trained with backpropagation and gradient descent like other neural networks.
Reference: arXiv: Attention Is All You Need (suggested reading on the NCA-GENL page) (opens in a new tab)

