Transformer Model Families: One Architecture, Six Ways to Use It

Introduction: One Block, Six Families

The original Transformer, introduced in Attention Is All You Need, was a single architecture built for one job: machine translation. It had an encoder that read a source sentence and a decoder that wrote a target sentence, connected by attention. Almost everything that has happened in the years since is the result of researchers taking that architecture apart, keeping the piece that mattered for their problem, and throwing away the rest.

That's the idea behind this article. Every transformer-based model you've heard of — BERT, GPT, T5, ViT, CLIP, LLaVA, Mamba — is a variation on the same core block: multi-head self-attention plus a feed-forward network, stacked and wrapped in residual connections and normalization. The differences between them are not differences in the fundamental building block. They're differences in which half of the original architecture survived, and what data type it was pointed at.

Understanding the six families this way — as deliberate subtractions and extensions from one shared design — makes it much easier to answer the question that actually matters in practice: given this task, which family should I even be looking at?

The Core Architecture: What Every Family Shares

Before splitting into families, it's worth being precise about the block itself, because every variant below is built from repetitions of it.

        ┌─────────────────┐
        │   Add & Norm     │
        ├─────────────────┤
        │  Feed Forward    │
        ├─────────────────┤
        │   Add & Norm     │
        ├─────────────────┤
        │ Multi-Head       │
        │ Self-Attention   │
        └─────────────────┘
               ↑
             Input

Multi-head self-attention lets every token in a sequence look at every other token and decide how much to weight it when building its own representation. Run several of these "heads" in parallel, each free to specialize in a different kind of relationship (syntax, coreference, position), and concatenate the results.

The feed-forward network is a small two-layer MLP applied independently to each token's representation, giving the model a place to do per-token nonlinear transformation after attention has mixed information across tokens.

Residual connections and layer normalization ("Add & Norm") are what let you stack dozens of these blocks without the gradients vanishing or the training dynamics falling apart — unglamorous, but arguably as important to the transformer's success as attention itself.

Stack this block N times and you have a transformer. What turns that generic stack into BERT, GPT, or ViT is a small number of design decisions layered on top: does attention see the whole sequence or only the past, is there a separate encoder and decoder, and what gets tokenized in the first place — words, or patches of an image.

Why So Many Families Exist

It's tempting to assume more architectural diversity means the field hasn't converged on an answer yet. The opposite is closer to the truth: the diversity is the answer. Four factors push different tasks toward different subtractions from the same block.

• Different data needs. Understanding an existing sentence and generating a new one are not the same problem, even though both operate on text.
• Different tasks and goals. Classification wants a single, rich representation of an input. Generation wants a model that never looks ahead. Translation wants both.
• Different constraints and resources. A 4,000-token context is cheap; a 400,000-token context with standard attention is not, because attention cost grows quadratically with sequence length.
• Different ideas for better learning. Bidirectional context, causal masking, cross-attention, and patch embeddings are all different bets on what signal helps a model learn the right thing.

With that framing, the six families below aren't six unrelated inventions — they're six answers to those four questions, all built from the same block.

1. Encoder-Only Transformers: Understanding, Not Generating

Encoder-only models keep just the encoder half of the original architecture. Every token attends to every other token in both directions — a word near the end of a sentence can influence the representation of a word near the beginning, and vice versa. This is what "bidirectional" means in a model card, and it's the single defining trait of this family.

Representative models: BERT, RoBERTa, ALBERT, DeBERTa, ELECTRA.

What Encoder-Only Models Are Trained For

The canonical pretraining objective is masked language modeling: hide 15% of the tokens in a sentence and ask the model to predict them using context from both directions. That's only possible because the model isn't trying to generate text left to right — it already has the whole sentence in front of it and just needs to fill in the blanks.

Input:  "The [MASK] sat on the mat."
Target: "cat"

The model uses "The", "sat", "on", "the", "mat" — tokens on
BOTH sides of the mask — to predict the missing word.

What They're Best At

• Classification — sentiment analysis, topic labeling, intent detection
• Named entity recognition — labeling spans of text with a category
• Semantic similarity and retrieval — the embeddings covered in production RAG pipelines almost always come from an encoder-only model, because a single bidirectional pass produces a richer sentence representation than a decoder ever needs to compute
• Extractive question answering — pointing at the span of an existing document that answers a question, rather than writing new text

A Minimal Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)

inputs = tokenizer("This product completely changed how I work.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

prediction = torch.argmax(logits, dim=-1).item()
print("positive" if prediction == 1 else "negative")

What an encoder-only model cannot do natively is generate free-form text — there's no mechanism stopping it from "seeing the future," so using it autoregressively would leak the answer into its own input. If a task needs a model to write new sentences rather than label or embed existing ones, this is the wrong family.

2. Decoder-Only Transformers: Generation, One Token at a Time

Decoder-only models keep just the decoder half, and drop the encoder-decoder cross-attention that isn't needed when there's no separate source sequence to translate from. The defining trait is the opposite of the encoder family: attention is causally masked, so a token can only attend to itself and the tokens before it — never the future.

Representative models: GPT, LLaMA, Mistral, Gemma, Qwen — effectively every modern general-purpose chat and coding LLM.

Causal Masking

Token:        The   cat   sat   on   the   mat
Can attend to:
  "The"        ✓     ✗     ✗     ✗    ✗     ✗
  "cat"        ✓     ✓     ✗     ✗    ✗     ✗
  "sat"        ✓     ✓     ✓     ✗    ✗     ✗
  "on"         ✓     ✓     ✓     ✓    ✗     ✗
  "the"        ✓     ✓     ✓     ✓    ✓     ✗
  "mat"        ✓     ✓     ✓     ✓    ✓     ✓

This triangular attention pattern is what makes autoregressive generation coherent: at training time the model learns to predict each token from only what came before it, which is exactly the situation it will be in at inference time, generating one token after another. The mechanics of that generation loop — why the first token is slow and every token after it is fast — are covered in detail in How LLM Inference Really Works.

What They're Best At

• Open-ended text generation — chat, drafting, creative writing
• Code generation — writing new code token by token, one line building on the last
• Reasoning and multi-step problem solving — chain-of-thought works because each generated token can condition on everything reasoned so far
• Agentic tool use — the perceive–reason–act–remember loop behind every modern AI agent runs on a decoder-only model deciding, one token at a time, what to do next

A Minimal Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("The transformer architecture works by", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))

The tradeoff for that generative freedom is that a decoder-only model builds its understanding of the input as it goes, left to right — it never gets the encoder's advantage of revising an early token's representation in light of something that appears later. For pure understanding tasks over a fixed input, this is usually more model than you need.

3. Encoder-Decoder Transformers: The Original Design, Still Useful

Encoder-decoder models are the closest thing to the 2017 original: a bidirectional encoder reads the full input and builds a representation of it, and a causally-masked decoder generates the output, using cross-attention to look back at the encoder's representation at every generation step.

Representative models: T5, BART, mT5, FLAN-T5.

Why Keep Both Halves

The case for this family is narrow but clear: tasks where the output is a genuinely different sequence, conditioned on fully understanding a separate input, benefit from giving the model a dedicated bidirectional pass over that input before it starts generating anything. A decoder-only model has to build its understanding of the source sentence and generate the translation in the same left-to-right pass; an encoder-decoder model gets to fully digest the source first.

Encoder (bidirectional)          Decoder (causal + cross-attention)
"Le chat est noir"      →        "The" → "cat" → "is" → "black"
        │                              ↑        ↑       ↑       ↑
        └──────────── cross-attention ─┴────────┴───────┴───────┘

What They're Best At

• Machine translation — the task the architecture was invented for
• Summarization — compressing a long input into a short, faithful output
• Text-to-text reformatting — T5's core idea was to cast every NLP task, including classification, as text-in-text-out, which made a single architecture reusable across tasks that used to need separate model heads
• Structured generation grounded in a fixed input — e.g. generating a form or a table from a paragraph of prose

A Minimal Example

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")

inputs = tokenizer(
    "summarize: The transformer architecture, introduced in 2017, "
    "replaced recurrence with self-attention and became the foundation "
    "for nearly every modern large language model.",
    return_tensors="pt",
)
output = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The cost of keeping both halves is exactly what you'd expect: more parameters doing conceptually separate jobs, and a training setup that's more involved than either encoder-only or decoder-only pretraining alone. For general-purpose chat and reasoning, the field has largely converged on decoder-only models instead — but for translation and summarization specifically, encoder-decoder models remain a strong, sometimes more efficient choice.

4. Vision Transformers: Attention Beyond Text

Vision Transformers (ViT) apply the exact same encoder block used by BERT to images, with one crucial preprocessing change: instead of tokenizing words, the image is cut into fixed-size patches, and each patch is flattened and linearly projected into a vector — a "visual token."

Representative models: ViT, DeiT, Swin Transformer, BEiT.

From Pixels to Tokens

224×224 image, 16×16 patches
        ↓
196 patches (14 × 14 grid)
        ↓
Each patch flattened → linear projection → 196 "tokens"
        ↓
+ learnable [CLS] token + positional embeddings
        ↓
Standard transformer encoder (same block as BERT)
        ↓
[CLS] token's final representation → classification head

Everything after patch embedding is identical to an encoder-only text transformer: bidirectional self-attention across all patches, feed-forward layers, residuals, norm. The insight behind ViT wasn't a new attention mechanism — it was demonstrating that attention over patches, given enough training data, outperforms the convolutional inductive bias that had dominated computer vision for a decade.

What They're Best At

• Image classification — the original ViT benchmark task
• Object detection and segmentation — with detection-specific heads on top of the same backbone
• Medical imaging — X-ray and scan classification, where long-range spatial relationships (a shadow on one side of an image informing interpretation of the other) matter more than convolution's local receptive field

A Minimal Example

from transformers import ViTImageProcessor, ViTForImageClassification
from PIL import Image

processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")

image = Image.open("photo.jpg")
inputs = processor(images=image, return_tensors="pt")
logits = model(**inputs).logits

predicted_class = logits.argmax(-1).item()
print(model.config.id2label[predicted_class])

The main practical constraint is data. Convolutional networks build in an assumption — nearby pixels are related — that a plain ViT has to learn from scratch, which is why the original ViT needed very large pretraining datasets to outperform CNNs; architectures like Swin Transformer reintroduce some of that locality bias (via windowed attention) to make vision transformers competitive on smaller datasets too.

5. Multimodal Transformers: More Than One Kind of Token

Multimodal transformers combine two or more of the families above so a single model can reason across data types — typically an encoder-only image branch, a text branch, and some mechanism for connecting the two representation spaces.

Representative models: CLIP, BLIP, Flamingo, LLaVA.

Two Common Patterns

Contrastive alignment (CLIP) trains an image encoder and a text encoder separately, then pulls the embeddings of matching image-caption pairs close together in a shared vector space while pushing non-matching pairs apart. There's no cross-attention between modalities at all — just two encoders learning to land in the same space.

Image → Vision Encoder → image_embedding  ┐
                                            ├─ cosine similarity → aligned?
"a photo of a cat" → Text Encoder → text_embedding ┘

Fusion into a language model (LLaVA) takes a pretrained vision encoder's patch embeddings, projects them into the same dimensional space as a decoder-only LLM's token embeddings, and feeds both image tokens and text tokens into the same causal decoder — letting the model "read" an image the same way it reads a sentence, then generate text about it.

What They're Best At

• Zero-shot image classification and retrieval — CLIP can rank arbitrary text labels against an image without ever being fine-tuned on that specific classification task
• Visual question answering — "what color is the car in this photo?"
• Image captioning — generating a natural-language description of an image
• Multimodal assistants — agents that need to read a screenshot, a chart, or a document scan alongside a text instruction

A Minimal Example

from transformers import CLIPProcessor, CLIPModel
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open("photo.jpg")
candidate_labels = ["a photo of a dog", "a photo of a cat", "a photo of a car"]

inputs = processor(text=candidate_labels, images=image, return_tensors="pt", padding=True)
logits_per_image = model(**inputs).logits_per_image
probs = logits_per_image.softmax(dim=1)

print(dict(zip(candidate_labels, probs.tolist()[0])))

The hard part of this family isn't any single attention mechanism — it's alignment: getting representations that were learned from very different data distributions to land in a space where distance and direction mean the same thing across modalities. Most of the research differentiation between CLIP, BLIP, Flamingo, and LLaVA is really differentiation in how that alignment is trained.

6. Efficient and Specialized Transformers: Fighting the Quadratic

Standard self-attention compares every token to every other token, which means its compute and memory cost grows with the square of sequence length. Double the context, quadruple the attention cost. This family exists entirely to fight that one number.

Representative models: Longformer, BigBird, Linformer, Performer, Mamba.

The Problem in One Line

Standard attention cost:  O(n²)   — n = sequence length

n = 1,000 tokens   →   1,000,000 pairwise comparisons
n = 100,000 tokens →  10,000,000,000 pairwise comparisons

That quadratic wall is also the reason the KV cache discussed in LLM inference serving grows the way it does — this family attacks the same underlying cost from the architecture side rather than the serving side.

Three Different Fixes

• Sparse attention (Longformer, BigBird) — instead of attending to every token, each token attends to a fixed local window plus a small number of globally important tokens, cutting the quadratic term down to something closer to linear.
• Low-rank / kernel approximations (Linformer, Performer) — approximate the full attention matrix with a lower-dimensional projection or a kernel trick that avoids ever materializing the full n×n matrix, trading a small amount of exactness for a large speedup.
• State-space models (Mamba) — step furthest from the original design: replace attention entirely with a selective state-space mechanism that processes tokens sequentially with a compressed hidden state, giving linear-time sequence processing at the cost of losing attention's ability to directly compare any two tokens regardless of distance.

What They're Best At

• Long-document processing — legal contracts, books, long codebases, where standard attention's context limit or cost becomes the bottleneck
• Memory-constrained deployment — serving long-context models on hardware where the KV cache would otherwise dominate available memory
• High-throughput, latency-sensitive inference — Mamba-style models trade the flexibility of attention for the predictable, linear-time generation profile that some production workloads specifically need

This family is the one place where "still a transformer" starts to become a looser claim — Mamba in particular keeps the residual-and-normalization scaffolding but drops attention as the core mixing mechanism entirely. Whether that still counts as a transformer or as its own architecture is more a semantic question than an engineering one; what matters is that it was built specifically in response to the same quadratic bottleneck that motivates the rest of this family.

One Idea, Many Expressions

Step back from the individual families and a pattern falls out: every one of them is answering the question "what should a token be allowed to see?" in a different way.

Family	What a token can attend to	Consequence
Encoder-only	Every token, both directions	Rich understanding, no generation
Decoder-only	Itself and everything before it	Coherent generation, one-pass understanding
Encoder-decoder	Encoder: bidirectional. Decoder: causal + cross-attends to encoder	Best of both, more parameters and complexity
Vision transformer	Every image patch, both directions	Same understanding advantage, applied to pixels
Multimodal	Tokens from more than one modality, aligned or fused	Reasoning across data types
Efficient / specialized	A restricted, approximated, or compressed subset	Longer sequences, lower cost, some loss of exactness

The right model for a task, in other words, is really the right attention visibility pattern for that task — everything else about picking a transformer follows from that one decision.

How to Choose: A Practical Decision Guide

Understanding a fixed input (classify, embed, extract)?
    → Encoder-only (BERT, RoBERTa, DeBERTa)

Generating open-ended text, chat, code, or agent actions?
    → Decoder-only (GPT-class, LLaMA, Mistral, Qwen)

Transforming one full sequence into another (translate, summarize)?
    → Encoder-decoder (T5, BART)

Understanding or classifying images?
    → Vision Transformer (ViT, Swin, DeiT)

Reasoning across text and images together?
    → Multimodal (CLIP for alignment/retrieval, LLaVA for generative VQA)

Processing very long sequences under tight compute or memory limits?
    → Efficient / specialized (Longformer, BigBird, Mamba)

Most production systems don't pick just one. A modern RAG-backed agent, for instance, typically uses an encoder-only model to generate the embeddings behind retrieval (see Building a Production RAG Pipeline), a decoder-only model to reason and generate the final answer, and — if it needs to read screenshots or scanned documents — a multimodal model somewhere in the ingestion path, as in the OCR-plus-LLM system described in this document-AI case study. The families aren't competing architectures so much as complementary tools that get composed.

Common Misconceptions

• "Decoder-only models can't understand text well." They understand text through the same self-attention mechanism as encoders — they just build that understanding incrementally, left to right, instead of in one bidirectional pass. In practice, at sufficient scale, this gap mostly closes for the tasks decoder-only models are used for.
• "Bigger context always needs an efficient/specialized architecture." Standard attention with a well-engineered serving stack (chunked prefill, PagedAttention — see this deep dive on LLM inference) has pushed usable context lengths far beyond what seemed feasible a few years ago. Reach for the specialized family when the quadratic cost is the actual bottleneck you've measured, not preemptively.
• "Vision Transformers replaced CNNs entirely." ViTs need more data or stronger augmentation to match CNNs at small scale, and hybrid architectures that reintroduce convolutional locality (Swin) are common precisely because pure global attention isn't automatically better for every vision task.
• "Multimodal means one giant unified architecture." Most multimodal systems in production are two or more single-modality backbones connected by a comparatively small alignment or fusion layer, not one model trained from scratch on mixed data.

Frequently Asked Questions

What is the difference between an encoder and a decoder in a transformer?

An encoder uses bidirectional self-attention, so every token's representation is informed by every other token in the input, in both directions. A decoder uses causal (masked) self-attention, so a token can only attend to itself and the tokens before it — the requirement for coherent, left-to-right text generation.

Why do most modern LLMs use decoder-only architectures instead of encoder-decoder?

Decoder-only models are simpler to scale, unify understanding and generation in a single pass, and turned out to generalize well across tasks once trained at sufficient scale — removing much of the practical advantage encoder-decoder models had for tasks beyond translation and summarization specifically.

Is a Vision Transformer the same architecture as BERT?

Structurally, yes — a ViT is an encoder-only transformer, using the same self-attention and feed-forward blocks as BERT. The only architectural difference is what gets tokenized before the transformer blocks: image patches instead of words.

What makes a model "multimodal"?

A multimodal model processes and relates more than one type of data — typically text and images — either by aligning separately encoded representations into a shared space (CLIP-style contrastive learning) or by projecting one modality's tokens into another modality's model and processing them jointly (LLaVA-style fusion).

Why does attention cost grow quadratically with sequence length?

Standard self-attention computes a similarity score between every pair of tokens in the sequence, so the number of comparisons grows with the square of the sequence length. Efficient transformer variants exist specifically to avoid computing or storing that full pairwise matrix.

Is Mamba a transformer?

Mamba keeps some of the surrounding scaffolding — residual connections, normalization, stacked blocks — but replaces self-attention itself with a selective state-space mechanism. Whether it counts as a transformer is mostly a definitional question; what matters practically is that it was built to solve the same long-sequence cost problem that motivates the rest of the efficient-transformer family, using a different mechanism than attention.

Which transformer family should I use for a RAG system?

Two families, working together: an encoder-only model to generate the embeddings used for retrieval, and a decoder-only model to generate the final grounded answer from the retrieved context. See Building a Production RAG Pipeline for the full pipeline.

Final Thought

The transformer revolution didn't happen because researchers kept inventing unrelated new architectures. It happened because one attention block turned out to be general enough that almost every subsequent advance could be framed as a subtraction, restriction, or extension of it: keep only the encoder and you get understanding; keep only the decoder and you get generation; keep both and you get translation; change what counts as a token and you get vision; align two token spaces and you get multimodality; restrict what a token can see and you get efficiency at scale.

Once you see the families this way, the question "which model should I use?" stops being a search through a long list of unfamiliar names, and becomes a much simpler question: what do I need a token to be able to see?