Deep Learning for Biological Sequences: CNNs, Transformers, and Pretrained Models

CNN · RNN · Transformer · DNABERT · ESM · Transfer learning · Protein language models

Published 2026 · Last updated 2026-09-27 · Hailu Zhou

Biology is a sequence science, and the last decade has shown that the architectures built for language and vision transfer remarkably well to DNA, RNA, and protein. The winning formula is now familiar: pretrain a large model on unlabeled sequences from a broad corpus, then fine-tune on the specific task. What follows is a tour of the architectures, why they work on biological sequences, and where the analogy to natural language breaks down.

Sequences as language, with caveats

A protein is a string over a 20-letter alphabet; DNA over 4; RNA over 4. Masked language modeling — predict the masked token from context — maps cleanly onto both. The differences matter: the vocabulary is tiny, but sequences are enormously long (a human chromosome is 250 Mb), and the "grammar" is not syntax but physics and evolution. Evolutionary constraints leave detectable covariation between residues, which is precisely the signal a masked model can learn. This is why protein language models work so well without any labels.

Convolutional models: motifs and local grammar

The first wave of deep learning on sequences used one-dimensional CNNs. Convolution filters of width 8–20 learn position-weight-matrix-like motifs, and stacking layers builds hierarchical features. DeepBind and Basset demonstrated this for protein–DNA binding and chromatin accessibility from raw sequence. CNNs excel at local patterns and are cheap, but their effective receptive field is limited and they cannot easily model long-range dependencies — a fatal limitation for promoters interacting with distal enhancers.

Recurrent models and why they faded

RNNs and LSTMs model sequences left to right and were the standard for protein structure prediction before attention. They handle variable length naturally but suffer from vanishing gradients over long spans, are inherently sequential (slow to train), and are strictly directional. On long biological sequences they underperformed, and the field largely moved on. They remain useful in specific settings — RNA secondary structure prediction with bidirectional LSTMs still appears in pipelines — but they are no longer the default.

Transformers: attention over biology

Self-attention computes pairwise interactions across all positions in one shot, which suits the long-range interactions in biology: residue contacts in protein folding, distal regulatory loops in chromatin, base-pairing in RNA. The Transformer architecture (Vaswani et al. 2017) replaced recurrence entirely. The cost is quadratic attention, which is why long-context genomic models use approximations — sparse attention, linear attention, or convolutional downsampling — to reach chromosome scale. AlphaFold2 showed that a carefully engineered attention-based architecture could solve protein structure to near-experimental accuracy (Jumper et al. 2021).

DNA and RNA pretrained models

DNABERT pretrained a BERT-style model on the human genome using k-mer tokenization and fine-tuned it for promoters, splice sites, and transcription-factor binding (Ji et al. 2021). Nucleotide Transformer and the Hyena-based models extended context to tens of thousands of bases and used multi-species corpora. For RNA, RNA-FM and RiNALMo learn structure-aware embeddings from hundreds of millions of ncRNA sequences. The common thread: pretraining on unlabeled sequence yields representations that transfer to many downstream tasks with modest labeled data — invaluable in biology, where labels are expensive.

Protein language models: ESM and friends

ESM-1b showed that a masked language model trained on 250 million protein sequences learns structure and function implicitly, and that its embeddings even capture contacts (Rives et al. 2021). The ESM-2 family scaled this to billions of parameters, and ESMFold derived structure directly from the language model without a multiple sequence alignment. ProtTrans, ProtBERT, and AlphaFold's own evolutionary-scale representations followed. For a bench scientist, the practical takeaway is that a frozen protein language model plus a light classifier often beats task-specific training from scratch.

Transfer learning and fine-tuning strategy

The recipe is now standard:

Report both the frozen and fine-tuned baselines. Fine-tuning a 650M-parameter model on 200 labels overfits aggressively; frozen embeddings or a shallow head is usually the right call at that scale.

What remains hard

Three honest limitations. First, benchmark leakage: many "held-out" sets overlap with pretraining corpora, inflating numbers. Second, interpretability: attention maps are not mechanistic explanations, and a high-attribution residue does not mean a causal one. Third, distribution shift: a model trained on human proteins degrades on viral or engineered sequences. For real deployment, report external validation and per-class performance, not just a single aggregate metric. Deep learning has genuinely transformed sequence biology; it has not made careful experimental validation optional.

References

  1. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998–6008.
  2. Rives A, Meier J, Sercu T, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc Natl Acad Sci USA. 2021;118(15):e2016239118.
  3. Ji Y, Zhou Z, Liu H, Davuluri RV. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics. 2021;37(15):2112–2120.
  4. Jumper J, Evans R, Pritzel A, et al. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596(7873):583–589.
  5. Lin Z, Akin H, Rao R, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science. 2023;379(6637):1123–1130.
  6. Elnaggar A, Heinzinger M, Dallago C, et al. ProtTrans: toward understanding the language of life through self-supervised learning. IEEE Trans Pattern Anal Mach Intell. 2022;44(10):7112–7127.
  7. Alipanahi B, Delong A, Weirauch MT, Frey BJ. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol. 2015;33(8):831–838.
  8. Kelley DR, Snoek J, Rinn JL. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res. 2016;26(7):990–999.
  9. Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. Proc NAACL-HLT. 2019:4171–4186.