Linux and Shell Scripting for Bioinformatics

Filesystem · permissions · processes · awk · sed · grep · pipes · SLURM

Published 2026 · Last updated 2026-09-27 · Hailu Zhou

Almost every bioinformatics tool is a command-line program that reads files and writes files, and almost every cluster is Linux. That means the shell is not a legacy skill you can substitute away with a Python wrapper — it is the layer where data actually moves. Researchers who are fluent here finish, in an afternoon, analyses that take others a week of manual shuffling. This article covers the four competencies that carry the most weight.

The filesystem: paths, trees, and where data lives

Hierarchical, case-sensitive, and everything is a file. That is the entire model. The practical consequences are what matter: absolute paths are unambiguous and belong in scripts; relative paths are convenient but break the moment you move a script. Mount points and symlinks mean the same file can have several names, and ls -l shows you which is which. For large genomics work the fast local scratch disk and the slow shared network filesystem behave very differently — copying a 60 GB BAM to scratch and processing it there is often several times faster than streaming it over NFS. Learn to check with df -h, and to inspect the real size of a directory tree with du -sh before you assume a copy has room to land.

Users, groups, and permissions

Permissions are a triplet of read/write/execute for user, group, and others, and understanding them prevents the two failure modes that waste the most time: a script that cannot run because the execute bit is unset, and a directory tree that collaborators cannot read because the mode is too restrictive. Use symbolic mode with chmod — u+x to make a script runnable — and recursive mode carefully. On shared clusters the group model matters: files you create should typically be group-readable so your lab and your pipelines can access them.

Processes: what is running and how to stop it

A process is an execution context with an ID, a parent, and a state. The three tools that solve most problems are ps to list processes, top (or htop) to watch resource use live, and kill to send a signal. The nuance that trips people up is differences between signals: a SIGTERM (kill default) asks politely and lets a well-written tool clean up its temporary files, while SIGKILL (kill -9) terminates immediately and can leave partial output behind. Long-running jobs survive your logout only if you detach them properly — with nohup, tmux, or the cluster's own scheduler. Never run a multi-hour job in a bare SSH session and forget the session timeout.

Text processing: grep, sed, and awk

This is where the shell earns its keep. Biological data is overwhelmingly text — FASTA, FASTQ, SAM, VCF, GFF, TSV — and these three tools turn hours of scripting into one-liners.

A concrete example: to sum a numeric column across a large TSV, awk streams the file in constant memory and finishes in seconds. The equivalent naive Python loop would spend minutes in interpreter overhead. awk's field-oriented model was designed for exactly this kind of records-processing and remains hard to beat (see the AWK book below).

Pipes and redirection: composing small tools

The Unix philosophy — small programs that do one thing and compose through pipes — is why bioinformatics pipelines exist in the form they do. The pipe operator connects the standard output of one command to the standard input of the next, so a multi-stage process becomes a single readable line. Redirection sends output to files, > to overwrite and >> to append, with 2> for errors. The crucial discipline is that well-behaved commands separate data on stdout from progress messages on stderr; if a tool mixes them, your downstream parsing breaks. Filter early and often, so that expensive steps downstream of a pipe see only the records they need.

Job scheduling with SLURM

A cluster is a shared resource, and the scheduler is the negotiating layer. SLURM is the dominant implementation, with a documented command set for submitting, inspecting, and cancelling jobs 2. The concepts are few:

The recurring mistake is script-local assumptions: a job runs on a compute node that is not your login node, with a possibly different environment. Always module load (or activate) your tools inside the batch script, use absolute paths, and write logs to a known directory so a failed array task can be inspected after the fact.

Scripting that survives contact with real data

Three habits separate fragile one-off commands from scripts you can trust. First, start every non-interactive script with set -euo pipefail: exit on error, treat unset variables as errors, and fail a pipeline if any of its stages fail. Second, log what you ran, with which inputs, and where the output went — a job that silently produces an empty file is worse than one that crashes. Third, quote your variables ("$file") so that paths containing spaces do not split into arguments. None of these are exotic; together they remove most of the silent-corruption failure modes in shell pipelines, and they are the same reproducibility principles that apply across computational science 3.

References

  1. Aho AV, Kernighan BW, Weinberger PJ. The AWK Programming Language. Reading, MA: Addison-Wesley; 1988.
  2. Yoo AB, Jette MA, Grondona M. SLURM: simple Linux utility for resource management. Job Scheduling Strategies for Parallel Processing (JSSPP). 2003;2862:44-60.
  3. Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9(10):e1003285.
  4. Raymond ES. The Art of UNIX Programming. Boston: Addison-Wesley; 2003.
  5. Kerrisk M. The Linux Programming Interface. San Francisco: No Starch Press; 2010.
  6. Nekrutenko A, Taylor J. Next-generation sequencing data interpretation: enhancing reproducibility and accessibility. Nat Rev Genet. 2012;13(9):667-672.
  7. Perkel JM. Ten simple rules for responsible use of high-performance computing clusters. PLoS Comput Biol. 2018;14(3):e1005888.
  8. Starmer C. Linux Command Line and Shell Scripting Bible. Indianapolis: Wiley; 2011.