Almost every bioinformatics tool is a command-line program that reads files and writes files, and almost every cluster is Linux. That means the shell is not a legacy skill you can substitute away with a Python wrapper — it is the layer where data actually moves. Researchers who are fluent here finish, in an afternoon, analyses that take others a week of manual shuffling. This article covers the four competencies that carry the most weight.
The filesystem: paths, trees, and where data lives
Hierarchical, case-sensitive, and everything is a file. That is the entire model. The practical consequences are what matter: absolute paths are unambiguous and belong in scripts; relative paths are convenient but break the moment you move a script. Mount points and symlinks mean the same file can have several names, and ls -l shows you which is which. For large genomics work the fast local scratch disk and the slow shared network filesystem behave very differently — copying a 60 GB BAM to scratch and processing it there is often several times faster than streaming it over NFS. Learn to check with df -h, and to inspect the real size of a directory tree with du -sh before you assume a copy has room to land.
Users, groups, and permissions
Permissions are a triplet of read/write/execute for user, group, and others, and understanding them prevents the two failure modes that waste the most time: a script that cannot run because the execute bit is unset, and a directory tree that collaborators cannot read because the mode is too restrictive. Use symbolic mode with chmod — u+x to make a script runnable — and recursive mode carefully. On shared clusters the group model matters: files you create should typically be group-readable so your lab and your pipelines can access them.
- Permissions: chmod to grant or revoke, chown to change ownership when you have the privilege.
- Sharing: set the group sticky bit on shared directories so new files inherit your team's group.
- Umask: a single setting that governs default permissions for every file you create going forward.
Processes: what is running and how to stop it
A process is an execution context with an ID, a parent, and a state. The three tools that solve most problems are ps to list processes, top (or htop) to watch resource use live, and kill to send a signal. The nuance that trips people up is differences between signals: a SIGTERM (kill default) asks politely and lets a well-written tool clean up its temporary files, while SIGKILL (kill -9) terminates immediately and can leave partial output behind. Long-running jobs survive your logout only if you detach them properly — with nohup, tmux, or the cluster's own scheduler. Never run a multi-hour job in a bare SSH session and forget the session timeout.
Text processing: grep, sed, and awk
This is where the shell earns its keep. Biological data is overwhelmingly text — FASTA, FASTQ, SAM, VCF, GFF, TSV — and these three tools turn hours of scripting into one-liners.
- grep selects lines by pattern. Anchor patterns with ^ and $, use -v to invert, and count with -c.
- sed performs stream edits, most often substitution with s/old/new/g. It is the right tool for normalising headers and fixing malformed lines.
- awk is a small programming language for field-structured records. Column selection ($1, $3), conditional filtering, and per-column sums or means are one expression away.
A concrete example: to sum a numeric column across a large TSV, awk streams the file in constant memory and finishes in seconds. The equivalent naive Python loop would spend minutes in interpreter overhead. awk's field-oriented model was designed for exactly this kind of records-processing and remains hard to beat (see the AWK book below).
Pipes and redirection: composing small tools
The Unix philosophy — small programs that do one thing and compose through pipes — is why bioinformatics pipelines exist in the form they do. The pipe operator connects the standard output of one command to the standard input of the next, so a multi-stage process becomes a single readable line. Redirection sends output to files, > to overwrite and >> to append, with 2> for errors. The crucial discipline is that well-behaved commands separate data on stdout from progress messages on stderr; if a tool mixes them, your downstream parsing breaks. Filter early and often, so that expensive steps downstream of a pipe see only the records they need.
Job scheduling with SLURM
A cluster is a shared resource, and the scheduler is the negotiating layer. SLURM is the dominant implementation, with a documented command set for submitting, inspecting, and cancelling jobs 2. The concepts are few:
- sbatch submits a batch script; srun launches steps inside an allocation; salloc requests an interactive session.
- Resource requests are explicit: --cpus-per-task, --mem, --time, and --gres for GPUs. Ask for only what you need — over-requesting delays your own job and wastes the shared allocation.
- squeue shows queue state and scancel cancels; --array runs one job per sample, the standard idiom for embarrassingly parallel bioinformatics.
- Job dependencies (--dependency) let you chain stages, so a downstream step only starts after its input finished successfully.
The recurring mistake is script-local assumptions: a job runs on a compute node that is not your login node, with a possibly different environment. Always module load (or activate) your tools inside the batch script, use absolute paths, and write logs to a known directory so a failed array task can be inspected after the fact.
Scripting that survives contact with real data
Three habits separate fragile one-off commands from scripts you can trust. First, start every non-interactive script with set -euo pipefail: exit on error, treat unset variables as errors, and fail a pipeline if any of its stages fail. Second, log what you ran, with which inputs, and where the output went — a job that silently produces an empty file is worse than one that crashes. Third, quote your variables ("$file") so that paths containing spaces do not split into arguments. None of these are exotic; together they remove most of the silent-corruption failure modes in shell pipelines, and they are the same reproducibility principles that apply across computational science 3.
References
- Aho AV, Kernighan BW, Weinberger PJ. The AWK Programming Language. Reading, MA: Addison-Wesley; 1988.
- Yoo AB, Jette MA, Grondona M. SLURM: simple Linux utility for resource management. Job Scheduling Strategies for Parallel Processing (JSSPP). 2003;2862:44-60.
- Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9(10):e1003285.
- Raymond ES. The Art of UNIX Programming. Boston: Addison-Wesley; 2003.
- Kerrisk M. The Linux Programming Interface. San Francisco: No Starch Press; 2010.
- Nekrutenko A, Taylor J. Next-generation sequencing data interpretation: enhancing reproducibility and accessibility. Nat Rev Genet. 2012;13(9):667-672.
- Perkel JM. Ten simple rules for responsible use of high-performance computing clusters. PLoS Comput Biol. 2018;14(3):e1005888.
- Starmer C. Linux Command Line and Shell Scripting Bible. Indianapolis: Wiley; 2011.