3. Command Line, Shells and Text Processing

Introduction and Context

Linux is frequently administered through graphical tools, but its deepest power is exposed through the command line. The command line provides a precise interface to the operating system, while the shell supplies interpretation, composition, automation, and access to system facilities. For postgraduate computer science students, command-line competence is not merely an operational skill. It is a way of expressing computational procedures, examining system state, transforming data, and building reproducible administrative and software-engineering workflows.

A Linux command normally reads input from standard input, writes ordinary results to standard output, and reports errors through standard error. These streams can be redirected, combined, filtered, and connected through pipelines. Shell features such as quoting, expansion, variables, functions, job control, and exit statuses allow simple commands to become robust programs. Text-processing utilities then provide a compact language for extracting information from logs, configuration files, data streams, and program output.

This chapter concentrates on Bash, the most widely encountered shell in Linux environments. The principles apply more broadly to POSIX shells, although individual features may differ. The chapter assumes familiarity with files, directories, permissions, and basic process concepts.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain the roles of the terminal, shell, command, process, and environment.

2. Use Bash syntax for commands, expansion, quoting, variables, functions, and control flow.

3. Combine commands through pipelines and redirect standard streams safely.

4. Apply grep, sed, awk, sort, uniq, cut, tr, and related filters to structured and unstructured text.

5. Construct and interpret regular expressions.

6. Manage foreground and background jobs and understand exit statuses.

7. Write shell scripts that are readable, portable where appropriate, and resistant to common failures.

8. Evaluate security, maintainability, and operational risks in shell-based automation.

The Shell as an Interpreter and Composition Environment

A terminal emulator is a program that provides a text interface. It is not itself the shell. The shell is a command interpreter, such as Bash, Zsh, Dash, or KornShell. When a user types a line, Bash performs several stages: it parses the line, performs expansions, identifies redirections, starts commands, and waits for or controls the resulting processes.

A command may be a shell builtin, an external executable, a function, an alias, or a compound command. Builtins such as `cd`, `export`, `read`, and `umask` must often be implemented inside the shell because they alter the shell's own state. By contrast, utilities such as `ls`, `grep`, and `sort` are normally separate executable files.

Command lookup is influenced by the `PATH` variable. Bash searches each colon-separated directory in `PATH` from left to right. The commands `type`, `command -v`, and `which` (where available) help identify what will be executed. A secure operational practice is to avoid placing the current directory implicitly in `PATH`, because an unintended executable could then be run in preference to a trusted system command.

The command `pwd` reports the current working directory, while `cd` changes it. A relative pathname is interpreted relative to the working directory; an absolute pathname begins at the root directory. Special pathname components include `.` for the current directory and `..` for its parent.

Streams, Descriptors, Redirection, and Pipelines

Unix-like systems represent input and output using file descriptors. By convention, descriptor 0 is standard input, descriptor 1 is standard output, and descriptor 2 is standard error. A command can therefore be connected to files or other commands without changing its internal algorithm.

Common redirections include:

• `command > file` writes standard output to a file, replacing its previous contents.
• `command >> file` appends standard output.
• `command < file` takes standard input from a file.
• `command 2> errors.log` redirects standard error.
• `command > output.log 2>&1` sends both standard output and standard error to the same destination.
• `command &> all.log` is a Bash shorthand for combining both streams.

Redirection order matters. In `command >file 2>&1`, standard output is first attached to `file`, and standard error is then duplicated to the same destination. In `command 2>&1 >file`, standard error remains attached to the original standard output while standard output is redirected to the file.

A pipeline uses `|` to connect the standard output of one command to the standard input of another:

```bash

journalctl -b | grep -i failure | sort | uniq -c

```

Pipelines encourage a filter-oriented style. Each program performs one transformation, making the overall procedure inspectable and reusable. By default, the exit status of a pipeline is the status of its last command. Bash's `pipefail` option changes this behavior so that a pipeline fails when any component fails, returning the rightmost nonzero status. Robust scripts commonly use:

```bash

set -o errexit

set -o nounset

set -o pipefail

```

These options are useful but not magical. `set -e` has contextual exceptions, `set -u` requires care with optional variables, and failures still need meaningful handling. A script should be tested rather than made reliable merely by adding a standard option set.

Here documents provide multiline input:

```bash

cat <<'EOF'

Configuration begins below.

No variable expansion occurs because the delimiter is quoted.

EOF

```

Here strings, written as `command <<< "$value"`, supply one string as standard input. Process substitution, such as `diff <(sort left.txt) <(sort right.txt)`, presents command output as a file-like input and is particularly useful in Bash, although it is not POSIX shell syntax.

Quoting and Expansion: The Source of Many Shell Errors

Bash expands a command before execution. Important expansions include parameter expansion, command substitution, arithmetic expansion, pathname expansion, and word splitting. Their interaction explains many surprising results.

Double quotes preserve spaces and most special characters while allowing parameter and command substitution. Single quotes preserve every character until the next single quote. Thus:

```bash

name='Asha Rao'

printf '%s\n' "$name"

```

prints the complete name as one argument. Without quotes, a variable containing spaces may become multiple arguments. Quoting also prevents wildcard characters in data from being interpreted as filename patterns.

Command substitution uses `$(command)`:

```bash

kernel=$(uname -r)

printf 'Kernel: %s\n' "$kernel"

```

The older backtick form is harder to nest and should generally be avoided. Arithmetic expansion uses `$(( expression ))`, for example `total=$((a + b))`.

Globbing is pathname expansion. `.log` matches names ending in `.log`, while `file?.txt` matches one arbitrary character. Globbing is different from regular-expression matching: glob syntax uses `` for any sequence, whereas basic regular expressions use `.*` for that meaning. If no filename matches a pattern, Bash may pass the pattern unchanged unless its behavior is altered with options such as `nullglob`.

A robust script should quote expansions by default:

```bash

for file in "$directory"/*.log; do

[ -e "$file" ] || continue

printf '%s\n' "$file"

done

```

For filenames containing arbitrary characters, line-oriented methods such as `for file in $(find ...)` are unsafe because command substitution and word splitting lose filename boundaries. Prefer NUL-delimited interfaces where available, for example `find ... -print0` combined with `while IFS= read -r -d '' file`.

Variables, Environment, and Functions

A shell variable is assigned without spaces around `=`:

```bash

project='compiler-lab'

count=3

```

`export project` places the variable in the environment inherited by subsequently created child processes. The shell's variables and the environment are related but not identical: an unexported variable is visible to the current shell but normally not to external commands.

Important variables include `HOME`, `PATH`, `PWD`, `OLDPWD`, `SHELL`, `USER`, `LANG`, and `IFS`. `IFS` influences word splitting and should not be changed casually. Positional parameters are referenced as `$1`, `$2`, and so forth; `$#` is their count, `$@` represents them individually when quoted, and `$?` is the previous command's exit status. `"$@"` is usually the correct way to forward script arguments.

Functions support local abstraction:

```bash

require_file() {

local file=$1

if [[ ! -f "$file" ]]; then

printf 'Missing file: %s\n' "$file" >&2

return 1

fi

}

```

The `local` declaration prevents accidental modification of global shell variables. A function should return meaningful statuses and write diagnostic messages to standard error.

Shell configuration is commonly divided between startup files for interactive shells and login shells. Scripts should not assume that interactive aliases, prompts, or personal configuration are available. A script should explicitly establish the environment it requires.

Regular Expressions and Text Filters

Regular expressions describe sets of strings. In basic regular expressions, commonly used by `grep`, `^` anchors the beginning of a line, `$` anchors the end, `.` matches one character, `*` repeats the preceding atom zero or more times, and bracket expressions such as `[[:digit:]]` describe character classes. Extended regular expressions, enabled by `grep -E` or used by `awk`, add operators such as `+`, `?`, and alternation with `|`.

For example, the pattern `^[[:space:]]#` identifies lines whose first nonblank character is `#`. The command `grep -E '^[[:space:]][^#[:space:]]' config` selects nonblank, noncomment lines. Regular expressions are patterns, not complete parsers; they become fragile when applied to nested, quoted, or context-sensitive formats.

The principal text filters have different strengths:

• `grep` selects lines matching a pattern; `grep -v` selects nonmatching lines.
• `cut` extracts columns or character ranges when delimiters are simple and fixed.
• `tr` translates or deletes characters and is useful for normalization.
• `sort` orders lines; `sort -n` performs numeric ordering and `sort -u` removes duplicate lines after sorting.
• `uniq` collapses adjacent duplicate lines, so sorting is normally required before global deduplication.
• `wc` counts lines, words, bytes, or characters.
• `head` and `tail` inspect beginning and end portions; `tail -f` follows a growing file.
• `sed` performs stream editing, including substitution, deletion, and selected printing.
• `awk` is a small pattern-action language particularly suited to records and fields.

Consider a log whose fields are separated by spaces. The following command counts requests by the seventh field, assuming it contains a path:

```bash

awk '{count[$7]++} END {for (path in count) print count[path], path}' access.log | sort -nr

```

This illustrates associative arrays, an `END` action, and composition with `sort`. It is useful only if the log format is known; quoted fields or spaces inside values may invalidate the field positions.

A `sed` substitution has the form `sed 's/old/new/g' file`. The delimiter need not be `/`; using `|` can reduce escaping when paths are involved. The `-i` option edits files in place, but its behavior differs among implementations and careless patterns can destroy data. A safer workflow writes to a temporary file, validates the result, and replaces the original atomically when appropriate.

Job Control and Process Interaction

Appending `&` starts a command asynchronously. The shell reports a job number and process identifier. `jobs` lists active jobs; `fg` brings a job to the foreground; `bg` resumes a stopped job in the background. `Ctrl-C` normally sends an interrupt to the foreground process group, while `Ctrl-Z` suspends it. `kill` sends a signal, with `TERM` generally preferred before `KILL`.

Job control is primarily an interactive feature. For long-running production tasks, use a service manager, scheduler, container platform, or terminal multiplexer rather than relying on a user's terminal. `nohup` and `disown` can detach work in limited situations but do not replace proper service supervision.

The `wait` builtin is essential when a script launches background jobs and must detect their outcomes:

```bash

process_a & pid_a=$!

process_b & pid_b=$!

wait "$pid_a" || exit 1

wait "$pid_b" || exit 1

```

Without `wait`, a script may finish successfully while child processes continue or fail unnoticed.

Writing Robust Shell Scripts

A shell script begins with a shebang such as `#!/usr/bin/env bash` when Bash is required. It should state assumptions, validate arguments, quote expansions, and use functions to separate concerns. Prefer `printf` over `echo` for predictable formatting, especially when data may begin with a hyphen or contain backslash sequences.

A typical script structure is:

```bash

#!/usr/bin/env bash

set -o errexit -o nounset -o pipefail

usage() {

printf 'Usage: %s DIRECTORY\n' "$0" >&2

}

if (( $# != 1 )); then

usage

exit 64

fi

dir=$1

if [[ ! -d "$dir" ]]; then

printf 'Not a directory: %s\n' "$dir" >&2

exit 66

fi

find "$dir" -type f -name '*.log' -print0 |

while IFS= read -r -d '' file; do

printf '%s\n' "$file"

done

```

The `[[ ... ]]` conditional is safer and more expressive than the older `[ ... ]` command for Bash scripts. Numeric comparisons use arithmetic syntax, such as `(( count > 0 ))`; string comparisons use `[[ "$a" == "$b" ]]`. ShellCheck is a valuable static-analysis tool for detecting quoting errors, suspicious tests, and portability problems.

Temporary files should be created with `mktemp`, permissions should be restrictive, and cleanup should be arranged with a trap:

```bash

tmp=$(mktemp)

trap 'rm -f -- "$tmp"' EXIT

```

The `--` convention protects utilities from arguments beginning with a hyphen, although not every command supports it identically. Scripts that process untrusted input must avoid `eval`, must not construct command strings unnecessarily, and must treat filenames and data as data rather than executable shell syntax.

Applied Case Study: Automating a Service-Health Report

Background

An operations group maintains several Linux application servers. Each server writes an access log containing a timestamp, response status, and request path. The team needs a daily report showing failed requests and the most frequent paths associated with them.

Problem

Manual inspection is slow and inconsistent. A first attempt uses `grep 500 access.log | cut -d' ' -f7 | uniq -c`, but it produces incorrect counts because `uniq` only combines adjacent records and because the status text may occur in an unrelated field.

Analysis

The team first documents the log format and confirms that the status code is field 9 and the request path is field 7. It decides to use `awk` for field-aware selection and counting, then `sort -nr` for ranking. The report must preserve filenames safely and must return a failure status when input is missing.

Approach

An illustrative Bash script is:

```bash

#!/usr/bin/env bash

set -o errexit -o nounset -o pipefail

log=${1:?Usage: report.sh LOGFILE}

[[ -r "$log" ]] || { printf 'Cannot read %s\n' "$log" >&2; exit 1; }

printf 'Failed requests in %s\n' "$log"

awk '$9 ~ /^5[0-9][0-9]$/ {failed[$7]++}

END {for (path in failed) print failed[path], path}' "$log" |

sort -nr

```

The script validates the input, uses a regular expression anchored to the complete status field, counts by path, and ranks numerically. In a production implementation, the format would be parsed according to the actual logging specification rather than assumed whitespace fields. The output might then be written to a controlled report directory and scheduled through a service scheduler.

Outcome and Learning Points

The automated report is repeatable, auditable, and easy to modify. The case demonstrates that a short pipeline is not automatically correct: field assumptions, ordering requirements, exit statuses, quoting, and input validation determine reliability. It also illustrates a boundary of shell tools: if the log becomes JSON, contains nested structures, or requires complex correlation across records, a dedicated parser may be safer.

Risks, Limitations, and Professional Practice

Shell is excellent for orchestration and text-oriented administration, but it is not a universal programming language. It has weak data structures, subtle expansion rules, limited error types, and portability differences between Bash, POSIX `sh`, GNU utilities, and BSD utilities. Large scripts often become difficult to test and maintain; Python, Go, or another general-purpose language may be preferable for complex parsing, concurrency, or transaction management.

Common mistakes include unquoted variables, parsing `ls`, confusing globs with regular expressions, assuming `uniq` performs global deduplication, ignoring standard error, losing pipeline failures, using `for line in $(command)`, and deleting files with insufficient validation. Shell injection is a serious risk when external input is interpolated into commands. Privileged scripts require particularly strict path validation, controlled environments, least privilege, and careful review of `PATH` and temporary-file handling.

Operational scripts should be version-controlled, peer-reviewed, tested with representative and adversarial inputs, and accompanied by usage documentation. Logs should avoid exposing passwords, tokens, personal data, or confidential content. Access controls and retention policies apply to generated reports just as they apply to source logs.

Current Developments and Emerging Practice

Modern Linux operations increasingly combine shell with configuration management, containers, CI/CD pipelines, and observability platforms. Shell remains a valuable glue language for invoking tools and expressing deployment steps, while structured formats such as JSON, YAML, and machine-readable APIs reduce the reliability of ad hoc text parsing. Tools such as ShellCheck, shfmt, Bats, and continuous integration tests support quality assurance.

AI-assisted coding can suggest shell commands, but generated commands require human verification because a superficially plausible command may overwrite files, leak secrets, or mishandle unusual filenames. A sound practice is to test commands in isolated environments, inspect expansions, use dry-run modes where available, and retain human approval for destructive or privileged operations.

Best Practices

Use the narrowest shell language required; declare Bash explicitly when using Bash features. Quote expansions, validate inputs, distinguish data from commands, and send diagnostics to standard error. Design pipelines so that each stage has a clear contract, enable and test appropriate failure handling, and preserve filenames with NUL-delimited processing when necessary. Prefer atomic updates, least privilege, version control, static analysis, and automated tests. Finally, choose a general-purpose language when the problem exceeds shell's strengths.

Chapter Summary

The Linux command line is a compositional environment in which shells interpret commands and connect processes through streams. Bash provides redirection, pipelines, expansion, quoting, variables, functions, control structures, and job control. Correct quoting and awareness of expansion order are central to safe command construction. Text filters such as `grep`, `sed`, `awk`, `sort`, `cut`, `tr`, and `uniq` support powerful transformations, but their assumptions must match the data format. Robust scripts validate inputs, preserve exit statuses, handle filenames safely, manage temporary resources, and avoid injection risks. Shell is highly effective for orchestration and system administration, while complex parsing and application logic may require other languages. Professional shell practice therefore combines concise composition with testing, security, documentation, and informed tool selection.

Key Terms

Shell: A command interpreter that parses and executes commands.

Bash: A widely used Unix shell implementing POSIX features plus extensions.

Terminal emulator: A program that provides an interactive text interface to a shell.

File descriptor: An integer handle through which a process accesses an input or output stream.

Standard input: File descriptor 0, conventionally used for input.

Standard output: File descriptor 1, conventionally used for ordinary output.

Standard error: File descriptor 2, conventionally used for diagnostics.

Pipeline: A connection in which one command's output becomes another command's input.

Redirection: Reassignment of a command's input or output to a file or descriptor.

Quoting: Shell syntax that controls interpretation, expansion, and word splitting.

Globbing: Shell pathname expansion using patterns such as `*` and `?`.

Environment variable: An exported variable inherited by child processes.

Regular expression: A pattern describing a set of matching strings.

Filter: A command that reads input, transforms or selects it, and produces output.

Exit status: An integer result indicating whether a command completed successfully.

Job control: Facilities for suspending, resuming, foregrounding, and backgrounding processes.

Here document: Multiline shell input delimited by a specified marker.

Command substitution: Expansion that replaces `$(command)` with command output.

Shell injection: Execution of unintended commands caused by unsafe handling of input.

Review Questions

Short Answer Questions

1. Distinguish a terminal emulator from a shell.

2. What are file descriptors 0, 1, and 2 used for?

3. Why is `uniq` commonly preceded by `sort`?

4. Explain the difference between single and double quotes in Bash.

5. What purpose does `pipefail` serve?

Long Answer Questions

1. Explain Bash expansion stages and show how quoting prevents common errors.

2. Describe pipelines and redirection, including the significance of redirection order.

3. Compare `grep`, `sed`, and `awk` in terms of their appropriate uses.

4. Discuss the design principles of a robust shell script.

5. Explain job control and the role of `wait` in scripts that launch background processes.

Analytical / Application Questions

1. Design a safe pipeline to count unique error codes from a log and explain every stage.

2. A script works for simple filenames but fails when filenames contain spaces and newlines. Diagnose the cause and propose a corrected approach.

3. Evaluate whether Bash or a general-purpose language is more suitable for processing a nested JSON log with concurrent network requests. Justify your decision.

22. Multiple Choice Questions with Answers

Question 1

Which redirection sends both standard output and standard error to the same file in the stated order?

A. command 2> file 1> file

B. command > file 2>&1

C. command < file 2>&1

D. command | file

Correct Answer: B. command > file 2>&1

Question 2

Why is `sort` commonly used before `uniq` when counting duplicate lines?

A. uniq sorts input automatically

B. sort converts text into numbers

C. uniq only collapses adjacent duplicates

D. sort removes all blank lines

Correct Answer: C. uniq only collapses adjacent duplicates

Question 3

Which quoted expansion correctly forwards all positional parameters while preserving argument boundaries?

A. $*

B. "$*"

C. $@

D. "$@"

Correct Answer: D. "$@"

Question 4

What is the principal effect of enabling Bash `pipefail`?

A. It disables pipelines containing errors

B. A pipeline can fail when a non-final component fails

C. It redirects errors to standard output

D. It makes every command run in the background

Correct Answer: B. A pipeline can fail when a non-final component fails

Question 5

Which regular expression matches a line whose first nonblank character is a comment marker?

A. ^[[:space:]]*#

B. #[[:space:]]*$

C. [[:space:]]+#$

D. ^#*[[:space:]]

Correct Answer: A. ^[[:space:]]*#

Question 6

A script uses `for file in $(find . -type f)`. What is the principal reliability problem?

A. find cannot search recursively

B. Command substitution may split filenames on whitespace

C. The loop always runs in the background

D. Shell variables cannot contain filenames

Correct Answer: B. Command substitution may split filenames on whitespace

Question 7

Which tool is most naturally suited to counting records by a selected field using an associative array?

A. awk

B. head

C. tr

D. wc

Correct Answer: A. awk

Question 8

What does `2>&1` mean when used after `>output.log`?

A. It sends standard input to output.log

B. It duplicates standard output's current destination for standard error

C. It discards standard error

D. It appends standard error to standard input

Correct Answer: B. It duplicates standard output's current destination for standard error

Question 9

Which practice most directly reduces shell injection risk?

A. Using eval for every constructed command

B. Interpolating untrusted input into a command string

C. Treating input as data and avoiding eval

D. Adding the current directory to PATH

Correct Answer: C. Treating input as data and avoiding eval

Question 10

Why should `wait` be used after launching background jobs in a script?

A. It converts background jobs into aliases

B. It allows the script to observe child-process completion and status

C. It prevents all signals from reaching the jobs

D. It automatically sorts their output

Correct Answer: B. It allows the script to observe child-process completion and status