Pages

Fixing PowerHA Error: "an incomplete cluster migration has been detected"

When you attempt to verify or synchronize a PowerHA cluster, you may encounter the following error:

clmgr: ERROR: an incomplete cluster migration has been detected.
Cluster verification/synchronization cannot be performed until the migration is
completed and all nodes are running the same version of the product.

This message means that the PowerHA cluster is in an inconsistent migration state — one or more nodes are not running the same PowerHA version or fix level, or the upgrade process was started but not completed on all nodes.

This issue typically appears after:
  • A PowerHA version upgrade (for example, from 7.2.4 to 7.2.5).
  • A fix pack or TL/SP installation on only one cluster node.
  • A partially completed migration — upgrade completed on one node but not the others.
  • An interrupted cluster migration process (clmigcheck, clmigstart, etc.).
  • Until all nodes run the same PowerHA version and migration level, cluster synchronization and verification commands (clmgr verify cluster, clmgr sync cluster) will fail with this message.
Root Cause:
PowerHA stores the cluster version and migration status in its object repository and internal configuration files under:

  • /usr/es/sbin/cluster/etc/objrepos/
  • /etc/es/objrepos/
If one node reports a newer or different version of PowerHA, the cluster recognizes this as an incomplete migration.
For example:
Node1: PowerHA 7.2.5.3
Node2: PowerHA 7.2.4.4

In this case, the cluster cannot synchronize because the internal structures may differ between versions.

Step-by-Step Fix:

Step 1: Check PowerHA Version on All Nodes
Run the following on each cluster node:

# lslpp -l | grep cluster.es.server
Typical output:
# cluster.es.server 7.2.5.3 COMMITTED PowerHA SystemMirror for AIX
If the versions differ, that’s the cause.

Step 2: Check Migration Status
You can use:
# clmgr query cluster
If a migration is incomplete, you’ll see status indicators like:
Cluster: mycluster
Migration: INCOMPLETE

Step 3: Complete or Roll Back the Migration

Option A: Complete the Migration
If you were upgrading PowerHA, ensure that all nodes are upgraded to the same level.
Use smitty update_all or installp to install the same PowerHA filesets across nodes.
For example:
# installp -acXYgd /path/to/PowerHA_7253_images all
Verify installation:
# lslpp -l | grep cluster.es.server
After all nodes have the same version, complete migration:
# clmgr migrate cluster complete

Option B: Roll Back (if migration failed)
If migration was accidentally started but should not continue, you can roll it back:
# clmgr migrate cluster rollback
Then ensure all nodes are running the same previous version.

Step 4: Verify Cluster Consistency
Once all nodes are synchronized in version and migration state:
# clmgr verify cluster
If successful, you’ll see:
Verification has completed normally.
Then synchronize:
# clmgr sync cluster <cluster_name>

Fixing PowerHA Error: Cluster IPC error: The cluster manager on node is in ST_INIT or NOT_CONFIGURED state

In IBM PowerHA SystemMirror (formerly HACMP), you may encounter an error message like this during cluster operations, synchronization, or while running clmgr commands:

Cluster IPC error: The cluster manager on node note1 is in ST_INIT or NOT_CONFIGURED state and cannot process the IPC request.

This error prevents cluster commands from completing successfully and indicates that inter-node communication (IPC) within the cluster has failed.

What This Error Means:

PowerHA nodes communicate through IPC (Inter-Process Communication) between the clstrmgrES daemons on each node.

This message tells you that the target node’s cluster manager process (clstrmgrES) is either:

  • ST_INIT The cluster manager is initializing — services haven’t fully started yet.
  • NOT_CONFIGURED The node’s local PowerHA configuration (ODM repository) isn’t set up or is corrupted.

So when you try to synchronize, verify, or query the cluster, that node cannot respond.

Root Causes:

This error can happen for several reasons:

  • Cluster manager service not running (clstrmgrES daemon stopped or failed).
  • ODM or repository corruption in /etc/es/objrepos/HACMP* files.
  • Network communication issue — PowerHA communication interfaces down or hostname mismatch.
  • Cluster configuration mismatch between nodes.
  • Partial node configuration (node added but not yet synchronized).


Step-by-Step Fix:


Step 1: Check the Cluster Manager Daemon

On the affected node (note1):

# ps -ef | grep clstrmgrES

If not running, start it:

# startsrc -s clstrmgrES

Confirm it’s active:

# lssrc -s clstrmgrES

Expected output:

Subsystem         Group            PID          Status

clstrmgrES       cluster          123456       active

If the status shows “inoperative,” check /tmp/hacmp.out and /var/hacmp/log/clstrmgr.debug for startup errors.


Step 2: Verify Network and Hostname Resolution

PowerHA depends on stable communication between cluster nodes.

Check interface status:

# netstat -in

# lsdev -Cc if

Verify that all cluster nodes resolve correctly:

# host note1

# host <other_node>

If any hostnames fail, fix /etc/hosts or DNS before continuing.


Step 3: Validate ODM (Object Repository) Integrity

Check that the cluster repository files exist and are consistent:

# ls -l /etc/es/objrepos/HACMP*

If they are missing, empty, or corrupted, restore them from a healthy node:

# scp root@healthy_node:/etc/es/objrepos/HACMP* /etc/es/objrepos/

Then verify:

# clmgr verify cluster


Step 4: Restart PowerHA Cluster Services

Once network and configuration are healthy, restart PowerHA services on the affected node:

# stopsrc -s clstrmgrES

sleep 5

# startsrc -s clstrmgrES

Monitor startup logs:

# tail -f /tmp/hacmp.out

# tail -f /var/hacmp/log/clstrmgr.debug

You should see messages showing the node transitioning from ST_INIT → UP.


Step 5: Synchronize Cluster Configuration

Once all nodes are up and communicating:

# clmgr sync cluster <cluster_name>

If successful, you’ll see:

Synchronization completed successfully.


Step 6: Confirm Cluster Health

Finally, verify cluster and node status:

# clmgr query cluster

# clmgr status cluster

Expected healthy output:

Cluster: mycluster

  State: UP

  Node note1: UP

  Node note2: UP

Fixing PowerHA Cluster Synchronization Error: "Updating the cluster in AIX failed"

Observed Error Messages:

WARNING: There is no split and merge policy set.
You may also configure a quarantine policy using disk fencing or active node halt policy.
The quarantine policy can be used instead of a split and merge policy or both can be used at the same time for added protection.
Completed 100 percent of the verification checks
Verification has completed normally.
Committing any changes, as required, to all available nodes...
cl_dspmsg: $ missing from %n$ format in message
cl_dspmsg: $ missing from %n$ format in message
ERROR: Updating the cluster in AIX failed. Check output for errors in local
cluster configuration, correct them, and try synchronization again.
cldare: Error detected during synchronization.


Understanding the Messages:

Message Meaning:
“No split and merge policy set” The cluster has no defined tie-breaker or node arbitration rule in a split-brain scenario.
cl_dspmsg: $ missing from %n$ format A localization/message catalog formatting issue in PowerHA utilities. Usually harmless but signals a mismatch in message catalog files.
“Updating the cluster in AIX failed” Cluster repository sync failed—PowerHA couldn’t commit configuration changes to ODM (AIX Object Data Manager).
cldare: Error detected during synchronization. The low-level PowerHA daemon reported the sync failure, often due to repository corruption or permission issues.


Root Cause Summary:
This situation usually happens when:
  • Cluster object repository files (/etc/es/objrepos/HACMP*) are corrupted or out of sync.
  • Split and merge policy is missing — not fatal, but it causes warnings and sometimes sync halts.
  • Message catalog files are mismatched due to incomplete PowerHA updates.
  • The node’s local configuration is inconsistent with the cluster definition.

Step-by-Step Fix Procedure:

Step 1: Backup Repository Files
Always protect your existing configuration first.
# mkdir -p /tmp/hacmp_repo_backup
# cp -p /etc/es/objrepos/HACMP* /tmp/hacmp_repo_backup/

Step 2: Verify Cluster Configuration
Run:
# clmgr verify cluster
If it passes (as it did in your log: “Verification has completed normally”), proceed to fix the synchronization step.

Step 3: Restore Local Repository from Active Configuration
If the sync fails, the active repository copy under PowerHA’s working directory is usually still valid.
Restore it:
# cp -p /usr/es/sbin/cluster/etc/objrepos/active/HACMPcluster /etc/es/objrepos/HACMPcluster
Do the same for other HACMP-related files if needed:
# cp -p /usr/es/sbin/cluster/etc/objrepos/active/HACMP* /etc/es/objrepos/

Step 4: Check File Ownership and Permissions
Make sure all repository files are owned correctly:

# chown root:system /etc/es/objrepos/HACMP*
# chmod 644 /etc/es/objrepos/HACMP*

Step 5: Define a Split and Merge Policy (Recommended)
This removes the warning and stabilizes cluster arbitration.
For example:
# clmgr modify cluster <cluster_name> split_merge_policy=HIGHEST_PRIORITY
Or if you prefer using SMIT:
# smitty hacmp
→ Extended Configuration
→ Extended Topology Configuration
→ Configure Cluster
→ Change/Show Cluster Configuration
Then set:
Split/Merge Policy: HIGHEST_PRIORITY
Quarantine Policy: disk_fencing (optional)

Step 6: Synchronize the Cluster Again
Once the repository is restored and policies set, run:
# clmgr sync cluster <cluster_name>
If successful, you’ll see:
Synchronization completed successfully.

Step 7: Confirm Cluster Status
# clmgr status cluster
# clmgr query cluster
All nodes should show UP and Stable.

Common shell script errors

1. Missing Shebang (#!/bin/bash)
Error: Script fails or behaves unexpectedly.
Example:
echo "Hello World"
Running ./script.sh may fail if no interpreter is specified.
Fix:
#!/bin/bash
echo "Hello World"
Always include a shebang at the top.
Make the script executable:
chmod +x script.sh

2. Permission Denied
Error:
bash: ./script.sh: Permission denied
Cause: The script file is not executable.
Fix:
chmod +x script.sh
./script.sh

3. Syntax Errors
Error:
syntax error near unexpected token `do'
Common Causes:
Missing do or done in loops
Missing then in if statements
Unmatched quotes (" or ')
Example:
if [ $num -eq 5 ]
echo "Number is 5"
fi
Fix:
if [ $num -eq 5 ]; then
echo "Number is 5"
fi

4. Variable Errors
Symptoms: Variables don’t expand, or you get errors like unary operator expected.
Common Causes:
Not using $ when referencing variables
Using spaces incorrectly
Example:
name=John
echo "Hello name"
Fix:
name=John
echo "Hello $name"
Another example (spaces):
count = 5 # 
Fix:
count=5 #  No spaces around =

5. Command Not Found
Error:
./script.sh: line 3: somecommand: command not found
Causes:
Typo in command
Command not installed
Command not in $PATH
Fix:
Check spelling
Install the command
Use full path (/usr/bin/somecommand)

6. File or Directory Not Found
Error:
./script.sh: line 4: /path/to/file: No such file or directory
Fix:
Make sure the file exists
Use correct paths
Use quotes for paths with spaces:
file="/path/to/My File.txt"
cat "$file"

7. Infinite Loops
Symptoms: Script never ends, CPU usage spikes.
Example:
while [ $count -le 5 ]
do
echo $count
# Missing increment
done
Fix:
while [ $count -le 5 ]
do
echo $count
((count++))
done

8. Quoting Issues
Symptoms: Unexpected word splitting or globbing.
Example:
name="John Doe"
echo $name # Works
touch $name # Creates two files: John and Doe 
Fix: Always quote variables:
touch "$name" #  Creates file "John Doe"

9. Using the Wrong Test Syntax
[ ... ] vs [[ ... ]]
-eq for numbers, = for strings
Example (error):
if [ $num = 5 ]; then # 
Fix:
if [ $num -eq 5 ]; then # For numbers

10. Missing exit Codes
Scripts sometimes fail silently.
Always use exit 0 for success, exit 1 for failure if needed.
Example:
#!/bin/bash
if ! grep "something" file.txt; then
echo "Not found"
exit 1
fi
echo "Done"
exit 0

Tips to Avoid Errors
Always use #!/bin/bash or appropriate shebang.
Quote variables when in doubt.
Test scripts incrementally.
Use set -e (exit on error) and set -x (debug mode).
#!/bin/bash
set -e # Exit on first error
set -x # Print commands as they run
Check return codes with $?.

Shell scripting string manipulation tools

Comparing Values in Shell Scripting:
In shell scripting, comparing values—numbers, strings, or files—is very common. The method depends on the type of data you’re comparing.

Compare Numbers
Use test, [ ], or [[ ]] with numeric operators:
Operator Meaning Example
-eq equal to [ $a -eq $b ]
-ne not equal to [ $a -ne $b ]
-lt less than [ $a -lt $b ]
-le less than or equal to [ $a -le $b ]
-gt greater than [ $a -gt $b ]
-ge greater than or equal to [ $a -ge $b ]
Example:
#!/bin/bash
a=10
b=20
if [ $a -lt $b ]; then
  echo "$a is less than $b"
else
  echo "$a is greater or equal to $b"
fi
Alternative Arithmetic Comparison

if (( a < b )); then
  echo "a is less than b"
fi

Compare Strings
Use = and !=. With [[ ]], you can also use pattern matching or lexicographical comparison.
Operator Meaning Example
= strings are equal [ "$a" = "$b" ]
!= strings are not equal [ "$a" != "$b" ]
< less than (lexicographical) [[ "$a" < "$b" ]]
> greater than [[ "$a" > "$b" ]]
-z string is empty [ -z "$a" ]
-n string is not empty [ -n "$a" ]
Example:
#!/bin/bash
str1="hello"
str2="world"
if [ "$str1" = "$str2" ]; then
  echo "Strings are equal"
else
  echo "Strings are different"
fi

Compare Files
Compare timestamps or identity:
Operator Meaning Example
-nt file1 is newer than file2 [ file1 -nt file2 ]
-ot file1 is older than file2 [ file1 -ot file2 ]
-ef same file (hard link) [ file1 -ef file2 ]
Example:
#!/bin/bash
if [ file1.txt -nt file2.txt ]; then
  echo "file1.txt is newer"
fi

Compare File Contents
Use diff or cmp:
diff → shows line-by-line differences
diff file1.txt file2.txt
cmp → compares byte-by-byte, returns 0 if identical
if cmp -s file1.txt file2.txt; then
  echo "Files are identical"
else
  echo "Files differ"
fi

Combine Conditions
&& → AND
|| → OR

Example:
if [ $a -gt 0 ] && [ $b -gt 0 ]; then
  echo "Both are positive"
fi
# Using (( )) for cleaner syntax
if (( a > 0 && b > 0 )); then
  echo "Both positive"
fi

Conditional Statements and Loops in Shell Scripting:

If Statement
Used to execute code conditionally based on a test.
Syntax:
if [ condition ]; then
  # code if condition is true
fi
Example:
#!/bin/bash
a=10
b=20
if [ $a -lt $b ]; then
  echo "$a is less than $b"
fi

If-Else Statement
Executes alternative code when the condition is false.
Syntax:
if [ condition ]; then
  # code if true
else
  # code if false
fi
Example:
#!/bin/bash
num=5
if [ $num -gt 0 ]; then
  echo "Positive number"
else
  echo "Zero or negative number"
fi

If-elif-else Statement
Allows multiple conditions to be checked sequentially.
Syntax:
if [ condition1 ]; then
  # code1
elif [ condition2 ]; then
  # code2
else
  # default code
fi
Example:
#!/bin/bash
num=0
if [ $num -gt 0 ]; then
  echo "Positive"
elif [ $num -lt 0 ]; then
  echo "Negative"
else
  echo "Zero"
fi

For Loop
Repeats code for a known number of iterations or items.
Syntax (numeric):
for (( i=0; i<5; i++ )); do
  # code
done
Example (numeric loop):
#!/bin/bash
for (( i=1; i<=5; i++ )); do
  echo "Count: $i"
done
Syntax (iterating over list):
for item in apple banana cherry; do
  echo $item
done

While Loop
Repeats code while a condition is true.
Syntax:
while [ condition ]; do
  # code
done
Example:
#!/bin/bash
count=1
while [ $count -le 5 ]; do
  echo "Count: $count"
  ((count++))
done

Until Loop
Repeats code until a condition becomes true (opposite of while).
Syntax:
until [ condition ]; do
  # code
done
Example:
#!/bin/bash
count=1
until [ $count -gt 5 ]; do
  echo "Count: $count"
  ((count++))
done

Do-While Simulation in Bash
Bash does not have native do-while, but it can be simulated using while true + break.
Syntax:
while true; do
  # code
  [ condition_to_stop ] && break
done
Example:
#!/bin/bash
count=1
while true; do
  echo "Count: $count"
  ((count++))
  [ $count -gt 5 ] && break
done

Nested Loops
Loops can be nested inside each other.
Example:
#!/bin/bash
for i in 1 2 3; do
  for j in a b; do
    echo "$i-$j"
  done
done

Tips:
Always quote variables in tests: [ "$var" = "value" ]
Use (( )) for arithmetic comparisons.
Use break to exit a loop early.
Use continue to skip the current iteration.

cut Command – Technical Reference:
The cut command is a Linux/Unix utility used to extract sections of text from each line of input, typically from a file or piped data. It allows cutting by delimiter, character position, or byte position, making it ideal for processing structured text like CSV files, logs, or system data.

Basic Syntax:
cut OPTION... [FILE]...
Common Options
Option   Description
-b   Select bytes
-c   Select characters
-d   Specify delimiter (default TAB)
-f   Select fields (requires -d)
--complement         Select everything except the specified selection
--output-delimiter Specify delimiter for output

Cutting by Character Position (-c):
echo -e "abcdefg\nhijklmn" | cut -c 1-5
# Output:
# abcde
# hijkl
1-5 → characters 1 through 5
-c 1,3,5 → characters 1, 3, and 5 only
-c 1-5,8-10 → multiple ranges

Cutting by Byte Position (-b):
echo "abcdef" | cut -b 1-3
# Output: abc
Counts bytes, not characters
Important for multi-byte encodings like UTF-8

Cutting by Field (-f):
Use with -d to extract columns from delimited files.
Example: data.txt

Alice,25,USA
Bob,30,UK
cut -d "," -f 1 data.txt
# Output:
# Alice
# Bob
-f 1,3 → fields 1 and 3
-f 2- → field 2 to the end
-f -2 → first 2 fields

Additional Options:
Complement
echo "apple,banana,cherry" | cut -d "," -f 2 --complement
# Output: apple,cherry
Output Delimiter
echo "apple,banana,cherry" | cut -d "," -f 1,3 --output-delimiter="|"
# Output: apple|cherry
Suppress Lines Without Delimiter (-s)
echo -e "a:b\nc\nd:e" | cut -d ":" -f 2 -s
# Output:
# b
# e
# (skips "c" because no ":")

Practical Examples:
Extract usernames from /etc/passwd
cut -d ":" -f 1 /etc/passwd | head -5
Extract first and last columns of a CSV
cut -d "," -f 1,3 data.csv
Extract specific characters
echo "hello world" | cut -c 1-5
# Output: hello
Extract multiple fields and change delimiter
echo "apple,banana,cherry" | cut -d "," -f 1,3 --output-delimiter=";"
# Output: apple;cherry
Exclude a field using complement
echo "1,2,3,4" | cut -d "," -f 2 --complement
# Output: 1,3,4

Tips & Best Practices:
Always quote variables if they contain spaces:
cut -d "," -f 1 <<< "$mystring"
cut is not ideal for complex patterns—use awk or sed instead.
-f is 1-indexed: first field is $1.
Use with pipes for dynamic input:
ps aux | cut -d " " -f 1 | sort | uniq

AWK Command – Technical Reference:
awk is a powerful text processing tool for pattern scanning, field extraction, and reporting. It is extremely flexible for files, piped input, or strings. Unlike cut, awk supports conditions, calculations, formatting, and loops, making it ideal for advanced text processing.

Basic Syntax
awk 'pattern { action }' file
pattern → what to search for (optional; defaults to all lines)
action → what to do with matching lines (optional; defaults to {print $0})
Example: Print first column of every line:
awk '{ print $1 }' file.txt

Fields and Records
$1, $2, $3 → first, second, third field
$0 → entire line
Default field separator: whitespace (spaces or tabs)
Custom separator:
awk -F ":" '{ print $1 }' /etc/passwd

Common Patterns
Pattern Description
/regex/    Matches lines containing regex
NR==n        Matches line number n
NR>=2        Matches lines from line 2 onwards
condition  Matches lines where condition is true (e.g., $2>10)
Examples:
awk '/error/ {print $0}' logfile.txt
awk 'NR==1 {print $0}' file.txt
awk '$3 > 50 {print $1, $3}' data.txt

Actions
{ print $1, $2 } → print specific fields
{ sum += $2 } END { print sum } → accumulate and print at the end
{ gsub(/pattern/, "replacement"); print } → replace text
Examples:
awk '{ print $1, $3 }' data.txt
awk '{ sum += $2 } END { print "Total:", sum }' data.txt
awk '{ gsub(/foo/, "bar"); print }' file.txt

Predefined Variables
Variable Meaning
NF Number of fields in current record
NR Number of records (lines) read so far
FS Input field separator
OFS Output field separator
RS Input record separator (default newline)
ORS Output record separator (default newline)
Example: Print last column of each line
awk '{ print $NF }' file.txt

BEGIN and END Blocks
BEGIN → executes before reading input
END → executes after all input is processed
Example: Print header and sum
awk 'BEGIN { print "Name Score" }
{ sum += $2; print $1, $2 }
END { print "Total:", sum }' data.txt

Field and String Operations
Concatenate strings:
awk '{ print $1 " lives in " $2 }' data.txt
Substring: substr($1,1,3)
Length: length($1)

Mathematical Operations
awk '{ total += $2; avg = total/NR } END { print "Average:", avg }' data.txt
Supports + - * / % ^

Conditional Statements
awk '{ if ($2 > 50) print $1, $2; else print $1, "low" }' data.txt
awk '{ print ($2 > 50) ? "High" : "Low" }' data.txt

Loops
For loop:
awk '{ for(i=1;i<=NF;i++) print $i }' data.txt
While loop:
awk '{ i=1; while(i<=NF){ print $i; i++ } }' data.txt

Multiple Files
awk '{ print FILENAME, $0 }' file1.txt file2.txt
FILENAME → current file being processed

Practical Examples
Extract usernames from /etc/passwd
awk -F ":" '{ print $1 }' /etc/passwd
Print lines containing "error"
awk '/error/ { print NR, $0 }' logfile.txt
Calculate total of second column
awk '{ sum += $2 } END { print "Total:", sum }' data.txt
Print lines with more than 3 fields
awk 'NF>3 { print $0 }' file.txt
Replace text
awk '{ gsub(/apple/, "orange"); print }' file.txt

AWK Options
Option Description
-F Input field separator
-v var=value Assign variable before execution
-f scriptfile Execute AWK script from a file
--version Show version info
Example with variable:
awk -v threshold=50 '$2 > threshold {print $1, $2}' data.txt

Tips
Use awk instead of cut when conditions, calculations, or formatting are needed.
Quote variables if they contain spaces:
awk '{print $1}' <<< "$myvar"
Combine with pipes for dynamic input:
ps aux | awk '{print $1, $2}'
Use BEGIN and END for headers, footers, totals, or setup.

SED Command – Technical Reference:
sed (Stream Editor) is a text processing utility for efficiently searching, replacing, deleting, inserting, and modifying text from files or standard input. Unlike awk, which can perform calculations and field extraction, sed focuses on line-oriented transformations using scripts and regular expressions.

Basic Syntax
sed [options] 'script' file
script → commands like s/pattern/replacement/, d, p, etc.
file → input file (optional; reads from stdin if omitted)

Common Sed Commands
Command Description
s/pattern/replacement/ Substitute pattern with replacement
d Delete the line
p Print the line (use with -n)
i\ text Insert text before the line
a\ text Append text after the line
c\ text Replace the entire line with text

Substitution (s)
sed 's/apple/orange/' file.txt
# Replace first occurrence in each line
sed 's/apple/orange/g' file.txt
# Replace all occurrences in the line
sed '2s/apple/orange/' file.txt
# Replace only on line 2
sed 's/apple/orange/2' file.txt
# Replace 2nd occurrence in line

Deleting Lines (d)
sed '3d' file.txt           # Delete line 3
sed '/apple/d' file.txt     # Delete lines containing "apple"
sed '1,5d' file.txt         # Delete lines 1 to 5

Printing Lines (p)
sed -n '3p' file.txt        # Print only line 3
sed -n '/apple/p' file.txt  # Print lines containing "apple"
-n suppresses default printing

Inserting, Appending, Changing
sed '2i\This is inserted line' file.txt  # Insert before line 2
sed '2a\This is appended line' file.txt  # Append after line 2
sed '2c\This line is replaced' file.txt # Change entire line 2

Multiple Commands
Using -e:
sed -e 's/apple/orange/' -e '/banana/d' file.txt
Using { } block:
sed '2,4{ s/apple/orange/; s/banana/mango/ }' file.txt
8. Regular Expressions
Symbol Meaning
^ Start of line
$ End of line
. Any character
* Zero or more occurrences
[a-z] Character class
Examples:
sed 's/^/START-/' file.txt          # Add START- at beginning
sed 's/$/-END/' file.txt            # Add -END at end
sed 's/[0-9]/#/g' file.txt          # Replace digits with #

In-place Editing (-i)
sed -i 's/apple/orange/g' file.txt   # Edit file directly
sed -i.bak 's/apple/orange/g' file.txt  # Backup original
10. Using Sed with Pipes
echo "apple banana apple" | sed 's/apple/orange/g'
cat file.txt | sed '/^$/d'          # Delete empty lines

Advanced Examples
Delete empty lines: sed '/^$/d' file.txt
Delete lines starting with #: sed '/^#/d' file.txt
Replace text between patterns: sed '/start/,/end/s/foo/bar/g' file.txt
Add text after matching line: sed '/pattern/a\This is appended' file.txt
Print line numbers: sed '=' file.txt | sed 'N;s/\n/ /'

Tips
Always quote sed scripts to avoid shell interpretation
Use -i.bak for safe in-place editing
Combine with awk and grep for complex processing
Use sed -r or sed -E for extended regex

Quick Reference Table
Operation        Sed Command
Substitute      s/pattern/replacement/
Global substitute s/pattern/replacement/g
Delete line      Nd or /pattern/d
Print line      Np or /pattern/p
Insert line      Ni\text
Append line      Na\text
Change line         Nc\text
In-place edit         -i
Range of lines  2,5d or 2,5s/foo/bar/g

READ Command – Technical Reference:
The read command is used to take input from the user or standard input and store it in a variable. It is fundamental for interactive shell scripts, allowing scripts to prompt users, read files, or handle input dynamically.

Basic Syntax
read [options] variable_name
Reads a line and stores it in the variable
If no variable is specified, input is stored in REPLY
Example:
read name
echo "Hello, $name"

Prompting the User
read -p "Enter your name: " name
echo "Hello, $name"
-p "prompt" → displays a prompt before reading input

Silent Input (Password)
read -s -p "Enter password: " password
echo
echo "Password accepted"
-s → input not displayed
Use echo to move to a new line after input

Timeout Option
read -t 5 -p "Enter your name within 5 seconds: " name
if [ -z "$name" ]; then
  echo "Timeout!"
else
  echo "Hello, $name"
fi
-t seconds → timeout in seconds

Reading Multiple Variables
read first_name last_name
echo "First: $first_name, Last: $last_name"
Input is split by whitespace
Extra words are stored in the last variable
read a b
# Input: 1 2 3 4 → a=1, b="2 3 4"

Reading Into an Array
read -a arr
# Input: 1 2 3 → arr[0]=1, arr[1]=2, arr[2]=3
-a → store input into an array

Specifying Input Field Separator
IFS="," read var1 var2
# Input: apple,banana → var1=apple, var2=banana
IFS → Internal Field Separator used to split input

Reading a Single Character
read -n 1 -p "Press any key: " key
echo
echo "You pressed $key"
-n N → read N characters without waiting for Enter

Reading Without Backslash Interpretation
read -r line
-r → prevents \ from being treated as escape characters

Checking Exit Status
if read -t 3 -p "Enter value: " val; then
  echo "You entered $val"
else
  echo "No input or timeout!"
fi
Returns 0 on success, non-zero on EOF or timeout

Reading From a File
while IFS= read -r line; do
  echo "Line: $line"
done < file.txt
IFS= prevents trimming of spaces
-r prevents backslash escapes

Common Options
Option Description
-p Prompt message
-s Silent mode (password input)
-t Timeout in seconds
-n Read N characters
-r Raw input, no backslash interpretation
-a Read input into an array
-d Use a custom delimiter (default newline)
Examples
Basic input:
read name
echo "Hello, $name"
Prompt with silent input:
read -s -p "Password: " pwd
echo
echo "Password received"
Timeout input:
read -t 5 -p "Enter your age: " age
Multiple variables:
read fname lname
echo "First: $fname, Last: $lname"
Reading into an array:
read -a colors
echo "Colors: ${colors[0]}, ${colors[1]}"
Reading file line by line:
while IFS= read -r line; do
  echo "$line"
done < file.txt
Single character input:
read -n 1 -p "Press Y to continue: " ans
echo

CAT Command – Technical Reference:
cat (concatenate) is a versatile command used to display file contents, combine files, create new files, and number lines. It works with stdin, stdout, and file redirection, making it essential for file inspection and quick text manipulation in shell scripts.

Basic Syntax
cat [options] [file1 file2 ...]
Reads files sequentially; if no files are provided, reads from stdin.
Multiple files are concatenated in the order specified.

Displaying File Contents
cat file.txt
Displays the entire contents of a file to the terminal.

Concatenating Multiple Files
cat file1.txt file2.txt
Outputs the contents of file1.txt followed by file2.txt.
Combine into a new file:
cat file1.txt file2.txt > combined.txt
Append to an existing file:
cat file1.txt >> combined.txt

Creating a New File
cat > newfile.txt
# Type content, press Ctrl+D to save
Ctrl+D signals EOF to end input.

Numbering Lines
Number all lines:
cat -n file.txt
Number only non-blank lines:
cat -b file.txt

Showing Non-Printing Characters
Option Description
-T Show tabs as ^I
-E Show $ at end of lines
-v Show non-printable characters
-A Show all non-printing characters (-vET)
Example:
cat -vET file.txt
Reading from Standard Input
cat
# Type text, press Ctrl+D to exit
echo "Hello World" | cat
# Output: Hello World

Appending to Files
cat >> file.txt
# Adds input to the end of file (Ctrl+D to finish)

Advanced Examples
Combine multiple files and number lines:
cat -n file1.txt file2.txt > combined.txt
View file with line endings:
cat -E file.txt
Pipe through other commands:
cat file.txt | grep "pattern"
cat file.txt | sort | uniq

Tips and Best Practices
Use cat for small files; for huge files, prefer less or more.
Avoid unnecessary cat with pipes (e.g., grep pattern file.txt instead of cat file.txt | grep pattern).
Combine options for advanced display: cat -bET file.txt.

Common Options
Option Description
-n Number all lines
-b Number only non-blank lines
-E Show $ at end of lines
-T Show tabs as ^I
-v Show non-printing characters
-A Show all non-printing characters (-vET)
> / >> Redirect output to file

FIND Command – Technical Reference:
find is a powerful command to search for files and directories in a directory hierarchy based on name, type, size, permissions, time, and other criteria. It is widely used in automation, scripting, and system administration.

Basic Syntax
find [path...] [options] [expression]
path → directories to search (default: .)
expression → conditions like name, type, size, permissions
options → control search behavior, e.g., -maxdepth, -mindepth

Searching by Name
find /path/to/dir -name "file.txt"      # case-sensitive
find /path/to/dir -iname "file.txt"     # case-insensitive
find /path/to/dir -name "*.txt"         # wildcard support

Searching by Type
-type f   # regular files
-type d   # directories
-type l   # symbolic links

Searching by Size
-size 10M      # exactly 10 MB
-size +10M     # larger than 10 MB
-size -10M     # smaller than 10 MB

Suffixes: c=bytes, k=KB, M=MB, G=GB

Searching by Time
-mtime 0      # modified today
-mtime +7     # modified more than 7 days ago
-mtime -7     # modified within last 7 days
-atime        # last accessed
-ctime        # last metadata change

Searching by Permissions
-perm 644         # exact mode
-perm -u=x        # executable by user
-perm /mode       # any bit matches

Combining Conditions
AND (default):
find /dir -type f -name "*.txt" -size +1M
OR:
find /dir -type f -name "*.txt" -o -name "*.md"
NOT:
find /dir ! -name "*.txt"
Grouping:
find /dir \( -name "*.txt" -o -name "*.md" \) -size +1M
Executing Commands on Found Files
find /dir -name "*.log" -exec rm {} \;
find /dir -type f -exec chmod 644 {} \;
find /dir -type f -exec ls -l {} \;
{} → placeholder for found file
\; → ends command
+ → pass multiple files at once for efficiency

Limiting Depth
-maxdepth 1  # only current directory
-mindepth 2  # start search from depth 2

Finding Empty Files or Directories
-type f -empty  # empty files
-type d -empty  # empty directories

Deleting Files
find /dir -type f -name "*.tmp" -delete
Use with caution; -delete removes files immediately

Recently Accessed or Modified
-type f -mtime -1  # modified within 1 day
-type f -atime -7  # accessed within 7 days

Using find with xargs
find /dir -name "*.log" | xargs rm
find /dir -name "*.log" -print0 | xargs -0 rm  # handles spaces safely

Common Options Quick Reference
Option Description
-name Search by file name (case-sensitive)
-iname Search by file name (case-insensitive)
-type f=file, d=directory, l=symlink
-size + greater, - less, c/k/M/G
-mtime Modification time (days)
-atime Access time
-ctime Metadata change time
-perm Match file permissions
-exec Execute command on found files
-delete Delete found files
-maxdepth N Limit search depth
-mindepth N Start search from depth N
! NOT operator
-o OR operator
\( \) Group expressions

GREP Command – Technical Reference:
grep (Global Regular Expression Print) is a powerful text search tool that searches for patterns, words, or regular expressions in files or input streams. It is widely used for filtering, pattern matching, and scripting pipelines.

Basic Syntax
grep [options] pattern [file...]
pattern → text or regex to search for
file → file(s) to search (if omitted, reads from stdin)
Examples:
grep "hello" file.txt
echo "hello world" | grep "hello"

Case-Insensitive Search
grep -i "hello" file.txt
-i → ignore case (Hello, HELLO, hello)

Searching for Whole Words
grep -w "word" file.txt
-w → match only whole words, not substrings

Displaying Line Numbers
grep -n "pattern" file.txt
-n → show line number of matching lines

Counting Matches
grep -c "pattern" file.txt
-c → only shows the number of matching lines

Inverting Match
grep -v "pattern" file.txt
-v → prints lines that do NOT match the pattern

Recursive Search
grep -r "pattern" /path/to/dir
grep -r --include="*.txt" "pattern" /dir
-r / -R → search recursively
--include → limit search to specific files

Using Regular Expressions
Type Example Option
Basic regex grep "he..o" file.txt -
Extended regex `grep -E "cat dog" file.txt`
Fixed string grep -F "a+b*c" file.txt -F

Showing Context Around Matches
grep -C 2 "pattern" file.txt  # 2 lines before & after
grep -B 3 "pattern" file.txt  # 3 lines before
grep -A 4 "pattern" file.txt  # 4 lines after
-C → context, -B → before, -A → after

Display Only Matching Part
grep -o "pattern" file.txt

-o → prints only matched text, not whole line

Quiet Mode
grep -q "pattern" file.txt
if [ $? -eq 0 ]; then echo "Found!"; else echo "Not found!"; fi

-q → suppress output; only sets exit code

Multiple Patterns
grep -e "pattern1" -e "pattern2" file.txt
grep -f patterns.txt file.txt
-e → specify multiple patterns
-f → read patterns from a file

Highlighting Matches
grep --color=auto "pattern" file.txt
Highlights the matching text in output

Combining with Other Commands
ps aux | grep "ssh"
ls -l | grep "^d"
Filters command output in pipelines, essential in shell scripting

Common Options Quick Reference
Option Description
-i Case-insensitive
-w Match whole word
-n Show line numbers
-c Count matching lines
-v Invert match
-r / -R Recursive search
-E Extended regex
-F Fixed string (no regex)
-o Print only matching part
-C N Show N lines before & after
-B N Show N lines before
-A N Show N lines after
-q Quiet mode, exit code only
-e Specify multiple patterns
-f Read patterns from a file
--color=auto Highlight matching text

EGREP Command – Technical Reference:
egrep (Extended Global Regular Expression Print) is a variant of grep that supports extended regular expressions (ERE) without needing to escape special characters like +, ?, |, or ().

Note: Modern Linux systems treat egrep as equivalent to grep -E.

Basic Syntax
egrep [options] pattern [file...]
pattern → extended regular expression (ERE)
file → file(s) to search (optional; reads stdin if omitted)
Examples:
egrep "cat|dog" file.txt
echo "hello world" | egrep "hello|world"

Extended Regular Expressions (ERE)
Character Meaning Example
+ One or more occurrences egrep "go+gle" → matches google, gooogle
? Zero or one occurrence egrep "colou?r" → matches color or colour
() Grouping `egrep "(cat
{n,m} At least n, at most m occurrences egrep "a{2,4}" file.txt
Unlike grep, special characters do not need escaping.

Case-Insensitive Search
egrep -i "pattern" file.txt
-i → ignore case

Display Line Numbers
egrep -n "pattern" file.txt
-n → shows line numbers of matches

Invert Match
egrep -v "pattern" file.txt
-v → shows lines not matching the pattern

Counting Matches
egrep -c "pattern" file.txt
-c → counts number of matching lines

Recursive Search
egrep -r "pattern" /path/to/dir
-r / -R → search recursively in directories

Show Context Around Matches
egrep -C 2 "pattern" file.txt  # 2 lines before & after
egrep -B 3 "pattern" file.txt  # 3 lines before
egrep -A 4 "pattern" file.txt  # 4 lines after

Multiple Patterns
egrep -e "pattern1" -e "pattern2" file.txt
egrep -f patterns.txt file.txt
-e → multiple patterns
-f → patterns read from file

Highlight Matches
egrep --color=auto "pattern" file.txt
Highlights matching text in terminal

Only Matching Part of Line
egrep -o "pattern" file.txt
-o → prints only matched text, not the entire line

Quiet Mode
egrep -q "pattern" file.txt
if egrep -q "error" logfile.txt; then echo "Error found!"; fi
-q → suppresses output; only sets exit status
Useful in shell scripts and conditional statements

Combining Options
egrep -i -n -C 2 "pattern" file.txt
Example: case-insensitive, show line numbers, and 2 lines of context

Comparing grep, grep -E, and egrep
Command Supports ERE? Escaping Needed?
grep No Yes, for +, ?, ()
grep -E Yes No
egrep Yes No
Modern Linux: egrep is deprecated, replaced by grep -E. Behavior is identical.

Common Options Quick Reference
Option Description
-i Case-insensitive
-v Invert match (lines not matching)
-c Count matching lines
-n Show line numbers
-r / -R Recursive search
-o Print only matching text
-C N Show N lines of context
-B N Show N lines before match
-A N Show N lines after match
-e Multiple patterns
-f Patterns from a file
--color=auto Highlight matches
-q Quiet mode (exit code only)

ECHO Command – Technical Reference:
The echo command is used to display text, variables, or command output to standard output (screen). It is widely used in shell scripting for printing messages, debugging, formatting output, or logging.

Basic Syntax
echo [options] [string...]
string → text, variables, or command substitution
By default, echo adds a newline at the end
Examples:
echo "Hello World"
name="Alice"
echo "Hello, $name"
echo $((5 + 3))  # Output: 8

Escape Sequences
Use -e to interpret escape sequences:
Escape Meaning
\n Newline
\t Horizontal tab
\\ Backslash
\" Double quote
\a Alert (bell)
\b Backspace
\r Carriage return
Example:
echo -e "Line1\nLine2\nLine3"
echo -e "Column1\tColumn2\tColumn3"

Suppressing Newline
echo -n "Hello "
echo "World"
-n → prevents newline at the end
Output: Hello World on the same line

Displaying Variables
user="Bob"
echo "Current user is $user"
echo "Home directory: $HOME"
$HOME → built-in environment variable

Command Substitution
echo "Today is $(date)"
echo "Files: $(ls)"
Executes commands in $(...) and prints the output

Using Quotes
Double quotes → preserve spaces, allow variable expansion:
name="Alice"
echo "Hello $name"
Single quotes → preserve literal text, no expansion:
echo 'Hello $name'  # Prints: Hello $name

Redirection with echo
echo "Hello World" > file.txt   # overwrite file
echo "Another line" >> file.txt # append to file
> → overwrite
>> → append

Printing Special Characters
echo -e "Tab\tNewline\nBackslash\\Quote\""
Handles tabs, newlines, backslashes, quotes, etc.

Using echo in Scripts
#!/bin/bash
echo "Script started"
name="Alice"
echo "Hello, $name"
echo "Listing files:"
echo "$(ls)"
Prints messages, variables, and command output

Common Options Quick Reference
Option Description
-e Enable interpretation of backslash escapes
-n Do not print trailing newline
-E Disable interpretation of backslash escapes (default)

Practical Examples
Multi-line message:
echo -e "Welcome\n================\nUser: $USER\nDate: $(date)"
Table with tabs:
echo -e "Name\tAge\tCity\nAlice\t25\tNY\nBob\t30\tLA"
Debugging in scripts:
echo "Variable x = $x"
Appending to a log file:
echo "$(date): Script executed successfully" >> script.log

PRINT / printf Command – Technical Reference:
Shell scripting does not have a universal print command like Python. Most printing is done using echo or printf. Some shells (e.g., ksh, zsh) also support a print command.

Printing options in shell scripting:
echo – simple printing, widely used
printf – formatted, portable, similar to C's printf
print – shell-specific (ksh, zsh)

Using echo
echo "Hello World"
echo -e "Hello\tWorld\nNew Line"
-e → interpret escape sequences
-n → suppress newline
Example:
echo -e "Name:\tAlice\nAge:\t25"

Using printf (Advanced / Portable)
printf is more powerful than echo and preferred for formatted output.
Syntax
printf "format" [arguments...]
format → string with placeholders (%s, %d, %f, etc.)
arguments → values replacing placeholders
Examples
printf "Name: %s\nAge: %d\n" "Alice" 25
# Output:
# Name: Alice
# Age: 25
printf "Pi: %.2f\n" 3.14159
# Output: Pi: 3.14
printf "%-10s %5d\n" "Alice" 25
# Output: Alice       25
# %-10s → left-align text, width 10
# %5d   → right-align integer, width 5

Using print (ksh / zsh)
Not POSIX standard; only in ksh or zsh
Can replace echo in these shells
Syntax
print [options] string
Examples
print "Hello World"
print -n "Hello"    # suppress newline
print " World"
print -r "Text with \n no escape"
-n → suppress newline
-r → raw, do not interpret backslash escapes

Comparison Table
Command Supports Escape Sequences Formatted Output Notes
echo Yes, with -e No Simple, most used
printf Yes, always Yes Preferred for complex formatting
print Optional (-r) Limited Shell-specific (ksh/zsh)

Examples in Shell Scripting
#!/bin/bash
# Using echo
name="Alice"
echo "Hello, $name"
# Using printf
age=25
printf "Name: %s, Age: %d\n" "$name" "$age"
# Using print (ksh/zsh)
print "Hello $name"

DIFF Command – Technical Reference:
The diff command compares two files line by line and shows their differences. It is widely used for version control, debugging, and script automation.

Basic Syntax
diff [options] file1 file2
file1, file2 → files to compare
Default output shows lines that differ using < and > symbols
Example:
# file1.txt
apple
banana
cherry
# file2.txt
apple
blueberry
cherry
diff file1.txt file2.txt
# Output:
2c2
< banana
---
> blueberry
2c2 → line 2 changed
< → line from file1
> → line from file2

Common Symbols in Output
Symbol Meaning
a added line
d deleted line
c changed line
< line from first file
> line from second file

Ignore Differences
Case-insensitive:
diff -i file1.txt file2.txt
Ignore whitespace:
diff -w file1.txt file2.txt  # all spaces
diff -b file1.txt file2.txt  # changes in number of spaces
diff -B file1.txt file2.txt  # blank lines

Side-by-Side Comparison
diff -y file1.txt file2.txt
diff -y --suppress-common-lines file1.txt file2.txt  # hide unchanged lines
-y → side-by-side display

Unified and Context Formats
Unified format:
diff -u file1.txt file2.txt
Shows context lines around changes; used in git patches
@@ -1,3 +1,3 @@
 apple
-banana
+blueberry
 cherry
Context format:
diff -c file1.txt file2.txt
Older style, shows surrounding lines

Recursive Comparison
diff -r dir1 dir2
Compares directories recursively
Shows differences for all files inside directories

Suppressing Common Lines
diff --suppress-common-lines file1.txt file2.txt
Only shows differing lines, reduces clutter

Exit Status Codes
Can be used in scripts to detect differences:
if diff file1.txt file2.txt > /dev/null; then
  echo "Files are identical"
else
  echo "Files differ"
fi

Exit codes:
Code Meaning
0 files are identical
1 files differ
>1 error

Comparing Binary Files
diff --binary file1.bin file2.bin
Compares binary files; often cmp is better for strict binary comparison

Common Options Quick Reference
Option Description
-i Ignore case
-w Ignore all whitespace
-b Ignore changes in number of spaces
-B Ignore blank lines
-y Side-by-side comparison
--suppress-common-lines Hide unchanged lines
-u Unified diff format
-c Context diff format
-r Recursive directory comparison
--brief Only report if files differ

Practical Examples
Check if two files are different:
if diff -q file1.txt file2.txt > /dev/null; then
  echo "Files are identical"
else
  echo "Files differ"
fi
Create a patch file:
diff -u old_version.txt new_version.txt > patch.diff
Compare directories recursively:
diff -r dir1/ dir2/

EXIT Command – Technical Reference:
The exit command is used in shell scripting to terminate a script or function and return a numeric exit status to the shell or calling process. Exit codes help indicate success or failure.

Basic Syntax
exit [n]
n → optional exit status (integer 0–255)
0 → success
Non-zero → failure
If omitted, exit returns the status of the last executed command

Examples
Exit with Success
#!/bin/bash
echo "Script started"
exit 0
echo "This will not run"
Output:
Script started
Script stops immediately after exit 0
Exit with Failure
#!/bin/bash
echo "Something went wrong!"
exit 1
Check exit status afterward:
echo $?
# Output: 1
Check Exit Status of Previous Command
ls /tmp > /dev/null
echo "Exit status of ls: $?"
Exit Codes:
Code Meaning
0 Success
1–255 Error / failure
Exit After an Error
Automatically terminate the script when a command fails:
#!/bin/bash
set -e
cp file1.txt /nonexistent/path
echo "This line will not execute"
set -e → exits immediately if a command fails
Exit from a Function
return → exit function only
exit → stops the entire script

#!/bin/bash
my_function() {
  echo "Inside function"
  exit 2  # stops script
  echo "This won't run"
}
my_function
echo "This won't run either"
Custom Exit Codes
SUCCESS=0
ERR_NO_FILE=10
ERR_INVALID=20

if [ ! -f "$1" ]; then
  echo "File not found!"
  exit $ERR_NO_FILE
fi

Best Practices
Use exit 0 for success
Use exit 1–255 for specific errors
Document exit codes clearly in scripts
Example Header:
Exit codes:
0 – Success
1 – Invalid input
2 – File not found