SEA Failover on Dual VIOS with VLAN Tagged Ethernet Adapters

To configure Shared Ethernet Adapter (SEA) failover on a dual Virtual I/O Server (VIOS) setup while using IEEE 802.1Q VLAN tagging. Follow these steps to provide HA network connectivity for AIX client LPARs using VLAN-tagged networks.

Prerequisites:

• Two VIOS instances (VIOS1 and VIOS2) managed in an HMC-managed system.
• Physical NIC(s) available on each VIOS and the physical switch configured to support required VLANs.
• HMC access (for DLPAR adapter creation) and root access to each VIOS.
• Ensure PowerVM hypervisor supports VLAN tagging (IEEE 802.1Q).
• Decide on VLAN IDs:
• PVID (trunk PVID) for SEA trunk adapters (example: 1).
• Control channel VLAN for SEA heartbeat (example: 100).
• External (tagged) VLAN(s) to carry client traffic (example: 1000).
• If using EtherChannel, configure switch ports for EtherChannel before creating EtherChannel on VIOS.

SEA (Shared Ethernet Adapter): Layer‑2 bridge on VIOS that connects virtual and physical networks.

ha_mode=auto (Failover): Configures two SEAs (one active, one standby) with a control channel for heartbeat.

Trunk (Access external network): Virtual adapters used by the SEA to bridge to external network; must share the same PVID on both VIOSes but use different priorities (lower value = higher priority).

Control channel: Dedicated virtual adapter on a unique VLAN (not exposed to external network) used for SEA heartbeats; must exist on both VIOSes and be specified in the SEA configuration.

High-level:

• Create trunk virtual adapter on VIOS1 (PVID=1) and mark "access external network".
• Create control-channel virtual adapter on VIOS1 (PVID=100), do not mark external access.
• Create SEA on VIOS1 with ha_mode=auto and set ctl_chan to the control adapter.
• Create VLAN subinterface on SEA for tagged VLAN (tag 1000) and assign IP to it.
• Repeat steps 1–4 on VIOS2 using the same PVID values but set a higher trunk priority (so VIOS1 is primary).
• On client LPAR(s), create virtual adapter(s) with the same PVID as SEAs and add VLAN subinterface(s) with the same tag(s).
• Verify failover and connectivity.

Detailed Steps — VIOS1 (Primary):

1. Create trunk virtual Ethernet adapter (via HMC DLPAR):
Tasks → Dynamic Logical Partitioning → Virtual Adapters → Actions → Create → Ethernet Adapter.
Adapter ID: choose slot (e.g., ent2).
VLAN ID (PVID): 1 (example).
Select IEEE 802.1Q compatible adapter and enter VLAN tags you will use (e.g., 1000).
Check Access external network.
Set trunk priority: 1 (lower = higher priority).
Save.

2. Create control-channel virtual adapter (via HMC DLPAR):
Adapter ID: another slot (e.g., ent3).
VLAN ID (PVID): 100 (example control VLAN).
Do NOT check Access external network.
Save and if using DLPAR you may need to cfgdev or reboot to make adapters available.

3. Create SEA on VIOS1 (ha_mode auto):
On VIOS1 shell run:
# mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1 -attr ha_mode=auto ctl_chan=ent3
ent0 is the physical adapter or the target SEA device name; adjust according to your system.
-vadapter ent2 uses the trunk virtual adapter created earlier.
-default ent2 -defaultid 1 sets the default trunk PVID for SEA traffic.
-attr ha_mode=auto ctl_chan=ent3 enables failover and points to control channel adapter.

4. Create VLAN sub interface on the SEA for external tagged VLAN (e.g., 1000):
On VIOS1 shell run:
# mkdev -vlan ent4 -tagid 1000
Note: The device name ent4 is an example — the actual name displayed may differ (the command returns the available device).

5. Assign IP and start TCP/IP on the SEA VLAN interface:
Example using mktcpip:
# mktcpip -hostname vio1 -interface en5 -inetaddr 192.168.10.10 -netmask 255.255.255.0 -gateway 9.3.5.41 -nsrvaddr 9.3.4.2 -nsrvdomain itsc.austin.ibm.com -start
 Replace interface (en5) and addresses with your network specifics.

Detailed Steps — VIOS2 (Backup)

1. Create trunk virtual adapter (match PVID):
Create a virtual adapter (e.g., ent2) with VLAN ID 1 and Access external network checked.
Set trunk priority: 2 (higher value than primary so it's secondary).
Enable IEEE 802.1Q and include VLAN tags (e.g., 1000).

2. Create control-channel virtual adapter (match control VLAN):
Create adapter (e.g., ent3) with VLAN ID 100 and do NOT check Access external network.
Activate adapter (cfgdev or reboot if needed).

3. Create SEA on VIOS2 with failover attribute(Optional) :

If you want SEA objects present on both VIOSes, run the same mkvdev command on VIOS2:
# mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1 -attr ha_mode=auto ctl_chan=ent3
This makes the SEA available in standby mode.

4. Create VLAN sub interface on VIOS2 SEA for tagged VLAN (e.g., 1000):
Run:
# mkdev -vlan ent4 -tagid 1000

6. Assign IP to the SEA VLAN adapter on VIOS2:
Example:
# mktcpip -hostname vio2 -interface en5 -inetaddr 9.3.5.137 -netmask 255.255.255.0 -gateway 9.3.5.41 -nsrvaddr 9.3.4.2 -nsrvdomain itsc.austin.ibm.com -start

Client LPAR Steps (AIX)

1. Create virtual adapter on client LPAR (via HMC DLPAR):
Tasks → Dynamic Logical Partitioning → Virtual Adapters → Create → Ethernet Adapter.
Adapter ID: choose slot (e.g., ent0).
VLAN ID (PVID): 1 (must match SEA trunk PVID).
Select IEEE 802.1Q compatible adapter and include VLAN tags (e.g.,1000).
Do NOT check Access external network on client adapters.
Set trunk priority: 1.
Save.

2. Create VLAN subinterface on client LPAR (AIX):
On AIX, run smitty vlan → Add a VLAN → Select ent0 → Specify VLAN ID 1000.
The VLAN device (e.g., ent1) becomes available.

3. Add TCP/IP configuration for VLAN interface on client:
# smitty mktcpip → Select the VLAN device (e.g., en1) → Enter hostname, IP address, netmask, gateway, nameserver, start TCP/IP daemons.

Verification & Testing:

1. Check SEA status on VIOS:
# lsdev -dev entX -attr
# entstat -d entX
Confirm which SEA is active and which is standby.

2. Validate VLAN tagging:
From client LPAR, ping gateway and external hosts on VLAN 1000.

3. Force failover test:
Shut down the active VIOS SEA interface or simulate failure and ensure the standby SEA becomes active and client LPARs retain network access.

Create SEA on VIOS (example)

# mkvdev -sea ent0 -vadapter ent2 -default ent2 -defaultid 1 -attr ha_mode=auto ctl_chan=ent3
Create VLAN subinterface for tagged VLAN
# mkdev -vlan ent4 -tagid 1000
Assign TCP/IP to SEA VLAN interface
# mktcpip -hostname vio1 -interface en5 -inetaddr 192.168.10.10 -netmask 255.255.255.0 -gateway 9.3.5.41 -nsrvaddr 9.3.4.2 -nsrvdomain itsc.austin.ibm.com -start

Deploying VIOS and Configuring IBM Power Systems

Setting up a Virtual I/O Server (VIOS) environment correctly is essential for achieving performance, scalability, and high availability on IBM Power Systems. This step-by-step guide walks you through the complete process — from installing VIOS (single or dual) to configuring networking, SEA failover, and client LPAR connectivity using HMC or IVM.

• Install and register VIOS partitions for virtualized network and storage access.
• Configure Shared Ethernet Adapters (SEA) with VLAN tagging for resilient networking.
• Map virtual storage to clients via vSCSI or NPIV.
• Create and configure AIX or Linux client LPARs, attach virtual adapters, and test network failover.

This practical workflow aligns with IBM Redbooks best practices and helps administrators build a robust PowerVM virtualization layer that supports flexible, high-availability enterprise workloads.

1. Obtain VIOS Installation ISO

Download the VIOS DVD ISO from IBM "https://www.ibm.com/servers/eserver/ess/landing/landing-page" for the desired version.

2. Upload ISO to HMC

On the HMC, navigate to:

HMC Management → Templates and OS Images → Add/Upload OS Image

Upload the downloaded ISO. The HMC will store it for VIO installations.

3. Activate VIO LPAR and Start Installation

Power on the Virtual I/O Server LPAR from HMC.

When prompted, choose "VIO Install".

Installation may take over an hour.

The progress bar may hang near the end for ~20 minutes — this is normal.

4. First Login and Initial Setup

# Login as padmin

# Set password for padmin

# Accept license

$ oem_setup_env

$ license -accept

5. Configure Network and Hostname

Configure primary interface

Example: en0

# chdev -l en0 -a netaddr=172.16.10.100 -a netmask=255.255.255.0 -a state=up

Set default route

# chdev -l inet0 -a route=0,172.16.10.1

Set hostname interactively

# smitty hostname

Update /etc/hosts and /etc/netsvc.conf if needed.

Update /etc/resolv.conf with domain, search list, and nameservers.

6. Creating new paging space (if you want a dedicated LV):

# mkps -s <size_in_MB> -n <paging_lv_name> <volume_group>

Example:

# mkps -s 6144 -n paging01 rootvg   # 6 GB on rootvg

Activate Paging Space

# swapon -a

Activates all paging spaces listed in /etc/swap.

Verify Paging Space

# lsps -a

7. Configure Dump Devices

Create lv_dump01 on primary disk

# mklv -t sysdump -y lv_dump01 rootvg 8 hdisk0

# sysdumpdev -Pp /dev/lv_dump01

Create lv_dump02 on second disk

# mklv -t sysdump -y lv_dump02 rootvg 8 hdisk1

# sysdumpdev -Ps /dev/lv_dump02

8. NTP and Time Zone

Create missing NTP files

# touch /home/padmin/config/ntp.conf /home/padmin/config/ntp.drift /home/padmin/config/ntp.log /home/padmin/config/ntp.trace

Sample ntp.conf

# cat > /home/padmin/config/ntp.conf <<EOF

server ntp.mydomain.com

driftfile /home/padmin/config/ntp.drift

tracefile /home/padmin/config/ntp.trace

logfile /home/padmin/config/ntp.log

EOF

Enable NTP at startup

# vi /etc/rc.tcpip

Uncomment/start line:

# start /usr/sbin/xntpd -a '-c /home/padmin/config/ntp.conf' "$src_running"

Set timezone

# vi /etc/environment

TZ=Europe/Vienna

9. Mirror Root Volume

# extendvg rootvg hdisk1

# mirrorios -defer hdisk1

# bosboot -ad hdisk0

# bosboot -ad hdisk1

10. Configure syslog

# vi /etc/syslog.conf

Example:

*.debug  /var/log/messages  rotate size 1m files 10

auth.debug /var/log/auth.log  rotate size 1m files 10

Create log files

# touch /var/log/messages /var/log/auth.log

# refresh -s syslogd

11. SSH Configuration

• Add needed public key to authorized_keys

12. Adding Adapters

• Add network cards or FC adapters as needed.
• Verify device names.

13. Apply Recommended VIOS Rules

View differences

# rules -o diff -s -d

Deploy recommended settings

# rules -o deploy -d

14. Network Configuration (Optional: Dual VIOS / SEA / LACP)

HMC GUI

• Configure link aggregation and SEA.
• Create virtual networks and VLANs via GUI.

Command Line:

Create virtual Ethernet adapters

# chhwres -r virtualio -m p950 -p vio01 -o a -s 100 --rsubtype eth -a "ieee_virtual_eth=1,port_vlan_id=4000,\"addl_vlan_ids=3030,1871\",is_trunk=1,trunk_priority=1"

# chhwres -r virtualio -m p950 -p vio02 -o a -s 100 --rsubtype eth -a "ieee_virtual_eth=1,port_vlan_id=4000,\"addl_vlan_ids=3030,1871\",is_trunk=1,trunk_priority=2"

Save profile

# mksyscfg -r prof -m p950 -p vio01 -o save -n default --force

# mksyscfg -r prof -m p950 -p vio02 -o save -n default --force

Create LACP Etherchannel

# mkvdev -lnagg ent4 ent5 -attr mode=8023ad hash_mode=src_dst_port

Create SEA

# mkvdev -sea ent18 -vadapter ent12 ent14 ent16 ent17 -default ent12 -defaultid 4000

chdev -dev ent20 -attr ha_mode=sharing

15. Performance Tuning

Increase SEA buffer sizes:

# chdev -l ent9 -a max_buf_huge=128 -a min_buf_huge=64 -a max_buf_large=128 -a min_buf_large=64 -a max_buf_medium=512 -a min_buf_medium=256 -a max_buf_small=4096 -a min_buf_small=2048 -a max_buf_tiny=4096 -a min_buf_tiny=2048 -P

Enable large_receive:

# chdev -l ent10 -a large_receive=yes

Adjust FC queue_depth if needed:

# chdev -l hdiskX -a queue_depth=32 -P

Pre & Post Reboot Validation Automation for AIX Systems

This script connects to a remote host, validates reachability (ping + SSH), and runs a sequence of system checks to capture critical configuration and status data into a single snapshot file.

Features:

• Works across AIX, VIOS, Linux, Solaris
• Safe remote execution using SSH heredoc (no temp files)

Captures:

• Host identity
• Screen saver / timeout config
• NTP / TZ info
• Default route and network interfaces
• HACMP resource group
• Paging space, PowerPath, MPIO
• Cluster services, IPSec status
• NFS/GPFS mounts
• errpt, lppchk, exportfs, showmount, etc.

Saves results locally under /tmp/pre_reboot_snapshots/<hostname>_pre_snap.txt

verify_post_reboot.ksh → compares post-reboot state against the baseline and highlights differences

Purpose:

After reboot, this script collects a new snapshot and compares it with the pre-reboot one.

It highlights configuration changes, missing devices, or service failures.

Features:

• Automatically detects pre-snapshot file
• Collects post-reboot snapshot using same logic as pre-reboot
• Performs line-by-line diff
• Provides summary of differences
• Logs results to /tmp/post_reboot_verification

How It Works:

• Validates SSH connection to target.
• Executes same snapshot logic (reusing collect_pre_reboot_snapshot.ksh function set).
• Stores results in /tmp/post_reboot_verification/<hostname>_post_snap.txt.
• Runs diff between pre and post files.
• Marks [ OK ] if no differences, [ WARN ] if differences exist.

Sample Folder Layout

/usr/local/syscheck/

├── collect_pre_reboot_snapshot.ksh

├── verify_post_reboot.ksh

├── /tmp/pre_reboot_snapshots/

│   └── aixlpar01_pre_snap.txt

└── /tmp/post_reboot_verification/

    ├── aixlpar01_post_snap.txt

    └── aixlpar01_diff_report.txt

Both scripts use ksh for maximum portability across enterprise Unix platforms.

Script 1 — collect_pre_reboot_snapshot.ksh

==========================================================================

#!/usr/bin/ksh

###############################################################################

# collect_pre_reboot_snapshot.ksh <hostname>

# Author: adminCtrlX

#

# Purpose:

#   Collects a comprehensive pre-reboot system snapshot from an AIX or VIOS host.

#   Runs remote commands over SSH and stores the output locally in a timestamped file.

#

# Features:

#   - Network and SSH connectivity tests

#   - Time sync, route, HACMP, paging, and storage validation

#   - Collects system identity and configuration details before reboot

#

# Compatible with: AIX / VIOS servers

###############################################################################

EXIT_OK=0

EXIT_ERR=1

EXIT_WARN=2

SSH_USER="root"

if [[ $# -ne 1 ]]; then

  echo "Usage: $0 <hostname>"

  exit $EXIT_ERR

fi

HOST="$1"

DATESTAMP=$(date +%Y%m%d_%H%M%S)

SNAPDIR="/tmp/pre_reboot_snapshots"

OUTFILE="${SNAPDIR}/${HOST}_pre_snap_${DATESTAMP}.txt"

mkdir -p "$SNAPDIR" || {

  echo "Failed to create snapshot directory: $SNAPDIR"

  exit $EXIT_ERR

}

print_line() { printf '%s\n' "-------------------------------------------------------------"; }

print_status() {

  case "$1" in

    ok)   printf '[ OK ]\n' ;;

    warn) printf '[ WARN ]\n' ;;

    fail) printf '[ FAIL ]\n' ;;

  esac

}

section_header() {

  print_line

  printf '%s\n' "$1"

  print_line

}

# --- Function: Ping detection (portable) ---

ping_host() {

  if ping -c1 -W1 "$HOST" >/dev/null 2>&1; then

    return 0

  elif ping -c1 "$HOST" >/dev/null 2>&1; then

    return 0

  elif ping -n 1 "$HOST" >/dev/null 2>&1; then

    return 0

  else

    return 1

  fi

}

###############################################################################

# Connectivity Checks

###############################################################################

section_header "Testing ping to $HOST"

if ping_host; then

  print_status ok

else

  print_status fail

  echo "Host $HOST is not responding to ping."

  exit $EXIT_ERR

fi

section_header "Testing SSH to $HOST"

if ssh -o BatchMode=yes -o ConnectTimeout=10 -q "$SSH_USER@$HOST" "echo ok" >/dev/null 2>&1; then

  print_status ok

else

  print_status fail

  echo "Unable to SSH to $SSH_USER@$HOST"

  exit $EXIT_ERR

fi

###############################################################################

# Collect Snapshot

###############################################################################

section_header "Collecting pre-reboot snapshot from $HOST"

ssh -o BatchMode=yes -o ConnectTimeout=30 "$SSH_USER@$HOST" 'ksh -s' <<'REMOTE' > "$OUTFILE" 2>&1

printf '--- Host identity ---\n'

uname -a || echo "Unable to get uname output"

printf '\n--- dtsession saverTimeout/lockTimeout ---\n'

if [ -d /etc/dt/config ]; then

  grep -H -E 'saverTimeout|lockTimeout' /etc/dt/config/* 2>/dev/null || echo "No saver/lockTimeout settings found"

else

  echo "/etc/dt/config not present"

fi

printf '\n--- TZ / NTP check ---\n'

if command -v lssrc >/dev/null 2>&1; then

  lssrc -s xntpd 2>/dev/null

fi

if command -v ntpq >/dev/null 2>&1; then

  ntpq -p 2>/dev/null

fi

printf '\n--- Default route ---\n'

if command -v netstat >/dev/null 2>&1; then

  netstat -rn 2>/dev/null | grep -E 'default|^0.0.0.0' || echo "No default route found"

else

  ip route show 2>/dev/null | grep default || echo "No default route command available"

fi

printf '\n--- HACMP resource group ---\n'

clRGinfo 2>/dev/null || echo "clRGinfo not available"

printf '\n--- Paging space ---\n'

if command -v lsps >/dev/null 2>&1; then

  lsps -s 2>/dev/null

else

  swapon -s 2>/dev/null || free -h 2>/dev/null || echo "Paging/Swap info not available"

fi

printf '\n--- Network interfaces ---\n'

ifconfig -a 2>/dev/null || ip addr show 2>/dev/null || echo "ifconfig/ip not present"

printf '\n--- NFS/GPFS mounts ---\n'

mount | egrep 'nfs|gpfs' 2>/dev/null || echo "No NFS/GPFS mounts found"

printf '\n--- Error report (errpt) ---\n'

if command -v errpt >/dev/null 2>&1; then

  errpt -a | head -n 20 2>/dev/null || echo "No errors in errpt"

else

  echo "errpt not present"

fi

printf '\n--- Package validation (lppchk) ---\n'

if command -v lppchk >/dev/null 2>&1; then

  lppchk -v 2>/dev/null || echo "No lppchk issues found"

else

  echo "lppchk not present"

fi

printf '\n--- Disk multipathing ---\n'

lsdev -Cc disk 2>/dev/null || echo "lsdev not available or no disks listed"

printf '\n--- PowerPath status ---\n'

if [ -x /usr/sbin/powermt ]; then

  /usr/sbin/powermt display dev=all 2>/dev/null || echo "powermt returned no data"

else

  echo "PowerPath not present"

fi

printf '\n--- exportfs ---\n'

exportfs 2>/dev/null || echo "No exports or exportfs not available"

printf '\n--- showmount ---\n'

showmount -a 2>/dev/null || echo "No NFS clients or showmount not available"

printf '\n--- PowerPath reserve_policy ---\n'

if command -v lsattr >/dev/null 2>&1; then

  lsattr -El powerpath0 -a reserve_policy 2>/dev/null || echo "powerpath0 not present"

else

  echo "lsattr not available"

fi

printf '\n--- MPIO Other detection ---\n'

lsdev -Cc disk 2>/dev/null | grep 'Other' || echo "No MPIO 'Other' devices found"

printf '\n--- Cluster service status ---\n'

if command -v lssrc >/dev/null 2>&1; then

  lssrc -ls clstrmgrES 2>/dev/null || echo "Cluster service not active"

else

  echo "lssrc not available"

fi

printf '\n--- IPSec check ---\n'

if command -v lssrc >/dev/null 2>&1; then

  lssrc -a 2>/dev/null | grep -i ipsec || echo "No IPSec service found"

else

  echo "lssrc not present"

fi

REMOTE

SSH_EXIT=$?

if [ $SSH_EXIT -eq 0 ]; then

  print_status ok

  echo "Snapshot saved: $OUTFILE"

  exit $EXIT_OK

else

  print_status fail

  echo "SSH or remote command failed (exit code: $SSH_EXIT). See $OUTFILE for details."

  exit $EXIT_ERR

fi

==========================================================================

Example Console Output
Command:
# ./collect_pre_reboot_snapshot.ksh aixlpar01
Example Output:
-------------------------------------------------------------
Testing ping to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Testing SSH to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Collecting pre-reboot snapshot from aixlpar01
-------------------------------------------------------------
[ OK ]
Snapshot saved: /tmp/pre_reboot_snapshots/aixlpar01_pre_snap_20251108_143212.txt

If the host is unreachable or SSH fails:
-------------------------------------------------------------
Testing SSH to aixlpar01
-------------------------------------------------------------
[ FAIL ]
Unable to SSH to root@aixlpar01

Example Snapshot File Output

File: /tmp/pre_reboot_snapshots/aixlpar01_pre_snap_20251108_143212.txt
--- Host identity ---
AIX aixlpar01 7 7100-05-02-1810 powerpc
--- dtsession saverTimeout/lockTimeout ---
/etc/dt/config/Xconfig: saverTimeout: 600
/etc/dt/config/Xconfig: lockTimeout: 900
--- TZ / NTP check ---
Subsystem Group PID Status
xntpd tcpip 12345 active
remote refid st t when poll reach delay offset jitter
==========================================================
*time1.ntp.ibm.co 192.168.1.1 2 u 256 1024 377 0.54 0.03 0.05
--- Default route ---
default 192.168.10.1 UG 0 36 en0 1500
--- HACMP resource group ---
Resource Group Name: rg_oracle01
State: Online
Node: aixlpar01
--- Paging space ---
Size %Used Physical Volume
2048MB 12% hd6
--- Network interfaces ---
en0: flags=4e080863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,GROUPRT>
inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255
ether 0a:1b:2c:3d:4e:5f
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST>
inet 127.0.0.1 netmask 0xff000000
--- NFS/GPFS mounts ---
server01:/exports/data on /data type nfs (rw,soft,intr,proto=tcp)
--- Error report (errpt) ---
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BFE4C025 1108225315 P H hdisk1 DISK OPERATION ERROR
BFE4C025 1108224915 P H hdisk0 DISK OPERATION ERROR
--- Package validation (lppchk) ---
lppchk: The following filesets are ok:
bos.rte, bos.mp64, devices.pci.14103302.rte
--- Disk multipathing ---
hdisk0 Available 00-00-01 MPIO Other FC Disk
hdisk1 Available 00-00-02 MPIO Other FC Disk
--- PowerPath status ---
Pseudo name=hdiskpower0
Symmetrix ID=000192600218
Logical device ID=1A00
state=alive; policy=SymmOpt; priority=0; queued-IOs=0
--- exportfs ---
/exports/data
--- showmount ---
All mount points on aixlpar01:
server01:/exports/data
--- PowerPath reserve_policy ---
reserve_policy no_reserve
--- MPIO Other detection ---
None
--- Cluster service status ---
Subsystem Group PID Status
clstrmgrES cluster 43120 active
--- IPSec check ---
ipsecconf tcpip 5319 inoperative

Script 2 — verify_post_reboot.ksh

==========================================================================

#!/usr/bin/ksh

###############################################################################

# verify_post_reboot.ksh <hostname>

# Author: adminCtrlX

#

# Comprehensive system validation script for AIX & VIOS servers.

# Collects a post-reboot snapshot from the target and compares it against the

# pre-reboot snapshot saved under:

#   /tmp/pre_reboot_snapshots/<hostname>_pre_snap.txt

###############################################################################

EXIT_OK=0

EXIT_ERR=1

EXIT_WARN=2

VERBOSE=1

TMPDIR="/tmp/verify_snap.$$"

SSH_USER="root"

SNAP_BASE="/tmp/pre_reboot_snapshots"

usage() {

  echo "Usage: $0 <hostname>"

  exit $EXIT_ERR

}

# -------------------- Argument validation --------------------

if [[ $# -ne 1 ]]; then

  usage

fi

HOST="$1"

PRE_SNAP="${SNAP_BASE}/${HOST}_pre_snap.txt"

POST_SNAP="${SNAP_BASE}/${HOST}_post_snap.txt"

DIFF_OUT="${SNAP_BASE}/${HOST}_verify_diff.txt"

# -------------------- Directory setup --------------------

mkdir -p "$SNAP_BASE" || {

  echo "Failed to create $SNAP_BASE"

  exit $EXIT_ERR

}

mkdir -p "$TMPDIR" || {

  echo "Failed to create $TMPDIR"

  exit $EXIT_ERR

}

trap 'rm -rf "$TMPDIR"' EXIT INT TERM

# -------------------- Helper functions --------------------

print_line() { printf '%s\n' "-------------------------------------------------------------"; }

print_status() {

  case "$1" in

    ok)   printf '[ OK ]\n' ;;

    warn) printf '[ WARN ]\n' ;;

    fail) printf '[ FAIL ]\n' ;;

  esac

}

section_header() {

  print_line

  printf '%s\n' "$1"

  print_line

}

# Portable ping helper (tries different syntax options)

ping_host() {

  if ping -c 1 -W 1 "$HOST" >/dev/null 2>&1; then return 0

  elif ping -c 1 "$HOST" >/dev/null 2>&1; then return 0

  elif ping -n 1 "$HOST" >/dev/null 2>&1; then return 0

  else return 1

  fi

}

# -------------------- Connectivity tests --------------------

section_header "Testing ping to $HOST"

if ping_host; then

  print_status ok

else

  print_status fail

  echo "Host $HOST not responding to ping."

  exit $EXIT_ERR

fi

section_header "Testing SSH to $HOST"

# BatchMode avoids interactive password prompts; ConnectTimeout limits wait time

if ssh -o BatchMode=yes -o ConnectTimeout=10 -q "$SSH_USER@$HOST" "echo ok" >/dev/null 2>&1; then

  print_status ok

else

  print_status fail

  echo "Unable to SSH to $SSH_USER@$HOST"

  exit $EXIT_ERR

fi

# -------------------- Snapshot collection --------------------

section_header "Collecting post-reboot snapshot from $HOST"

# Run remote commands via SSH heredoc (quoted to avoid local variable expansion)

ssh -o BatchMode=yes -o ConnectTimeout=30 "$SSH_USER@$HOST" 'ksh -s' <<'REMOTE' >"$POST_SNAP" 2>&1

printf '--- HOST ---\n'

uname -a || true

printf '\n--- dtsession saverTimeout/lockTimeout ---\n'

if [ -d /etc/dt/config ]; then

  grep -H -E 'saverTimeout|lockTimeout' /etc/dt/config/* 2>/dev/null || echo 'Not found'

else

  echo '/etc/dt/config not present'

fi

printf '\n--- TZ / NTP ---\n'

if command -v lssrc >/dev/null 2>&1; then lssrc -s xntpd 2>/dev/null || true; fi

if command -v ntpq >/dev/null 2>&1; then ntpq -p 2>/dev/null || true; fi

printf '\n--- Default route ---\n'

if command -v netstat >/dev/null 2>&1; then

  netstat -rn 2>/dev/null | grep -E "default|^0.0.0.0" || true

else

  ip route show 2>/dev/null | grep default || true

fi

printf '\n--- HACMP resource group ---\n'

clRGinfo 2>/dev/null || echo 'clRGinfo not present'

printf '\n--- Paging space / Swap ---\n'

if command -v lsps >/dev/null 2>&1; then

  lsps -s 2>/dev/null || true

else

  swapon -s 2>/dev/null || free -h 2>/dev/null || echo 'Swap info unavailable'

fi

printf '\n--- Network interfaces ---\n'

ifconfig -a 2>/dev/null || ip addr show 2>/dev/null || echo 'No ifconfig/ip'

printf '\n--- NFS/GPFS mounts ---\n'

mount | egrep 'nfs|gpfs' 2>/dev/null || echo 'No NFS/GPFS mounts'

printf '\n--- Error report (errpt) ---\n'

if command -v errpt >/dev/null 2>&1; then

  errpt -a | head -n 20 2>/dev/null || true

else

  echo 'errpt not present'

fi

printf '\n--- Package validation (lppchk) ---\n'

if command -v lppchk >/dev/null 2>&1; then

  lppchk -v 2>/dev/null || true

else

  echo 'lppchk not present'

fi

printf '\n--- Disk multipathing ---\n'

lsdev -Cc disk 2>/dev/null || echo 'lsdev not present'

printf '\n--- PowerPath status ---\n'

if [ -x /usr/sbin/powermt ]; then

  /usr/sbin/powermt display dev=all 2>/dev/null || true

else

  echo 'powermt not present'

fi

printf '\n--- exportfs ---\n'

exportfs 2>/dev/null || echo 'exportfs not present'

printf '\n--- showmount ---\n'

showmount -a 2>/dev/null || echo 'showmount not present'

printf '\n--- PowerPath reserve_policy ---\n'

if command -v lsattr >/dev/null 2>&1; then

  lsattr -El powerpath0 -a reserve_policy 2>/dev/null || echo 'powerpath0 not present'

fi

printf '\n--- MPIO Other detection ---\n'

lsdev -Cc disk 2>/dev/null | grep 'Other' || echo 'None'

printf '\n--- Cluster service status ---\n'

if command -v lssrc >/dev/null 2>&1; then

  lssrc -ls clstrmgrES 2>/dev/null || echo 'clstrmgrES not present'

else

  echo 'lssrc not present'

fi

printf '\n--- IPSec check ---\n'

if command -v lssrc >/dev/null 2>&1; then

  lssrc -a 2>/dev/null | grep -i ipsec || echo 'No IPSec listed'

else

  echo 'lssrc not present'

fi

REMOTE

SSH_EXIT=$?

if [ $SSH_EXIT -ne 0 ]; then

  print_status fail

  echo "Remote collection failed (SSH exit code $SSH_EXIT). See $POST_SNAP for details."

  exit $EXIT_ERR

fi

print_status ok

# -------------------- Diff comparison --------------------

section_header "Comparing pre and post snapshots"

if [ ! -f "$PRE_SNAP" ]; then

  echo "Pre-reboot snapshot not found: $PRE_SNAP"

  print_status fail

  exit $EXIT_ERR

fi

# Normalize transient lines before diff (remove timestamps, uptime, etc.)

grep -v -E '^(--- HOST ---|^Date:|^Uptime:|^uptime:|^Last login:|^login:)' "$PRE_SNAP"  >"$TMPDIR/pre_norm"

grep -v -E '^(--- HOST ---|^Date:|^Uptime:|^uptime:|^Last login:|^login:)' "$POST_SNAP" >"$TMPDIR/post_norm"

diff -u "$TMPDIR/pre_norm" "$TMPDIR/post_norm" >"$DIFF_OUT" 2>/dev/null

DIFF_RC=$?

case $DIFF_RC in

  0)

    print_status ok

    echo "No differences detected."

    exit $EXIT_OK

    ;;

  1)

    echo "Differences detected (excerpt):"

    awk '/^(\+|\-)/ && !/^\+\+\+|^---/' "$DIFF_OUT" | head -n 50

    print_status warn

    echo "Full diff saved: $DIFF_OUT"

    exit $EXIT_WARN

    ;;

  *)

    print_status fail

    echo "Diff failed (rc=$DIFF_RC). See $DIFF_OUT and $POST_SNAP for details."

    exit $EXIT_ERR

    ;;

esac

==========================================================================

Example 1 — Successful run (No differences found)
Command:
# ./verify_post_reboot.ksh aixlpar01
Console Output:
-------------------------------------------------------------
Testing ping to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Testing SSH to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Collecting post-reboot snapshot from aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Comparing pre and post snapshots
-------------------------------------------------------------
[ OK ]
No differences detected.

Files generated:
/tmp/pre_reboot_snapshots/aixlpar01_pre_snap.txt
/tmp/pre_reboot_snapshots/aixlpar01_post_snap.txt
/tmp/pre_reboot_snapshots/aixlpar01_verify_diff.txt

Contents of aixlpar01_verify_diff.txt:
# Empty file – no differences detected

Example 2 — Differences detected (e.g. IP or NFS change)
Command:
# ./verify_post_reboot.ksh aixlpar01
Console Output:
-------------------------------------------------------------
Testing ping to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Testing SSH to aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Collecting post-reboot snapshot from aixlpar01
-------------------------------------------------------------
[ OK ]
-------------------------------------------------------------
Comparing pre and post snapshots
-------------------------------------------------------------
Differences detected (excerpt):
+--- Network interfaces ---
+en0: flags=4e080863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,GROUPRT>
+ inet 192.168.10.25 netmask 0xffffff00 broadcast 192.168.10.255
---- Network interfaces ---
-en0: flags=4e080863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,GROUPRT>
- inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255
+--- NFS/GPFS mounts ---
+server02:/exports/data on /data type nfs (rw,soft,intr,proto=tcp)
---- NFS/GPFS mounts ---
-server01:/exports/data on /data type nfs (rw,soft,intr,proto=tcp)
[ WARN ]
Full diff saved: /tmp/pre_reboot_snapshots/aixlpar01_verify_diff.txt
Contents of aixlpar01_verify_diff.txt (first few lines):
--- /tmp/verify_snap.10231/pre_norm 2025-11-08 14:32:12.000000000 +0530
+++ /tmp/verify_snap.10231/post_norm 2025-11-08 14:32:56.000000000 +0530
@@ -145,7 +145,7 @@
--- Network interfaces ---
-en0: flags=4e080863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,GROUPRT>
- inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255
+en0: flags=4e080863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,GROUPRT>
+ inet 192.168.10.25 netmask 0xffffff00 broadcast 192.168.10.255
@@ -198,7 +198,7 @@
--- NFS/GPFS mounts ---
-server01:/exports/data on /data type nfs (rw,soft,intr,proto=tcp)
+server02:/exports/data on /data type nfs (rw,soft,intr,proto=tcp)
Interpretation:
Network IP changed from .15 → .25
NFS mount source changed from server01 → server02
These are flagged as [ WARN ], but not fatal errors.

Example 3 — SSH or ping failure
Command:
$ ./verify_post_reboot.ksh aixlpar01
Console Output:
-------------------------------------------------------------
Testing ping to aixlpar01
-------------------------------------------------------------
[ FAIL ]
Host aixlpar01 not responding to ping.
or
-------------------------------------------------------------
Testing SSH to aixlpar01
-------------------------------------------------------------
[ FAIL ]
Unable to SSH to root@aixlpar01

In these cases, the script exits immediately with:
Exit code: 1 (EXIT_ERR)

Automating AIX TL/SP Patching with alt_disk_copy

Patching AIX servers can be a complex and risky process if done manually, especially when upgrading Technology Levels (TLs) and Service Packs (SPs). A safer approach is to use alternate rootvg cloning with alt_disk_copy, which allows you to patch a clone of the root volume group without affecting the live system.

In this post, I’ll share a production-ready KornShell script (aix_patching.ksh) that automates TL/SP patching remotely from a NIM server.

Features of the Script:

• Works remotely via SSH on a target AIX server.
• Removes any existing alternate rootvg (altinst_rootvg) safely.
• Reduces rootvg only if mirrored.
• Automatically mounts NFS share containing TL/SP packages.
• Creates a rootvg clone, removes old EFIXes, commits filesets, applies updates, and validates the system.
• Includes logging and a DRY_RUN mode for safe testing.

How It Works:

• Step 0: Checks for an existing altinst_rootvg and removes it.
• Step 1: Reduces rootvg disk if mirrored.
• Step 2: Mounts the NFS share containing TL/SP packages.
• Step 3: Creates an alternate rootvg clone using alt_disk_copy -P1.
• Step 4: Removes any existing EFIXes from the alternate rootvg.
• Step 5: Commits all applied filesets.
• Step 6: Performs an LPP consistency check.
• Step 7: Applies TL/SP updates from the mounted NFS share.
• Step 8: Validates the patched alternate rootvg including OS level, missing ML/SP, and package consistency.

Usage:

./aix_patching_v2.ksh <target_server> -d <target_disk>

<target_server> → Hostname or IP of the AIX server.

<target_disk> → Disk to use for alternate rootvg cloning.

Example:

./aix_patching_v2.ksh aix01 -d hdisk5

The Script

===============================================================================

#!/usr/bin/ksh

#==============================================================================

# Script Name: aix_patching.ksh

# Description: Automates AIX TL/SP patching using alt_disk_copy for a remote server

# Author: Tasleem A Khan

# Usage: aix_patching.ksh <target_server> -d <target_disk>

#==============================================================================

# Parameters

if [ $# -ne 3 ]; then

    echo "Usage: $0 <target_server> -d <target_disk>"

    exit 1

fi

TARGET_SERVER=$1

TARGET_DISK=$3

NFS_SERVER="aixnimserver"

NFS_PATH="/exports/software/aix_72.05.10"

MOUNT_POINT="/mnt"

ALT_ROOT="/alt_inst"

LOGFILE="/tmp/aix_patch_$(date +%Y%m%d%H%M%S).log"

DRY_RUN=0   # Set to 1 for testing without removing EFIXes

# Logging function

log() {

    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOGFILE

}

log "==== AIX Patching Script Started ===="

log "Target Server: $TARGET_SERVER"

log "Target Disk  : $TARGET_DISK"

# Remote execution

ssh root@$TARGET_SERVER bash <<EOF

log() { echo "\$(date '+%Y-%m-%d %H:%M:%S') - \$1" | tee -a $LOGFILE; }

# Step 0: Remove existing alternate rootvg if exists

log "Step 0: Checking existing alternate rootvg"

ALTVG_EXISTS=\$(lsvg -o | grep altinst_rootvg)

if [ "\$ALTVG_EXISTS" ]; then

    log "Alternate rootvg exists, removing..."

    alt_disk_copy -d $TARGET_DISK -C

fi

# Step 1: Reduce rootvg if mirrored

log "Step 1: Checking if rootvg is mirrored"

MIRROR_COUNT=\$(lsvg -l rootvg | grep -v PV | awk '{print \$2}' | wc -l)

if [ \$MIRROR_COUNT -gt 1 ]; then

    log "Rootvg is mirrored, reducing disk $TARGET_DISK"

    unmirrorvg rootvg $TARGET_DISK

    reducevg -df rootvg $TARGET_DISK

    chpv -c $TARGET_DISK

    ipl_varyon -i

    bootlist -m normal -o

    bosboot -ad /dev/$TARGET_DISK

else

    log "Rootvg not mirrored, skipping disk reduction"

fi

# Step 2: Mount NFS Share

log "Step 2: Mounting NFS Share"

if ! mount | grep -q "$MOUNT_POINT"; then

    log "Mount point $MOUNT_POINT not mounted. Mounting..."

    mkdir -p $MOUNT_POINT

    mount $NFS_SERVER:$NFS_PATH $MOUNT_POINT

else

    log "Mount point $MOUNT_POINT already mounted"

fi

# Step 3: Create Alternate Root Disk Clone (P1)

log "Step 3: Creating alternate root disk clone"

alt_disk_copy -d $TARGET_DISK -P1

# Step 4: Remove existing EFIXes

log "Step 4: Removing existing EFIXes (if any) from $ALT_ROOT"

if [ -d $ALT_ROOT ]; then

    EFIX_LIST="\$(chroot $ALT_ROOT /usr/sbin/emgr -l 2>/dev/null | grep -v 'No efix' | awk '{print \$1}' || true)"

    if [ -n "\$EFIX_LIST" ]; then

        for efix in \$EFIX_LIST; do

            log "Removing efix \$efix..."

            if [ "$DRY_RUN" -eq 1 ]; then

                log "DRY RUN: would run: chroot $ALT_ROOT /usr/sbin/emgr -r -L \$efix"

            else

                chroot $ALT_ROOT /usr/sbin/emgr -r -L "\$efix" || log "Warning: failed to remove \$efix"

            fi

        done

    else

        log "No EFIXes to remove in $ALT_ROOT."

    fi

else

    log "$ALT_ROOT does not exist, skipping EFIX removal."

fi

# Step 5: Commit applied filesets

log "Step 5: Committing applied filesets"

chroot $ALT_ROOT /usr/sbin/installp -c all

# Step 6: LPP Check

log "Step 6: Running LPP check"

chroot $ALT_ROOT /usr/bin/lppchk -vm3

# Step 7: Apply TL/SP Update

log "Step 7: Applying TL/SP updates"

alt_disk_copy -d $TARGET_DISK -P23 -l $MOUNT_POINT -b update_all

# Step 8: Validate patched alternate rootvg

log "Step 8: Validating patched alternate rootvg"

alt_root_op -W -d $TARGET_DISK

chroot $ALT_ROOT /usr/bin/oslevel -s

chroot $ALT_ROOT /usr/bin/instfix -i | grep ML

chroot $ALT_ROOT /usr/bin/instfix -i | grep SP

chroot $ALT_ROOT /usr/bin/lppchk -vm3

alt_root_op -S -d $TARGET_DISK

EOF

log "==== AIX OS Patching Script Completed ===="

===============================================================================

# ./aix_patching_v2.ksh aix01 -d hdisk5

2025-11-08 17:45:12 - ==== AIX Patching Script Started ====

2025-11-08 17:45:12 - Target Server: aix01

2025-11-08 17:45:12 - Target Disk  : hdisk5

2025-11-08 17:45:13 - Step 0: Checking existing alternate rootvg

2025-11-08 17:45:13 - Alternate rootvg exists, removing...

Verifying and removing altinst_rootvg on hdisk5.......................

alt_disk_copy completed successfully.

2025-11-08 17:45:20 - Step 1: Checking if rootvg is mirrored

2025-11-08 17:45:20 - Rootvg is mirrored, reducing disk hdisk5

unmirrorvg executed successfully.

reducevg executed successfully.

chpv -c hdisk5 completed.

Boot configuration updated and bosboot executed.

2025-11-08 17:45:40 - Step 2: Mounting NFS Share

2025-11-08 17:45:40 - Mount point /mnt not mounted. Mounting...

Mount successful: aixnimserver:/exports/software/aix_72.05.10 mounted on /mnt

2025-11-08 17:45:45 - Step 3: Creating alternate root disk clone

alt_disk_copy -P1 executed on hdisk5. Alternate rootvg clone created.

2025-11-08 17:46:10 - Step 4: Removing existing EFIXes (if any) from /alt_inst

2025-11-08 17:46:10 - Removing efix 7200-05-1234...

EFIX 7200-05-1234 removed successfully.

2025-11-08 17:46:12 - Removing efix 7200-05-5678...

EFIX 7200-05-5678 removed successfully.

2025-11-08 17:46:15 - No additional EFIXes to remove in /alt_inst.

2025-11-08 17:46:20 - Step 5: Committing applied filesets

installp -c all executed successfully.

2025-11-08 17:46:25 - Step 6: Running LPP check

lppchk -vm3 completed. All packages are consistent.

2025-11-08 17:46:30 - Step 7: Applying TL/SP updates

alt_disk_copy -P23 -b update_all executed successfully from /mnt.

2025-11-08 17:47:15 - Step 8: Validating patched alternate rootvg

alt_root_op -W executed. Alternate rootvg is ready for validation.

OS Level: 7200-05-10-2446

Missing ML Fixes: None

Missing SP Fixes: None

LPP check: All packages consistent

alt_root_op -S executed. Alternate rootvg is now ready to boot.

2025-11-08 17:47:20 - ==== AIX OS Patching Script Completed ====

Log file saved at: /tmp/aix_patch_20251108174512.log

Automated OpenSSH and OpenSSL Upgrade with IFIX Installation on AIX

Upgrade SSH/SSL and apply IFIXes on AIX remotely:

==========================================================================
#!/bin/ksh
# =========================================================================
# upgrade_ssh_ifix.ksh - Upgrade SSH/SSL and apply IFIXes on AIX remotely
# Usage : ./upgrade_ssh_ifix.ksh <hostname>
# Author : Tasleem A Khan
# =========================================================================
# --- Usage Check ---
if [ $# -ne 1 ]; then
echo "Usage: $0 <hostname>"
exit 1
fi

HOST=$1
SSH="ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -q"
SCP="scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 -q"
NIM=$(hostname)
SOFTSRC="/software/openssh-ssl"

echo "=============================================================="
echo " Starting OpenSSH/OpenSSL upgrade on host: $HOST"
echo " NIM Server: $NIM"
echo "=============================================================="
echo

# --- Step 1: Check remote connectivity ---
echo "[CHECK] Testing SSH connectivity to $HOST..."
if ! ${SSH} ${HOST} "hostname" >/dev/null 2>&1; then
echo "[ERROR] Unable to connect to $HOST via SSH. Aborting."
exit 2
fi
echo "[OK] SSH connectivity verified."
echo

# --- Step 2: Prepare temporary directory for SSH/SSL install ---
echo "[INFO] Preparing /tmp/openssh-ssl directory..."
${SSH} ${HOST} "rm -rf /tmp/openssh-ssl && mkdir -p /tmp/openssh-ssl"

# --- Step 3: Copy installation files ---
echo "[INFO] Copying installation files from $SOFTSRC to $HOST..."
${SCP} ${SOFTSRC}/* ${HOST}:/tmp/openssh-ssl/ >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "[ERROR] File copy failed. Check $SOFTSRC path and permissions."
exit 3
fi
echo "[OK] Files copied successfully."
echo

# --- Step 4: Install OpenSSH and OpenSSL base packages ---
echo "[INFO] Installing OpenSSH and OpenSSL base packages..."
${SSH} ${HOST} "installp -aXYd /tmp/openssh-ssl \
openssh.base openssh.license openssh.man.en_US \
openssl.base openssl.license openssl.man.en_US \
openssh.msg.EN_US openssh.msg.en_US" >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "[WARNING] installp encountered warnings or errors. Verify manually."
fi
echo "[OK] Installation phase complete."
echo

# --- Step 5: Apply IFIX packages (epkg.Z) ---
echo "[INFO] Applying IFIX packages..."
for FIX in 3013ma.240923.epkg.Z 973013sa.250306.epkg.Z; do
echo " -> Applying IFIX: $FIX"
${SSH} ${HOST} "emgr -e /tmp/openssh-ssl/$FIX" >/dev/null 2>&1
done
echo "[OK] IFIXes applied successfully."
echo

# --- Step 6: Restart SSHD service ---
echo "[INFO] Restarting SSHD service..."
${SSH} ${HOST} "stopsrc -s sshd >/dev/null 2>&1; sleep 3; startsrc -s sshd >/dev/null 2>&1"
if [ $? -eq 0 ]; then
echo "[OK] SSHD restarted successfully."
else
echo "[WARNING] SSHD restart failed. Please check manually."
fi
echo

# --- Step 7: Cleanup temporary files ---
echo "[INFO] Cleaning up temporary files..."
${SSH} ${HOST} "rm -f /tmp/openssh-ssl/* /tmp/openssh-ssl/.toc; rmdir /tmp/openssh-ssl" >/dev/null 2>&1
echo "[OK] Cleanup completed."
echo

# --- Step 8: Post-upgrade validation ---
echo "[INFO] Post-upgrade validation:"
echo "--------------------------------------------------------------"
${SSH} ${HOST} "oslevel -s"
${SSH} ${HOST} "lslpp -L | grep -E 'openssh|openssl' | grep -v fileset"
${SSH} ${HOST} "emgr -l | grep -E 'State|Label|Description' | head -n 15"
${SSH} ${HOST} "lppchk -v >/dev/null 2>&1; echo 'lppchk -v: completed successfully'"
echo "--------------------------------------------------------------"
echo
echo "===== OpenSSH/OpenSSL Upgrade Completed Successfully on $HOST ====="
exit 0

==========================================================================
Example Output:
Here’s what you’ll typically see when you run:
# ./upgrade_ssh_ifix.ksh aixlpar01

Example output:

==============================================================

 Starting OpenSSH/OpenSSL upgrade on host: aixlpar01

 NIM Server: nim-master01

==============================================================

[CHECK] Testing SSH connectivity to aixlpar01...

[OK] SSH connectivity verified.

[INFO] Preparing /tmp/openssh-ssl directory...

[OK] Created temporary directory.

[INFO] Copying installation files from /software/openssh-ssl to aixlpar01...

[OK] Files copied successfully.

[INFO] Installing OpenSSH and OpenSSL base packages...

[OK] Installation phase complete.

[INFO] Applying IFIX packages...

   -> Applying IFIX: 3013ma.240923.epkg.Z

   -> Applying IFIX: 973013sa.250306.epkg.Z

[OK] IFIXes applied successfully.

[INFO] Restarting SSHD service...

[OK] SSHD restarted successfully.

[INFO] Cleaning up temporary files...

[OK] Cleanup completed.

[INFO] Post-upgrade validation:

--------------------------------------------------------------

7200-05-09-2346

openssh.base              9.0.100.250923  COMMITTED  OpenSSH Secure Shell Server

openssl.base              3.0.10.250923   COMMITTED  OpenSSL Cryptography Library

State = Applied  Label = 3013ma.240923.epkg.Z

State = Applied  Label = 973013sa.250306.epkg.Z

lppchk -v: completed successfully

--------------------------------------------------------------

===== OpenSSH/OpenSSL Upgrade Completed Successfully on aixlpar01 =====