Pages

Join a RHEL Linux Server to Active Directory Using SSSD

This guide explains how to join a Linux server to an Active Directory (AD) domain, configure authentication using SSSD, allow AD groups to log in, and grant sudo privileges to a specific AD group.

Prerequisites
  • RHEL Linux Servers 8/9/10
  • Proper DNS connectivity to the AD server
  • AD administrator credentials
  • Root or sudo access on the Linux server
Step 1: Install Required Packages
Install all necessary packages for AD integration.
# dnf install -y realmd sssd oddjob oddjob-mkhomedir adcli samba-common-tools krb5-workstation

Step 2: Configure DNS Resolution
Edit /etc/resolv.conf
# vi /etc/resolv.conf
Add the following entries:
search ppc.com
nameserver 192.168.10.100
nameserver 192.168.20.100
Save and exit:
Press Esc
Type :wq!
Press Enter

Step 3: Update /etc/hosts

# vi /etc/hosts
Add the AD server entry:
192.168.10.100 inddcpads01.ppc.com inddcpads01
Save and exit using :wq!.

Step 4: Configure Kerberos Encryption Types
Edit the crypto policy file:
# vi /etc/krb5.conf.d/crypto-policies
Add the following:
[libdefaults]
permitted_enctypes = aes256-cts-hmac-sha1-96 aes256-cts-hmac-sha384-192 camellia256-cts-cmac aes128-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 camellia128-cts-cmac rc4-hmac
Save and exit.

Step 5: Configure Authselect and Oddjob
If you see the message:
Directory [/etc/authselect] does not exist, please create it!
Run the following commands:
# mkdir /etc/authselect
# authselect select sssd with-mkhomedir --force
# systemctl enable --now oddjobd.service
This enables automatic home directory creation for AD users.

Step 6: Join the Linux Server to the AD Domain
Join the system to the domain using an AD administrator account:
# realm join -v -U administrator inddcpads01.ppc.com
Enter the AD Administrator password when prompted.

Step 7: Verify Domain Join Status
Check whether the system successfully joined the domain:
# realm list
Confirm that:
The domain is listed
permitted-groups includes the intended AD group

Step 8: Permit an AD Group to Log In
Allow the AD group unix_admin to access the Linux server:
# realm permit -g unix_admin

Step 9: Configure SSSD
Edit the SSSD configuration file:
# vi /etc/sssd/sssd.conf
Update or add the following configuration:
[domain/ppc.com]
ad_server = inddcpads01.ppc.com
ad_domain = ppc.com
krb5_realm = ppc.com
realmd_tags = manages-system joined-with-adcli
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = False
fallback_homedir = /home/%u
access_provider = simple
simple_allow_groups = unix_admin

Save and exit.
Important: Ensure correct file permissions:
# chmod 600 /etc/sssd/sssd.conf

Step 10: Restart SSSD Service
# systemctl restart sssd

Step 11: Create AD Group and Add Users (On AD Server)
Create the AD group: unix_admin
Add required AD users (e.g., sysadm) to this group
Allow time for AD replication if needed

Step 12: Configure Sudo Access for AD Group
Edit the sudoers file safely:
# visudo
Add the following line:
%unix_admin ALL=(ALL) NOPASSWD: ALL
This grants passwordless sudo access to all members of the unix_admin group.

Step 13: Test Login and Root Access

Log in using an AD user (example: sysadm)
Verify sudo access:
$ sudo su -
If successful, the user now has root privileges.

Conclusion
You have successfully:
  • Joined  Linux server to Active Directory
  • Configured SSSD authentication
  • Enabled automatic home directory creation
  • Restricted access to a specific AD group
  • Granted sudo/root privileges to AD users
This setup provides centralized authentication, improved security, and easier user management across Linux servers.

Grafana Enterprise Installation

This document provides step-by-step guidance for installing Grafana Enterprise on RHEL/CentOS, configuring NGINX as a reverse proxy with HTTPS (Let’s Encrypt), and enabling high availability (HA) for Grafana, Prometheus, Alertmanager, and the load balancer.

1. Install Grafana Enterprise
1.1 Install RPM Package
# yum install -y https://dl.grafana.com/grafana-enterprise/release/12.3.1/grafana-enterprise_12.3.1_20271043721_linux_amd64.rpm
# OR
# dnf install -y https://dl.grafana.com/grafana-enterprise/release/12.3.1/grafana-enterprise_12.3.1_20271043721_linux_amd64.rpm
1.2 Enable and Start Grafana
# systemctl enable --now grafana-server

2. Configure Grafana Server
2.1 Update Grafana Configuration
# vi /etc/grafana/grafana.ini
Edit [server] section:
[server]
http_addr = localhost
http_port = 3000
domain = www.grafana.ppc.com
Binding Grafana to localhost ensures external access only via NGINX.
2.2 Restart Grafana
# systemctl restart grafana-server

3. Install and Configure NGINX Reverse Proxy
3.1 Install NGINX
# dnf install nginx -y
3.2 NGINX Configuration for Grafana

Create /etc/nginx/conf.d/grafana.conf:

# Proxy WebSocket connections for Grafana Live
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}
server {
    listen 80;
    server_name grafana.example.io;
    rewrite ^ https://$server_name$request_uri? permanent;
}
server {
  listen 443 ssl http2;
  server_name grafana.example.io;

  root /usr/share/nginx/html;
  index index.html index.htm;

  ssl_certificate /etc/letsencrypt/live/grafana.example.io/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/grafana.example.io/privkey.pem;

  access_log /var/log/nginx/grafana-access.log;
  error_log /var/log/nginx/grafana-error.log;

  location / {
    proxy_pass https://localhost:3000/;
  }

  location /api/live {
    rewrite ^/(.*) /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass https://localhost:3000/;
  }
}
3.3 Validate and Start NGINX
# nginx -t
# systemctl enable --now nginx
# systemctl status nginx

4. Access Grafana
Open browser at:
https://www.grafana.ppc.com
Default credentials:
Username: admin
Password: admin (change on first login)

5. SELinux Configuration for Grafana + NGINX
5.1 Verify SELinux Status
# getenforce
Expected: Enforcing
5.2 Allow NGINX Network Connections
# setsebool -P httpd_can_network_connect 1
5.3 WebSocket Support
WebSockets for Grafana Live are allowed via the same boolean above. No additional SELinux policies required.
5.4 Optional: Verify Grafana Port Context
# semanage port -l | grep 3000
# semanage port -a -t http_port_t -p tcp 3000  # if required
5.5 Check SELinux Denials
# ausearch -m AVC -ts recent
# journalctl -t setroubleshoot

6. Grafana High Availability (HA)
6.1 HA Requirements
Component Requirement
Database PostgreSQL or MySQL (not SQLite)
Sessions Shared DB
Storage Local disks
Load Balancer NGINX or HAProxy
Grafana Version Enterprise / OSS
6.2 Shared Database Configuration
[database]
type = postgres
host = dbserver.example.io:5432
name = grafana
user = grafana
password = strongpassword
ssl_mode = disable
Restart Grafana on each node:
# systemctl restart grafana-server
6.3 Node Configuration
[server]
http_addr = 0.0.0.0
http_port = 3000
domain = grafana.example.io

[unified_alerting]
enabled = true

Each node listens locally; external access only via the load balancer.

7. Load Balancer Configuration
Option A: NGINX

Create /etc/nginx/conf.d/grafana-ha.conf:

upstream grafana_backend {
    least_conn;
    server indrxgraf01:3000;
    server indrxgraf02:3000;
}

server {
    listen 443 ssl http2;
    server_name grafana.example.io;

    ssl_certificate /etc/letsencrypt/live/grafana.example.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana.example.io/privkey.pem;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://grafana_backend;
    }

    location /api/live {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_pass http://grafana_backend;
    }
}
# nginx -t
# systemctl reload nginx
Option B: HAProxy (Alternative)
frontend grafana_https
    bind *:443 ssl crt /etc/haproxy/certs/grafana.pem
    default_backend grafana_nodes

backend grafana_nodes
    balance roundrobin
    server graf01 indrxgraf01:3000 check
    server graf02 indrxgraf02:3000 check

8. Keepalived Load Balancer HA
8.1 Install Keepalived
# dnf install keepalived -y
# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
# sysctl -p
8.2 Configure MASTER Node

Edit /etc/keepalived/keepalived.conf:

vrrp_script chk_nginx {
    script "/usr/bin/pidof nginx"
    interval 2
    weight -20
}

vrrp_instance VI_GRAFANA {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication { auth_type PASS auth_pass StrongPass }
    virtual_ipaddress { 192.168.20.50/24 }
    track_script { chk_nginx }
}
8.3 Configure BACKUP Node
Same file, change:
state BACKUP
priority 100
8.4 Enable Keepalived
# systemctl enable --now keepalived
# systemctl status keepalived
# ip a | grep 192.168.20.50
8.5 SELinux & Firewall
# setsebool -P keepalived_connect_any 1
# firewall-cmd --add-service=keepalived --permanent
# firewall-cmd --reload

9. Prometheus High Availability
9.1 Install Prometheus
# dnf install prometheus -y
# systemctl enable --now prometheus
9.2 Prometheus Configuration
Edit /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'grafana'
    static_configs:
      - targets:
          - indrxgraf01:3000
          - indrxgraf02:3000
Restart Prometheus:
# systemctl restart prometheus

10. Alertmanager High Availability
10.1 Install Alertmanager
# dnf install alertmanager -y
# systemctl enable --now alertmanager
10.2 Configure Clustering
Node 1:
ALERTMANAGER_OPTS="--cluster.listen-address=0.0.0.0:9094 --cluster.peer=alert01.example.io:9094"

Node 2:
ALERTMANAGER_OPTS="--cluster.listen-address=0.0.0.0:9094 --cluster.peer=alert02.example.io:9094"
10.3 Prometheus Alertmanager Integration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alert01.example.io:9093
            - alert02.example.io:9093
Restart Prometheus:
# systemctl restart prometheus

11. Grafana Integration
Add Prometheus HA endpoints as data sources
Add Alertmanager HA endpoints for alerting
Grafana handles failover automatically

12. Result
Secure Grafana Enterprise installation with HTTPS
NGINX handles SSL termination and WebSocket connections
Active-active HA for Grafana, Prometheus, Alertmanager, and load balancer
SELinux remains enforcing
No single point of failure

DB2 GPFS Cluster LUN Setup Using iSCSI and LVM on RHEL/CentOS

End-to-End High Availability Storage & Database Architecture

IBM DB2 high availability deployments require consistent shared storage, predictable fencing, and cluster-aware filesystems. When SAN infrastructure is unavailable or cost-prohibitive, iSCSI + GPFS (IBM Spectrum Scale) provides a fully supported, enterprise-grade alternative.

This guide covers the complete stack:
  • iSCSI shared LUNs (targetcli + LVM)
  • Multipath-secured access
  • GPFS cluster installation and quorum design
  • DB2 installation on GPFS
  • Performance tuning and HA validation
Reference Architecture
┌───────────────────────────────┐
│  iSCSI Target Server          │
│  RHEL/CentOS                  │
│  LVM VG: datavg               │
│  ├─ db2disk01 (3G)  → Data    │
│  ├─ db2disk02 (7G)  → Indexes │
│  ├─ db2disk03 (3G)  → Logs    │
│  └─ db2disk04 (7G)  → Temp    │
│  targetcli / LIO              │
└──────────────┬────────────────┘
               │ iSCSI (TCP 3260, MTU 9000)
 ┌─────────────┴─────────────┐
 │                           │
┌────────────────────┐   ┌────────────────────┐
│ DB2 Node 1         │   │ DB2 Node 2         │
indrxldb201        │   │ indrxldb202        │
│ GPFS Node          │   │ GPFS Node          │
│ DB2 Instance       │   │ DB2 Instance       │
└────────────────────┘   └────────────────────┘

Why GPFS for DB2?
RequirementGPFS Capability
Concurrent accessDistributed lock manager
HA fencingQuorum + tiebreaker disks
Write orderingJournaled metadata
Fast failoverSub-30s remount
IBM supportFully certified

ext4/XFS are NOT supported for DB2 HA shared-disk clusters

Environment Summary
ComponentValue
OSRHEL 8/9 or CentOS Stream
GPFSIBM Spectrum Scale 5.1+
DB211.5.x
Network10Gb iSCSI VLAN (MTU 9000)
Cluster Nodesindrxldb201, indrxldb202
QuorumTiebreaker disk required

Part 1 – GPFS Installation (Both Nodes)

1.1 Prerequisites
# dnf install -y \
  kernel-devel \
  gcc \
  cpp \
  elfutils-libelf-devel \
  numactl \
  chrony \
  net-tools
Ensure time synchronization:
# systemctl enable --now chronyd

1.2 Install IBM Spectrum Scale Packages
Upload Spectrum Scale RPMs to both nodes.
# rpm -ivh gpfs.base*.rpm gpfs.gpl*.rpm gpfs.msg*.rpm
Build kernel module:
/usr/lpp/mmfs/bin/mmbuildgpl
Verify:
# lsmod | grep mmfs
# echo "export PATH=$PATH:/usr/lpp/mmfs/bin" >> /root/.bash_profile

1.3 Create GPFS Cluster
Run once from primary node:
# mmcrcluster -N indrxldb201:manager-quorum,indrxldb202:manager-quorum -p indrxldb201 -s indrxldb202 -r /usr/bin/ssh -R /usr/bin/scp -C GPFS_DB2

# /usr/lpp/mmfs/bin/mmstartup -N indrxldb201
# /usr/lpp/mmfs/bin/mmstartup -N indrxldb201

Start GPFS:
# mmstartup -a
Verify:
# mmgetstate -a

1.4 Configure Quorum & Tiebreaker
For 2-node clusters, quorum is mandatory.
# mmadddisk tb -F /dev/mapper/mpatha
# mmchconfig tiebreakerDisks=mpatha
# mmchconfig quorum=4

Vote SourceCount
Node 11
Node 21
Data disk1
Tiebreaker1

Part 2 – GPFS Filesystem Design for DB2
FilesystemPurpose
gpfs_db2dataTablespaces
gpfs_db2indexIndexes
gpfs_db2logsTransaction logs
gpfs_db2tempTemp tables
Create File /tmp/nsdlist
# vi /tmp/nsdlist
%nsd:
device=/dev/sdb
nsd=nsd_01
servers=indrxldb201,indrxldb202
usage=dataAndMetadata 
failureGroup=-1 
pool=system 

Create NSDs
# mmcrnsd -F /tmp/nsdlist
Create Filesystems
# mmcrfs gpfs_db2data  -F /dev/mapper/mpathb -A yes -Q yes
# mmcrfs gpfs_db2index -F /dev/mapper/mpathc -A yes -Q yes
# mmcrfs gpfs_db2logs  -F /dev/mapper/mpathd -A yes -Q no
Mount:
# mmmount all -a

Part 3 – DB2 Installation (Both Nodes)

3.1 OS Kernel Tuning
# sysctl -w kernel.shmmni=8192
# sysctl -w kernel.shmmax=$(($(getconf _PHYS_PAGES) * 4096))
# sysctl -w kernel.shmall=$(($(getconf _PHYS_PAGES) * 4096 / 4096))
# sysctl -w vm.swappiness=1
Persist in /etc/sysctl.conf.

3.2 Create DB2 User & Groups
# groupadd -g 1001 db2iadm1
# groupadd -g 1002 db2fadm1
# useradd -u 1001 -g db2iadm1 -G db2fadm1 db2inst1
# passwd db2inst1

3.3 Install DB2 Software
# tar -xvf DB2_Svr_11.5_Linux_x86-64.tar.gz
# cd server_dec
# ./db2_install
Select:
Server Edition
Typical install

3.4 Create DB2 Instance
# /opt/ibm/db2/V11.5/instance/db2icrt \
  -u db2fadm1 db2inst1

Part 4 – DB2 Database on GPFS
Directory Structure
# mkdir -p /gpfs_db2data/db2inst1
# mkdir -p /gpfs_db2logs/db2inst1
# chown -R db2inst1:db2iadm1 /gpfs*

Create Database
# su - db2inst1
db2 create database PRODDB \
  on /gpfs_db2data \
  dbpath on /gpfs_db2data \
  using codeset UTF-8 territory us

Move logs:
# db2 update db cfg for PRODDB using NEWLOGPATH /gpfs_db2logs/db2inst1

Part 5 – High Availability Behavior
EventResult
Node crashGPFS remounts
Network splitQuorum prevents corruption
iSCSI path lossMultipath reroutes
Storage restartDB2 recovers logs

Typical DB2 recovery time: 20–45 seconds

Performance Tuning
GPFS
# mmchconfig pagepool=80%RAM
# mmchconfig maxFilesToCache=1000000
DB2
# db2 update db cfg using LOGFILSIZ 8192
# db2 update db cfg using LOGPRIMARY 20
# db2 update db cfg using LOGSECOND 100
iSCSI
# echo 256 > /sys/block/sdX/queue/nr_requests

Validation Checklist
  • iSCSI sessions persistent
  • Multipath active
  • GPFS quorum healthy
  • DB2 database starts on either node
  • Forced node failure recovers cleanly
  • No GPFS fencing events
Operational Commands Cheat Sheet
# mmgetstate -a
# mmlscluster
# mmlsdisk gpfs_db2data
# db2pd -db PRODDB -logs
# iscsiadm -m session -P 3
# multipath -ll

Final Thoughts
This architecture delivers:
  • IBM-supported DB2 HA
  • SAN-like behavior using software-defined storage
  • Strong fencing and split-brain prevention
  • Predictable performance at scale
The success factors are discipline and testing:
  • Quorum
  • Multipath
  • Dedicated network
  • Regular failover drills

MQ GPFS Cluster LUN Setup Using iSCSI and LVM

Enterprise Design for Shared Storage–Based MQ Multi-Instance Clusters

High availability (HA) for IBM MQ multi-instance queue managers depends on shared, consistent, low-latency storage. While SAN or enterprise NAS is common, iSCSI-backed LUNs combined with LVM and IBM Spectrum Scale (GPFS) provide a flexible, software-defined alternative that is production-proven when designed correctly.

This guide walks through a full-stack HA storage architecture:
  • LVM-backed iSCSI LUNs
  • Secure export using targetcli
  • Multipath iSCSI initiators
  • GPFS clustered filesystems
  • IBM MQ HA compatibility and failover guarantees
Architecture Overview
                 ┌──────────────────────┐
                 │  iSCSI Target Server │
                 │  RHEL / CentOS       │
                 │                      │
                 │  LVM (datavg)        │
                 │  ├─ mqsdisk01 (LUN)  │
                 │  └─ mqsdisk02 (LUN)  │
                 │                      │
                 │  targetcli / LIO     │
                 └──────────┬───────────┘
                            │ iSCSI (TCP 3260)
           ┌────────────────┴────────────────┐
           │                                 │
┌────────────────────┐        ┌────────────────────┐
│ MQ Node 1          │        │ MQ Node 2          │
indrxlmqs01        │        │ indrxlmqs02        │
│                    │        │                    │
│ iSCSI Initiator    │        │ iSCSI Initiator    │
│ Multipath          │        │ Multipath          │
│ GPFS Node          │        │ GPFS Node          │
│ MQ Instance        │        │ MQ Instance        │
└────────────────────┘        └────────────────────┘

Why GPFS for IBM MQ?
  • IBM MQ multi-instance queue managers require:
  • Shared filesystem
  • POSIX locking
  • Fast failover
  • Guaranteed write ordering
Why GPFS (Spectrum Scale)?
FeatureBenefit
Distributed lock managerSafe concurrent access
Quorum + tiebreakerSplit-brain prevention
High-performance journalingMQ log safety
Fast mount failover<30s recovery
Certified by IBMSupported configuration

GPFS is explicitly supported for MQ HA. ext4/XFS are not.

Environment Specifications
ComponentValue
iSCSI Target ServerRHEL/CentOS, IP: 192.168.20.20
Volume Groupdatavg (/dev/sdb)
Logical Volumesmqsdisk01 (5G), mqsdisk02 (5G)
iSCSI Target IQNiqn.2025-08.ppc.com:mqsservers
Initiator Nodesindrxlmqs01, indrxlmqs02
OSRHEL 8/9 or CentOS 8 Stream
GPFS VersionIBM Spectrum Scale 5.1+

Why LVM Under iSCSI?
Online resize (future MQ growth)
Snapshot capability
Alignment control
Striping across RAID

GPFS Optimal Alignment
PE size: 64KB
Stripe size: 64KB
Avoid 4MB defaults
Misalignment = silent performance loss.

Step 1: Prepare Storage on iSCSI Target
Disk Partitioning
# parted /dev/sdb mklabel gpt
# parted /dev/sdb mkpart primary 1MiB 100%

Physical Volume & VG Creation
# pvcreate --dataalignment 1m /dev/sdb1
# vgcreate -s 64K datavg /dev/sdb1

Create GPFS-Optimized Logical Volumes
# lvcreate -L 5G -i 4 -I 64K -n mqsdisk01 datavg
# lvcreate -L 5G -i 4 -I 64K -n mqsdisk02 datavg


Why striping?
GPFS issues parallel I/O
MQ log writes benefit from stripe width
Underlying RAID must support it

Step 2: iSCSI Target Configuration
Create Block Backstores
/backstores/block create MQS_LUN01 /dev/datavg/mqsdisk01
/backstores/block create MQS_LUN02 /dev/datavg/mqsdisk02

Create Target
/iscsi create iqn.2025-08.ppc.com:mqsservers

Map LUNs
/iscsi/iqn.2025-08.ppc.com:mqsservers/tpg1/luns create /backstores/block/MQS_LUN01
/iscsi/iqn.2025-08.ppc.com:mqsservers/tpg1/luns create /backstores/block/MQS_LUN02

ACL Configuration
/acls create iqn.2025-08.ppc.com:indrxlmqs01
/acls create iqn.2025-08.ppc.com:indrxlmqs02

Security Hardening
set attribute authentication=1
set auth userid=mqchap password=StrongSecret!

Never use demo_mode_write_protect in production

Step 3: iSCSI Initiators (MQ Nodes)
Install Required Packages
# dnf install iscsi-initiator-utils device-mapper-multipath -y

Configure IQN (Unique per Node)
# echo "InitiatorName=iqn.2025-08.ppc.com:indrxlmqs01" \
> /etc/iscsi/initiatorname.iscsi

Persistent Discovery
# iscsiadm -m discoverydb -t sendtargets \
  -p 192.168.20.20 --discover

Login
# iscsiadm -m node --login

Multipath Configuration (Strongly Recommended)
/etc/multipath.conf
defaults {
  user_friendly_names yes
  path_grouping_policy multibus
  rr_min_io 100
  rr_weight uniform
  failback immediate
}

# systemctl enable --now multipathd
# multipath -ll

GPFS must use multipath devices.

Step 4: GPFS Cluster Setup
Create Cluster
# mmcrcluster -N indrxlmqs01,indrxlmqs02 \
  -C mqsgpfscluster \
  --admin-interface eth0

Add Tiebreaker Disk (Required for 2-Node)
# mmadddisk tb -F /dev/mapper/mpatha

Create GPFS Filesystems
# mmcrfs gpfsdata -F /dev/mapper/mpathb \
  -A yes -Q no -T /ibm/mqdata
# mmcrfs gpfslogs -F /dev/mapper/mpathc \
  -A yes -Q no -T /ibm/mqlogs
Mount
# mmmount all -a
Quorum & Fencing Logic
# mmchconfig tiebreakerDisks=mpatha
# mmchconfig quorum=4

ComponentVote
Node11
Node21
Data disk1
Tiebreaker1

Prevents split-brain if one node loses storage or network.

IBM MQ Integration Notes
Recommended Layout
MQ ComponentGPFS FS
QM datagpfsdata
Logsgpfslogs
Error logsgpfsdata
Tracegpfsdata

MQ HA Behavior
  • Only one instance active
  • Standby monitors lock files
  • GPFS ensures fast lock transfer
  • Typical failover: 15–30s
Performance Tuning
iSCSI
# echo 128 > /sys/block/sdX/queue/nr_requests
GPFS
# mmchconfig pagepool=80%RAM
# mmchconfig maxFilesToCache=500000
MQ
  • Use linear logging
  • Separate data and logs
  • Avoid filesystem compression
Failure Scenarios & Behavior
FailureOutcome
MQ active node crashStandby takes over
iSCSI path lossMultipath reroutes
Storage server rebootGPFS retries
Network partitionQuorum prevents split-brain

Validation Checklist
  • LUNs visible on both nodes
  • Multipath active
  • GPFS quorum healthy
  • MQ starts on either node
  • Forced failover succeeds
  • No split-brain warnings
Operational Commands Cheat Sheet
# iscsiadm -m session -P 3
# multipath -ll
# mmgetstate -a
# mmlscluster
# strmqcsv QM1
# dspmq -x

Final Thoughts
This architecture delivers:
  • Enterprise-grade HA
  • SAN-like behavior with software-defined flexibility
  • IBM-supported MQ configuration
  • Predictable failover
The key is discipline:
  • Quorum
  • Fencing
  • Multipath
  • Testing
Done right, this design rivals expensive SAN solutions at a fraction of the cost.

RHEL 7, 8, 9, 10 – Storage Issues

Storage issues in Red Hat Enterprise Linux (RHEL) are among the most critical problems administrators face. They can cause boot failures, application downtime, data corruption, or performance degradation.

This guide provides a structured troubleshooting approach that works consistently across RHEL 7 through RHEL 10.

1. Identify the Storage Problem
Start by understanding what type of storage issue you are facing.
Symptom                                                     Likely Cause
Filesystem full                                          → Disk usage or log growth
Mount fails at boot                                            → /etc/fstab error
Disk not detected                                               → Hardware or driver issue
LVM volumes missing                                    → VG/LV not activated
Read-only filesystem                               → Filesystem corruption
Slow I/O                                                                → Disk or SAN performance
iSCSI/NFS not mounting                               → Network or auth issue

2. Check Disk Detection and Hardware Status
List Block Devices

# lsblk
Check Disk Details
# blkid
# fdisk -l
Check Kernel Disk Messages
# dmesg | grep -i sd
If disks are missing, verify:
  • SAN mapping
  • VM disk attachment
  • Hardware health
3. Filesystem Full Issues
Check Disk Usage

# df -h
Find Large Files
# du -sh /* 2>/dev/null
Clear Logs Safely
# journalctl --vacuum-time=7d

4. Read-Only Filesystem Issues
This usually indicates filesystem corruption.
Verify Mount Status
# mount | grep ro
Remount (Temporary)
# mount -o remount,rw /
Permanent Fix
Boot into rescue mode
Run:
# fsck -y /dev/mapper/rhel-root
Never run fsck on mounted filesystems.

5. Fix /etc/fstab Mount Failures
Incorrect entries cause boot into emergency mode.
Check fstab
# vi /etc/fstab
Verify UUIDs
# blkid
Test fstab
# mount -a
Comment out invalid entries if necessary.

6. LVM Issues (Most Common in RHEL)
Check LVM Status

# pvs
# vgs
# lvs
Activate Volume Groups
# vgchange -ay
Scan for Missing Volumes
# pvscan
# vgscan
# lvscan

7. Extend LVM Filesystem (Low Space)
Extend Logical Volume

# lvextend -L +10G /dev/rhel/root
Resize Filesystem
# xfs_growfs /
# resize2fs /dev/rhel/root

8. Recover Missing or Corrupt LVM
Rebuild LVM Metadata

# vgcfgrestore vg_name
List backups:
# ls /etc/lvm/archive/

9. Boot Fails Due to Storage Issues
Check initramfs

# lsinitrd
Rebuild initramfs
# dracut -f
Verify Root Device
# blkid

10. NFS Storage Issues
Check Mount Status

# mount | grep nfs
Test Connectivity
# showmount -e server_ip
Restart Services
# systemctl restart nfs-client.target

11. iSCSI Storage Issues
Check iSCSI Sessions

# iscsiadm -m session
Discover Targets
# iscsiadm -m discovery -t sendtargets -p target_ip
Login to Target
# iscsiadm -m node -l

12. Multipath Issues (SAN Storage)
Check Multipath Status
# multipath -ll
Restart Multipath
# systemctl restart multipathd

13. Storage Performance Issues
Check Disk I/O

# iostat -xm 5
Identify Slow Processes
# iotop

14. SELinux Storage-Related Issues
SELinux may block access to mounted volumes.
Check Denials
# ausearch -m avc -ts recent
Fix Context
# restorecon -Rv /mount_point

15. Backup and Data Safety (Before Fixes)
Always verify backups before major storage changes.

# rsync -av /data /backup

16. Best Practices to Prevent Storage Issues
  • Monitor disk usage proactively
  • Validate /etc/fstab changes
  • Use LVM snapshots
  • Keep rescue media available
  • Monitor SAN/NAS health
  • Perform regular filesystem checks
Conclusion
Storage troubleshooting in RHEL 7, 8, 9, and 10 follows consistent principles:
  • Verify hardware and detection
  • Fix filesystem and LVM issues
  • Validate mounts and network storage
  • Monitor performance and prevent recurrence
Using this step-by-step approach ensures data integrity, stability, and minimal downtime in enterprise Linux environments.