IBM Spectrum Scale, previously known as GPFS (General Parallel File System), is a powerful, high-performance clustered file system designed for demanding enterprise and HPC environments. It delivers scalable, high-throughput, and highly available access to shared data across multiple nodes—ideal for analytics, AI, and SAP workloads.
Spectrum Scale supports advanced features such as parallel I/O, data replication, fast failover, snapshots, and policy-based data management. When integrated with IBM PowerHA SystemMirror, it ensures continuous availability by automating monitoring and failover of GPFS resources within clustered environments.
With support for up to 16,384 nodes in the latest versions, Spectrum Scale offers unmatched scalability and flexibility for both AIX and Linux systems—making it a cornerstone technology for organizations that need performance, reliability, and resilience in their data infrastructure.
IBM GPFS (Spectrum Scale) Cluster - Logical Overview:
GPFS Cluster Nodes:
- Multiple nodes (can be physical or virtual)
- Run the GPFS daemon (mmfsd) on each node
- Nodes can be clients and servers simultaneously
Shared Storage:
- Block storage (SAN, NAS, or local disks)
- Storage is organized as disks in GPFS managed storage pools
- GPFS handles striping, replication, and redundancy
Metadata Servers (MDS):
- Manage file system metadata (directory structure, file attributes)
- Can be single or multiple for redundancy
GPFS Clients:
- Nodes mounting the GPFS file system
- Access files without being metadata servers
Cluster Interconnect:
- High-speed network connecting all nodes
- Handles heartbeat, messaging, and data transfer
Optional NSD (Network Shared Disk) Daemons:
- Present on nodes managing disk access
- Nodes can act as NSD servers to share storage across the cluster
Distributed Metadata – GPFS metadata servers manage directories and file attributes, optionally replicated for HA.
Concurrent Access – All nodes can read/write data simultaneously.
High Availability – NSD servers and multiple MDS nodes ensure no single point of failure.
Data Striping & Redundancy – GPFS distributes data across disks for performance and reliability.
Spectrum Scale Server Nodes – The core of the cluster, providing file system access and data services.
Shared Storage (NSDs) – Network Shared Disks that offer shared, distributed storage to all participating nodes.
Clients – Nodes that perform parallel I/O operations, accessing data across multiple NSD servers for maximum throughput.
Cluster Manager – Oversees cluster membership, detects failures, and manages recovery operations.
Configuration Servers – Maintain consistent configuration data for all nodes in the cluster.
Filesystem Managers – Handle filesystem-specific tasks such as configuration, space allocation, and quotas.
Token Managers – Manage distributed locks and consistency tokens for synchronized, concurrent access.
Gateways (CES Nodes) – Provide protocol services (NFS, SMB, Object) for broader client access.
Utility Nodes – Support administration, monitoring, backup, and data management tasks.
IBM Spectrum Scale (GPFS) Deadlock:
Deadlocks in IBM Spectrum Scale (formerly GPFS) occur when cluster file system operations become indefinitely blocked due to conflicting resource waits—often involving distributed locks or metadata operations. Such conditions can cause nodes to hang and impact overall cluster availability.
Common causes include complex locking dependencies, slow or failing I/O, network delays, or resource contention. To address this, Spectrum Scale introduced automated deadlock detection starting with version 4.1. This feature identifies “long waiter” threads based on configurable thresholds, logs detailed warnings, and collects diagnostic data to help administrators pinpoint issues.
In severe cases, recovery may require coordinated node reboots or fencing to restore stability. Spectrum Scale also employs a Deadman Switch (DMS) timer to crash unresponsive nodes, preventing data corruption from stalled I/O.
IBM Spectrum Scale Tiebreaker Disk:
A tiebreaker disk in IBM Spectrum Scale (formerly GPFS) is a special shared disk resource designed to help maintain cluster quorum and prevent split-brain conditions—particularly in small or even-numbered node clusters.
When network partitions occur, the tiebreaker disk acts as a shared arbitration point. Cluster nodes attempt to reserve this disk, and the node subset that controls the majority of tiebreaker disks gains quorum to continue operations safely.
Tiebreaker disks are designated from existing Network Shared Disks (NSDs) using the IBM.TieBreaker resource class. They are not used for file storage but exclusively for cluster decision-making. Connectivity to all tiebreaker disks is mandatory for quorum nodes, as disk access determines which partition stays active.
The mechanism relies on SCSI persistent reservations to enable quick and reliable cluster manager elections. Administrators can configure one to three tiebreaker disks using the mmchconfig command with the tiebreakerDisks parameter.
In small clusters—especially two-node configurations—tiebreaker disks play a crucial role in ensuring high availability by allowing one surviving node with disk access to maintain cluster operations without risk of data corruption.
IBM Spectrum Scale (GPFS) Cluster Installation:
1. Prerequisites
- Ensure all cluster nodes are running compatible IBM AIX or Linux operating system versions.
- Verify network connectivity between all nodes.
- Ensure disks intended for GPFS are visible to the OS.
Check available disks:
AIX:
# lspv
Linux:
# lsblk
2. Configure Passwordless SSH
Set up passwordless SSH between the admin node and all cluster nodes (root user).
Generate SSH Key (on node1)
AIX & Linux:
# ssh-keygen -t rsa -b 4096
Configure Authorized Keys
On node1:
# cp $HOME/.ssh/id_rsa.pub $HOME/.ssh/authorized_keys
Copy public key from node2 to node1:
# scp node2:/.ssh/id_rsa.pub /tmp/id_rsa.pub
# cat /tmp/id_rsa.pub >> $HOME/.ssh/authorized_keys
Copy authorized_keys from node1 to node2:
# scp $HOME/.ssh/authorized_keys node2:/.ssh/authorized_keys
Test SSH Connectivity
From node1:
# ssh node1 date
# ssh node2 date
From node2:
# ssh node1 date
# ssh node2 date
3. Install IBM Spectrum Scale Packages
Install required packages on each node:
AIX:
# installp -acgXYd /mnt gpfs.base gpfs.docs.data gpfs.ext gpfs.gskit gpfs.msg.en_US
Linux:
cd /tmp/gpfs
# dnf or yum install gpfs.base gpfs.docs.data gpfs.ext gpfs.gskit gpfs.msg.en_US
4. Configure GPFS Environment Path
Log in as root.
AIX:
Backup existing profile:
# cp /root/.profile /root/.profile.bak
Edit profile:
# vi /root/.profile
Add:
export PATH=/usr/lpp/mmfs/bin:$PATH
Activate changes:
# . /.profile
Linux:
# cp /root/.bash_profile /root/.bash_profile.bkp
# echo "export PATH=$PATH:/usr/lpp/mmfs/bin" >> /root/.bash_profile
# source /root/.bash_profile
5. Create Cluster
Create Node Definition File
# vi /tmp/node.list
Contents:
node1:quorum-manager
node2:quorum-manager
Create Cluster
# mmcrcluster -N /tmp/node.list --ccr-enable -r /usr/bin/ssh -R /usr/bin/scp -C <Cluster_Name>
or
# mmcrcluster -N node1:manager-quorum,node2:manager-quorum -p node1 -s node2 -r /usr/bin/ssh -R /usr/bin/scp -C <Cluster_Name>
6. Verify Cluster Creation
# mmlscluster
Example:
# mmlscluster
GPFS cluster information
========================
GPFS cluster name: GPFS_CLUSTER1.ppc.com
GPFS cluster id: 7380396095829822664
GPFS UID domain: GPFS_CLUSTER1.ppc.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------
1 indaixgps01.ppc.com 192.168.122.14 indaixgps01.ppc.com quorum-manager
2 indaixgps02.ppc.com 192.168.122.15 indaixgps02.ppc.com quorum-manager
# mmlsconfig
Configuration data for cluster GPFS_CLUSTER1.ppc.com:
-----------------------------------------------------
clusterName GPFS_CLUSTER1.ppc.com
clusterId 7380396095829822664
autoload no
dmapiFileHandleSize 32
minReleaseLevel 4.1.1.0
ccrEnabled yes
adminMode central
File systems in cluster GPFS_CLUSTER1.ppc.com:
----------------------------------------------
7. Apply License
Apply server license on each node:
# mmchlicense server --accept -N node1
# mmchlicense server --accept -N node2
Change cluster service auto start after reboot:
# mmchconfig autoload=yes
# mmlsconfig
Example:
# mmchconfig autoload=yes
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
# mmlsconfig
Configuration data for cluster GPFS_CLUSTER1.ppc.com:
-----------------------------------------------------
clusterName GPFS_CLUSTER1.ppc.com
clusterId 7380396095829822664
dmapiFileHandleSize 32
minReleaseLevel 4.1.1.0
ccrEnabled yes
autoload yes
adminMode central
File systems in cluster GPFS_CLUSTER1.ppc.com:
----------------------------------------------
8. Start GPFS
Start GPFS services on all nodes:
# mmstartup -a
Tue Feb 24 23:10:56 CST 2026: 6027-1642 mmstartup: Starting GPFS ...
9. Verify Cluster Status
# mmgetstate -a
Example:
# mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 indaixgps01 down
2 indaixgps02 down
or
# mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 indaixgps01 2 2 2 active quorum node
2 indaixgps02 2 2 2 active quorum node
10. Create NSD Definition File
On node1:
# vi /tmp/nsd_disk.txt
Contents:
%nsd:
device=/dev/hdiskX
nsd=nsd_name
servers=node1,node2
usage=dataAndMetadata
failureGroup=-1
pool=system
Example:
%nsd:
device=/dev/hdisk3 --This is AIX for Linux Change here /dev/sdc or sdc1 if its created partition
nsd=nsd_02
servers=indaixgps01,indaixgps02
usage=dataAndMetadata
failureGroup=-1
pool=system
11. Create NSDs
# mmcrnsd -F /tmp/nsd_disk.txt
mmcrnsd: Processing disk hdisk3
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
To Verify
# mmlsnsd
# lspv
Example:
# lspv | grep nsd
hdisk2 00a69bd09336280c nsd_01
hdisk3 00a69bd093362993 nsd_02
or
# mmlsnsd
File system Disk name NSD servers
---------------------------------------------------------------------------
(free disk) nsd_01 indaixgps01.ppc.com,indaixgps02.ppc.com
(free disk) nsd_02 indaixgps01.ppc.com,indaixgps02.ppc.com
12. Create File System
# mmcrfs <gpfs_fs_mountpoint> <gpfs_fs_name> -F <nsd_disk_file> -B 64k
Example:
# mmcrfs /ibmmq gpfsmq -F /tmp/disk1.list -B 512K
GPFS: 6027-531 The following disks of gpfsmq will be formatted on node indaixgps01:
mmls nsd_01: size 20480 MB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 218 GB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
GPFS: 6027-572 Completed creation of file system /dev/gpfsmq.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
13. Verify File System
Check file system configuration:
# mmlsnsd
# mmlsconfig
Example:
# mmlsnsd
File system Disk name NSD servers
---------------------------------------------------------------------------
gpfsmq nsd_01 indaixgps01.ppc.com,indaixgps02.ppc.com
(free disk) nsd_02 indaixgps01.ppc.com,indaixgps02.ppc.com
# mmlsconfig
Configuration data for cluster GPFS_CLUSTER1.ppc.com:
-----------------------------------------------------
clusterName GPFS_CLUSTER1.ppc.com
clusterId 7380396095829822664
autoload no
dmapiFileHandleSize 32
minReleaseLevel 4.1.1.0
ccrEnabled yes
adminMode central
File systems in cluster GPFS_CLUSTER1.ppc.com:
----------------------------------------------
/dev/gpfsmq
Mount all the file systems all gpfs nodes:
# mmmount all -a
or mount single filesystem
# mmmount <filesystem>
Example:
# mmmount all -a
Wed Feb 25 00:04:02 CST 2026: 6027-1623 mmmount: Mounting file systems ...
[indaixgps01] / # df -g | grep /dev/gpfsmq
/dev/gpfsmq 20.00 19.55 3% 4038 7% /ibmmq
[indaixgps01] / #
[indaixgps02] / # df -g | grep /dev/gpfsmq
/dev/gpfsmq 20.00 19.55 3% 4038 7% /ibmmq
[indaixgps02] / #
Check file system disk usage:
# mmdf <gpfs_fs>
Example:
# mmdf /dev/gpfsmq
disk disk size failure holds holds free KB free KB
name in KB group metadata data in full blocks in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 203 GB)
nsd_01 20971520 -1 yes yes 20500480 ( 98%) 1120 ( 0%)
------------- -------------------- -------------------
(pool total) 20971520 20500480 ( 98%) 1120 ( 0%)
============= ==================== ===================
(total) 20971520 20500480 ( 98%) 1120 ( 0%)
Inode Information
-----------------
Number of used inodes: 4038
Number of free inodes: 61754
Number of allocated inodes: 65792
Maximum number of inodes: 65792
IBM Spectrum Scale (GPFS) Administration Commands:
View Cluster Info:
# mmlsconfig → Show GPFS version
# mmlscluster → Display cluster configuration
Change Cluster Info:
# mmchcluster -C ClusterName → Change cluster Name
# mmchcluster --ccr-enable → CCR enable
# mmchcluster --ccr-disable → CCR disable
Manage Nodes:
# mmaddnode -N <nodefile> → Add nodes
# mmdelnode -N <node> → Remove nodes
# mmlsnode -L → List all nodes and details
# mmchnode -N <node> -a <attr=value> → Modify node attributes
Cluster Parameters:
# mmlsconfig → Display configuration
# mmchconfig <parameter>=<value> → Change parameter
# mmlsconfig > /tmp/gpfs_config.txt → Export configuration
# mmchconfig tiebreakerDisks=no → Configure the CCR without tiebreaker disks
# mmchconfig tiebreakerDisks=<ORIGINAL_TIEBREAKER_DISKS> → Reconfigure the CCR with the original tiebreaker disks
Start/Stop GPFS:
# mmstartup -a → Start on all nodes
# mmshutdown -a → Stop on all nodes
# mmstartup -N <node> → Start cluster one node
# mmshutdown -N <node> → Stop cluster one node
Check Status:
# mmgetstate -a → Cluster status
# mmgetstate -aL → Daemon status
List Filesystems:
# mmlsfs all → List all filesystems
# mmlsfs <fsname> → Show details
Manage Filesystems:
# mmcrfs <Device> -F <stanzaFile> -A yes -T /mountpoint → Create filesystem
# mmmount all -a → Mount all filesystems
# mmmount <filesystem> -a → Mount one filesystems
# mmmount <filesystem> -a -N <node> → Mount one filesystems one node
# mmumount <filesystem> -a → Unmount one filesystems
# mmumount <filesystem> -a -N <node> → Unmount filesystems perticuler node
# mmumount all -a → Unmount all filesystems
# mmlsmount all -L → Check mount status
# mmdelfs <fsname> → Delete filesystem
View & Manage Disks:
# mmlsdisk all -L → List all disks
# mmadddisk <fsname> <diskdesc> -r <pool> → Add disks
# mmdeldisk <fsname> <disk> → Remove disks
# mmdf <fsname> → Check disk usage
# mmlspool <fsname> → View pool info
Enable/Disable Quotas:
# mmquotaon <fsname> / mmquotaoff <fsname>
View & Set Quotas:
# mmlsquota -a -j all → Show quota status
# mmsetquota -u <user>:<fsname> --block <limit> → Set user quota
Manage NSDs:
# mmlsnsd → List NSDs
# mmcrnsd -F <nsdfile> → Define new NSDs
# mmadddisk <fsname> -F <nsdfile> → Add NSD to filesystem
# mmdeldisk <FilesystemName> <DiskName> → Delete the NSD to filesystem
Cluster Health & Logs:
# mmhealth node show -a → Health check
# mmlslog -g → View GPFS event logs
# mmperfmon query → Performance metrics
# mmfsadm dump → Collect diagnostic data
Role Management:
# mmlscluster –roles → View roles
# mmchconfig minQuorumNodes=<n> → Configure quorum nodes
Maintenance:
# mmfsck <fsname> → Filesystem check
# mmdefragfs <fsname> → Defragment filesystem
Snapshots:
# mmcrsnapshot <fsname> <name> → Create snapshot
# mmlssnapshot <fsname> → List snapshots
# mmdelsnapshot <fsname> <name> → Delete snapshot
Check Inode Limits:
# mmdf gpfs0 -F
Increase Inode Limits:
# mmchfs gpfs0 --inode-limit 10M:8M
Log Locations:
Daemon Logs: /var/adm/ras/mmfs.log.latest
Kernel Logs: /var/adm/ras/mmfs.log.*
Trace Logs: /var/mmfs/traces/
Health Logs: /var/mmfs/gen/mmhealth.log
Quick One-Liners:
# mmlscluster; mmlsnode; mmgetstate -a → Cluster summary
# mmdf all → Disk usage summary
# mmlsnsd -m → Verify NSD mapping
# mmcheck config → Verify configuration consistency
Tip:
Regularly monitor health (mmhealth node show -a), keep configurations backed up (mmlsconfig > /tmp/gpfs_config.txt), and maintain reliable network and storage connectivity to ensure a stable and performant Spectrum Scale environment.
